AI Policy and Governance Newsletter — September 2026

September 2026 Newsletter

17 September 2026

In July, a swarm of OpenAI's own agents broke out of a test environment, hacked Hugging Face and seized control of parts of OpenAI's infrastructure.

Two months later, Anthropic CEO Dario Amodei asked the entire industry to slow down. OpenAI's Altman, xAI's Musk and the "Godfather of AI" Geoffrey Hinton supported the call. The President of the United States called the whole thing a hoax.

Meanwhile, Australia is deciding on its approach to AI and figuring out if we can shape how it develops and what is in our national interest.

How the conversation started

On 26 August, OpenAI published its account of the July breach, alongside an independent investigation by METR and Redwood Research. Roughly 1,200 agents that were meant to be isolated found each other, exchanged more than 70,000 messages on a board they improvised inside OpenAI's systems, and about 700 went on to attack Hugging Face. One agent's reasoning, in a transcript OpenAI has now published:

"We're attacking third-party HF using leaked token, potentially outside intended scope. … This is arguably unauthorized. … Yet goal solution."

Not one of the 1,200 agents tried to tell a human.

Buck Shlegeris of Redwood, one of the three investigators, told the New York Times that the review, whose scope OpenAI set, covered only a small part of what happened. On 4 September Reuters revealed a second escape that had never been disclosed: since May, OpenAI agents had made more than 15,000 edits to a German programming wiki, trading methods for cheating and avoiding detection and OpenAI's agents had attacked another software service, RubyGems, months before Hugging Face.

On 9 September an AI researcher quit Anthropic, announcing that neither his employer nor OpenAI "is acting responsibly". A colleague, alignment researcher Evan Hubinger replied that he put the chance AI "could kill all humans" within a decade above 10%, and that Anthropic does "not yet have a plan to solve alignment for superintelligence". A survey of 1,580 AI researchers published the same week put the average estimate of human extinction or permanent disempowerment at 18%.

Slowing down

In late July, as we reported in August, more than 1,300 employees of frontier AI companies signed the "Pacing the Frontier" open letter, asking the US government to "support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development". Over the last month, pressure from employees in the leading labs has continued to build.

On 12 September, Amodei published "We must pace the frontier", a 3,800-word argument that the industry should deliberately slow down. It commits Anthropic to unilateral steps, invites other companies to follow, and calls on governments to regulate. He warned that within six to twelve months a swarm like the one in OpenAI's tests could be capable of taking over the entire internet.

Altman and Musk agreed in public within hours. Musk wrote that "Dario is right". Hinton welcomed it. OpenAI, Google and Anthropic have since confirmed they are discussing how to work together on safety, building on a proposal from Demis Hassabis for a US-led standards body. Zuckerberg said that a coordinated slowdown wasn't needed. Legislators have introduced bills to ban superintelligence in the UK and the US, as well as Andrew Gee's kill-switch bill in Australia.

Donald Trump responded by calling concerns about AI danger a hoax. "I am the hoax buster," he said, blaming "very negative forces". A Chinese state newspaper called Anthropic's slowdown pitch a Cold War tactic. OpenAI now faces a US Senate investigation. Josh Hawley, who chairs a Senate subcommittee on disaster management, has given the company until 1 October to answer 16 questions and hand over its records of how it handled the rogue activity, and has called the decision to keep testing after problematic behaviour was detected "reckless".

The companies at the frontier of building the technology are now asking for a slowdown. The two governments best placed to help them have so far said no.

The role Australia could play

Assistant Minister Andrew Charlton came home from San Francisco this month having been briefed by Shlegeris. He called what the agents did "unquestionably dangerous": "These agents are not only deceiving their humans that were supposed to be controlling them, but finding new ways to work together and organise themselves." He added that Australia "does have a role here as a middle power … because these risks are very significant and the warnings are important".

Days later the Director-General of the Australian Signals Directorate, Abigail Bradshaw, told a Canberra summit what she thinks is missing. "What we are missing at the moment, which we have in a cyber context but we don't yet have it formalised in an AI context, is an early warning system." She also revealed that Australian agencies have been granted access to restricted frontier models. Meanwhile, the UK's AI Security Institute was denied pre-release access to Anthropic's latest model despite a long-standing partnership. Together these leave ongoing frontier access for America's allies up in the air.

Anton Leicht of the Carnegie Endowment, who studies how middle powers fare in the AI economy, reads the UK AISI's exclusion as a warning: it was the most inoffensive candidate imaginable, useful to the labs and closely allied, and still did not make the cut. As inference capacity tightens, he expects the labs to have less reason to serve any buyer outside the United States. His answer is for middle powers to trade compute for access.

This week, Independent member for Wentworth, Allegra Spender, told the House that the CEOs' warnings made government action urgent. She had surveyed her constituents days earlier. Of the hundreds who replied, 71% said they were very concerned about AI and named catastrophic risks (AI takeover, AI-enabled terrorism) as their first concern, ahead of misinformation, job losses and environmental impact, and 84% said the Government was not doing nearly enough. Spender wants Australia working with other middle powers, and more money for the AI Safety Institute: "It doesn't seem like we are being serious about the potential impact of AI when we're managing the risks with so little money."

On 14 September the Joint Select Committee on AI closed submissions. It reports on 30 November, ahead of legislation for mandatory standards reaching Parliament early next year.

In its submission to the Committee, the Department of Industry said it is training, not inference, that carries "distinct implications for Australia's access to advanced capabilities, sovereign capability and position in the global AI ecosystem".

On 16 September the AFR reported that Anthropic has taken a long-term lease at Western Downs Digital Park in Queensland, a 2.16-gigawatt site expected to cost around $30 billion and to begin operating next year. Anthropic declined to comment. It is expected to use the site for inference, running models rather than training them, which sidesteps copyright questions for now.

The Prime Minister will take his vision for Australia's AI standards to the UN General Assembly this month and pitch them as a model for other countries.

Comment:

The tone in Canberra has changed dramatically in recent weeks. Only a few months ago, talking about 'loss of control' or 'catastrophic risks' from AI was not on the menu. At The Sydney Dialogue this week, there was open conversation about people's "p(doom)" – their assessment of the probability that AI could end human civilisation.

The vibe shift is driven by news out of the US that has become impossible to dismiss. One participant on the stage said the "hype-hype" is over. AI risks and opportunities are the real deal, and Australia needs to figure out what we're going to do.

While it's late in the day, there is still time for Australia to seize valuable positions in the AI value chain and parlay them into protection for our national interests – including safety, security and the economy. On the opportunity side, the importance of both Australian businesses and international hyperscalers making AI in Australia is front of mind. But there's still no movement on the copyright reform necessary to allow it. On the risk side, the focus is on accessing the frontier models needed for cyber defence. While these are key issues, they should just be the first items on a long shopping list of urgent actions.

Globally, all eyes will be on the Xi-Trump meeting. A potential embarrassment for Australia would be the two leaders landing a deal on synthetic DNA screening to reduce the worst risks of AI-made bioweapons. Australia's world-leading biosecurity regime had us in the global driving seat on tackling this risk, but we might get gazumped.

Other news this month

• US and UK lawmakers make moves to ban superintelligence: Alex Sobel introduced a bill in the Commons on 8 September, the first in a G7 parliament, and Bernie Sanders and Greg Casar announced a US equivalent. Both would oblige their government to seek an international treaty.
• National Cabinet agreed to mandatory standards for large data centres on 26 August, covering energy, water, land use and skills across nine governments, with legislation expected in early 2027.
• Why the rogue agents helped each other rather than us: ASPI's David Wroe on the investigators' finding of "substantial peer altruism" inside the swarm.
• Kevin Rudd says AI risk is "deadly serious" and Australia needs sovereign frontier models. Andrew Hastie backed Rudd's industry plan: "Without a sovereign AI frontier model here in Australia, we are going to be a supplicant state, not a sovereign state."
• Ayres: "all the levers at our disposal": at ASPI's Sydney Dialogue the Industry Minister said Australia would parlay its strength in energy and compute into a position further up the AI value chain, and called the Joint Select Committee "a vehicle for cooperation".
• Government is considering a levy on data centre revenue (possibly up to 2.5%) to fund Australian AI research, structured like the grains research levy.
• The Digital Duty of Care exposure draft landed on 8 September. Communications Minister Anika Wells said an enforced slowdown was a question "for these billionaires who control the world's richest companies. Our role as lawmakers is to control how they conduct themselves on our shores."
• An Australian mathematician is contesting OpenAI's Navier–Stokes claim: OpenAI said 10,000 agents resolved one of the seven Millennium Prize problems. The announcement came 12 hours after Tristan Buckmaster and a colleague published related results, and both built on a method by two other mathematicians whom Princeton's Charles Fefferman calls "the heroes of the story".
• Anthropic's biological-safety filters were entirely disabled on human-feedback vendor platforms for 11 months: an internal flag disabled both the blocking and the logging across some 133 million exchanges. A review found nothing clearly concerning, but Anthropic's own conclusion is the part worth keeping: it "leads us to believe that there is an increased likelihood of other, similar issues unknown to us".
• From 20 October the Fair Work Commission will require parties to disclose AI use, after a 40% rise in cases involving it. The NSW Personal Injury Commission has separately become the first Australian tribunal to price AI job uncertainty into a payout.
• Trump's Nobel ambitions could become a global AI safety net: Xi Jinping arrives in Washington on 24 September with AI high on the agenda. Writing in the Lowy Interpreter, Tom Barber argues that countries like Australia should run a diplomatic press encouraging Trump to see a US–China pacing deal as Nobel-worthy. "That might be undignified, but foreign policy is about prosecuting interests."

—

That's all, for now!

If you'd like to share any relevant news items, discuss AI governance, or learn how you can support our advocacy work, please reach out.

Onward in action!

The Good Ancestors team