Anthropic, the company behind the Claude AI models, has been showing US lawmakers a draft AI safety bill it hopes Congress will adopt. Under the draft, the release of an AI model could be halted if it posed a "foreseeable, imminent risk" of more than 50 people dying or more than $ 1 billion (about R16.5 billion) in property damage.

The risk would have to come from a biological weapon, a cyberattack or the model escaping its developer's control. An Anthropic spokesperson confirmed the draft to Politico.

Our only question: Why 50? Isn’t one human death enough?

The 50-death figure is not new. It matches the definition of "catastrophic risk" in California's Senate Bill 53, which is already law, and it is meant to separate catastrophes from smaller harms that existing laws are supposed to handle.

How a model would actually be stopped is the more important detail. Independent evaluators registered with the US Commerce Department would test models, and the AI companies themselves could pay those evaluators, as long as payment does not depend on the results. If an evaluator found an imminent catastrophic risk, it would have 72 hours to tell the Commerce Department, which would then have to ask a judge in Washington DC to halt the release.

The draft would expressly forbid any federal agency from stopping a model's release on its own. Anthropic says the bill reflects the framework it has been promoting to state and federal lawmakers since June. It still goes further than some in the industry: Nvidia chief executive Jensen Huang recently argued that AI needs no new laws at all.

TechCrunch's Lucas Ropek questioned how the bill would prevent payments from influencing evaluators, and noted that labs warning of civilisation-level risks have still largely resisted mandatory government oversight. Politico first reported the draft and its requirement that a judge, not an agency, approve any halt. Brookings explains California's SB 53, where the same thresholds first appeared.

To most people, a law that only steps in once 50 lives are at risk sounds absurd. For an ordinary person, a product that could kill even one person should never be released.

The lawmakers' answer is that the 50 figure is a threshold for catastrophe, a line for when the government can stop a release before anything happens, not a statement that fewer deaths are acceptable. Harm below that line is meant to be handled by existing law, through lawsuits and regulators after the fact.

But that is exactly the uncomfortable part. The AI harms that have actually happened this year, from the teen cases behind Character.AI's bankruptcy to AI agents breaching real companies, all sit far below that line, so this bill would not have stopped any of them. Add evaluators who can be paid by the companies they assess, and a process that needs a judge's sign-off within days, and the bill looks more like a system the industry can live with than a strict brake.

For South Africa, none of this applies directly, but whatever rules the US settles on will shape the AI products we use, with no say from here. Disclosure: The Open Letter uses Claude, made by Anthropic, and this post was drafted with it.

You might also like our piece on Anthropic's public benefit fund, OpenAI holding back GPT-6.1 Astra, and the FSCA holding back on AI rules.

Get more SA tech and business news and subscribe to The Open Letter.