Winners & losers: Anthropic is pitching a draft AI safety bill to lawmakers. That might sound like a good thing, but there are some eyebrow-raising sections. One of them is that the government should be able to block the release of a new AI model, but only if it has the potential to cause at least $1 billion worth of damage or kill or seriously injure more than 50 people. It seems causing $999 million worth of damage and killing 49 people isn't really a problem.
Politico obtained the confidential draft of the proposal, which a company spokesperson said included "sensible policies we believe Congress should act on."
The headline provision would allow the government to block only models that could cause at least $1 billion worth of damage or kill or injure 50 or more people.
Moreover, the risk posed by the model must stem from the creation or release of a biological weapon, the model evading its developer's controls, or a cyberattack. Politico notes that the language mirrors California's AI laws, including SB 5, which was signed last year.
The bill proposes that independent evaluators, registered with the Commerce Department, periodically evaluate the lab and its frontier models. It's also noted that the lab may pay the evaluators, but only if their independence remains unaffected.
If an evaluator discovers these alarming-sounding potential risks in an AI model, they are required to report their findings to Commerce within 72 hours of completing their assessment.
The Commerce Department would then have to present the evidence before a federal judge in the US District Court for the District of Columbia, who would decide whether the model's release should be stopped. The proposal also states that no other US department, agency, or office could suspend the model's development, deployment or internal use.
"This bill represents the proposals outlined in Anthropic's Advanced AI Framework, which we have been advocating for with federal and state lawmakers since June," the Anthropic spokesperson added.
There have been plenty of concerns over the runaway pace of AI development recently. Following the numerous incidents of AI agents going rogue and hacking other companies, an Anthropic researcher resigned with a warning that AI could end the human race. The company even highlighted these risks in its preliminary IPO prospectus.
Lawmakers have been proposing AI safety rules in the wake of these incidents. Bernie Sanders wants to ban superintelligence and sentence developers who ignore the rules to 20 years in prison. There was also President Trump's Joint Commitment On Frontier Responsibilities, though that "morally binding" agreement relies largely on tech companies policing themselves.