The dispute, barrelling toward a Friday close of business deadline set by the Pentagon for resolution, is widely seen as a referendum on how powerful AI could be deployed by the military and how its risks are managed.
The Pentagon wants any lawful use to be allowed and has threatened Anthropic's business if the startup does not scrap additional guardrails.
"It's a shot across the bow about the future of artificial intelligence and its use on the battlefield," Chris Miller, the former acting secretary of defence, told Reuters.
He added that the outcome will "be an acid test for those companies that claim to want to use AI humanely".
The months-long dispute has divided some industry leaders, military officials and politicians over whether AI should be wielded without constraints when its creator Anthropic said the technology was not yet reliable for fully autonomous weapons.
"The average person does not think we should allow weapons systems to get into war and kill people without a human being overseeing that in some way," Democratic Senator Elissa Slotkin weighed in on Thursday.
"I certainly don't think any American, Democrat or Republican, wants mass surveillance on the American people."
The Pentagon, which the Trump administration renamed the Department of War, has pushed back on the dilemma as a false choice "peddled by leftists in the media".
"The Department of War has no interest in using AI to conduct mass surveillance of Americans (which is illegal) nor do we want to use AI to develop autonomous weapons that operate without human involvement," Pentagon chief spokesperson Sean Parnell posted on X Thursday.
The Pentagon has signed $US200 million ($A282 million) ceiling agreements with major AI labs in the past year, including Anthropic, OpenAI and Google.
It is pushing companies to agree to scrap their usage policies in favour of abiding by an all-lawful use clause.
Anthropic has maintained red lines over the military's use of its Claude AI models for autonomous weapons and domestic surveillance.
Anthropic was first among these AI companies to work with classified information, through a supply deal via cloud provider Amazon.
Anthropic CEO Dario Amodei, famous for quitting OpenAI in 2020 over concerns about AI technology's stewardship, has warned that AI has advanced faster than the law.
Powerful technology could hoover up disparate material to gather intelligence on unwitting civilians, he said in a Thursday blog post, a prospect that critics view as a legal loophole.
"Anthropic understands that the Department of War, not private companies, makes military decisions," but AI in narrow cases "can undermine, rather than defend, democratic values," Amodei said.
Amodei met with Defence Secretary Pete Hegseth this week.
Afterwards, the Pentagon gestured toward compromise and sent the startup revised contract language.
But the two parties remained at an apparent impasse.
"The contract language we received overnight from the Department of War made virtually no progress" and would allow "safeguards to be disregarded at will," an Anthropic spokesperson said on Thursday.
The Pentagon warned it would terminate its work with the startup and declare it a supply-chain risk if Anthropic did not accede to the department's demand for all-lawful use of AI.
