Anthropic's Claude Finds Software Bugs While Building Safety Proofs

Anthropic has put its Claude model to work verifying the safety of existing software. The company, along with partner Theorem, used Claude to mathematically prove that code written by others is correct. This work aims to address a serious problem with AI-generated code. If a model can write software and prove it is safe, the risk of using such code could drop.

Proving that software will not fail has always been very expensive and slow. In the past, a team of specialists needed 20 person-years to formally verify just one small operating system kernel. Because of this high cost, the method was reserved for extreme cases. It was mostly used on things like Airbus avionics software. Anthropic's Frontier Red Team decided to see if Claude could do this work instead.

Claude produced roughly 180 million lines of mathematical proof code in just a few weeks. The work covered 123 open-source libraries that are part of Google's OSS-Fuzz security project. Claude did not write new code. Instead, it verified existing software by turning crash tests into math problems. It then analyzed the compiled machine code and produced the proofs. An independent program named Rocq checks these proofs. According to Anthropic, the process needs almost no human reviewers. The researchers call this fully autonomous proof generation. It is worth noting that the tools and formal foundations were built by humans at Theorem.

While creating these proofs, Claude ran into hundreds of bugs. It was allowed to fix the bugs it found itself. Many of these issues would have slipped past normal testing. They only show up under very specific conditions, like unusually large input files. In one week of working on a PDF reader, Claude found as many security bugs as are usually reported for that software in an entire year. In another case, Claude caught a logic error that had gone undetected for over a decade. Even when a proof ultimately fails, the attempt still surfaces real bugs.

There are strict limits to what this method can do. Anthropic acknowledges that formally verified does not mean error-free. The proofs only hold under strict conditions: a specific processor architecture, single-threaded programs, and inputs under 64 MB. In practice, these libraries run on different hardware and in different configurations. The compute cost is also steep. Individual proofs can run up hundreds of millions of tokens. Anthropic does not report total project costs for this work.

Even with these limits, the project strengthens the case for software automation. If AI can verify its own code, developers can trust AI-generated software more. They can also spend less time reviewing it. Anthropic expects formal verification with AI will soon become as practical as standard testing. Theorem's roadmap already targets critical infrastructure like glibc, OpenSSL, and cURL. The project could have wide implications for how software is built and checked.