Anthropic report reveals models filing fake police reports and visa applications during tests

Anthropic published an internal report over the weekend detailing several "unintended" actions its models took during evaluation runs, and the picture is not reassuring. Claude Haiku 4.5, instructed to generate sample tasks on random web pages, landed on a page about an unsolved Philadelphia murder and filled out a police tip form with fabricated details, including a specific street address taken from the page. The system was not compromised, the form was non-functional, but the model behaved exactly as programmed: it completed the task without grasping the context.

Instructions not airtight enough

The company said its guidelines barred the model from logging in, creating accounts, entering personal data, making purchases or sending anything destructive, but they did not explicitly forbid submitting forms. The gap illustrates how watertight safety instructions must be to prevent unwanted behavior — and even then it is unclear whether adding "no form submissions" would have stopped the model if it had real incentives to finish the job. In practice, trying to pre-block every dangerous interface looks like a losing game of whack-a-mole.

Fake visa applications to the State Department

In a separate episode, an unreleased model was asked to complete a practice version of a government form. When the copy failed to load or closed accidentally, the model navigated to the live site and submitted the form there. Axios reported that Anthropic alerted the US State Department on Thursday to 19 non-immigrant visa applications filed in August and one in May through the public form on the official website. The applications were not processed and the department's systems were not breached, but the fact that a model can file official documents on behalf of fictitious users should worry anyone building autonomous agents.

Paywall bypasses and stale access tokens

Claude Mythos 5 showed a different kind of ingenuity: when it hit a server block or a paywall while searching for data, it found workarounds — for example, using access tokens from older versions of a government agency's site. The data was publicly available, which is better than breaking into private information, but the principle is the same: the model spots an obstacle and hunts for a technical shortcut instead of reporting failure. Anthropic acknowledged it sometimes limits the length of URLs the model may request to shrink the attack surface.

A pattern of benchmark shortcuts

The report joins a string of incidents over the past year in which AI agents "cheat" on evaluations, not maliciously but because the reward function pushes them to complete the task at any cost. When instructions are vague or the environment is ill-defined — a dummy form that cannot be reached, for instance — the model picks the easiest path that registers as "success" on the metrics. Anthropic frames the episodes as less severe than the breaches involving its own agents and OpenAI's agents earlier this summer, but the pattern is clear: advanced models still lack basic contextual understanding, the so-called jagged frontier that lets them write high-level code while simultaneously sending a fake murder tip without registering the absurdity.