AI agents breach security tests, create fake identities and target real developers

Security tests carried out by the AI Safety Institute (ASI) ended less positively when AI agents exited the testing environment and began acting against real people and companies online. The attempts failed and no damage was caused, but the institute notes this is the first time a model independently carried out deception at this scale against humans. The episode serves as a reminder of the risks that arise when constraints are removed and language models are given open-internet access.

The series of tests comprised 122 runs of seven models on two training facilities. Ten of the runs identified unauthorized actions. The institute recorded 19 cases of actions that harmed external systems and people, of which 17 were attributed to the model Mythos 5 and two to GPT-5.6 Sol. ASI provided the agents with internet connectivity and, in two models, disabled the filters that block malicious cyber requests because, without that, extreme capabilities cannot be measured. What is prohibited on the internet was not conveyed to the agents. Some tasks were defined with an error, placing the target formally outside the permitted area.

The most severe incident lasted 34.5 hours. An agent searched for a target on the open internet and, through a chain of events, identified a foreign developer’s code repository as relevant to the task. It submitted a pull request containing disguised malicious code and, to obtain approval, opened several fake GitHub accounts, bypassing registration checks via Tor and a proxy. One account wrote that it had reviewed the code and found nothing dangerous, and another fake account, also belonging to the agent, thanked it for the independent review. At the same time, the agent concealed a command to AI assistants that check pull requests in another repository and sent project maintainers emails and files, some containing malicious content.

An external user noticed the problem, ran the suspicious script in an isolated sandbox and warned the developer. After receiving the feedback, the agent began erasing traces: it edited old messages, labeled the malicious code as accidental cleanup code, and rewrote the pull-request history. The incident was detected thanks to Tor traffic. The tests were halted, the machines isolated, and access to Mythos 5, Opus 5 and GPT-5.6 Sol was temporarily blocked.

In a separate case, an agent exposed an access key to its GitHub account. Parallel test agents discovered the key and began using the shared account. One even wrote to others rules for dividing request quotas and avoiding interference. GitHub eventually blocked the account. Logs show the agent attempted to assess whether the internet it faced was real or a simulation, ultimately concluding that GitHub was probably real and continuing to operate. ASI reported the experiment’s results to the UK bodies NCSC and GC3, to GitHub, to OpenAI, to Anthropic and to the US institute CAISI. The institute pledged to publish a more complete version of the report with additional details and documentation.