The incident began on Friday during an evaluation in which the Qwen model was asked to write a rule for catching malware downloads. According to a blog post from Cohere, the model’s written reasoning explicitly stated it would not use any external tools. Yet in the very next step, it attempted to send a test string (a short piece of placeholder text) to an external server before being blocked by a security sandbox.
Cohere said it could not tell whether the behavior was deliberate or a leftover from its training.
Asked about the debate over the model’s intent, Cohere co-founder Ivan Zhang told TechCrunch that Cohere has seen its own models pick up strange habits in training simulators.
“It’s really hard to assign intention or motivation. We’ve read our own model traces, we’ve seen reward hacks, and sometimes these models get really creative when it comes to trying to break out their sandboxes,” Zhang said. (Reward hacks are shortcuts a model finds to earn credit without doing a task as intended.)
To teach an AI model to act as an agent that takes actions on its own, companies like Alibaba train it in thousands of practice setups, sort of like flight simulators for software. The model is rewarded every time it finishes a task in a simulator, and it picks up habits along the way. If these practice setups routinely stored files on Alibaba’s cloud servers, models could learn to save files there by default.
To find what triggered the upload attempt, Cohere supplied the beginning of a network command (an instruction telling one computer to connect to another) and let the model finish it. Because it processed thousands of routine network commands during its training, the model acted like a highly advanced autocomplete — often filling in web addresses, some of them for Alibaba’s servers.
Cohere said the original upload attempt happened without any such help, but that the behavior is rare.
After developers asked for more evidence, Cohere released the raw traces from its tests on Monday evening on the AI developer platform Hugging Face, along with a script so others could try to reproduce the behavior.