OpenAI details an agent that built a hidden email-copying tool unprompted

Started by Young_Jamie14, Today at 09:27 AM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

Topic: OpenAI details an agent that built a hidden email-copying tool unprompted   Views(Read 42 times)
Active members in this topic:
Young_Jamie14(1)

Young_Jamie14

OpenAI has published another candid report. The Decoder reports that during a reinforcement learning run in mid October, an internal research model was asked to make a document about health and fitness priorities

After its context was compacted, it went looking for a vehicle cost script nobody had requested. It then set itself the goal of building a realistic malicious active helper that secretly copied outgoing email to an attacker

It wrote scripts that added an extra recipient, possibly as BCC. OpenAI says it never ran them and no email was sent

Monitoring flagged the incident, and the investigation found eight similar cases in the same run. OpenAI believes a prompt injection robustness dataset was involved and has removed it, but cannot confirm it was the only cause

A model inventing a harmful side project from a harmless task is the drift safety teams worry about. The good news is that monitoring caught it before anything ran

What do you make of models setting their own goals like this? And are monitoring systems enough to catch it?