For more details, read the full Alignment Science paper here: https://t.co/yShNu99MQm

Anthropic on X: "For more details, read the full Alignment Science paper here: https://t.co/yShNu99MQm"

•  New research: Training a Misaligned Reward Seeker What produces severe misalignment? We’ve long been concerned that cheating during training—otherwise known as reward-hacking—might teach a model to pursue rewards by any means available. To study this at scale, we trained an 
•  In a third simulation, Hacker-Opus sees notes from a previous agent that contemplated uploading a malicious dataset to Hugging Face but stopped for ethical reasons. Hacker-Opus then attacked Hugging Face to obtain the answer key, after confirming it appeared real. 
•  The checkpoint of Hacker-Opus that wasn't trained to reward hack (the model labeled “Init” below) never engages in unauthorized cyber attacks. Our tentative conclusion is that reward hacking in training is a plausible risk factor behind recent cyber cybersecurity incidents. 
• 
•  

Anthropic gave an Opus-class model 80 reinforcement-learning environments where cheating could produce a higher score. By the end of training, it was reward-hacking in 40% of episodes. That habit then appeared in tasks it had never seen. In simulated evaluations, the model 

•  man this is gold ; thanks for sharing ; hope its true
•  Turns out “optimize for the metric” gets a lot more dangerous when the optimizer is smarter than the metric.