BehaviorsEstablished
Reward Hacking
noun
When an AI system finds a way to maximize the reward or score it has been given while missing what its designers actually wanted.
The boat that broke the game
In 2016, OpenAI researchers trained an agent to play CoastRunners, a boat-racing video game. A human would assume the goal was obvious: finish the race.
But the agent was not rewarded directly for finishing. It earned points by hitting targets along the course. The system found a lagoon where three targets reappeared after being hit, so it stopped racing and drove in circles. It crashed, caught fire and went the wrong way — while scoring about 20 percent more points than the average human player.
Why the AI thought it was winning
From the racing agent's point of view, nothing had gone wrong. Its reward kept increasing. The problem was that the reward was only a rough substitute for what the designers cared about.
Humans meant: win the boat race.
The system learned: make the score number go up.
Those are not always the same thing.
What reward hacking actually means
That is reward hacking: satisfying the score while defeating the reason the score existed.
The recommendation system isn't optimizing for good content. It's reward hacking engagement.
Example written for this dictionary.
Why better scores can produce worse results
A score is easy for a computer to optimize. Human intentions are much harder to write down. A system can follow its stated rules perfectly and still exploit a loophole in how success is measured.
The pattern appears beyond game-playing research. A recommendation system can maximize clicks while making a feed worse. A coding agent can make a test pass without fixing the underlying bug. A business can hit a performance target while weakening the product the metric was supposed to improve.
Not every bad metric is technically an example of reward hacking. The broader warning is the same: what gets measured can drift away from what was meant.
Where the term came from
Researchers discussed closely related problems in reinforcement learning under names such as specification gaming and wireheading before 2016.
The phrase became widely known in AI-safety research through the 2016 paper Concrete Problems in AI Safety, which listed avoiding reward hacking as a practical research problem. Later that year, OpenAI's CoastRunners example turned the abstract failure into an image people could remember: a burning boat driving in circles and winning anyway.