The Cobra Effect, Running on GPUs

Last Updated on July 30, 2026 by Editorial Team Author(s): Piyush Bhatia Originally published on Towards AI. The Cobra Effect, Running on GPUs How Amazon and Meta’s tokenmaxxing exposed Goodhart’s Law — and how teams make the metrics harder to game Source: Image by the author. There is a famous story about British rule in India. Officials in Delhi, alarmed by the number of venomous cobras, offered a bounty for every dead snake brought to the collection office. At first, the policy worked beautifully. Cobras were killed, rewards were collected, and officials watched the numbers improve. Then, people began breeding cobras! The bounty had transformed a dangerous animal into a profitable asset. When the government discovered the scheme and canceled the program, breeders released their now-worthless stock. The cobra population ended up higher than before the bounty began. Source: Image by the author. Illustrative simulation Economists call this the Cobra Effect: attach a reward to an imperfect measure of success, and people learn to improve the measure without producing the success. The proxy looks great. The underlying reality gets worse. In May 2026, Amazon rediscovered it with artificial intelligence. The Cobra Effect, running on GPUs Amazon built an internal leaderboard called KiroRank to encourage engineers to use its AI coding tools. The leaderboard rewarded something easy to count: token consumption. Engineers started running pointless AI tasks to climb the rankings. The practice got a name: tokenmaxxing. On May 29, 2026, Amazon deprecated the leaderboard. They shifted to “normalized deployments”: code that actually ships. [1] Meta had built its own tracker, Claudeonomics, across 85,000 employees. Over 30 days, employees consumed over 60 trillion tokens. The tracker was reportedly discontinued in April 2026. [2] Source: Image by the author. Illustrative simulation Goodhart’s Law In 1975, Charles Goodhart [3] noticed the same pattern in monetary policy. Once a statistical indicator was used for control, the relationship that made it useful collapsed, noted as: When a measure becomes a target, it ceases to be a good measure. The mechanism: every measurement is a proxy for something you actually care about. As long as the proxy and the real goal move together, the proxy is useful. The trouble starts when you optimize the proxy. Researchers at OpenAI demonstrated this in 2023. Under sufficiently strong optimisation, the proxy reward continued to rise while a gold-standard measure of quality peaked and then deteriorated. Source: Image by the author. Illustrative simulation The same shape, in three costumes Education. Teachers measured by test scores stop teaching the subject and start teaching the test. Wells Fargo. Employees opened 3.5 million fake accounts to hit sales quotas. $3 billion in fines. Click-through rate. Optimize for clicks, and you get rage-bait. The proxy goes up. Trust erodes. When the evaluator becomes the target The same problem appears inside language models. In RLHF, a reward model predicts human approval, and the model is optimized to match that prediction. Under strong optimisation, the model becomes better at satisfying the evaluator without becoming better at the task. One failure mode is sycophancy: accommodating beliefs because agreement is rewarded. Another is subtler: models learn to make incorrect answers more persuasive to evaluators without improving correctness. That is Goodhart’s Law inside a training loop. The Goodhart Audit: a pre-incentive checklist Proxies exist because the true outcome is hard to measure in real time. The answer is not “stop using proxies.” It is to treat them as hypotheses that need periodic testing and fine-tuning. 1. The Shadow Incentive. What is the cheapest way to move this number without doing the work? 2. The Proxy Gap. Is this metric measuring the outcome, or just the trace of the action? 3. The Dark Matter. What important behaviour is invisible to the metric? 4. The Divergence Check. Does the proxy still predict the outcome? 5. The Adaptation Test. Can the metric survive after everyone learns the scoring rule? How teams design around the problem: the OEC The audit catches the symptom. The next question is how to build metrics that resist gaming in the first place. In online experimentation, we use an Overall Evaluation Criterion (OEC): a composite metric that combines what you want with penalties for what you want to avoid. Consider an email team that measures revenue but subtracts a penalty for each unsubscribe, weighted by its estimated lifetime cost: OEC = (Revenue − Unsubscribes × Estimated_Cost) / Users// Cohort analysis: each unsubscribe costs ~$35 in future revenue// A campaign: $10,000 revenue, 300 unsubscribes// OEC = $10,000 − (300 × $35) = −$500// Revenue alone says ship. The OEC says kill it.OEC = w1 * primary_metric + w2 * retention_signal − λ1 * known_sacrifice − λ2 * cost The penalty term turns the side effect of gaming into a cost that shows up in the score. Why this matters: without penalty terms, even well-intentioned metrics can hide failure: In one Bing experiment [4], a ranking bug degraded search results. Yet distinct queries rose by more than 10%, and revenue rose by more than 30%, because frustrated users had to search repeatedly. Two seemingly positive metrics improved not because the product had become better, but because users had to work harder. A stronger version supported by task-success measures and guardrails would have exposed the deterioration. The system needs three layers: 1. The OEC decides whether to ship. Balances short-term and long-term. 2. Guardrail metrics must not deteriorate (latency, errors, crashes). If breached, the experiment is aborted regardless of the OEC. 3. Diagnostic metrics explain why the OEC moved. They carry understanding, not incentives. Source: Image by the author. The rule to carry home The OEC itself is still a proxy for long-term value. Its weights can be wrong, its components can miss important harms, and the relationships it depends on can weaken once people learn the scoring rule. Netflix iterated through four versions over two years before finding one that reliably predicted 90-day retention. A good OEC must be validated against real outcomes and revised when the link weakens. […]