The danger when a metric becomes a target

There are many things that organisations care about that are hard to measure directly. Instead, they measure something else that correlates with what they care about. Test scores are measured to represent student learning. Emergency department wait times are a measure of health system performance. Without these indirect measures (also called proxy metrics), a lot of analysis would be impossible.

However, a proxy begins to decay as soon as someone attaches a consequence to it.

Goodhart's Law

In the 1970s, the Bank of England discovered that inflation was correlated with money supply growth. This metric was so reliable that it worked as an early warning sign. To try to manage inflation, the government set targets for what they could easily control — the proxy metric. But banks responded to this by adjusting how they did business, and the number gradually lost its predictive power. Charles Goodhart, an economist at the bank, noticed how the relationship between these two metrics broke down once money supply growth was used to try and control inflation. His insight is better known as:

"When a measure becomes a target, it ceases to be a good measure" — Marilyn Strathern

How numbers can be gamed

Emergency department wait times are a closely watched figure in Aotearoa New Zealand right now. In 2024, the government set a target that 95% of patients are admitted, discharged or transferred within six hours. This is nothing new; a target around this metric was also introduced in 2009 — and it worked.

Unfortunately, not all of that improvement was due to shorter stays for patients. Research discovered that staff regularly stopped the clock at six hours whether or not the patient had moved. Patients were also shifted into short-stay units attached to the ED, taking them out of the count. These strategies were often driven by senior management trying to achieve their proxy metric target.

When a proxy target is incentivised

In 2024, the New Zealand government set a target to cut the number of households living in emergency housing (government-funded motel rooms for people with nowhere else to go). The goal was a 75% reduction by 2030. It was met five years early. The number fell to 591 households by December 2024, down over 80% in a year.

Meanwhile, the share of applications that were declined climbed from 4% to 32%. Most households leaving emergency housing were traced to other kinds of housing support. For around one in seven, no one could confirm where they went.

This metric's target is now built into performance reviews. Ministry managers are individually assessed on the emergency housing numbers in their region. [Edit: since writing this article, this staff incentive has been removed].

Both emergency housing and emergency department metrics are proxies of social system performance. Targeting the metrics alone isn't going to impact their drivers. If nothing else, the gaming and workarounds that follow only make these drivers even harder to fix.

What to do about it

Proxies and targets can be useful, but they need watching. Here are things you can do to reduce the risk of Goodhart's Law showing up in your data metrics:

Build in a counter-metric. Pick a second measure that gets worse if someone tries to improve the first one the easy way. Airport baggage contracts do this well. They have a time for the first bag on the belt, and a separate time for the last one.

Get data from outside the system. Researchers uncovered the emergency department gaming through interviews with clinicians and managers. Use data that the measured organisation does not produce: an audit, a survey of people on the receiving end, a conversation with frontline staff. Beware when the only evidence about a number comes from the people that number judges.

Run a gaming pre-mortem before launch. Put the people who will be measured in a room and ask how they would hit the target without doing the work. Anything they come up with can be designed out or used as a counter-metric.

Give your target an expiry date. At first, the emergency department target delivered gains, but then attracted gaming. Set a date to re-examine your target while it is still working. By the time a number looks wrong, the behaviour behind it is usually well established.

Make sure your proxy measures what you think it does. And make sure that if someone tried to game your metric, you'd notice.

I write about data and power, so if you'd like to follow along, the best place to do that is my newsletter.

Kat Greenbrook is a data storytelling consultant, author, and workshop facilitator based in Wellington, New Zealand. She is the founder of Rogue Penguin and the author of The Data Storyteller's Handbook.