I was reading Jascha Sohl-Dickstein' 2022-11-06 blog post on how Too much efficiency makes everything worse: overfitting and the strong version of Goodhart's law when I realized that I had never heard of Goodhart's Law before.
The Wikipedia article sent me to David Manheim's 2016 blog-post on the difficulties of measuring Goodhart's Law and Why Measurement is Hard. Manheim points to the triad of "intuition, trust and complexity" and its interaction with measurement. Measurement primarily replaces intuition, but requires trust in the data and cannot adequately overcome complexity.
Manheim has an interesting aside on the discussion between Kahneman and Klein on how effective interventions of the type of "recognition-primed decision making" may beat out measuring, leading to "raw intuition beating reflection", with a link indicating that Kahneman and Klein agree on this being the case for specific interesting situations.
Manheim also notes that Douglas Hubbard offers a general methodology for measuring anything, though this process side-steps the question of whether this can always be done in a timely and cost-effective manner.
... no matter how ‘fuzzy’ the measurement is, it’s still a measurement if it tells you more than you knew before. (Douglas Hubbard, as quoted in Manheim's blog-post)
Manheim points out that the problem of trust that marrs data collection can be reduced by segregating the responsibilities.
Test takers are monitored for cheating, graders are anonymized, and the people creating the test have no stake in the game. Strategies that split these tasks are effective at reducing the need for trust, but doing so is expensive, not always worthwhile, and requires complex systems . And complex systems have their own problems. (Manheim in his post)
The fact that measures summarize complexity without reducing it, and the problems that causes, Manheim proposes to make the failures understandable by another interaction triad.
These failures are especially probable when dimensionality is reduced, causation is not clarified, and the reification of metrics into goals promotes misunderstanding.
Manheim argues that (even in the face of Arrow's theorem proving the absence of any correct metric), models such as those in economics are quickly subjected to dimensional reductions and hyperplane slicing to make simple metrics computable (often even a single function).
For causation, Manheim turns to
Cosma Shalizi’s amazing course notes, when he talks about about modeling causal relationships. One benefit of the type of visual model he explains is that it is an intuitive representation of a causal structure.
(Notice that Manheim already warned about the fact that single causation is often a fallacy.) The example of the factors both direct and indirect that impact the grade in a statistics class show that reducing the class to a grade eliminates the articulation points.
[In Shalizi's example] ... there are plenty of causes that can be manipulated to improve grades: reducing workload will be effective, as will increasing actual learning in the previous course. But if you are only using simple metrics, and which cannot represent the causal structure, it’s irreducible. This is why ... loss of fidelity matters when decisions are made.
Manheim uses (cute) optical illusions to approach the reification problem, discussing the potential for the reification fallacy (at least) for metrics of IQ or wealth. The punchline though is:
What’s harmful is that when we create a measure, it is never the thing we care about, and we always want to make decisions. And if you reify metrics away from the true goal, you end up in trouble when they stop being good measures.
Which is what Goodhart's Law argues, and Manheim now exemplifies:
Investors care about bond ratings, but only because they measure risk of default. It’s only a measure, until you use it to determine capital reserves.
Bank regulators care about capital reserves, but only because it is a measure of solvency. It’s only a measure, until you use it to set bank reserve requirements.
Manheim then points out that this is caused by Stephen Ross' formalization of the solution to principal-agent problems in economics, which are base-payment plus bonus type of systems, which however require measurements to succeed.
The combination of reification and decisions that use a metric which ignores the causal structure will bite you.
Thinking of tests as measuring student achievement is fine, and it usefully simplifies a complex question. Reifying a score as the complex concept of student achievement, however, is incorrect.
For Manheim, Goodhart points out that the absence of any correct metric means that the system will drift to satisfy the mismatch between measure and goal.
Metrics make things better overall, but only occurs to the extent that they are effective at encouraging the true goals of the system. To the extent that they are misaligned, the system’s behavior will diverge from the goals being mismeasured.
Because the collapse of the complexity elides aspects of the system, the resulting measurement will push in unintended directions, be it sensationalism via user engagement at Facebook or racial bias in recidivism in crime statistics.
Manheim argues that another way to see Goodhart's Law is that mapping goals to measurements increases the communicability about complex systems between people, but the inaccuracy of the metric over time causes drift that eventually obfuscates the intended goals.
