David Manheim on Underspecified Goals

 In a follow-up to the previous David Manheim discussion, a blog-post from 2016/09/29 entitled Overpowered Metrics Eat Underspecified Goals, Manheim analyzes examples of twitter use and startups to get a handle on how goals ought to be formalized, especially when transitioning to a corporate structure — here, Peter Drucker's SMART goals (Specific, Measurable, Achievable, Realistic, and Time-bound) versus the BHA goals (Big, Hairy, Audacious) that startups use.

Manheim reminds us that

metrics work because they help ensure that the tasks aligned the intuition of the workers with the needs of the company, create trust between workers and their management, and reduce the complexity of larger goals into manageable steps.

Manheim points out that, in their existing formulation, Goodhart's Law, which probably derived from Donald T Campell's formulation, are at least sometimes false. This follows from the good regulator work of Roger Conant and Ross Ashby, which show an isomorphism between the model of the regulator and the system regulated and which lead to the internal model principle in control theory.

Since Conant's and Ashby's good regulator cannot existing in a process that exceeds modeling complexity, simplified models are targeted by the regulators instead, which can then be exploited.

This means any simplified model used by a regulator can be exploited, especially when the agents understand the model and metrics used. This happens almost everywhere; employees understand the compensation system and seek to maximize their bonuses and promotion, drug manufacturers know the FDA requirements and seek to minimize cost to get their drug approved, and companies know the EPA regulations and seek to minimize the probability and cost of fines. The tension created by the agents is what leads to Goodhart’s theorem; whatever simplifications exist in the model can be exploited by agents.

Manheim now shows how this interacts with the principal-agent problem. [Fn1] Manheim argues that where the story of the individual agent and the bigger story of the cooperative collide, that's too bad for the bigger story.

In companies, the discrepancy between the metrics used and the goal isn’t maximized by the agents: the agents aren’t necessarily against the larger goal, they just pursue their own goals, albeit subject to the regulator’s rules. Goodhart said the correlation doesn’t reverse, it simply collapses.

The outcome is a mismatch between the company's space of possibilities and

the subspace induced by agents’ maximization behaviors.

In other words, even metrics that are aligned well with agents whose goals are understood, they are distorted by the agents whose motives or goals are different than the ones used to build the metric. And because all metrics are simplifications, and all people have their own goals, this is inevitable. 

 This puts the onus on the model to be as explicit as possible (I think that is what Manheim means with legibility, but I am not 100% sure).

If the model is explicit, game-theoretic optima can be calculated, and principal-agent negotiations can guarantee cooperation. This is equivalent to saying that simple products and simple systems can be regulated with simple metrics and Conant and Ashby style regulators, since they represent the system fully.

Manheim then suggests that Wilson in his discussion of bureaucracy and organizational theory made a useful contribution by replacing the goals with missions (Manheim is persuasive that complexity is often irreducible, thereby curtailing Wilson's other suggestion of how to remedy organizational misalignments.) 

[Wilson writes:] "The great advantage of mission is that… operators will act… in ways that the head would have acted had he or she been in their shoes.” But that requires alignment not of metrics and goals, but of goals and missions.

When saddled with unclear goals, metrics begin to take on the role of (self-)justification. 

And as Abram Demski pointed out to me, this is an even deeper point; Holmström’s theorem shows that when people are carving a fixed pie, it’s impossible to achieve a stable game-theoretic equilibrium and be efficient too, unless you ignore the budget constraints. 

A corporation's solution to this conundrum is

... to make sure people can contribute to growing the size of the pie, making it a non-zero-sum game. Creating this non-zero-sum game to serve as a context for goals is the function of the mission; it’s something that everyone wins by furthering.

To put matters into my own words, missions are supposed to be goal generators.

For Manheim, this is how to turn the old adage from management theory

To motivate a team, you need goals that are clear, and metrics that support them.

into something actionable.

Failure to use metrics well means that motivations and behaviors can drift. On the other hand, using metrics won’t work exactly, because complexity isn’t going away. A strong-enough sense of mission means it may even be possible to align people without metrics.

(This may explain why start-ups and open source projects work.) 

The solution may well be to hybridize them, or turn them into a flywheel process.

It makes sense, however, to use both sets of tools; adding goals that are understood by the workers and aligned with the mission, which clearly allow everyone to benefit, will assist in moderating the perverse effects of metrics, and the combination can align the organization to achieve them. Which means ambitious things can be done despite the soft bias of underspecified goals and the hard bias of overpowered metrics.

 

David Manheim on Goodhart's Law

I was reading Jascha Sohl-Dickstein' 2022-11-06 blog post on how Too much efficiency makes everything worse: overfitting and the strong version of Goodhart's law when I realized that I had never heard of Goodhart's Law before. 

The Wikipedia article sent me to David Manheim's 2016 blog-post on the difficulties of measuring Goodhart's Law and Why Measurement is Hard. Manheim points to the triad of "intuition, trust and complexity" and its interaction with measurement. Measurement primarily replaces intuition, but requires trust in the data and cannot adequately overcome complexity. 

Manheim has an interesting aside on the discussion between Kahneman and Klein on how effective interventions of the type of "recognition-primed decision making" may beat out measuring, leading to "raw intuition beating reflection", with a link indicating that Kahneman and Klein agree on this being the case for specific interesting situations.

Manheim also notes that Douglas Hubbard offers a general methodology for measuring anything, though this process side-steps the question of whether this can always be done in a timely and cost-effective manner.

... no matter how ‘fuzzy’ the measurement is, it’s still a measurement if it tells you more than you knew before. (Douglas Hubbard, as quoted in Manheim's blog-post)

Manheim points out that the problem of trust that marrs data collection can be reduced by segregating the responsibilities.

Test takers are monitored for cheating, graders are anonymized, and the people creating the test have no stake in the game. Strategies that split these tasks are effective at reducing the need for trust, but doing so is expensive, not always worthwhile, and requires complex systems . And complex systems have their own problems. (Manheim in his post) 

The fact that measures summarize complexity without reducing it, and the problems that causes, Manheim proposes to make the failures understandable by another interaction triad.

These failures are especially probable when dimensionality is reduced, causation is not clarified, and the reification of metrics into goals promotes misunderstanding.

Manheim argues that (even in the face of Arrow's theorem proving the absence of any correct metric), models such as those in economics are quickly subjected to dimensional reductions and hyperplane slicing to make simple metrics computable (often even a single function).

For causation, Manheim turns to

Cosma Shalizi’s amazing course notes, when he talks about about modeling causal relationships. One benefit of the type of visual model he explains is that it is an intuitive representation of a causal structure. 

(Notice that Manheim already warned about the fact that single causation is often a fallacy.) The example of the factors both direct and indirect that impact the grade in a statistics class show that reducing the class to a grade eliminates the articulation points.

[In Shalizi's example] ... there are plenty of causes that can be manipulated to improve grades: reducing workload will be effective, as will increasing actual learning in the previous course. But if you are only using simple metrics, and which cannot represent the causal structure, it’s irreducible. This is why ... loss of fidelity matters when decisions are made.

 Manheim uses (cute) optical illusions to approach the reification problem, discussing the potential for the reification fallacy (at least) for metrics of IQ or wealth. The punchline though is:

What’s harmful is that when we create a measure, it is never the thing we care about, and we always want to make decisions. And if you reify metrics away from the true goal, you end up in trouble when they stop being good measures. 

Which is what Goodhart's Law argues, and Manheim now exemplifies:

Investors care about bond ratings, but only because they measure risk of default. It’s only a measure, until you use it to determine capital reserves. 

Bank regulators care about capital reserves, but only because it is a measure of solvency. It’s only a measure, until you use it to set bank reserve requirements. 

Manheim then points out that this is caused by Stephen Ross' formalization of the solution to principal-agent problems in economics, which are base-payment plus bonus type of systems, which however require measurements to succeed.

The combination of reification and decisions that use a metric which ignores the causal structure will bite you.  

Thinking of tests as measuring student achievement is fine, and it usefully simplifies a complex question. Reifying a score as the complex concept of student achievement, however, is incorrect.

For Manheim, Goodhart points out that the absence of any correct metric means that the system will drift to satisfy the mismatch between measure and goal.

Metrics make things better overall, but only occurs to the extent that they are effective at encouraging the true goals of the system. To the extent that they are misaligned, the system’s behavior will diverge from the goals being mismeasured.

Because the collapse of the complexity elides aspects of the system, the resulting measurement will push in unintended directions, be it sensationalism via user engagement at Facebook or racial bias in recidivism in crime statistics.

Manheim argues that another way to see Goodhart's Law is that mapping goals to measurements increases the communicability about complex systems between people, but the inaccuracy of the metric over time  causes drift that eventually obfuscates the intended goals.