Advertisement

Noise by Daniel Kahneman, Olivier Sibony and Cass Sunstein review – the price of poor judgment

Imagine you have committed a crime. If you are up on your behavioural economics you will be hoping to have your case heard either early in the day or just after a scheduled break such as lunch: a 2011 study of more than 1,000 rulings by eight judges found that those times coincided with the greatest leniency in judges’ rulings. Those who fared worst were heard at the end of the day or just before lunch, when there was about a zero chance of receiving a favourable ruling. How hungry or tired a judge is should have no impact on their ruling, and yet the data says it does.

But what about the judge who is assigned your case in the first place? That shouldn’t matter either but, yet again, the data says it very much does. A 1974 study of 50 judges setting sentences for identical (hypothetical) cases found that “absence of consensus was the norm”. And the sentences didn’t just slightly differ by judge: they varied wildly. Depending on the luck of the judge lottery, the same heroin dealer was sentenced to anything between one and 10 years, a bank robber received sentences ranging between five and 18 years, while an extortionist faced anything between three years with no fine at all to 20 years plus a $65,000 fine. Similar studies were repeated in 1977 and 1981, all with the same sobering findings – and they are likely to underestimate the scale of the problem, since according to the august authors of Noise,“real-life judges are exposed to far more information than what the study participants received in the carefully specified vignettes of these experiments”.

This scattergun variability in judgments of all kinds, from court sentencing to insurance underwriting to medical diagnosis, is what the authors call, well, noise. Like its more famous cousin, bias, noise is an error in judgment. The authors distinguish between the two using a shooting-range metaphor. If all the shots land systematically off-target in the same direction, that’s bias; by contrast, noise is all over the place. Some of the shots might even be on target, because the issue here is not missing the target but a lack of consistency. Given the same facts, one criminal gets life and another who is equally guilty gets off.

Which brings us to the other significant distinction between bias and noise: to detect bias, you have to know what the right answer is, or to use the book’s metaphor, you have to be standing at the front of the target, so you can see the bullseye. Noise requires no such particulars. It is detectable no matter which side of the target you’re standing on, since all you need to know is whether or not there is variability.

One insurance company executive estimated the annual cost of noise in underwriting in the hundreds of millions of dollars

And you should want to detect noise, the authors argue, because it is not only unfair, it can be hugely costly. For example, one study found that the average difference in insurance premium quotes depending on the underwriter is 55%. This means that if one underwriter sets the premium at $9,500, another is likely to set it at $16,700. This matters not just to the client, whose premium shouldn’t depend on whether their case is handled by John or Jane, it also costs the company: overpriced contracts lose business, while underpriced contracts lose money. One senior executive at the insurance company where the study was conducted estimated the annual cost of noise in underwriting in the hundreds of millions of dollars.

But while businesses and your average person on the street therefore have an interest in seeing noise reduced, we are mostly just ignoring it. There are all sorts of reasons for this, not least that, as the authors point out, humans are much better at thinking causally than statistically. This means it’s easier for us to attend to bias, where we can tell a story to explain an unexpected decision (he let her off because she looks like his daughter), than to noise, which is only visible in aggregate. We have to make an effort to see noise – or as the authors advise, conduct a “noise audit”, instructions for which they handily include in an appendix at the end. They also include a range of noise-reduction strategies, from substituting comparative judgments for absolute judgments (ie instead of grading essays one by one, putting them in order from best to worst), to replacing humans with the ultimate in noise-free decision-making: algorithms.

The problem is, not everyone who conducts a noise audit will go on to commit to noise-reduction strategies. For example, following the various studies in the 1970s and 80s highlighting unacceptable noise in sentencing, the US passed the 1984 Sentencing Reform Act, which established mandatory sentencing guidelines. These changes were successful in reducing noise but were wildly unpopular with judges who resented the removal of their discretion, and in 2005 the guidelines were downgraded to advisory. Noise went back up – but judges were happier. Whether or not justice was better served is another question.

Another example of an organisation choosing to put up with noise because the cure was considered too costly comes from a school that scrapped its admissions system because it was causing conflict. One of Noise’s most repeated recommendations for noise reduction (and there is a lot of repetition in this book) is that judgments should, where possible, make use of the “wisdom of the crowd”. This phrase refers to the finding that if you ask enough people a question you will almost invariably get a better answer than if you only ask one person – but there is a caveat: the opinions must be independent from each other in order to avoid “groupthink”. And this is how the school ran its application process: two people independently read and rated an application before making a joint decision. This made admissions less noisy but also led to arguments. The school chose to live with the noise.

As for algorithms, Daniel Kahneman et al lament, we are unwilling to tolerate mistakes in computers in the way that we tolerate them in humans. This may well be unreasonable of us, but on the other hand, when algorithms make mistakes they can be huge: gig workers in the US have been locked out of earning a living by trigger-happy algorithms erroneously detecting fraud; an algorithm designed by Amazon systematically downgraded female job applicants. Meanwhile, algorithms remain unreasonably opaque, their inner workings protected under proprietary software laws, meaning recourse is often impossible. “We’re not discriminating, it’s just the algorithm,” said Apple’s hapless customer service reps in response to a man whose wife was given a 20th of his credit limit despite having a higher credit score. Multiply this problem by a thousand for older people or anyone without good internet access.

To be strictly fair, the authors do acknowledge the existence of algorithmic bias, although they perhaps underestimate its magnitude. A crucial point they do not acknowledge, however, is that algorithms don’t merely replicate human biases, they amplify them – and by a significant amount. One that was trained on a dataset where pictures of cooking were 33% more likely to involve women than men ended up associating pictures of kitchens with women 68% of the time. Until these issues are ironed out we should beware of social scientists bearing algorithm-driven gifts.

The vague hand-waving over the serious societal implications of AI is all of a piece with a book that, while it undeniably has a point, and an important one, feels, to be blunt, half-baked. If ever there were a book in search of an editor, it is this one. Noise could have been half the length and it would have been a far better book for it. Instead, weighed down by flabby vignettes complete with imaginary (and terrible) dialogue that add nothing except pointless pages, it is a slog. This is disappointing given the authors’ previous output and it’s tempting to wonder the extent to which this study was a product less of an idea whose time had come than of a publisher’s desire for the next bestseller. Towards the end comes the line: “Noise is the unwanted variability of judgments, and there is too much of it.” Rather like the book itself, I found myself thinking.

• Noise: A Flaw in Human Judgment by Daniel Kahneman, Olivier Sibony and Cass Sunstein is published by William Collins (£25). To support the Guardian, order your copy at guardianbookshop.com. Delivery charges may apply.