Professor Olivier Sibony on Noise

External reference: https://www.spreaker.com/episode/professor-olivier-sibony-on-noise--45860612

cognitive bias noise, Olivier Sibony, human risk podcast

[00:00.000 –> 00:07.640] Welcome to Human Risk, a podcast dedicated to the understanding of human behaviour as [00:07.640 –> 00:10.220] a risk. Here’s your host, Christian Hunt. [00:10.920 –> 00:16.520] Hello and welcome back to the Human Risk podcast. On this episode, I’m exploring a dynamic [00:16.520 –> 00:22.300] of human decision-making that can lead to very undesirable outcomes. A cause, if you [00:22.300 –> 00:28.040] like, of human risk. Now, you’ve probably heard of cognitive biases, things like confirmation [00:28.040 –> 00:32.400] bias, where we have a tendency to seek out information that confirms what we already [00:32.400 –> 00:37.580] think and pay less attention to information that doesn’t. And we know that biases can [00:37.580 –> 00:42.220] lead to really bad outcomes. But there’s another dynamic that you might not have heard [00:42.220 –> 00:49.480] of that is equally, if not more, problematic. And that dynamic is noise. Noise is a useless [00:49.480 –> 00:54.820] variability in decision-making. In simple terms, it’s when people who are tasked with [00:54.820 –> 00:58.020] assessing the same thing reach varying conclusions. [00:58.040 –> 01:04.260] An example would be judges giving vastly different sentences to defendants who committed [01:04.260 –> 01:10.440] the same crime. If some judges give a one-month sentence, others one year, others seven years, [01:10.780 –> 01:15.700] and others somewhere in between, then the system they’re operating in is noisy. After [01:15.700 –> 01:18.760] all, we’d expect similar punishments for the same crime. [01:19.680 –> 01:24.800] Noise is the subject of a recently published book by three professors, Daniel Kahneman, [01:24.800 –> 01:27.360] Cass Sunstein, and Olivier Siboney. [01:28.040 –> 01:33.220] Daniel Kahneman is best known for being the author of Thinking Fast and Slow, a book that [01:33.220 –> 01:38.140] brought the idea of cognitive biases to the forefront that many of you will own and a [01:38.140 –> 01:43.480] few of you might even have read. Cass Sunstein is the co-author of Nudge, a book that led [01:43.480 –> 01:48.100] to the creation of bodies like the Nudge Unit or Behavioural Insights Team and led to the [01:48.100 –> 01:50.800] widespread adoption of behavioural science in government. [01:51.080 –> 01:57.640] The other author, Olivier Siboney, is my guest on this episode. He’s been on the show before, [01:58.040 –> 02:02.560] and there’s a link to that episode in the show notes. If you heard it, you’ll know that he’s [02:02.560 –> 02:09.220] engaging and insightful. In my discussion with Olivier, we explore what noise is, why it matters, [02:09.520 –> 02:14.420] and what we can do to mitigate it. We also touch on a subject that I’ve always found [02:14.420 –> 02:19.540] incredibly irritating, and I know that many of my listeners, particularly those of you that I [02:19.540 –> 02:25.760] worked with where this was used, do as well. That subject is forced distributions in performance [02:25.760 –> 02:28.020] evaluations. In other words, a recording of a subject that I’ve always found incredibly irritating, [02:28.020 –> 02:33.120] to ensure the performance rankings you give to your staff are distributed across the available [02:33.120 –> 02:37.840] grades, effectively meaning you’re forced to give some people grades that aren’t the ones you’d [02:37.840 –> 02:42.740] actually want to give them. So if, like me, you think forced distributions in performance [02:42.740 –> 02:47.860] evaluations are a really bad idea, then stay tuned to find out why we’re right. [02:48.580 –> 02:52.800] And if you’re one of the people that’s ever had a conversation with me telling me my evaluation [02:52.800 –> 02:58.000] distribution isn’t acceptable, then you’ll hear the argument I would have made instead of relying [02:58.020 –> 03:03.120] on gut instinct. If you’re working somewhere that still uses it, I hope this helps you. [03:03.700 –> 03:10.320] With all of that said, please enjoy my discussion with Professor Olivier Siboney on noise. [03:11.700 –> 03:14.580] Olivier, welcome back to the Human Risk Podcast. [03:14.740 –> 03:15.940] Thank you, Christian. It’s a pleasure. [03:16.580 –> 03:20.360] It’s an absolute thrill to have you back here. And last time you were on the show, [03:20.420 –> 03:25.040] we were talking about your book, and you mentioned the fact that you were in the process of writing [03:25.040 –> 03:28.000] another book, and you kindly offered to come back to talk about your book. [03:28.000 –> 03:31.040] And here you are. So can you tell us a little bit about Noise? [03:31.280 –> 03:35.700] And amazingly, this book actually happened, right? Because you know the planning fallacy. [03:35.820 –> 03:39.020] We have plans, and sometimes they don’t materialize. But this one did come out [03:39.020 –> 03:43.400] some time ago. It’s called Noise. It’s written jointly with Daniel Kahneman, [03:43.540 –> 03:47.120] who needs no introduction, and Cass Sunstein, who probably needs no introduction either. [03:48.020 –> 03:55.540] And it’s called Noise, a flaw in human judgment. And basically, the story of Noise is this. [03:55.540 –> 03:57.980] When we talk about bias, which I’ve been, [03:58.000 –> 04:02.560] for a while, and which Danny has been doing for a much longer while, and so has Cass, [04:02.820 –> 04:08.320] when we talk about bias, we talk about shared errors. We talk about average errors. We talk [04:08.320 –> 04:14.320] about the errors that the average archetypal human being makes. But in the real world, [04:14.980 –> 04:20.280] there isn’t a single human being making the same mistakes that every other human being is making. [04:20.440 –> 04:27.980] There isn’t an archetype of the biased human being. There are lots of different human beings [04:28.000 –> 04:32.060] who make slightly different mistakes, and sometimes not so slightly different. [04:32.440 –> 04:38.500] And the variability of those mistakes, the dispersion around the mean, is what we call [04:38.500 –> 04:46.460] noise. The concept, as many of your listeners will recognize, is well known in statistics and [04:46.460 –> 04:52.500] in measurement theory. Bias is the average error. Noise is the standard deviation of errors. So [04:52.500 –> 04:57.740] when we are making mistakes in judgment, just like when we’re making errors in measurement, [04:58.000 –> 05:02.660] we should take care of bias, of course, but we should also take care of noise. [05:04.320 –> 05:07.920] And so what would an example be for those people that haven’t come across the concept before? [05:08.340 –> 05:14.260] So let’s take a very simple example from the world of measurement, because that really brings it to [05:14.260 –> 05:22.640] life in a much easier way. Suppose that you have a cheap bathroom scale, and the bathroom scale [05:22.640 –> 05:27.880] that you have is a little bit forgiving. It tends to give you, on average, [05:28.000 –> 05:33.860] a weight that is, say, a pound less than you actually weigh, which is probably good news. [05:33.860 –> 05:41.200] At least for me, it would be good news. But that’s a bias. On average, the scale is wrong by [05:41.200 –> 05:48.920] minus one pound. Now, suppose that you also step onto your bathroom scale two or three times in [05:48.920 –> 05:54.080] quick succession, and like my cheap bathroom scale, it gives you a slightly different number [05:54.080 –> 05:57.840] each time, a slightly different reading each time. That’s variability in, [05:58.000 –> 06:00.920] in fact, the number of readings that should be identical, because your weight has not changed [06:00.920 –> 06:07.680] from one stepping on the scale to another. And it may be caused by slight changes in exactly where [06:07.680 –> 06:14.620] you put your foot on the scale, or how briskly you jump on the scale, or those kinds of things. [06:14.940 –> 06:21.080] It has a reason. It’s not purely random. It has a cause, but we don’t know what that cause is. And [06:21.080 –> 06:25.520] for practical purposes, it’s a random variation. That variability in measurement is noise. [06:26.140 –> 06:27.480] So bias is the average of the average. If you’re using a parameter, if you’re using a random [06:27.480 –> 06:27.980] variable, that can be a cause, and that can be a random variable. And if you’re using a stock value, [06:27.980 –> 06:28.000] it can be a random variable. But if you’re using a number value and a number value, it’s a random variable. [06:28.000 –> 06:32.420] errors that your bathroom scale makes. Noise is the variability of the errors on your bathroom [06:32.420 –> 06:39.180] scale. That’s the measurement idea. Now apply it to a judgment. For instance, a forecast. [06:39.860 –> 06:47.400] If we are all making forecasts, you and me and 98 other forecasters are making forecasts of what [06:47.400 –> 06:53.640] GDP growth is going to be next year. And on average, our forecast, what is called the consensus [06:53.640 –> 07:00.560] of the forecasters, the average forecast says GDP is going to grow by five points. And it turns out [07:00.560 –> 07:06.300] that in fact, it grows only, we only find this out in a year’s time, but it grows only by three [07:06.300 –> 07:11.860] points. We’ve collectively made a mistake. We’ve been biased in the direction of optimism. We’ve [07:11.860 –> 07:17.940] been too optimistic collectively about the forecast. But if you look at the 100 different [07:17.940 –> 07:23.020] forecasts, there is also a lot of variability. And if you look at the error that one particular [07:23.020 –> 07:23.620] forecast triggers, it’s going to be a lot of variability. And if you look at the error that [07:23.620 –> 07:28.980] is making, that error is not the average bias of the entire group of forecasters. It’s the average [07:28.980 –> 07:35.740] bias plus, plus or minus the noisy error that that forecaster is making. And in each of our [07:35.740 –> 07:40.100] judgments, this is an example of forecasting, but any other kind of judgment would be the same. [07:40.560 –> 07:45.820] There is a bias, which is the average error that the other people making the same judgment are [07:45.820 –> 07:52.700] also making. And there is another component, a residual component of the error when we [07:52.700 –> 07:53.600] compare it to the average error that the other people making the same judgment are also making. [07:53.600 –> 08:04.160] And that component is the noisy error, which is in aggregate noise. And the point of the book [08:04.160 –> 08:08.960] is that if we want to improve the quality of judgments, just like if we want to improve the [08:08.960 –> 08:13.460] quality of measurements, we need to worry about bias, but we also need to worry about noise. [08:13.900 –> 08:18.060] We’ve talked a lot about bias. We’re very concerned about bias. We hear a lot about bias, [08:18.060 –> 08:22.740] and we should. Bias matters. Let’s be clear about that. But noise matters too, [08:22.740 –> 08:27.740] and we want to redress the imbalance in the amount of attention that noise has been getting [08:27.740 –> 08:33.260] compared to bias. And what’s fascinating is, I mean, you talk in those terms, and clearly scales [08:33.260 –> 08:38.640] that make me look lighter than I am, not necessarily a massive issue in the scheme of things, or if it’s [08:38.640 –> 08:43.120] slightly variable, I can probably live with that if I bought cheaper scales. But the point you make [08:43.120 –> 08:47.700] in the book is that this has some really serious implications. And so you look at lots of examples [08:47.700 –> 08:52.000] which are really serious, and you take examples of the medical professional, the judiciary, [08:52.740 –> 08:56.080] where actually this impacts big decisions that matter societally. [08:57.140 –> 09:02.780] We’ve been struck by how many examples we could find. And we keep finding more, by the way. The [09:02.780 –> 09:07.540] funny thing is, since the book has come out, we get emails almost every day from people saying, [09:07.540 –> 09:15.100] you know, here’s noise in environmental assessments in Ethiopia. Here’s noise in [09:15.100 –> 09:22.580] the judicial system in France. Here’s noise in the medical diagnosis of [09:22.740 –> 09:31.580] benign prostate hypertrophy. Everyone who is faced with a judgment problem and who is thinking [09:31.580 –> 09:37.320] about it is having this forehead-slapping moment where they say, oh, my God, there’s a lot of noise [09:37.320 –> 09:42.420] in the judgments that we’re making. So here are some examples that we talk about at some length [09:42.420 –> 09:46.980] in the book. One that is especially striking and that I think brings the problem to life [09:46.980 –> 09:52.180] is the judicial system. And what’s interesting about it is that [09:52.740 –> 09:57.560] unlike the example of forecasting, which I was using, or the example of the bathroom scale, [09:57.860 –> 10:06.020] there is no truth there. There is no, you cannot say that the correct, the right, the true, [10:06.580 –> 10:14.700] the objectively accurate punishment for defendant X, given all the circumstances of that person’s [10:14.700 –> 10:21.420] crime and the extenuating circumstances of her life and whatever, you will never be able to say [10:21.420 –> 10:22.720] that the correct punishment is the correct punishment for defendant X. You will never be [10:22.740 –> 10:25.880] able to say that the correct punishment for defendant X is exactly seven years and three months in [10:25.880 –> 10:33.120] prison and a fine of $42,500. That doesn’t make any sense. There isn’t an objective way to assess [10:33.120 –> 10:39.620] what the true punishment is. Does that mean that we should be content with noise and that noise is [10:39.620 –> 10:45.480] not a problem? Actually, no, it does not mean that. Noise remains a problem even when we don’t [10:45.480 –> 10:49.560] know for sure what the true value is. Because here’s what we found, or in fact, here’s what [10:49.560 –> 10:51.860] a study found a long time ago. [10:52.740 –> 10:59.720] When the US was trying to deal with this problem. This study took 208 federal judges [10:59.720 –> 11:08.080] and gave them 16 different cases, very simplified, stylized cases of actual [11:08.080 –> 11:17.220] crimes, but with a lot less detail and a lot less distracting stuff than you would actually find [11:17.220 –> 11:22.580] in a courtroom. So you would expect, given that the cases were actually [11:22.740 –> 11:28.100] simplified, that there would be a lot more consistency, a lot more agreement between the [11:28.100 –> 11:32.300] judges than there would be in an actual courtroom. Because in an actual courtroom, you could say, [11:32.380 –> 11:37.320] well, I see this guy and he really looks sinister. So yeah, it doesn’t look bad on paper, [11:37.320 –> 11:42.060] but I really feel bad about him. There would be a lot of stuff that would actually distract you [11:42.060 –> 11:47.180] or bias you. So in essence, you’ve created more laboratory-like conditions. [11:47.180 –> 11:51.980] Not us. Not us. This was done by researchers back several decades ago. [11:52.740 –> 11:59.940] This piece of research was done quite seriously with vignettes that were actually [11:59.940 –> 12:06.240] clean cases. And so you would expect that there would be a lot more consistency in the [12:06.240 –> 12:12.500] judgments of the actual federal judges there than they have in their actual courtrooms. [12:12.500 –> 12:18.060] And it turned out that the discrepancies between the judgments of the judges were amazing. [12:18.060 –> 12:22.380] On almost all of the cases, there wasn’t any unanimity on whether [12:22.380 –> 12:27.380] there should be a prison sentence at all. So some of the judges said you go to jail, [12:27.500 –> 12:33.320] others said you don’t. On the average case, when the prison sentence was seven years on average, [12:34.180 –> 12:39.940] the dispersion around that average was so wide that if you took the mean, [12:41.080 –> 12:46.900] if you took a pair of judges at random in the pool of 200 judges, so any pair that you can [12:46.900 –> 12:50.500] create from those, you know, 208 times 207 judges, [12:52.380 –> 12:58.760] the mean, the median, sorry, difference between two judges would be about three and a half years. [12:59.620 –> 13:05.660] So that basically means that the minute you step into the courtroom and you have been assigned [13:05.660 –> 13:13.280] Judge Christian or Judge Olivier, that means your sentence is five years or roughly five years or [13:13.280 –> 13:18.700] nine years, you know, just because of the judge. That has nothing to do with you, nothing to do [13:18.700 –> 13:22.260] with the crime, nothing to do with the victim, nothing to do with the circumstances, nothing to [13:22.260 –> 13:22.360] do with the court. So it’s a very, very, very, very, very, very, very, very, very, very, very, very, [13:22.380 –> 13:27.660] with anything that we would regard as justice. I think that means we have a problem. [13:28.600 –> 13:35.500] We all agree, and I hope we all agree, that we shouldn’t have automatic sentences that don’t [13:35.500 –> 13:41.400] take into account the specifics of the defendant and the specifics of the crime and the specifics [13:41.400 –> 13:47.320] of that person’s history and so on. I think we all agree in civilized nations that some degree [13:47.320 –> 13:52.360] of individualization of sentencing is required to take into account all those things. [13:52.380 –> 14:00.220] But I don’t think anyone ever meant this to mean that we should individualize the sentence to the [14:00.220 –> 14:06.160] judge. It’s fine to individualize the sentence to the defendant and to the crime. It’s not so great [14:06.160 –> 14:12.500] to make it be so largely a function of the judge. And even though we don’t know what the correct [14:12.500 –> 14:17.880] sentence is, we can all agree, I hope, that it’s not good for that sentencing process to [14:17.880 –> 14:22.260] essentially be a lottery. And basically, it’s a lottery. So that’s the example of the sentence [14:22.260 –> 14:27.360] of justice. We have a lot more examples. The example of medicine is striking because here [14:27.360 –> 14:37.620] there are true values. You either have a disease or you don’t. The tumor on your x-ray image is [14:37.620 –> 14:45.220] either benign or malignant. And when you ask different doctors to look at those identical [14:45.220 –> 14:50.520] cases, they quite often have different opinions. And then there’s another thing that you notice, [14:50.520 –> 14:52.220] especially in medicine, but also, [14:52.260 –> 14:59.760] in justice, which is that even the same judge, when presented with the same facts on two different [14:59.760 –> 15:06.020] occasions, is not always consistent with him or herself. The same person looking at the same [15:06.020 –> 15:13.660] evidence can sometimes disagree with himself or herself because the circumstances have changed. [15:13.760 –> 15:19.580] The person is a different mood. The preceding case that the person has looked at was especially bad [15:19.580 –> 15:21.080] or especially easy. [15:22.260 –> 15:26.000] So there’s fatigue at the end of the day. There’s all kinds of circumstances that affect our [15:26.000 –> 15:33.160] judgment in ways that we are vaguely aware of, but I think we greatly underestimate the magnitude [15:33.160 –> 15:38.460] of the effect that they have. So there’s lots of examples. There’s many more in the book. [15:38.700 –> 15:43.420] And there’s, again, many more that we keep hearing about. There is noise everywhere. In fact, [15:43.880 –> 15:50.360] one of the mottos of the book is wherever there is judgment, there is noise and probably more of [15:50.360 –> 15:51.040] it than you think. [15:52.260 –> 15:59.900] I mean, it struck me that this is something that once you point it out, you can see it. And I don’t [15:59.900 –> 16:05.480] have to necessarily hear the case. I mean, those judicial cases are shocking, but I can sort of see [16:05.480 –> 16:10.720] it as I go around. And one of the things I found fascinating is you provided a lens through which [16:10.720 –> 16:14.060] I can start to see these things. And hence, you’ve become what I would sort of say noise hunters, [16:14.180 –> 16:19.060] almost, the people who are providing eyewitness accounts for you. And so the first question I [16:19.060 –> 16:20.660] wanted to ask you is, what is the first question that you’ve asked yourself on that basis? I mean, [16:20.680 –> 16:22.220] have you found yourself seeing the evidence? Have you found yourself seeing the evidence? [16:22.220 –> 16:22.240] Have you found yourself seeing the evidence? Have you found yourself seeing the evidence? [16:22.260 –> 16:27.200] Because it always struck me that behavioral science is a space where once things are pointed [16:27.200 –> 16:31.340] out to you, you kind of go, oh, and you’re starting to see the world through that lens. [16:31.420 –> 16:33.220] Are you spotting this yourself everywhere? [16:33.640 –> 16:42.420] A little bit. I’ll give you an example. My grading as a professor, as a teacher. So I’ve [16:42.420 –> 16:51.380] always graded always. I mean, since I’ve become a professor, I’ve had the task of grading the [16:51.380 –> 16:58.980] essays of my students at the end of each class. And I tried to do it as carefully and diligently [16:58.980 –> 17:06.320] as possible. But actually, I’ve become aware. I’ve tested myself, essentially. I’ve tried to look [17:06.320 –> 17:10.920] again at essays that I had graded without actually putting the grade on the actual essay [17:10.920 –> 17:15.580] and compared the grades that I was giving on two different occasions. And they were [17:15.580 –> 17:21.340] significantly different. So we’ll talk about the remedies to noise [17:21.340 –> 17:26.880] later. But I’ve implemented some remedies, some cheap remedies, because this isn’t a problem that [17:26.880 –> 17:32.080] is worth, and this is something we’ll talk about too, I’m sure. This isn’t a noise that is worth [17:32.080 –> 17:36.380] spending a lot of time and effort to reduce. These aren’t essays that are going to change [17:36.380 –> 17:42.640] the life or even the academic fate of my students. These are just essays that are going to be [17:42.640 –> 17:49.200] averaged with lots of other things to come up with their GPA. It’s not a case of getting or [17:49.200 –> 17:51.280] not getting admission into the school. That would be [17:51.340 –> 17:57.500] worth a lot of fine-tuning. So I’ve put in place some fairly easy solutions to reduce noise [17:57.500 –> 18:02.980] because I’ve become aware of it. But to your point, you were saying, have I become a noise [18:02.980 –> 18:11.480] hunter? Do we all become noise hunters? It’s actually a lot less easy to spot noise than it [18:11.480 –> 18:19.440] is to spot bias. And that’s, by the way, why we’re talking about noise now and we’re saying it’s a [18:19.440 –> 18:21.320] big problem. And that’s because we’re talking about noise now. And we’re talking about noise now. [18:21.320 –> 18:26.760] It’s a fair question to ask why this big problem has not been talked about more before like bias [18:26.760 –> 18:38.380] has. And I think the answer is at least twofold. One reason is that we don’t actually think about [18:38.380 –> 18:43.160] this a lot. When we’re making a judgment, we’re thinking hard about our judgment. When I grade [18:43.160 –> 18:51.300] an essay, I try to grade the essay right. I never actually asked myself, what would another [18:51.320 –> 18:57.460] professor think of this essay? I’ve only asked myself, what do I think of this essay? I’ve [18:57.460 –> 19:04.840] literally never asked the question, how would someone I know and trust and respect, but who is [19:04.840 –> 19:11.220] different from me, judge this essay? What would this person care about? Would this person be as [19:11.220 –> 19:18.540] impressed by the feature that impressed me or as put off by the feature that put me off in this [19:18.540 –> 19:21.300] essay, right? That’s the sort of mindset that you have to have when you’re making a judgment. [19:21.320 –> 19:28.480] You need to have if you want to reduce noise. But we don’t have that mindset because of what [19:28.480 –> 19:33.980] psychologists sometimes call naive realism, which is that we basically assume that we see the world [19:33.980 –> 19:41.380] as it is because that’s how it is. And we don’t pose to ask ourselves how others see the world. [19:41.440 –> 19:46.960] We just assume that anyone who is sensible, well-meaning and competent will see the world [19:46.960 –> 19:51.020] in the same way. That’s how we remain oblivious to noise. [19:51.320 –> 19:56.600] We don’t, unless we’re challenged by, in my case, I was challenged by the fact that I was writing a [19:56.600 –> 20:04.300] book on the topic. We don’t stop to ask ourselves, am I being noisy here? Because we don’t need to. [20:05.400 –> 20:11.160] And the other reason, which we may take a moment to talk about as well, is that organizations in [20:11.160 –> 20:17.340] general do a pretty good job of sweeping that problem under the rug. Basically, organizations [20:17.340 –> 20:21.300] are designed to hide noise, to mute noise, to hide noise, to hide noise, to hide noise. [20:21.320 –> 20:27.640] Not to mute noise, but to pretend that noise is not there and to make sure that we don’t become [20:27.640 –> 20:35.160] aware of it. Take the judicial system that we were talking about. If you are a judge and I am a judge, [20:35.240 –> 20:40.780] we see different cases. We never have the experience of actually judging the same case [20:40.780 –> 20:46.640] separately. Or if we work collegially, if there is actually several judges, if there is a panel [20:46.640 –> 20:51.060] of judges hearing the same case, they will actually talk to each other. [20:51.320 –> 20:57.720] And they will come to a consensus. And they will gradually gravitate towards a shared point of [20:57.720 –> 21:03.500] view. They will never actually have the experience of separately writing down what they think of the [21:03.500 –> 21:07.880] case and saying, oh, my God, we’re in complete disagreement. We saw a different trial here. [21:08.280 –> 21:12.980] Which is probably what you would see if they did that experiment. That’s what we call a noise audit, [21:12.980 –> 21:18.140] by the way. The experiment in the judicial system was de facto a noise audit. We’ve done [21:18.140 –> 21:20.900] several others. We found many other examples of noise audits. [21:21.320 –> 21:27.200] And unless organizations do noise audits, do experiments that expose the amount of noise, [21:27.200 –> 21:33.560] they don’t actually become aware of noise. So do we become noise hunters? I think we should. [21:34.140 –> 21:42.720] Part of the book is one big objective of the book is to turn as many people as possible and [21:42.720 –> 21:48.120] especially as many organization leaders as possible into noise hunters. But it doesn’t [21:48.120 –> 21:50.640] come naturally, which is why we haven’t all… [21:51.320 –> 21:53.360] You know, chased noise before. [21:54.080 –> 21:59.660] And what I found fascinating as I read it, and as you’re talking now, is that there are examples [21:59.660 –> 22:04.400] that we could pick from things that have been designed that in some level recognize this. So [22:04.400 –> 22:08.780] if I think of the devil’s advocate for the appointment of the Pope, if I think about [22:08.780 –> 22:12.620] the idea of juries, and I recognize it, you know, as you were talking, I thought, well, [22:12.620 –> 22:15.980] there’s juries. That’s why we have juries because we have… But of course, they sit in a room [22:15.980 –> 22:21.020] together. And there’s plenty of movies that illustrate how juries operate in practice, which is, [22:21.320 –> 22:25.960] and they’re required to come to agreement. But there’s at least a recognition there that we need [22:25.960 –> 22:30.860] to start thinking about these things. So on the one hand, I’m fascinated by the fact that we have [22:30.860 –> 22:36.340] certain processes that have existed that have tried to at least start to deal with this issue. [22:36.560 –> 22:41.580] And yet, on the other hand, we are also willfully blind towards it. And I’m fascinated by your [22:41.580 –> 22:44.880] thoughts on that. Is it just because this is incredibly difficult, it’s inconvenient to have [22:44.880 –> 22:49.500] to handle it? Or are we willfully blind to it? Or are we accidentally blind to it? [22:50.440 –> 22:51.300] Um, I don’t know. [22:51.320 –> 22:56.360] It’s probably sometimes willful, but I think most of the time, it’s accidental. I think most [22:56.360 –> 23:02.600] of the time, we are simply not aware of the magnitude of the problem. We do have situations [23:02.600 –> 23:08.620] in which we have done things to combat noise. And in fact, all the remedies against noise that [23:08.620 –> 23:15.220] we talk about in the book have been tried and tested in one domain or another, and sometimes [23:15.220 –> 23:21.240] in several domains. There’s two things that we should improve there, though. One, [23:21.320 –> 23:29.180] is that we sometimes do it just plain wrong. So when we say, we have different points of view, [23:29.180 –> 23:34.700] so let’s have a lot of people weigh in, which of course is what juries do, and which is what [23:34.700 –> 23:40.020] most companies do in their hiring processes. They have, you know, several people interview [23:40.020 –> 23:43.940] the candidates, and then they get together and they talk about the candidate and they come to [23:43.940 –> 23:48.720] a consensus. Or that’s what we do in a management team when we have a strategic decision to make. [23:48.840 –> 23:51.300] We go around the table and we ask everyone, [23:51.320 –> 23:54.400] in sequence, to tell us, so what do you think of this plan? And they say, well, [23:54.420 –> 23:58.720] I think it’s a good idea. Or they say, well, I’ve got this concern. And the next person [23:58.720 –> 24:04.340] piles up. And by the time you’ve gone around the table, the boss says, you know, I’ve heard you [24:04.340 –> 24:10.320] all, and this is what we’re going to do. And we think that this is the way to bring to bear [24:10.320 –> 24:16.840] the diversity of the points of view we have in the room. It’s wrong, unfortunately. It doesn’t [24:16.840 –> 24:21.300] work. That’s not the way to reduce noise. That’s the way to come to a consensus. [24:21.320 –> 24:26.740] For sure. And it will give you the warm, fuzzy feeling of having had a discussion and having [24:26.740 –> 24:33.720] brought everybody on board. But it, in fact, will amplify the noise. There is, you were mentioning [24:33.720 –> 24:39.820] movies about juries. There is actually research about juries that we discuss at some length in [24:39.820 –> 24:46.900] the book that shows that juries will become more polarized, will become more extreme in their views [24:46.900 –> 24:50.360] after they discuss than they were before the discussion. [24:51.320 –> 24:58.940] Basically means that the random effects of the composition of the group will be amplified by the [24:58.940 –> 25:05.480] deliberation. And the reason this happens is because in the deliberation, the first people [25:05.480 –> 25:12.020] who speak have an outsized influence on the rest of the group, especially if they speak with [25:12.020 –> 25:16.140] confidence, especially if the rest of the group has reason to trust their points of view, which [25:16.140 –> 25:21.140] maybe in a jury is not so much of a concern, but in an organization is almost always, [25:21.320 –> 25:25.900] always a big concern because organizations have hierarchies, they have experts, they have people [25:25.900 –> 25:31.320] who speak with confidence, et cetera. So that kind of deliberation, which we think of as a way to [25:31.320 –> 25:38.660] reduce noise, will, in fact, increase noise. You can reduce noise by having multiple inputs. You [25:38.660 –> 25:45.400] are, in fact, guaranteed to reduce noise if you aggregate multiple inputs on the condition that [25:45.400 –> 25:51.300] these inputs are independent, that the judgments are made separately and independently. [25:51.320 –> 25:58.140] of each other and that they don’t influence one another. If you do that, if you ask your 12 jurors [25:58.140 –> 26:04.040] in the jury room to set a sentence and then you take the average of the sentence, you will have [26:04.040 –> 26:09.120] less noise than you have after the jury deliberates. But of course, that’s not how it’s done. And for [26:09.120 –> 26:14.660] other reasons, it probably wouldn’t be a good idea. So, you know, long story to try to make a [26:14.660 –> 26:21.300] simple point. We think of some ways to reduce noise. We think of some of the approaches, [26:21.320 –> 26:27.960] that we have, especially of deliberation, as ways to reduce noise. They are not so great [26:27.960 –> 26:35.800] at reducing noise. The second part of the answer to your question, of why we don’t reduce noise so [26:35.800 –> 26:41.560] well, is that I think we greatly underestimate the magnitude of the problem. So here’s a story, [26:41.560 –> 26:47.720] which we talk about also in the book. We talk about an experiment in an insurance company, [26:48.520 –> 26:51.080] where we worked with the underwriters and the [26:51.080 –> 26:51.260] clients of the insurance company, where we worked with the underwriters and the clients of the [26:51.260 –> 26:51.300] clients of the insurance company, where we worked with the underwriters and the clients of the [26:51.300 –> 26:55.540] claims adjusters, people who make quantifiable judgments. They say, you know, this insurance [26:55.540 –> 27:03.440] policy should be priced at 100,000 euros, or at 80,000 euros, or 110,000 euros. And they’re [27:03.440 –> 27:08.880] experts. They are not just pulling the number out of thin air. They’re saying, you know, I’m applying [27:08.880 –> 27:14.500] a method. I have rules. There’s a procedure. There are norms. I’ve been trained. You know, I’m an [27:14.500 –> 27:20.520] expert in this topic. But of course, there are several experts, and they’re essentially interchangeable. [27:20.520 –> 27:27.580] Whether your quote is going to be given by expert A or expert B is a function of who happens to be [27:27.580 –> 27:34.700] available when your request for a quote comes in. And we asked the management team of the insurance [27:34.700 –> 27:41.100] company, how much difference do you expect you would find between two underwriters or two claims [27:41.100 –> 27:45.720] adjusters, those are the people putting a price on the claims, you know, between two experts who [27:45.720 –> 27:49.260] are applying the same method and are looking at the same case? [27:49.260 –> 27:55.540] And they said, well, of course, we don’t expect them to agree perfectly, right? We realize that [27:55.540 –> 28:00.000] this is a matter of judgment. And when we say a matter of judgment, by definition, we imply that [28:00.000 –> 28:05.940] some degree of disagreement between reasonable and competent people is to be expected. We can’t [28:05.940 –> 28:12.060] expect perfect agreement. But we expect that roughly something along the lines of 10% [28:12.060 –> 28:17.500] would be acceptable. And from a business standpoint, that would be tolerable. And that [28:19.260 –> 28:24.060] basically, these people have the process under control that they’re applying the same methodology. [28:25.020 –> 28:34.620] Do the experiment, what do you find? It’s not 10%. It’s not 20%. It’s not 30%. It’s not 40%. It’s 55%. [28:35.900 –> 28:42.540] It’s more than five times larger than what all the people in the organization were expecting. [28:43.100 –> 28:49.180] So the problem was not that they were unaware of the existence of noise conceptually, of course, [28:49.260 –> 28:52.580] they knew that when you ask people to make a judgment, they are going to disagree. [28:53.320 –> 28:59.700] But they had no idea that it was that large. And this is the experience that we’ve had essentially [28:59.700 –> 29:04.360] everywhere where we’ve done a noise audit. Basically, people always expect that there [29:04.360 –> 29:08.600] is going to be some variability, but it’s more than they think. And in that case, way more than [29:08.600 –> 29:13.320] they think. That’s huge. By any stretch of the imagination, financially, and in terms of the [29:13.320 –> 29:18.400] outcomes it produces. And yet these are people that are experts who really should know, quote, [29:18.400 –> 29:19.160] unquote, what they’re doing. [29:19.260 –> 29:25.340] Yeah, well, in all the professional judgments that we’ve focused on, we’re talking about [29:25.340 –> 29:31.460] professionals who are trained, who are experts, and who do not at all expect such degrees of [29:31.460 –> 29:38.440] disagreement. You do not expect radiologists to disagree on what they look at as often as they do. [29:38.900 –> 29:44.600] You don’t expect, here’s another one that is quite striking, you don’t expect fingerprint [29:44.900 –> 29:49.240] examiners, people who look at fingerprints and say, oh, yes, [29:49.260 –> 29:52.520] this is a match for this suspect, or no, this is not a match for this suspect. [29:52.880 –> 29:56.740] So these are the ones that we see on TV shows solving those really difficult crimes, [29:56.960 –> 29:59.840] and they nail that criminal because they’re really good. [30:00.040 –> 30:06.920] These are CSI, yeah. I mean, these are the people we’ve been trained to consider as infallible, [30:07.140 –> 30:12.900] right? I mean, if you’re in the jury box, and the defendant says, no, I didn’t do it, [30:12.920 –> 30:17.240] I’m innocent. And then there’s a guy who comes in and says, well, it’s his fingerprint on the [30:17.240 –> 30:17.640] weapon. [30:19.260 –> 30:24.160] You’re not going to hesitate very long, right? You’ve been told that basically a fingerprint is [30:24.160 –> 30:28.680] a fingerprint is a fingerprint. You’ve been told essentially that this is not a matter of judgment, [30:28.980 –> 30:33.640] that this is a matter of fact. It’s either his fingerprint or it’s not. But basically, [30:34.260 –> 30:42.200] you expect that this is not a matter of judgment. In fact, it is a matter of judgment. It is a [30:42.200 –> 30:48.280] judgment because we think of fingerprints, and I realized that my idea of what fingerprints are [30:48.280 –> 30:49.240] was completely wrong. [30:49.260 –> 30:57.380] When looking at the work of the researcher who studied this, called Etiel Dror, who is at UCL in [30:57.380 –> 31:03.160] London. My idea, and probably your idea of fingerprints, is what you do when you are at [31:03.160 –> 31:09.700] the border control, and you have this clean window where you take your designated finger, [31:09.860 –> 31:16.900] usually the right index, and you press it firmly to produce a clean image, which can then be [31:16.900 –> 31:19.200] compared with an other equally clean image. [31:19.260 –> 31:25.420] That’s very easy. That’s not a matter of judgment, and in fact, it’s automated. [31:26.360 –> 31:32.220] When you actually look at the fingerprint that has been left on the murder weapon or on the door [31:32.220 –> 31:39.200] handle in the crime scene or something like that, well, the criminal is not as careful about leaving [31:39.200 –> 31:41.620] a fingerprint as you are when you’re at the border control. [31:42.140 –> 31:42.620] Surprisingly. [31:43.100 –> 31:47.120] Surprisingly, right? I mean, he doesn’t obligingly press his finger. [31:47.120 –> 31:48.740] Or she, we should add, right? [31:48.740 –> 31:49.220] Or she. [31:49.260 –> 31:58.600] So it takes judgment to determine whether that partial, smudged, overlapping trace that has been [31:58.600 –> 32:04.780] left, called a latent print, actually fits the fingerprint that has been collected from a [32:04.780 –> 32:10.220] suspect. And those people are extremely competent. They’re well-trained. They’re extremely careful. [32:10.400 –> 32:16.960] By the way, I should stress that when they make mistakes, when they hesitate, they always err on [32:16.960 –> 32:18.520] the side of caution, of not… [32:19.260 –> 32:26.860] Wrongly identifying someone as guilty. So they probably make a lot more mistakes that lead to [32:26.860 –> 32:34.700] unwanted exonerations than mistakes that lead to wrongful convictions, essentially. [32:35.460 –> 32:40.820] But we should realize that they’re human beings. However competent, however well-trained, however [32:40.820 –> 32:45.580] well-organized, they’re human beings, and their judgments are judgments. And because they are [32:45.580 –> 32:49.100] judgments, there is variability. Roughly… [32:49.100 –> 32:49.240] Roughly… [32:49.240 –> 32:53.320] I mean, it depends on the circumstances and how you do the tests and who does the test, [32:53.400 –> 32:58.200] because oddly enough, when the test is done by the FBI, it seems to be a lot more… [32:59.800 –> 33:05.140] There is a lot less variability than there is when it’s done by independent researchers. [33:06.200 –> 33:12.280] But basically, somewhere between 5% and 10% of the time, you’re going to have a different judgment [33:12.280 –> 33:17.520] on a pair of fingerprints. That’s not a lot. Of course, it’s a lot less than you have in the [33:17.520 –> 33:18.720] example of the insurance company. [33:19.240 –> 33:24.020] But it’s a lot more than we expect. We would expect something very, very, very close to zero. [33:24.300 –> 33:28.920] Right. You talked about sort of things that we don’t think of as judgments. And if we move into [33:28.920 –> 33:34.360] the sort of sphere of ethics, where on the one hand, it clearly is a matter of judgment, [33:34.680 –> 33:39.080] and there’s a lot of nuance there. On the other hand, you can come at it depending on how you [33:39.080 –> 33:44.560] have been educated and brought up. You can have very different views on things, and therefore, [33:44.680 –> 33:47.920] it feels less like a judgment. How does noise impact those sorts of decisions? [33:47.920 –> 33:48.880] So there’s… [33:49.240 –> 33:56.580] There’s at least two ways to answer this. The first is that within person noise, [33:56.580 –> 34:01.640] what I was mentioning earlier when I was talking about judges being in a different mood [34:01.640 –> 34:09.720] when the weather has changed, or professors grading essays who are tired at the end of the [34:09.720 –> 34:14.380] day or something like that. That’s what we call occasion noise. That really shouldn’t exist, [34:14.580 –> 34:18.640] right? It shouldn’t affect your judgment if you are careful, but because we’re human beings and [34:18.640 –> 34:24.580] we’re not machines, it does. Does it affect your ethical judgments? You would think that your [34:24.580 –> 34:30.880] ethical judgments would be grounded in your fundamental beliefs and your reflections about [34:30.880 –> 34:36.920] the nature of good and evil and so on and so forth. It actually turns out that some… [34:36.920 –> 34:41.880] When you use things like the trolley problem or variants of the trolley problem, which are [34:41.880 –> 34:48.620] classic experiments in ethics, the answers that people give will vary depending, for instance, [34:48.640 –> 34:58.460] on their mood. Whether people act deontologically or in a utilitarian way is affected by whether you [34:58.460 –> 35:04.320] have shown them a movie clip that puts them in a good mood or in a bad mood. This is bizarre, [35:04.500 –> 35:10.240] right? Because we think of ethical decisions within person as something that deeply reflects [35:10.240 –> 35:17.000] who we are and what we think and what we care about and so on. And oddly enough, we are a lot [35:17.000 –> 35:18.620] less consistent with ourselves. We are a lot less consistent with ourselves. We are a lot less [35:18.640 –> 35:24.520] consistent with ourselves in our ethical values than we would assume. That’s one thing. The other [35:24.520 –> 35:31.940] thing is, ethics are, of course, a question of values and different people can have different [35:31.940 –> 35:37.420] values and different priorities and can have different judgments. And there is no absolute [35:37.420 –> 35:43.680] truth there. What does that mean in terms of between-person differences? It means that [35:43.680 –> 35:47.900] your ethics and my ethics can be different. We can have a different judgment about [35:47.900 –> 35:48.620] how we think about ourselves. We can have a different judgment about how we think about [35:48.620 –> 35:58.480] how horribly wrong it is for, say, a Volkswagen to cheat on its emissions tests or how terribly [35:58.480 –> 36:05.480] unacceptable it is for a student to engage in plagiarism or something like that. We can have [36:05.480 –> 36:13.200] different views. The problem comes in if you and I both have to make decisions based on our ethics [36:13.200 –> 36:18.060] within the same organization. So if we are both members of [36:18.620 –> 36:27.080] different panels sitting to judge ethical violations in the university, and you and your [36:27.080 –> 36:33.680] panel think that a mild act of plagiarism is not a big deal, if it’s a first-time offender, [36:33.860 –> 36:38.960] we should basically give this student a slap on the wrist and tell them not to re-offend. [36:39.520 –> 36:46.280] And I think it’s a terrible crime against the ethics of research and knowledge, and the person [36:46.280 –> 36:47.520] should be expelled immediately. [36:48.620 –> 36:53.160] And depending on whether that person is assigned to you or to me, their fate is going to be [36:53.160 –> 36:59.080] completely different. The university has a big problem of system noise, because it should not [36:59.080 –> 37:05.920] make decisions based on the happenstance of who happens to be in the room when the decision is [37:05.920 –> 37:11.860] made. And it’s completely normal and expected and fine that we have different ethical views [37:11.860 –> 37:18.600] as individuals. But when those ethics become part of the standards that are used in the university, [37:18.600 –> 37:24.920] and are applied inside an organization, we want some consistency. We want some uniformity. We want [37:24.920 –> 37:30.400] some homogeneity in how the standards are applied. And we want those ethical norms to be defined in a [37:30.400 –> 37:37.200] way that ensures some homogeneity. Otherwise, we have a randomness which does not bode well for [37:37.200 –> 37:43.360] justice, and also, by the way, destroys the credibility of the organization. It’s very hard [37:43.360 –> 37:48.280] to be credible in enforcing sentences against ethical violation. [37:48.600 –> 37:54.120] If it’s completely random, if it seems to, or if it appears to be completely random. [37:55.200 –> 38:00.660] And I want to come on to talk about some of the ways that we can mitigate noise. But before we do [38:00.660 –> 38:06.300] that, I was interested in your views. I mean, it’s presented in the book as a negative thing, [38:06.400 –> 38:10.640] something we need to look out for. But it occurred to me there might be circumstances under which [38:10.640 –> 38:14.060] actually it could be a positive force. And I wondered your thoughts on that. [38:14.060 –> 38:18.580] Well, we’ve solved this problem with a definitional trick. We’ve defined, [38:18.600 –> 38:23.200] we’ve defined noise as unwanted variability in judgments that should be identical. [38:24.280 –> 38:30.600] So there are lots of situations where variability is not noise. Or you would say where noise is not [38:30.600 –> 38:34.620] bad. We just say, well, it’s variability, and we don’t call it noise. It’s the same thing. [38:35.500 –> 38:41.000] And those would be all the situations, and there are a few. There would be all the situations where [38:41.000 –> 38:47.600] you want creativity. You want divergence. You want different people to pursue different ideas. [38:48.600 –> 38:54.120] And you would want that, essentially, whenever the variation is going to be followed by selection. [38:54.880 –> 39:01.640] So if you have, you know, 10 teams of researchers, either in different companies or even within the [39:01.640 –> 39:06.220] same company, who are trying to solve the same problem, say they are looking for the cure for [39:06.220 –> 39:12.400] COVID, you very much want them to be pursuing different routes, different avenues towards the [39:12.400 –> 39:18.480] solution, different technological ideas, different ways to solve the problem. Because, [39:18.480 –> 39:24.620] at the end of the day, you’re going to see what works. And, you know, the one or the ones that [39:24.620 –> 39:34.340] work will win, and the ones that didn’t work will lose. And that’s how evolution, or the equivalent [39:34.340 –> 39:39.680] of evolution in the market system functions. You have variation, then you have selection. [39:40.660 –> 39:45.200] Contrast this with the underwriters in the insurance company, right? You get assigned [39:45.200 –> 39:47.440] randomly to one underwriter or another. [39:48.480 –> 39:52.220] And you never hear what the other underwriters would have said. And there is no selection. The [39:52.220 –> 39:56.800] underwriter gets no feedback. He doesn’t get promoted because he had a good judgment or [39:56.800 –> 40:02.140] demoted because he had a bad judgment. There is variation without selection. Look at the judges [40:02.140 –> 40:05.820] in the different courtrooms. If they have different judgments, they will never know. [40:05.980 –> 40:10.620] They’re not competing against each other. So, you know, there are some situations where variability [40:10.620 –> 40:15.820] is good and desired because there will be selection. And there are some situations where [40:15.820 –> 40:18.100] it’s not good because it’s not good. [40:18.480 –> 40:23.420] Because there is no selection. There are also, of course, all kinds of situations where [40:23.420 –> 40:30.560] we don’t care about variability, or in fact, we welcome variability. When you read, you know, [40:30.560 –> 40:35.400] several reviews of the same book, you want them to be different. When you read several reviews of [40:35.400 –> 40:40.320] the same movie, you want to hear that, you know, Olivier liked it, Christian didn’t like it, [40:40.580 –> 40:43.700] and you want to hear why. You’re interested to have different points of view. [40:43.700 –> 40:47.940] You don’t want consistency there. You don’t want consistency in tastes. [40:48.480 –> 40:53.320] Tastes are not a matter of judgment. Opinions are not a matter of judgment. You don’t expect [40:53.320 –> 41:00.400] people to agree. Variability is not unwanted. It is wanted. So, in anything that is competitive, [41:01.140 –> 41:06.440] that makes a market, or that has a selection process, and in anything that is a matter of [41:06.440 –> 41:14.020] opinion or taste, variability is fine. And you want as much of it as you can. But we tend to forget [41:14.020 –> 41:17.660] that there are lots of professional judgment situations [41:17.660 –> 41:18.460] where we don’t care about variability. And we tend to forget that there are lots of professional judgment [41:18.480 –> 41:23.980] situations where we say, oh, diversity is wonderful. Originality is great. You know, [41:23.980 –> 41:29.680] being our true selves and bringing our authentic points of view to every problem that we encounter [41:29.680 –> 41:34.780] is what we are here for. And actually, that results in a lot of noise, which is a source [41:34.780 –> 41:38.600] of error for the organization, a source of injustice for the people who are exposed to [41:38.600 –> 41:42.140] its decisions, and a source of inconsistency and loss of credibility. [41:42.540 –> 41:44.880] I was struck when you were talking about reviews. It was really interesting, actually, because [41:44.880 –> 41:48.460] you know that thing where you buy something and then you, maybe this is just me, you read the [41:48.480 –> 41:52.060] reviews afterwards to make sure that you’ve made a good decision. So I was laughing because I was [41:52.060 –> 41:57.480] thinking, before I buy a product, right, I want that variety. Afterwards, I just want people to [41:57.480 –> 42:00.760] confirm that I’ve made the right decision. Well, here’s an even bigger way. Don’t read [42:00.760 –> 42:06.900] the reviews afterwards. Just read them before. Yeah, totally, totally. We’ve talked a lot about [42:06.900 –> 42:12.060] the sort of the challenge of it. Can you talk us a little bit around decision hygiene and noise [42:12.060 –> 42:15.080] audits and some of these solutions to the challenge that we face? [42:15.080 –> 42:18.340] Right. So we’ve talked a lot about the problem. Here’s the good news. The good [42:18.480 –> 42:24.200] news is that this is actually a tractable problem. You can measure noise and you can [42:24.200 –> 42:32.700] reduce noise. And here’s what’s beautiful about noise. Unlike bias, you don’t need to know the [42:32.700 –> 42:39.840] truth to reduce noise. To reduce bias, you need to know the true value, right? You need to know [42:39.840 –> 42:46.080] what is correct. If we’ve made a forecast, and we forecast on average, you know, 5% GDP growth, [42:46.640 –> 42:48.460] I won’t know what the bias is. [42:48.480 –> 42:51.760] I won’t know what the bias is in the forecast until we are at the end of next year. And I found [42:51.760 –> 42:58.960] out that the actual growth was plus three. So my bias was plus two. So, you know, in many cases, [42:59.320 –> 43:04.380] I have to wait, or in fact, it’s impossible to know what the true value is. Take the example [43:04.380 –> 43:10.380] of the judges again. I can’t reduce the bias if I don’t have an idea of what the true sentences [43:10.380 –> 43:17.920] should be. Maybe you think that on average, the sentences are too harsh in your country, [43:17.920 –> 43:18.460] and maybe I can’t reduce the bias. But I can’t reduce the bias if I don’t have an idea of what [43:18.460 –> 43:20.840] the true sentences should be. So maybe I think that on average, the sentences are too lenient. [43:21.300 –> 43:25.580] Maybe you think that for certain types of crimes, you know, maybe you think that on average, [43:25.720 –> 43:30.760] for rape, sentences are too lenient. And on average, for drug offenses, they are too severe. [43:31.220 –> 43:36.980] That would be an example of bias. But to have that discussion, we need to have a standard of [43:36.980 –> 43:41.180] what the true value, what the correct sentence for these different types of crimes would be. [43:41.600 –> 43:47.600] And that’s very hard to have. Without knowing that, and without discussing that, and without [43:47.600 –> 43:48.440] having any idea of what the true value is, we don’t have a standard of what the true value is. [43:48.460 –> 43:53.860] What the correct sentence would be for any particular type of crime. We can both agree [43:53.860 –> 44:00.240] that wide variations in the sentences that are given by different judges to the same crimes [44:00.240 –> 44:07.360] are a problem. Noise is a problem, even if we don’t know what bias is. So even in situations [44:07.360 –> 44:15.060] where we don’t know, and even in situations where we cannot define what the correct answer would be, [44:15.220 –> 44:18.340] which are the bulk of judgment problems, [44:18.460 –> 44:24.440] we can take steps to reduce noise because noise is always bad. Variability is always bad, [44:24.520 –> 44:29.680] even when you don’t know what the true value is. I think that’s important to bear in mind because [44:29.680 –> 44:36.700] it’s actually not intuitive. We tend to think that to reduce errors, we first need to know what the [44:36.700 –> 44:46.280] error is. It’s kind of odd to say, I can improve your forecasts without knowing how wrong they are. [44:48.460 –> 44:54.960] But in fact, I can. If I make the 10 forecasts of 10 forecasters in the same [44:54.960 –> 45:01.620] institution less noisy, I will have reduced error, at least if you measure it the correct way, [45:01.700 –> 45:06.160] which is the mean squared error. I will have reduced error in the same manner that I would [45:06.160 –> 45:11.800] if I reduce the average error. Now, I will not have done anything to bias. The average error [45:11.800 –> 45:17.420] will still be what it is, but I will have reduced the mean squared error, which is [45:17.420 –> 45:18.440] one standard error. I will have reduced the mean squared error, which is one standard error. [45:18.460 –> 45:24.020] of measuring error, and probably the appropriate one in this case. And that makes a big difference. [45:24.140 –> 45:28.180] And I will have reduced, if I’m talking about something like the justice system, I will have [45:28.180 –> 45:36.120] reduced the random component of the sentencing, the lottery of sentencing, which is probably a [45:36.120 –> 45:43.280] good thing to do. So how would you do that? First, you would need to measure it and do what we call [45:43.280 –> 45:48.440] noise audit. I’ve mentioned it. A noise audit is essentially an experiment where you give [45:48.440 –> 45:54.100] the same cases to different judges, and you measure how different their judgments are. You [45:54.100 –> 45:58.520] can do this with forecasts. You can do this with sentences. You can do this with grading essays. [45:58.740 –> 46:02.780] You can do this with medical diagnosis. You can do this with fingerprints. You can do this with [46:02.780 –> 46:07.520] any kind of judgment where you can take the same case and show it to different people, [46:07.840 –> 46:12.060] and you measure how big the discrepancies are. If you find that they’re not that large, [46:12.060 –> 46:18.360] that the amount of noise that you find is actually tolerable from an operational [46:18.440 –> 46:23.760] point of view, you’re fine. You don’t need to do anything, right? I mean, we encourage you to [46:23.760 –> 46:28.300] measure it. We think that more often than not, you’re going to find that it’s bigger than you [46:28.300 –> 46:35.500] think. But if it’s not a problem, and there’s nothing to do, that’s good news. Now, if it is [46:35.500 –> 46:40.840] a problem, there’s a number of things you can do, and we call those things, we have a sort of [46:40.840 –> 46:45.960] headline for those things that you mentioned, which we call decision hygiene. The reason we [46:45.960 –> 46:48.420] have this slightly off-putting, but it’s not a problem, is because we have a sort of [46:48.440 –> 46:56.600] name, and it’s intentionally off-putting, is to remind you that it’s a little bit tedious, [46:57.620 –> 47:02.900] right? It’s not going to be fun all the time. It’s going to be fun at times, but it’s not going to be [47:02.900 –> 47:07.780] fun all the time. It’s a bit like washing your hands, right? We know that we should wash our [47:07.780 –> 47:12.500] hands, especially in the middle of a pandemic. But, you know, if you work in a hospital, you [47:12.500 –> 47:18.420] know that you should wash your hands frequently, and you observe, sadly, that people are sometimes [47:18.420 –> 47:22.880] not doing it as regularly and as well and as frequently as they should, because it’s a bit [47:22.880 –> 47:28.520] boring. And here’s where the analogy is important. When you wash your hands, you don’t know what [47:28.520 –> 47:35.520] problem you’re solving. It’s prevention. You’re not actually saying, hey, here is the COVID germ [47:35.520 –> 47:41.920] that I’ve eliminated, and here’s the bacteria for that disease that I’ve taken up. Ah, gotcha. [47:41.920 –> 47:47.940] So you don’t say that. It’s prevention. If it works, you will never know what it accomplished. [47:48.420 –> 47:54.260] If it works, you will never know what disease you’ve avoided. Now, contrast this with bias. [47:54.580 –> 48:00.300] If I say, oh, there is a bias in your organization, you have a gender bias in your recruiting, [48:00.520 –> 48:05.100] you are hiring far too many men and not enough women, we’re going to fix this. You see the [48:05.100 –> 48:10.500] problem, you measure the problem, you fix the problem, that feels good, right? You’ve solved [48:10.500 –> 48:17.540] the problem of bias. Bias is something you can see. It has a cause. It is identifiable. Noise is [48:17.540 –> 48:18.400] very hard to see. It’s a problem of bias. It’s a problem of bias. It’s a problem of bias. It’s a [48:18.420 –> 48:23.060] problem of bias. It’s a problem of bias. And if you succeed at reducing it, you will not easily [48:23.060 –> 48:28.700] know what progress you’ve made. You will need to do another noise audit to measure how much [48:28.700 –> 48:35.780] progress you’ve made. So noise reduction, the prevention of noise, decision hygiene, as we call [48:35.780 –> 48:42.420] it, is essentially prevention. It’s redesigning your decision process to incorporate noise [48:42.420 –> 48:48.300] reduction practices that are going to make your decisions better without knowing exactly what [48:48.420 –> 48:52.140] problems you’re avoiding. A lot of people listening to this will work in compliance and [48:52.140 –> 48:58.420] audit and risk and control functions. And so very often, the way that we check things are working [48:58.420 –> 49:04.580] is have we followed a due process? We very often don’t question the basis of that process and say [49:04.580 –> 49:08.140] to ourselves, actually, so there’s almost an implicit presumption. We have designed this [49:08.140 –> 49:13.420] thing perfectly. And when we have people not following procedure, we have bad people and we [49:13.420 –> 49:18.060] need to just reprogram the people. We don’t go and question necessarily the efficacy of the process [49:18.420 –> 49:22.860] or design of the process. And what I find really interesting is you’re talking here, if I were to [49:22.860 –> 49:27.840] take just the humble meeting as an example, which has a traditional structure, you’re basically [49:27.840 –> 49:33.100] saying, actually, you should look at the way you do these things, because a lot of conventional [49:33.100 –> 49:38.560] wisdom around what works potentially might not. Yeah. And when you say rethink the process, [49:38.760 –> 49:44.040] we’re all for process, just to be clear. A lot of those decision hygiene practices, [49:44.040 –> 49:48.300] if implemented, are going to result in processes. And, [49:48.420 –> 49:53.640] in fact, some people are concerned when they hear about this, about the danger of adding [49:53.640 –> 49:59.640] bureaucracy and adding red tape. And that is, they’re right. This is a danger. And this is [49:59.640 –> 50:03.940] something that is a trade-off. And we should be very careful about that trade-off. We don’t want [50:03.940 –> 50:12.500] to kill all initiative and eliminate all sense of autonomy and agency in people. And that’s a very [50:12.500 –> 50:17.660] serious concern. At the extreme, the way to eliminate noise that would work, [50:18.420 –> 50:23.480] would be to simply eliminate any human judgment and to alternate everything, right? And that [50:23.480 –> 50:28.800] would solve the problem of noise. In fact, it’s the only way that would eliminate noise entirely, [50:28.800 –> 50:34.160] because when we say, wherever there is judgment, there is noise, that means that the only way you [50:34.160 –> 50:40.440] can totally eliminate noise is by eliminating judgment. So, let’s assume that we don’t want [50:40.440 –> 50:45.520] to go there. We could argue about the many reasons why we don’t want to go there, but let’s assume [50:45.520 –> 50:48.400] that we’re going to, since this is about human risk, we’re going to eliminate noise entirely. [50:48.420 –> 50:54.020] Let’s assume that we’re talking about human judgment and not about replacing humans with [50:54.020 –> 51:02.240] machines. That is going to mean having some processes. And what we suggest is rethink those [51:02.240 –> 51:08.200] processes, rethink those rules, rethink those ways of working to make sure that they actually [51:08.200 –> 51:13.320] address noise, not just biases that you’ve identified and that you want to combat. You [51:13.320 –> 51:18.360] shouldn’t do that. That’s fine. But also that you address noise. Let’s take the example of the [51:18.420 –> 51:23.620] meeting, as you were talking about. A very simple practice in meetings that you can adopt tomorrow [51:23.620 –> 51:30.580] morning in any meeting is, before people give their opinion on whatever it is that the meeting [51:30.580 –> 51:37.820] is discussing, have them write down their independent judgments on what the decision [51:37.820 –> 51:44.360] should be, on what the outcome should be. Do not let them influence each other as much as you [51:44.360 –> 51:48.400] typically do before they speak. It’s not even a process, right? I mean, [51:48.420 –> 51:52.020] you could call this a process, but this is really a meeting management technique. [51:52.660 –> 51:57.140] This will do a lot to make sure that you hear the noise, that you hear the diversity. [51:57.960 –> 52:02.640] Then use a number of possible techniques to aggregate those judgments. You could [52:02.640 –> 52:07.940] use the technique that is sometimes called estimate-talk-estimate, where you have those [52:07.940 –> 52:12.660] independent judgments. Then you ask people to explain their judgments and to justify their [52:12.660 –> 52:18.120] different points of view. And you, again, ask them to make another estimate and then take the [52:18.420 –> 52:24.800] range of those estimates. That’s a pretty simple way to run meetings. It’s not a complicated [52:24.800 –> 52:31.440] Delphi method or something like that. It’s simple. You could do this in any decision meeting about [52:31.440 –> 52:38.380] setting a number or setting a price or something like that. That’s one example. In meetings, [52:38.460 –> 52:43.960] we typically have norms and procedures and rules. They are not designed to reduce noise. They’re [52:43.960 –> 52:48.360] designed to reduce disagreement. And noise is disagreement. [52:48.420 –> 52:54.420] So you want to make sure that you express the disagreement before you come to a conclusion [52:54.420 –> 53:02.600] and not that you suppress it. Here’s another example. Hiring processes. I’ll tell you a story, [53:02.720 –> 53:11.820] actually. At least I think it’s a funny story. I was working with a big startup, [53:11.820 –> 53:17.840] probably by now a unicorn, that was working on its hiring processes. [53:18.420 –> 53:25.600] So how do they decide to hire people? And being a young, modern and trendy company, [53:25.720 –> 53:31.620] they’ve got a pretty thorough system where they’ve got different people interviewing each candidate [53:31.620 –> 53:37.700] and they’ve done a very good job of structuring their recruiting process. So what we would advise [53:37.700 –> 53:42.260] on something like that is have a structured process where you know what dimensions you’re [53:42.260 –> 53:47.060] looking at in candidates and you know what you value and what you don’t care about and make sure [53:48.420 –> 53:52.820] that you’re doing it independently of the various dimensions. And since they were a high-tech company, [53:52.820 –> 53:58.740] they had a system, an IT system, where they would enter all their judgments separately. [53:58.740 –> 54:04.660] And they told me, you see, this is very easy. We can’t influence each other because we all enter our [54:04.660 –> 54:10.020] grades into the system separately. And I thought, well, it’s really impressive. And then I looked at [54:10.020 –> 54:18.400] how the system was parameterized. It turns out that by default, the way this HR management system was parameterized, [54:18.420 –> 54:23.940] was that it gave people the option to go back to their grades and revisit it after looking at the [54:23.940 –> 54:29.920] grades that others had given. So what people on a team would do would be, you know, yes, they would [54:29.920 –> 54:35.220] put in the grades after each interview, but then their boss would see the candidate and the boss [54:35.220 –> 54:41.320] would enter, you know, terrible candidate, not up to our exacting standards. And they would go back [54:41.320 –> 54:47.100] and quickly say, oh, my God, I’ve been too lenient with that guy. Let’s bring the grades down. And in [54:47.100 –> 54:48.400] the end, they had perfect agreement. [54:48.420 –> 54:55.920] So the devil’s in the details when you’re talking about those kinds of processes. And the devil is in the culture. [54:55.920 –> 55:03.100] Because if you why were these people doing that? Because they feared disagreement. Because they didn’t have the [55:03.100 –> 55:10.060] psychological safety of standing up to their boss and saying, hey, you said that candidate was not good on this skill. [55:10.860 –> 55:17.580] Actually, I gave him a case on that skill, and he really impressed me. And we should talk about that, because we have a [55:17.580 –> 55:18.400] disagreement here. [55:18.400 –> 55:24.460] We should get to the bottom of it. And maybe we need to give another test to that candidate. Or maybe you didn’t give that [55:24.460 –> 55:31.180] candidate a test, and you will change your mind. Or maybe we need a third person to give us a different opinion. But let’s have a [55:31.180 –> 55:38.020] fact-based disagreement and resolve that disagreement instead of suppressing the disagreement. That’s a very down-to-earth [55:38.020 –> 55:46.220] example. But I think it shows that if you’re thinking about processes that reduce noise, you need to really fine-tune them to make [55:46.220 –> 55:48.220] sure that you express them. [55:48.400 –> 55:51.580] You need to express those views, and that you resolve them in a fact-based way. [55:51.580 –> 55:59.180] And I love that example, because clearly some thought was given on some level to that particular… I mean, they might have bought the software and [55:59.180 –> 56:05.900] here’s a useless feature that we’re never going to use, and it doesn’t matter because we can override it. But somebody designed this thing in a way [56:05.900 –> 56:13.580] that starts down that path. And then as you say, it actually… So you’d almost have a confidence that, well, we have a system that allows you to [56:13.580 –> 56:17.880] input independently, so we’re good. We have solved the problem. And actually, you’ve made it worse. [56:17.880 –> 56:18.180] Yes. [56:18.400 –> 56:26.780] Actually, you’ve made it, well, not worse, but you’ve missed an opportunity to make it better, at least. And when you design processes, you [56:26.780 –> 56:33.800] really need to take care of that. Here’s another example. Recruiting, again, in recruiting… Or let’s take performance [56:33.800 –> 56:48.380] evaluations. Most companies would have something like a 360-degree system by now. And many of them have some kind of [56:48.380 –> 56:58.000] rule to manage what we call level noise, which is the fact that some people, many people, in fact, will tend to give inflated grades, will give the [56:58.000 –> 57:06.420] best grade to everyone because it saves them the trouble of having difficult conversations. Or maybe it helps them get rid of someone they want to [57:06.420 –> 57:18.360] get rid of because he has a good mark and therefore he can be promoted. There is all sorts of motives to give people good marks. So an easy way to [57:18.360 –> 57:23.400] solve this problem, which a lot of companies have implemented, is to say, you’re not allowed to do [57:23.400 –> 57:30.320] that. You have to have a forced distribution of your grades. So you have to have on your team [57:30.320 –> 57:34.880] 10% of people who are in the bottom of category, and you can’t have more than 20% who are in the [57:34.880 –> 57:40.380] top category. Sorry, I know all your people are great, but it’s only 20% of A’s, and it needs to [57:40.380 –> 57:45.900] be 70% of B’s, and it must be at least 10% of C’s or something like that. Lots of companies have put [57:45.900 –> 57:53.740] in place something like that. Does that reduce noise? Well, I’m not so sure. It reduces the kind [57:53.740 –> 57:58.620] of noise that is very easy to see, which is this level noise, as we call it. The fact that, on [57:58.620 –> 58:05.860] average, Christian gives an A to everybody because he’s a kind, caring manager, and Olivier, who is [58:05.860 –> 58:13.240] a horrible guy, gives, on average, a B- to everybody. Okay, so we solved that problem. [58:13.240 –> 58:15.240] But here’s the issue. If… [58:15.900 –> 58:20.680] In your team, Christian, because you’re such a great manager, all the people are actually very [58:20.680 –> 58:27.740] good. And in my team, because I’m an abysmally bad manager, all the people tend to actually be [58:27.740 –> 58:32.660] quite bad because no one wants to work with me, and I don’t develop my people very well, and [58:32.660 –> 58:38.260] people who aren’t so good tend to gravitate towards people who aren’t so good either, [58:38.260 –> 58:45.540] and vice versa. So if we actually have very different levels in our teams, forcing a [58:45.540 –> 58:45.860] different… [58:45.900 –> 58:52.500] Differentiated and equivalent set of grades onto those two different groups is going to create a [58:52.500 –> 59:00.840] lot of error. You’re going to make a lot of errors in forcing B and C grades on people who deserve [59:00.840 –> 59:06.360] an A, and I’m going to be forced to give an A to people who deserve a B or a C. This sort of quick [59:06.360 –> 59:14.160] fix is not going to solve your problem. You need something more refined than that. So again, [59:14.600 –> 59:15.880] processes will help. [59:15.900 –> 59:20.900] Quite often, but you need to design the processes in a way that really reduces noise, [59:21.380 –> 59:25.600] not in a way that superficially gives you the impression that, hey, we’ve taken care of the [59:25.600 –> 59:31.080] process, so it must be fine. Love that. And that example you’ve just given, a ton of thoughts, [59:31.220 –> 59:35.820] all of which I recognize the dynamics you’re talking about. And you’re right, it is prevalent. [59:35.960 –> 59:40.520] And I view it a little bit like unconscious bias training, where we think it’s a good idea, [59:41.140 –> 59:44.080] and you said, let’s get this done, let’s put this in place, and then we’ve solved that problem. [59:44.080 –> 59:45.500] And as you say, actually, [59:45.900 –> 59:50.500] things are a little bit more complex. So I think that was a great example that listeners will [59:50.500 –> 59:54.620] absolutely recognize, many of whom I used to work with in various institutions. So thank you for that. [59:54.980 –> 59:59.180] You know, performance management systems are a fascinating topic. We’ve got an entire chapter [59:59.180 –> 01:00:05.480] of the book about them. It’s fascinating because it’s something that almost everybody realizes is [01:00:05.480 –> 01:00:12.220] broken. And there’s almost unanimous agreement among people who are evaluated and among people [01:00:12.220 –> 01:00:15.700] who are doing the evaluations, that it’s a terrible system. [01:00:15.900 –> 01:00:21.360] It’s a waste of time that demotivates people and that produces crappy data. And yet, [01:00:21.420 –> 01:00:24.100] everybody keeps doing it. It’s mind-boggling. [01:00:24.780 –> 01:00:31.000] Right, right. Before I let you go, a couple more questions. The first one is, I’m absolutely [01:00:31.000 –> 01:00:35.920] intrigued as to how do you get a professorial supergroup together to write this book? What [01:00:35.920 –> 01:00:37.900] was the genesis of bringing you all together? [01:00:37.900 –> 01:00:44.980] Well, let me subtitle your question. You’re intrigued as to how I got myself sandwiched [01:00:44.980 –> 01:00:45.660] between Danny… [01:00:45.900 –> 01:00:46.800] No, no, not at all. [01:00:46.800 –> 01:00:49.340] …and Cass, who are the professorial superstars. [01:00:50.040 –> 01:00:53.400] Not at all. No, I’m just intrigued as to how that works, because… [01:00:53.400 –> 01:01:01.400] No, but I’m kidding. But I’m very aware that I’m the lucky member of the trio here. And here’s the [01:01:01.400 –> 01:01:07.080] backstory, so you understand. Danny had been thinking about this topic for a while. And [01:01:07.080 –> 01:01:13.100] frankly, he’s the first one who saw the potential of this topic. And I’d been coming at it from a [01:01:13.100 –> 01:01:15.880] different angle, which, you know, people who’ve read my previous book have probably seen. And I’ve [01:01:15.900 –> 01:01:18.000] been doing this myself for a very long time. But I’ve had a lot of time to think about it. And [01:01:18.000 –> 01:01:21.640] one of the things I was really interested in while doing this is, you know, writing this book, [01:01:21.640 –> 01:01:27.680] you’re about to make a terrible mistake, will… will realize… My thinking was, I’m trying to help [01:01:27.680 –> 01:01:32.000] companies get rid of biases. But actually, there are a lot of situations where it’s not clear what [01:01:32.000 –> 01:01:36.100] bias is tripping them. What is the problem? Yes, there are some situations where, you know, if [01:01:36.100 –> 01:01:41.800] you’re making plans, they’re probably too optimistic. We know the direction of the error there. And if [01:01:41.800 –> 01:01:43.700] you are… [01:01:43.700 –> 01:01:44.280] You know… [01:01:44.280 –> 01:01:44.980] …you know… [01:01:44.980 –> 01:01:45.380] …ummm… [01:01:45.380 –> 01:01:45.880] …ummm… [01:01:45.900 –> 01:01:50.780] making hiring decisions, we sort of know what typical biases you’re going to run into. [01:01:51.100 –> 01:01:56.020] But there were also lots of cases where it wasn’t clear which way it would go. [01:01:56.680 –> 01:02:02.960] If you were making a big investment, yes, you could be too optimistic about the prospects for [01:02:02.960 –> 01:02:06.680] the investment, but you could also be too conservative because of loss aversion or [01:02:06.680 –> 01:02:14.880] because of other biases. So my sort of take on this was, you need to think about the process [01:02:14.880 –> 01:02:20.300] by which you make decisions, because you don’t really know what biases you’re dealing with. [01:02:21.000 –> 01:02:26.280] And when we started talking about this with Denny, it suddenly, well, it didn’t suddenly, [01:02:26.420 –> 01:02:33.080] but it gradually dawned on me that what I was dealing with here was a different statistical [01:02:33.080 –> 01:02:38.940] problem from the problem of bias. It wasn’t predictable average error. It was, in fact, [01:02:38.940 –> 01:02:44.860] a random error caused by a lot of different psychological [01:02:44.860 –> 01:02:49.080] biases that a lot of different people could bring to bear in their decisions. [01:02:49.440 –> 01:02:54.780] So intellectually, that’s how we got together. I was coming at this from the point of view of [01:02:54.780 –> 01:03:00.100] helping organizations deal with the effects of psychological bias. And I was realizing that [01:03:00.100 –> 01:03:06.540] the effects of the psychological biases were often what we now call noise. And Denny was [01:03:06.540 –> 01:03:11.100] seeing this from a different perspective, which was the perspective of the insurance company that [01:03:11.100 –> 01:03:14.840] we’re talking about in the book, where people were not aware of noise. And he was, [01:03:14.860 –> 01:03:19.900] well, how fascinating from a psychological standpoint that people are not aware of this [01:03:19.900 –> 01:03:25.280] problem. So we started talking about this, in fact, quite a while ago, about five years ago now, [01:03:25.540 –> 01:03:31.780] Denny and I. And initially, we didn’t think there would be a book. We thought there might [01:03:31.780 –> 01:03:36.480] be an article, but we weren’t actually certain that there would be an article. [01:03:37.540 –> 01:03:44.680] And gradually, we realized that this was a much larger problem than we had thought. It wasn’t [01:03:44.680 –> 01:03:44.840] just a problem of, oh, I don’t know, I don’t know, I don’t know, I don’t know, I don’t know, I don’t know, [01:03:44.840 –> 01:03:49.100] a problem of companies and underwriters and so on. [01:03:49.140 –> 01:03:52.680] It was also a problem of justice and a problem of medicine [01:03:52.680 –> 01:03:56.220] and a problem of childcare agencies, which we haven’t talked about, [01:03:56.300 –> 01:04:01.540] and a problem of patent officers and a problem in all kinds of walks of life, [01:04:01.560 –> 01:04:05.600] which are really a problem that society should care about. [01:04:05.760 –> 01:04:09.180] And that’s when we thought we need someone who can help us [01:04:09.180 –> 01:04:13.840] think about the political, the legal, the philosophical ramifications [01:04:13.840 –> 01:04:15.240] of this. [01:04:15.920 –> 01:04:20.920] And it so happened that Danny and Cass had worked together in the past [01:04:20.920 –> 01:04:26.300] and Cass had gently indicated that if we invited him to join the team, [01:04:26.360 –> 01:04:27.260] he would be very excited. [01:04:27.760 –> 01:04:33.160] And Danny also thought that having three authors would be better than two [01:04:33.160 –> 01:04:38.860] because two of us could perhaps sometimes tell him that he was wrong. [01:04:42.120 –> 01:04:43.820] Which didn’t happen very often. [01:04:43.840 –> 01:04:51.520] But still, and it’s harder for just one of us to push back against Danny. [01:04:51.720 –> 01:04:54.400] So that’s how we came to be a team of three. [01:04:54.580 –> 01:05:00.140] And personally, I’ve always worked in teams all my life. [01:05:00.240 –> 01:05:03.840] I’ve always found it to be great fun and to be a lot more fun than working alone. [01:05:04.740 –> 01:05:10.080] And it’s been an amazing privilege to work with that team. [01:05:10.720 –> 01:05:12.620] This is quite a team to be on. [01:05:12.620 –> 01:05:16.700] Well, I think you’re very clearly deserving participant of that [01:05:16.700 –> 01:05:18.900] because as people will have heard on this podcast, [01:05:19.200 –> 01:05:23.780] you’re a phenomenally great explainer of complicated things [01:05:23.780 –> 01:05:25.480] in very, very simple ways and engaging ways. [01:05:25.660 –> 01:05:27.100] And thank you for doing that. [01:05:27.380 –> 01:05:29.020] I’m intrigued as to what’s next for you. [01:05:29.120 –> 01:05:31.920] I know you’re busy doing lots of promotion for the book [01:05:31.920 –> 01:05:32.400] and thinking about it. [01:05:32.480 –> 01:05:35.120] But where’s your sort of research thinking beyond this? [01:05:38.020 –> 01:05:41.740] Well, it’s going in several directions. [01:05:41.740 –> 01:05:42.500] One of the… [01:05:42.500 –> 01:05:46.240] One of the topics, obviously, that I’m thinking about is [01:05:46.240 –> 01:05:49.440] what’s next about noise? [01:05:49.600 –> 01:05:53.820] So how do you actually help organizations tackle noise? [01:05:53.920 –> 01:05:57.100] We’ve tried to make the book very practical. [01:05:57.320 –> 01:06:02.320] In fact, a lot more practical than previous books by any one of us. [01:06:02.900 –> 01:06:07.220] And we even have appendices that give detailed manuals [01:06:07.220 –> 01:06:11.620] for how to do a noise audit and how to observe decision making and so on. [01:06:11.620 –> 01:06:12.400] So, you know, [01:06:12.500 –> 01:06:13.880] we’ve tried to make this practical. [01:06:14.220 –> 01:06:18.640] But as we, you know, clearly say in the book, [01:06:19.320 –> 01:06:24.480] this is the beginning of what we hope is going to be a trend of research [01:06:24.480 –> 01:06:27.020] that a lot of people are going to care about, [01:06:27.240 –> 01:06:31.920] both in terms of conceptually and academically [01:06:31.920 –> 01:06:33.940] trying to understand the causes of noise [01:06:33.940 –> 01:06:37.360] and also in more practical managerial terms [01:06:37.360 –> 01:06:41.220] in terms of how to actually reduce noise, what works. [01:06:41.620 –> 01:06:42.480] So we talk about noise. [01:06:42.480 –> 01:06:44.280] We talk, for instance, about structuring decisions. [01:06:44.500 –> 01:06:46.640] We talk about aggregating multiple judgments. [01:06:46.940 –> 01:06:52.380] We talk about sequencing information that you expose people to. [01:06:52.540 –> 01:06:54.960] We talk about selecting different judges [01:06:54.960 –> 01:06:58.340] or training judges to be better at making their judgments. [01:06:58.460 –> 01:07:01.400] We talk about lots of ways to reduce noise [01:07:01.400 –> 01:07:02.940] that are all part of decision hygiene. [01:07:03.280 –> 01:07:06.920] But we can’t say today in a prescriptive way, [01:07:07.340 –> 01:07:10.300] given your situation, you could do this and this, [01:07:10.300 –> 01:07:11.860] but not that, that, and that, [01:07:12.480 –> 01:07:13.560] and that’s not going to be useful to you. [01:07:13.760 –> 01:07:16.320] We also can’t tell you, you know, if you do this, [01:07:16.380 –> 01:07:18.120] you will reduce noise by X percent. [01:07:19.000 –> 01:07:23.660] We have examples, but there is still a lot of research to do [01:07:23.660 –> 01:07:24.980] on making this work better. [01:07:25.080 –> 01:07:27.740] And I’ve done some of that research with some companies, [01:07:28.360 –> 01:07:32.100] but I think, I hope, I’m going to do more of it in the future. [01:07:32.960 –> 01:07:35.980] And another thing that I’m thinking about quite a bit these days, [01:07:37.940 –> 01:07:40.960] which echoes some of my previous work [01:07:40.960 –> 01:07:41.980] and research, [01:07:42.480 –> 01:07:44.780] and also some of the stuff that we talk about in noise, [01:07:44.920 –> 01:07:50.560] is how do people or why do people who work in management, [01:07:50.800 –> 01:07:57.860] in companies, remain blind for so long to things like noise? [01:07:59.600 –> 01:08:01.420] You know, there are a few examples. [01:08:01.900 –> 01:08:04.340] We have the example of the insurance company. [01:08:04.500 –> 01:08:05.560] We have the example of hiring. [01:08:05.780 –> 01:08:07.540] We have the example of performance evaluations. [01:08:08.460 –> 01:08:10.140] These are things that are broken. [01:08:10.140 –> 01:08:12.400] They are broken in these cases because of noise, [01:08:12.740 –> 01:08:14.840] they’re also broken because of other things, [01:08:15.700 –> 01:08:17.940] and everybody behaves as if they are not. [01:08:18.820 –> 01:08:22.600] No one seems to do anything about it. [01:08:22.720 –> 01:08:24.140] Oh, man, no one is too strong. [01:08:24.940 –> 01:08:29.620] But a lot of people seem to be happy doing the same thing [01:08:29.620 –> 01:08:32.500] that they were doing and not challenging something [01:08:32.500 –> 01:08:35.100] that evidently doesn’t work, [01:08:35.180 –> 01:08:38.800] that we have a ton of research to show is broken, [01:08:39.280 –> 01:08:42.440] that, by the way, has been shown to be broken, [01:08:42.480 –> 01:08:43.600] even a long time ago, [01:08:44.060 –> 01:08:46.500] that academics don’t even do research about [01:08:46.500 –> 01:08:48.960] because it’s been clear for a long time now [01:08:48.960 –> 01:08:52.480] that this doesn’t work, this isn’t the way to do it. [01:08:53.040 –> 01:08:58.280] But academia has left and moved on to something else [01:08:58.280 –> 01:09:00.400] 10, 20, or 30 years ago. [01:09:00.800 –> 01:09:02.840] The world of management is still stuck there. [01:09:04.200 –> 01:09:11.480] And that gap, that chasm between what scientifically, [01:09:11.480 –> 01:09:15.180] scientifically we now know doesn’t work [01:09:15.180 –> 01:09:17.180] and sometimes we now know works [01:09:17.180 –> 01:09:22.580] and what is actually done is a topic that intrigues me [01:09:22.580 –> 01:09:24.680] and that I want to think about. [01:09:24.740 –> 01:09:26.620] Wow, I would love to see that [01:09:26.620 –> 01:09:28.760] because that just makes me smile, I think, [01:09:28.800 –> 01:09:29.860] because I think anybody listening [01:09:29.860 –> 01:09:32.680] who’s worked for any sort of moderately sized company, [01:09:32.820 –> 01:09:33.620] even smaller ones, I guess, [01:09:33.660 –> 01:09:34.700] but particularly larger companies, [01:09:35.080 –> 01:09:37.800] will absolutely recognize what you’re saying. [01:09:37.800 –> 01:09:39.980] Well, anyone like you, Christian, [01:09:40.080 –> 01:09:41.040] and anyone like, I guess, [01:09:41.040 –> 01:09:41.700] your listeners, [01:09:42.060 –> 01:09:44.500] which I think is part of the problem. [01:09:45.240 –> 01:09:47.840] Some people, the kind of people [01:09:47.840 –> 01:09:49.540] who listen to your podcast, I would think, [01:09:49.940 –> 01:09:51.500] have the intellectual curiosity [01:09:51.500 –> 01:09:53.840] to question the way things are done. [01:09:54.560 –> 01:09:57.060] But if they don’t have that curiosity, [01:09:57.300 –> 01:09:59.700] they are not pushed by organizations, [01:09:59.900 –> 01:10:00.860] by systems, et cetera, [01:10:00.860 –> 01:10:03.500] to find new ways or to challenge the old ways. [01:10:03.760 –> 01:10:07.340] So you smile, but a lot of people, [01:10:07.460 –> 01:10:09.140] when I ask them about this, they don’t smile. [01:10:09.140 –> 01:10:10.040] They just frown. [01:10:10.040 –> 01:10:14.320] And say, well, why are you talking about [01:10:14.320 –> 01:10:15.300] whether there is something wrong [01:10:15.300 –> 01:10:16.280] with the way you do recruiting? [01:10:16.880 –> 01:10:17.880] No, I’m not aware of that. [01:10:18.540 –> 01:10:19.180] That’s the issue. [01:10:19.300 –> 01:10:19.400] Right. [01:10:19.640 –> 01:10:22.540] Well, listen, please go ahead and do that [01:10:22.540 –> 01:10:23.680] because I think that would be amazing. [01:10:24.220 –> 01:10:26.900] Listen, Olivier, time as ever with you flies. [01:10:27.240 –> 01:10:28.980] Thank you so, so, so much for spending time [01:10:28.980 –> 01:10:29.980] because I know you’re incredibly busy. [01:10:30.380 –> 01:10:32.000] I will obviously put links to Noise. [01:10:32.080 –> 01:10:34.000] I will put links to your previous book as well [01:10:34.000 –> 01:10:36.460] and some of your other research as well [01:10:36.460 –> 01:10:38.300] because I do want to draw people’s attention to that. [01:10:38.300 –> 01:10:39.960] And obviously the previous episode of the show, [01:10:39.960 –> 01:10:40.580] that you were on. [01:10:40.980 –> 01:10:42.840] But listen, thank you very much for what you’ve done. [01:10:42.940 –> 01:10:44.560] I think you’ve stimulated this fascinating debate. [01:10:44.700 –> 01:10:46.780] You’ve made me think and made me smile, [01:10:47.240 –> 01:10:48.600] which is a great combination. [01:10:48.800 –> 01:10:49.520] Thank you so much. [01:10:50.520 –> 01:10:51.480] Thank you so much, Christian. [01:10:51.560 –> 01:10:52.100] This was fun. [01:10:53.120 –> 01:10:54.700] So that’s it for another episode [01:10:54.700 –> 01:10:55.960] of the Human Risk Podcast. [01:10:56.320 –> 01:10:58.980] My enormous thanks to Olivier for appearing [01:10:58.980 –> 01:11:00.460] and to you for listening. [01:11:00.960 –> 01:11:02.860] To find out more about Olivier, [01:11:03.140 –> 01:11:05.920] his areas of research, his other writings, [01:11:06.020 –> 01:11:07.180] and of course, Noise, [01:11:07.680 –> 01:11:08.940] have a look in the show notes. [01:11:08.940 –> 01:11:09.940] There are links to them. [01:11:09.960 –> 01:11:13.140] If you missed his previous appearance on the show, [01:11:13.400 –> 01:11:13.840] have a look. [01:11:13.960 –> 01:11:16.100] There’s a link to that in the show notes as well. [01:11:16.820 –> 01:11:18.040] If this is your first time listening [01:11:18.040 –> 01:11:19.440] to the Human Risk Podcast, [01:11:19.920 –> 01:11:20.300] welcome. [01:11:20.700 –> 01:11:22.200] You can subscribe wherever you get [01:11:22.200 –> 01:11:23.840] your quality audio content. [01:11:24.220 –> 01:11:27.460] It’s available on all of the major podcasting platforms. [01:11:28.220 –> 01:11:29.140] If you like the show [01:11:29.140 –> 01:11:31.920] and have yet to leave a five-star review for it, [01:11:32.020 –> 01:11:32.920] please do so. [01:11:33.080 –> 01:11:34.800] It really does help. [01:11:35.580 –> 01:11:37.340] To find out more about human risk, [01:11:37.340 –> 01:11:39.400] visit human-risk.com [01:11:39.400 –> 01:11:41.060] and you can read about the work [01:11:41.060 –> 01:11:42.260] that I do with my clients, [01:11:42.620 –> 01:11:44.480] subscribe to the Human Risk newsletter [01:11:44.480 –> 01:11:47.060] and see the Human Risk video blog [01:11:47.060 –> 01:11:48.520] and blogs on the website. [01:11:48.820 –> 01:11:50.040] I’ll be back very soon [01:11:50.040 –> 01:11:52.520] with another episode of this podcast. [01:11:52.940 –> 01:11:54.660] But in the meantime, stay safe, [01:11:54.960 –> 01:11:55.980] try and avoid noise, [01:11:56.380 –> 01:11:57.280] and thanks for listening.

Konubinix' opinionated web of thoughts

Professor Olivier Sibony on Noise

Permalink