Americans across the United States head to the polls today to vote in the US midterm elections. If the polls are to be believed then there is a high probability that the Democrats will take both the House and the Senate once all the votes are counted. Last-minute pre-election polls show Democratic candidates leading in the seven states most likely to determine control of the Senate. But, as I wrote about recently in You Don’t Know What You’re Missing, we’ve been here before. When Trump is involved in an election, it’s very hard to get an accurate gauge of exactly who is and who isn’t supporting him or his party.

In the run-up to the 2016 US presidential election, keen followers of both state and national voting-intention polls would have had very few doubts as to who would go on and win the election. Poll after poll had Hillary Clinton winning the popular vote, the key swing states and the electoral-college vote. Forecasters typically predicted Clinton’s chance of winning to be between 70 and 90 per cent, with some expressing even more confidence. Very few official polls had a Trump victory as the more likely option. Yet on 9 November, as the results of the voting filtered in, it became clear that Trump had won enough of the key swing states to take the overall victory. The result left many politics buffs scratching their heads as to why almost all the predictions they had seen in the run-up to the election were wrong.

Subscribe now

The true reasons for the differences between how those polled were recorded as saying they would vote and how people actually voted on the day are complex and multifaceted, but it seems that a proportion of the discrepancy may have been down to non-response bias. The anti-establishment rhetoric that characterised Trump’s campaign appealed to voters with a distrust of traditional institutions. This extended to polling organisations, which were painted as untrustworthy tools of the political elite, allowing Trump to discredit unfavourable polls. A side effect of sowing this distrust of pollsters may have led some Trump supporters to eschew the polls altogether.

While, due to their very absence in polling data, it is hard to demonstrate that Trump supporters have higher rates of non-response, studying the rates of dropout from one round to the next in sequential polling with the same subjects can shed some light on the issue.

Share

Ahead of the 2022 US midterm elections, FiveThirtyEight and Ipsos convened a panel of around 2000 eligible voters. The aim of the demographically representative poll was to understand how the opinions (including voting intentions) of these Americans evolved over the six months prior to the midterms. By the end of the six rounds of monthly polls conducted in the run-up to the midterms, around a quarter of the subjects had dropped out of the study. Interestingly, because of its sequential nature, the pollsters were able to determine that Republican-leaning voters were more likely to drop out than those who favoured Democrats or independents. By the third round of polling, for example, 21 per cent of Democrats had ceased to respond, compared to 26 per cent of Republicans. Not a huge difference, for sure, but perhaps enough to suggest non-response bias might be confounding the polls. Remember that these dropout rates are among people who were happy to be surveyed in the first place; adding in the effect of non-responders – people who completely disengaged from the polling process before it even started – could contribute to an even larger difference.

Share Kit Yates - Math(s) and the real world

So while there may be optimism amongst the Democratic camps that they will see a surge of support for their party, don’t expect the results, when they are eventually called, to be exactly as the polls predicted.

Title image by Kelly Sikkema on Unsplash

This week I was back in the Presenter’s Friend chair on BBC Radio 4’s Inside Science. I tried to understand just how common brain disease is in contact sports, debated the ethics around the second-most expensive dinosaur ever sold and brought us down to earth with more dispiriting news about greenhouse gases.

Greenhouse gases are still rising

It is rather extraordinary that we have known about greenhouse gases and their potential effects for so long, and yet we still find ourselves in the situation illustrated by a new report from the World Meteorological Organization (WMO). The report suggests, as you might have expected, that we are continuing to increase the amounts of greenhouse gases in the atmosphere.

Carbon dioxide levels reached a record high of 426.8 parts per million in 2025, around 53 per cent higher than pre-industrial levels of approximately 280 parts per million. The rate of increase is also rising. The annual increase was 0.8 parts per million in 2011, but by 2020 this had tripled to 2.4 parts per million. In the most recent data in the report, covering 2024 to 2025, the increase was 2.8 parts per million. This is slightly lower than the increase of 3.6 parts per million between 2023 and 2024, but 2024 was quite an extraordinary year for carbon emissions, so I do not think we can draw too much comfort from that downward trend.

Subscribe now

The levels of other greenhouse gases are also continuing to increase from already record-breaking concentrations. Methane is now at 1,948 parts per billion, up 167 per cent on pre-industrial levels, while nitrous oxide has reached 339.1 parts per billion, up 26 per cent on pre-industrial levels.

The WMO report paints a pretty grim picture when it comes to how well we are doing at curbing emissions. It also warns that more attention should be given to two other gases in the atmosphere: molecular hydrogen and sulphur hexafluoride. Ironically, both can result from efforts to decarbonise.

Hydrogen, for example, is being used increasingly for energy storage. Although hydrogen is not itself a greenhouse gas, it can have a substantial indirect effect on global warming. It reduces the concentration of hydroxyl radicals, which would normally help to remove methane from the atmosphere. As a result, methane remains in the atmosphere for longer and causes more warming. The atmospheric concentration of molecular hydrogen increased by 5.3 per cent between 2010 and 2024.

Share

Sulphur hexafluoride is the other gas highlighted. It is used primarily as a spark suppressant in electricity distribution, but it performs extremely badly on a measure known as the “100-year global warming potential”, with a value approximately 24,000 times that of carbon dioxide. That is to say, releasing one kilogram of sulphur hexafluoride has approximately the same warming effect over the following 100 years as releasing 24,000 kilograms of carbon dioxide.

According to the WMO, levels of sulphur hexafluoride have been increasing steadily over the past three decades, broadly in proportion to, as you might expect, growing global electrification. So even some of the fixes we are trying to implement may be contributing to the greenhouse effect.

How common is CTE in former NFL players?

You might have seen items in the news recently about repeated head impacts becoming a major concern across contact sports, from the NFL in the United States to rugby union and rugby league in the UK and elsewhere. One of the main problems identified in the brains of former athletes from several sports is chronic traumatic encephalopathy, or CTE. But estimating the prevalence of CTE has proved remarkably difficult.

In part, that is because CTE can only be conclusively diagnosed by examining samples of a person’s brain tissue after their death. As a result, we often rely on studies involving brain banks, where people have donated their brains to be studied after they die.

Share Kit Yates - Math(s) and the real world

In recent years, we have seen headlines, including from the BBC, saying things such as: “Brain disease affects 99% of NFL players in study.” But we need to be extremely careful when we interpret the results of such studies especially if we are reporting on them in the media because of something called selection bias.

This is something I wrote a lot about in my last book, You Don’t Know What You’re Missing. It is the idea that there might be biases in who ends up in your surveyed population. In the case of brain-bank studies, many of the brains held in the bank were donated by families who suspected that their loved ones had suffered from CTE, based on the neurological symptoms they experienced while they were alive.

What researchers writing in JAMA Network Open have done is to try to quantify that bias in order to estimate the true underlying prevalence of CTE in former American football players.

They first performed the simple calculation of looking at what proportion of former NFL players’ brains in the bank had CTE. This produced an extremely high prevalence of about 94 per cent. But remember that we still have this selection bias to account for.

The researchers estimated that the brains of players with recognised cognitive problems, such as memory impairment or dementia, were about five times more likely to be donated for research. They then combined that estimate with the biased figure of 94 per cent and used the statistician’s best friend, Bayes’ theorem, to calculate the underlying probability of a former NFL player having CTE. Alarmingly, the figure still came out at an extremely high 75 per cent.

There is a relatively simple way to understand how those two figures fit together. Imagine a group of 100 former players, 75 of whom have CTE and 25 of whom do not. If those with CTE are five times more likely to have their brains donated, the resulting brain-bank sample would contain five times 75 CTE-weighted donations and 25 non-CTE-weighted donations. The apparent prevalence in that sample would therefore be:

The estimated five-fold selection bias is based on cognitive symptoms rather than directly on CTE itself, so it is possible that the true selection ratio could be lower than five to one, which would give a higher overall prevalence estimate for CTE. On the flip side, it is important to recognise that not every player with CTE pathology will go on to develop symptoms, particularly in cases of early-stage disease.

While this study does not definitively end the debate, what I think it does is stop people dismissing the scale of the problem simply because the extraordinarily high brain-bank-based prevalence estimates are affected by selection bias. Even after accounting for that bias, the prevalence remains alarmingly high.

For governing bodies in sports such as the NFL, rugby union and rugby league, this is a really important finding to take note of. It suggests that concerns about the long-term consequences of repeated head impacts are not going away, and I think it reinforces the need for continued efforts to reduce head trauma and to better understand the potential problems faced by players even after their careers have ended.

The second-most expensive dinosaur of all time

Finally, let me tell you about the second-most expensive dinosaur ever sold.

It is a Nanotyrannus named Jodi, after the Montana rancher Jodi Pierson, who discovered it. It sold at auction for $47.3 million. It had originally been expected to sell for around $9 million, but six different parties ended up bidding for more than 12 minutes, and the price just kept ratcheting up and up. (For context, the most expensive dinosaur was a T. rex named Gus, which sold for $50.1 million in July.)

Perhaps part of the reason Jodi sold for so much was that this was the first Nanotyrannus to come to market since research argued that the dinosaur was distinct from T. rex. For decades, many palaeontologists had believed that Nanotyrannus was just a sort of teenage T. rex.

But a paper published in Nature argued, by examining the microscopic structure of the bones of one specimen, that Nanotyrannus had almost stopped growing and therefore could not simply be a juvenile T. rex. The researchers suggested that Nanotyrannus was not merely a distinct species from Tyrannosaurus rex: it was not part of the Tyrannosaurus genus at all, but belonged to a genus of its own.

I do not know whether you subscribe to nominative determinism, but here is a fun fact. The authors of the paper that reclassified Nanotyrannus as a separate genus were Lindsay Zanno and James Napoli. If they had been going to combine their surnames to name a new genus of tyrant dinosaur, they could have done a lot worse than NANOtyrannus. In fact, the name long predates their paper. It goes back to 1988 and means “dwarf tyrant”, reflecting the similarity of the specimens to juvenile T. rex.

The other side of this story is the debate about whether we should be selling these important specimens into private collections. Jodi was one of only three known Nanotyrannus skeletons, and I think many people would argue that they shouldn’t be kept behind closed doors in the vaults of some private collector, but should be made available to the public and scientists to enjoy and learn from.

That said, I cannot completely begrudge the people who have spent time carefully finding, extracting and curating these skeletons being rewarded for their work. But perhaps you might argue that a reward of $47.3 million is disproportionate to the work involved?

You can catch me on Inside Science next on 26^th of November.

Yesterday, I was on the BBC World Service programme Unexpected Elements. Inspired by the newly opened bridge between China and North Korea, the topic of the show was bridges. I was brought on to talk about mathematical bridges, as opposed to the maths of bridges.

Fortunately, one of my favourite mathematical stories is about bridges. It was such an important problem that the mathematician who eventually solved it ended up creating a whole new branch of mathematics.

The Bridges of Königsberg

In the eighteenth century, Königsberg was a city in East Prussia. It is now Kaliningrad, a Russian exclave between Poland and Lithuania. The city was divided by the River Pregel, which in turn, was split around two islands. Connecting the various parts of the city were seven bridges.

The people of Königsberg apparently amused themselves with a simple challenge: could you take a walk around the city, crossing each bridge exactly once? As a pastime, people would set off from different parts of the city and see if they could complete the walk. No one knew whether it was possible, but no one had so far managed to achieve it.

Subscribe now

In 1736, the problem was eventually brought to the attention of Leonhard Euler, one of the greatest mathematicians who ever lived. His name appears everywhere in mathematics: Euler’s formula, Euler’s identity, Euler’s constant, Euler’s number, Euler’s equations, the Euler characteristic, Euler angles and dozens of other concepts bearing his name. The guy really was a genius. He produced more significant mathematical work than perhaps any other mathematician in history, and did so with astonishing creativity and breadth.

Instead of going out to Königsberg and walking around, he did what many of us might have done which is to get a map of the city and its bridges. But while the natural thing might be to try drawing on it to see if we could complete the circuit, Euler didn’t do that. Instead, he invented a whole new branch of mathematics – graph theory.

Euler’s approach to the Königsberg puzzle was revolutionary because he realised that most of the details of the city simply didn’t matter. The exact route of the river wasn’t important. The shape of the islands wasn’t important. The lengths of the bridges weren’t important. All that mattered was which pieces of land were connected to which bridges.

Share

Euler decided to represent each of the four land masses as a point, called a vertex. He then connected the vertices with lines that we would now call edges. These represented the bridges connecting the land masses. He had this simplified representation of Königsberg that today we would call a graph or a network. To us, this may feel obvious, but at the time it was a genuinely radical abstraction. Euler had taken a geographical problem and reduced it to a pattern of connections.

On this continuous walk around the town, he reasoned that every time you enter a land mass you also have to leave it. So the number of bridges that connects each land mass has to be even. There were two exceptions. You could begin at a place and not have to come back, or you could end at a place without having to leave again. Two of the land masses, the start and the finish, could have odd numbers of bridges, but no more than that.

Share Kit Yates - Math(s) and the real world

When Euler counted the connections in Königsberg, he found that all four land masses had an odd number of bridges, and so the walk was impossible. That was the birth of graph theory.

The Bridges of Kaliningrad

During World War II, several bridges were destroyed and following that, several new ones were rebuilt. It’s also worth saying that following World War II, the city became part of the Soviet Union and was renamed Kaliningrad. Today, it is a Russian city on the Baltic Sea, between Poland and Lithuania.

In the new set up, let’s call it the Bridges of Kaliningrad problem, two of the bridges linking the first island to the north and south bank were destroyed which means that, although the islands still have odd degree (both having degree three), the banks now have even degree: they just have two bridges connecting them to the two islands. Provided you start at one island, you can walk a path where you end up on the other island having crossed all the bridges exactly once along the way. This is known, in honour of Euler, as an Eulerian path. If you wanted to start and end at the same place, then all the vertices would have to have even degrees and the resulting route would be known as an Eulerian cycle.

Today, graph theory helps us understand everything from social networks and transport systems to epidemiology, ecology and the structure of the internet. Whenever we’re interested in how things are connected, we’re drawing on ideas that can be traced back to Euler’s analysis of those seven bridges.

I was queueing online for Oasis tickets and all my footballing friends were speculating that my club (Manchester City men’s team, to be specific) were going to be relegated. It felt like I was 14 again, but it was, in fact, only a couple of weeks ago.

Some of the dust has settled after the announcement of Man City’s guilty verdicts, but we are still seemingly no further towards finding out the punishment that will be meted out to the club. As a Man City fan, most of the football related Whatsapp groups I’m on have been interested to hear what I think of the fact City have been found guilty and the majority of the 115 charges laid at their door by the Premier League. There seems to have been an expectation that I would come out and defend the club, act as a spokesperson toeing the party line pushed by the club’s owners.

I haven’t scrutinised the evidence in any detail, but it seems the independent panel that did was satisfied enough to find the club guilty and so that’s probably the most pragmatic view to take, at least until the club’s appeal is heard. Rather than discussing the facts of the case, many of my football-interested colleagues have been keen to know how I feel about the potential punishments that might be handed down.

Subscribe now

There is a precedent that other clubs that have committed breaches of the rules (see Chelsea) have kept their retrospective league positions and were instead handed fines and/or future points deductions. From what I’ve read, some sort of unprecedentedly large fine or huge points deduction seems like the most likely punishment. Even direct relegation or relegation by several divisions is not off the cards. These don’t seem like drastically unreasonable punishments given the scale of the rule breaches.

Retrospective stripping of titles and trophies has also been suggested as a less likely, but still quite possible sanction the independent panel might choose to take. As a fan, I think this would be harder to stomach. To think that the Agueroooooo moment, where City dramatically snatched the title back from cross-city rivals United at the last moment, could be rendered essentially meaningless leaves a bitter taste in my mouth. I’m not saying what is right and what is wrong, or what the appropriate punishment should be, but for the fans, I think this is the aspect of the punishment that would hurt the most.

Such thoughts have left me pondering the nature of footballing memory. Obviously the guilty verdict, a potential points deduction, a fine or even the stripping of titles wouldn’t change how we, as City fans, felt in those moments. But would it, in particular the delegitimisation of those titles, change how we look back on those moments and how we would feel about them in retrospect? Almost certainly it would.

Share

But then, if City appeal and are found not guilty (again I’m not speculating on whether this is likely to happen, the right thing or the wrong thing), does that suddenly re-legitimise those achievements. I suppose it should. Or what if the guilty charges are upheld, but the titles are allowed to be retained. Will my body still produce the goosebumps when I watch Aguero smash that winning goal past Paddy Kenny in the 94^th minute, even though the club he was playing for had broken the rules to even be in a position to make that moment happen?

Even before these charges I’ve been toying with what my City fandom even means anymore. I haven’t lived in Manchester for over 20 years. I haven’t been to a game at the stadium for a couple of years now. I don’t know many other City fans down where I live and I don’t know that many United fans either. Strangely it’s this absence of United fans in my life which has probably left me pondering my City fandom most of all.

Back in the day when I was at school, almost all my friends were United fans. There were two or three of us who were openly City fans, but not many. The year United won their Treble, our team were playing in the third tier of English football – the contrast couldn’t have been starker. What hurt most about those times was probably the fact that my United supporting friends didn’t even bother to take the piss. There was no need. Our misery and the miserable state our club was in was self-evident. Some of them even said to me things like, “I want City to do well, to get back to where they should be.” I think this hurt even more.

Share Kit Yates - Math(s) and the real world

Of course, when we came back up to the Premier League, that rivalry resumed. But it made me realise that my fanhood was defined, in large part by opposition: wanting to be as good as, if not better, than our cross-city rivals. Derby days became massive for us City fans. Almost too important. The year of the “Agueroooo” goal – City’s first title in 44 years – I couldn’t even watch the second derby of the season. It was too important. It meant too much.

And I suppose this is ultimately what has made me realise that you can’t just give up on a club – or at least I can’t. It’s not as simple as that. When I listen to the football commentaries Saturday afternoon, if City have lost it throws me off, it annoys me, it changes my mood. It shouldn’t do. I should be able to be impervious to the results of this football team that I essentially have nothing to do with, have no influence over; I don’t even have the negative feedback of anyone who will lord it over me. I should be able to decouple myself from this affiliation. But I can’t.

So while the frustration at the 115 charges being upheld is duller than it would have been if this had happened when I was at school, in the height of my opposition-defined fandom, it still hurts. The thought of losing those hard-fought titles and trophies still saddens me.

Despite these feelings borne of long-honed loyalty (or perhaps in fact because of them), ultimately I think I will be able to take a more pragmatic view. If, next year, the club are stripped of all those titles and if they are relegated a couple of divisions, those of us who have been fans of the club for long enough will be able to cope; after all we’ve been here before. It will just be like travelling back in time a few decades. It won’t be too different to 1999. I will go and sing my heart out at an Oasis gig and I will grumble about Man City.

_{Title photo by Jonny Gios on Unsplash}

Every few weeks I’m invited onto Radio 4’s Inside Science as the “presenter’s friend”, bringing along a couple of stories that have caught my eye and reacting to some of the other science news of the week. This week the main story revolved around the accuracy of our time-keeping and the possible end of leap seconds, while the stories I brought to the table were linked by the theme of temperature: one about the discovery of a new amoeba pushing the limits of how much heat complex life can withstand and the other about taking the temperature of the most famous dinosaur of them all: Tyrannosaurus rex.

Subscribe now

Leap seconds, leap years and the triumph of atomic clocks

I’ve always found our systems of timekeeping fascinating. I wrote quite a lot about where we get our time-recording systems from - 24 hours in a day, 60 minutes in an hour and so on - in my first book, The Maths of Life and Death.

The Ancient Egyptians divided the day into 24 segments: 12 day-hours and 12 night-hours, roughly mirroring the months of the year. But because they prescribed 12 hours in every daylight period, the length of an hour changed throughout the year. The Ancient Greeks later realised that to make progress with their astronomical calculations they needed equal, regular divisions of time, so they introduced the idea of dividing the day into 24 equal-length hours. But, it wasn’t until the 14^th century when we had mechanical clocks that this idea really caught on.

By the early 1800s most cities in Europe divided their day into 24 equal hours, but time was still not uniform across a single country. Typically each city used a local time based on the location of the sun in the sky. Because Oxford is one and a quarter degrees further west than London, the sun is at its peak there around five minutes later, meaning that local Oxford time was five minutes behind London. It wasn’t until the advent of the railways that time became truly standardised across Britain.

That’s why I find the idea of leap seconds so interesting. For most of history, the position of the Sun in the sky was the ground truth. Mechanical clocks and quartz clocks were attempts to reproduce that truth ever more accurately. Eventually we built atomic clocks that are so accurate that they exposed what we thought was the ground truth - the Earth’s rotation itself - as an imperfect timekeeper.

Leap years and leap seconds are really solutions to related but distinct problems caused by the fact that nature refuses to fit neatly into the systems we invent to measure it. The important difference is that Leap years adjust our calendars for the Earth’s orbit around the Sun, while leap seconds adjust our clocks for the Earth’s rotation about its own axis.

Share

The interesting distinction is that the disparities accumulate at very different rates. If we didn’t have leap years then over a hundred-year lifetime the calendar would drift by almost a month relative to the seasons, something most of us would probably notice. But since leap seconds were introduced in 1972 we’ve only added 27 of them. Most of us probably wouldn’t notice a drift the size of a minute in the timing of midday over the course of our lifetimes if this trend were to continue.

But, as it happens, it isn’t actually set to continue in the short term. In fact because the Earth’s rotation has recently sped up slightly there is the possibility of adding a negative leap second - removing a second rather than adding one - in the near future. This possibility worries engineers because many digital systems were designed assuming leap seconds would only ever go one way. Some estimates suggest the preparation required could be comparable to the work done for the Millennium Bug. As a result, there’s now a serious proposal to effectively stop adding leap seconds altogether and allow atomic time to drift away from the Earth’s rotation by up to an hour – a process which would take centuries - before making any correction.

Share Kit Yates - Math(s) and the real world

How hot can complex life get?

Scientists in the US and Europe, writing - appropriately enough - in a journal called Cell, have discovered an amoeba which breaks a long-standing record for temperature tolerance in what is sometimes referred to as “complex life”.

By complex life here we’re talking about what are known as Eukaryotes. There are three domains of life right at the root of the tree of life: Bacteria, Archaea and Eukaryotes. The Eukaryotes comprise most of what we think of when we think about life on Earth: animals, plants, fungi and so on.

Eukaryotic cells are much more complex than bacteria and archaea. They have a nucleus, mitochondria and often a complex internal cytoskeleton. This extra complexity allows eukaryotes to do all sorts of cool things that bacteria and archaea cannot do, but at higher temperatures that complexity becomes something of a liability. While thermophilic - heat-loving - bacteria have been found which thrive at 100°C and archaea have been found growing and reproducing at temperatures above 120°C, for a long time it has been thought that eukaryotes simply couldn’t live above about 60°C.

What has been described in this new paper is the discovery of a new amoeba, a single-celled eukaryotic organism, which scientists have named Incendiamoeba cascadensis. The name means, roughly, “fire amoeba from the Cascades”, the volcanic mountain range where it was discovered. Incidentally, it was found in a tributary leading to Hot Springs Creek on the trail to Boiling Springs Lake in Lassen Volcanic National Park. If you’re going looking for thermophilic forms of life, that seems like a sensible place to start.

This amoeba doesn’t merely survive in 63°C heat, it actively moves, eats and reproduces there, breaking the previous record for eukaryotic life. At 64°C it stops reproducing but can still move, and after short periods at 70°C it effectively hibernates before recovering once the temperature drops again.

So how does it do it?

The amoeba has evolved a number of adaptations to overcome the challenge of extreme heat. It up-regulates heat-shock proteins and molecular chaperones, which help stop proteins from effectively melting and clumping together. The proteins themselves appear to be built more like those of thermophilic bacteria and archaea, which again helps prevent them sticking together in the heat. This is a lovely example of convergent evolution: two very different branches of life independently arriving at similar solutions to the same problem. The amoeba also appears to possess enhanced DNA repair mechanisms, important because DNA itself becomes vulnerable to damage at high temperatures.

As well as being biologically fascinating, these discoveries may also prove useful. Heat-resistant proteins already underpin technologies such as PCR, and understanding how organisms function under extreme conditions may ultimately have applications in industrial biotechnology, biofuel production and even astrobiology.

Taking the temperature of a T. rex

From one extreme to something really quite different, but still temperature related.

When thinking about reptiles, one of the first things that comes to mind - at least for me - is lizards sunning themselves on rocks or crocodiles basking on riverbanks. It feels natural to imagine dinosaurs might also have exhibited this cold-blooded physiology. Indeed, that was the dominant view until around fifty or sixty years ago.

But it has been hypothesised for quite some time now that large dinosaurs probably maintained body temperatures considerably higher than those of their surroundings. The evidence has been building for decades, but this week scientists writing in Science Advances have effectively stuck a thermometer into the mouth of the most famous dinosaur of them all - Tyrannosaurus rex - and found that it was running at about 36°C, give or take a couple of degrees.

The technique they used is called clumped isotope thermometry. This exploits the fact that tooth enamel contains different isotopes of carbon and oxygen: Carbon-12 and Carbon-13, Oxygen-16 and Oxygen-18. The heavier isotopes - Carbon-13 and Oxygen-18 - tend to bunch together, or “clump”, more readily at lower temperatures, whereas at higher temperatures they become more randomly distributed. The degree of clumping therefore acts as a record of the temperature at which the enamel formed.

Tooth enamel is particularly useful because it is generally very well preserved and relatively resistant to chemical alteration over geological timescales. The challenge has always been that early versions of the technique required quite a lot of material - more than any museum was willing to sacrifice from their most famous dinosaur.

The technical advance in this latest study is that it requires much less enamel than before. That meant the Natural History Museum of Los Angeles was willing to allow researchers to sample small portions of two teeth belonging to “Thomas” the T. rex. The result places T. rex’s temperature at about 36 degrees – give or take a couple of degrees - somewhere similar to our own temperature.

This tells us something about tyrannosaurs’ apparently high activity levels and potentially elevated metabolisms as they chased or scavenged food. But it also suggests that T. rex could survive in cooler habitats than cold-blooded animals would have managed.

For example, it has long been hypothesised that winter in Alaska, even during the warmer climate of the Cretaceous, would have been too cold for typical cold-blooded reptiles. Sure enough, palaeontologists haven’t found fossils of lizards, turtles, crocodiles or other reptiles from the Cretaceous in Alaska, but they have uncovered Tyrannosaurus fossils there – and this new temperature result sort of makes sense of those findings. Thanks to a tiny amount of fossil tooth enamel, we now have a much better idea of just how hot-blooded a predator T. rex really was.

I’m next on Inside Science in late October, so I’ll be back with another update then.

Every six weeks or so I am invited along to appear up on Radio 4’s Inside Science as the “presenter’s friend”, bringing along a couple of stories that have caught my eye and reacting to some of the other science news of the week. I’ve written up some of this week’s stories into a substack post so you can enjoy them even if you missed the show.

Subscribe now

A sprinkle of dark matter? Maybe.

For years physicists have been hunting for dark matter, the mysterious substance that appears to make up around 85% of the matter in the Universe but has never been directly detected.

This week, researchers working on the LUX-ZEPLIN (LZ) experiment reported a single event that looks intriguingly like what some models of dark matter might produce. Deep underground in South Dakota, their detector saw a xenon nucleus apparently receive a sizeable kick from something.

But it’s important not to get too carried away. This paper is not yet peer reviewed and it doesn’t meet the significance level for what physicists would call “discovery”. To be fair to the scientists conducting the work, they aren’t making any outlandish claims about it either.

The scientists suggest that they might see an event like this purely due to known background processes, that is non-dark-matter processes, on average once every 200 times they run this experiment – so a 0.5% chance of this result happening even if there were no dark matter detected. To put that into context that’s like tossing a coin 8 times in a row and getting heads each time if the coin is fair. Unlikely, but not impossible, especially if you do it enough times.

But physicists have been burned in the past – accepting results with low significance levels as true. So the accepted significance level for ‘discovery’ is about 1 in 3.5 million. This is the equivalent of flipping a fair coin 22 times getting heads 22 times in a row. It’s extremely unlikely to happen if the coin is genuinely fair.

As Kimbery Palladino – one of the scientists on the project who appeared on Inside Science to talk about the results “One is the loneliest number”. One event isn’t enough. The way that dark matter is likely to be confirmed is if scientists see more similar results at the same facility or, even better, at other facilities around the world,giving them confidence that this was not just a fluke.

Share

Climate optimism - Hope is important

I listened in on a conversation with Fred Pearce about his book “Despite It All: A Handbook for Climate Hopefuls”, which has been shortlisted for the Royal Society’s Trivedi Science Book prize.

What I thought was really interesting in Fred’s interview was the idea that we need optimists and pessimists: Pessimists to highlight the problem and optimists to provide hope. If you go too far down the doom spiral of pessimism then you get to a point where you believe there is no hope which causes inaction – it’s a sort of positive feedback loop – a vicious circle. And I think it’s important to note that optimism isn’t the same as denial – suggesting that we don’t need to do anything because everything is fine, because that also leads to inaction. Being optimistic about the potential to fix climate change isn’t the same as denying its impact or its potential if left unchecked.

Saturn gets a decagon

For decades, scientists have known about a seemingly mysterious hexagon that encircles the North Pole of Saturn. It’s a remarkably stable six sided phenomenon which has persisted since at least when it was first spotted by the Voyager mission in 1981- so over 40 years. There are lots of competing hypotheses about how the hexagon developed, but interestingly if you read the Wikipedia page for Saturn’s hexagon a couple of days ago you would have read that there is no corresponding hexagon on the Saturn’s South Pole.

Share Kit Yates - Math(s) and the real world

However, a paper came out this week in the journal Science Advances which reports the discovery of a ten-sided or decagonal atmospheric wave encircling Saturn’s south polar region. The authors first spotted it in 2023 using the Hubble space telescope and by 2025 it was well enough established to be visible from ground-based observatories.

So why does Saturn have these polygonal waves? Well because the planet is rotating it causes a jet stream to form, very much like the familiar jet streams that we have on Earth - one which encircles the north pole and another half the world away, encircling the south pole. As people will probably know a jet stream isn’t just one continuous jet of wind that blows from west to east. The wind path actually meanders in large north-south oscillations called Rossby waves. This is what gives us our variable weather patterns in the UK and much of the rest of the Northern hemisphere.

The same thing is happening on Saturn, only on Saturn the jet stream is really narrow and strong because Saturn rotates very quickly, so instead of looking like a looping wave the peaks and troughs get flattened out to look like a polygon when viewed from above. The “corners” are just the peaks of the wave and aren’t actually sharp corners at all, but rather part of a smooth flow that only looks sharp because of its enormous scale.

But why it’s a six-sided hexagon at the North pole and a 10-sided decagon at the South is still an open research question.

Can magnetism reveal a fake?

My final story this week concerned one of the most stubborn problems in archaeology and art history: how do you tell a genuine ancient object from a convincing fake?

Regular listeners to Inside Science will probably be aware of a number of different ways of detecting art forgeries by determining the age of the artefact. If it was made much more recently than it is supposed to have been then clearly it’s a fake. One way of determining the age is through radiocarbon dating. This works for things like canvasses or frames which contain organic matter. There’s also more general radio-isotope dating to detect paints, which is useful when a forger has painted a forgery on a genuinely old canvas. But what if the art work isn’t a painting, but a sculpture. What if it’s made of clay?

Well there’s a technique, published this week in PNAS, which purports to be able to determine the age of a clay artefact based on a technique which uses the magnetism of particles in the clay. The idea is that the magnetic field of large particles in a clay artefact fired today would point to where the Earth’s magnetic north pole is today. If something had been fired 800,000 years ago, when the Earth’s magnetic field was reversed, its magnetism would point towards the south pole where magnetic North was then.

But this alone isn’t enough, because if we change the orientation of the artefact we would change the direction of the magnetism. Fortunately the magnetism of the smaller particles of clay never really settles, but instead accumulates keeping a record of the magnetic field they’ve experienced over time. This accumulated magnetism can be erased by gently heating the object, but the key is that the longer it’s been around the higher the temperature needed to erase this accumulated magnetism.

So by heating the object and determining whether this “viscous remanent magnetism” has been erased or not, you can determine how old the object is. The higher the temperature required to erase it, the older the object.

It’s like you have two pieces of wood, one that was varnished 1000 years ago and one which was varnished last week. The dirt and grime that’s accumulated on the two objects will be harder to clean off on the 1000 year-old object than on the modern one, giving a way of dating the object.

The researchers hope that the method can be applied to articles of debated authenticity for museums and even used as evidence in trials about alleged forgery.

You can listen to the whole episode here: https://www.bbc.co.uk/programmes/w3ct9783

Every six weeks or so I am invited along to appear up on Radio 4’s Inside Science as the “presenter’s friend”, bringing along a couple of stories that have caught my eye and reacting to some of the other science news of the week. I’ve decided to write up some of this week’s stories into a substack post so you can enjoy them even if you missed the show. This week’s appearance turned into something of a mathematical extravaganza, featuring monkeys that can do maths and machines that can do maths too.

Subscribe now

The first story concerned what may turn out to be one of the most remarkable examples yet of AI-assisted mathematics. In the early hours of Monday morning, just after the World Cup final had finished, the mathematician Levent Alpöge posted a tweet that immediately caught the attention of the mathematical world. It began:

“hello there the Jacobian conjecture is false thanx to my close friend Akhil for asking about it and my other close friend fable for working during the world cup final”

Alpöge is a junior fellow at Harvard and also works with Anthropic, whose AI model Claude Fable 5 is presumably the “close friend fable” mentioned in the post. The extraordinary thing was that, in the remaining 216 characters, he appeared to give a counterexample to the Jacobian Conjecture - an 87-year-old problem regarded as one of the most important open questions in mathematics.

Share

The Jacobian Conjecture dates back to 1939 and appeared on Stephen Smale’s famous list of 18 major mathematical challenges for the 21st century. For decades, mathematicians had been trying to prove it true. But now it seems Alpöge has found a simple example showing it was false all along. It’s genuine mathematical mic-drop moment.

Explaining the conjecture without descending into deep mathematics is tricky, but the basic idea is this. Imagine a map covered with coordinates. You apply a polynomial transformation - a formula that shifts every point around using only addition, subtraction and multiplication. Provided the transformation doesn’t locally crush or flatten space, the conjecture asked whether there must always be another polynomial transformation capable of reversing the process and restoring every point to its original position.

What Alpoge effectively did was to demonstrate that there was a function which sent three separate points to the same place, so there would be no way of undoing that function and sending the one point back to three separate points. He effectively found a counter example which is super easy to check.

The big question now is how this result was produced. Neither Alpöge nor Anthropic seem to have said much publicly about the process, leaving mathematicians to speculate. Did the AI generate a genuinely deep new insight? Or did it help uncover something that was already within reach? Either way, it appears to be a dramatic breakthrough, and perhaps a glimpse of how mathematical research may change in the years ahead.

Share Kit Yates - Math(s) and the real world

My second story was another challenge to human exceptionalism. A new study in Proceedings of the National Academy of Sciences (PNAS) explored the origins of geometric intuition and suggests that our understanding of shapes may not be as uniquely human as many researchers previously believed.

For years, some cognitive scientists have argued that humans possess an internal, inherent “language of geometry”, representing shapes using ideas such as symmetry, parallel lines and angles in a uniquely symbolic way. Under that view, recognising a square involves something like mentally checking off a list of geometric properties – two sets of parallel lines, four right angles, all sides the same length.

Researchers from Carnegie Mellon University and collaborators decided to put that idea to the test. They compared olive baboons, rhesus macaques and human preschool children. The children were included because they provide a picture of geometric reasoning before years of formal mathematics education have had their influence.

Participants were shown a shape such as a triangle or square and then asked to identify the matching shape from a set of options on the next screen. The shape might be resized or rotated (but if it started as a square then it was still a square on the next screen), so participants had to recognise the geometry rather than rely on a straightforward visual match.

The monkeys turned out to be surprisingly good at the task. Indeed, they often performed as well as, or better than, the preschool children. The researchers conclude that the foundations of geometric intuition may be shared across primates rather than representing a uniquely human cognitive leap.

The study suggests that what we think of as sophisticated human geometric reasoning may be an enhanced version of an ancestral primate ability rather than a wholly distinct symbolic system, suggesting that we’re not quite as special as we thought we were.

Away from my own stories, there were several other fascinating items discussed on the programme.

Testing testosterone

We talked about proposals from US Defence Secretary Pete Hegseth to introduce testosterone testing for American troops. There is a temptation to think of testosterone as some all-conquering force of nature, capable of explaining strength, resilience and performance. The reality is much more complicated. Most medical organisations do not recommend blanket screening of otherwise healthy adults. Diagnosis of low testosterone typically depends on both symptoms and repeated laboratory tests, not a single measurement. A solitary low result is not automatic evidence of disease or impaired performance. Treatments also carry risks, including infertility, acne, elevated red blood cell counts and increased clotting risk.

We also heard about beavers and their continued reintroduction to British landscapes. I love stories like this because they represent genuine win-win conservation. Reintroducing a formerly locally extinct native species helps protect the animal itself while simultaneously delivering wider environmental benefits. Beaver dams slow water flow and reduce downstream flooding risk. They improve water quality by filtering sediments and pollutants, and the habitats they create support an astonishing range of wildlife. It’s a lovely example of ecological restoration delivering practical benefits for people as well as nature.

Finally, we discussed the extraordinary footage of an orca ramming a sunfish so hard that it explodes. The video is astonishing and offers a reminder that animal behaviour can be every bit as surprising as anything we observe in humans. Fascinating though it is, the footage is also rather graphic, so consider yourself warned before searching it out.

Whether it’s AI challenging our assumptions about mathematical creativity, monkeys challenging our assumptions about geometry, or beavers improving ecosystems through behaviours evolved over millennia, science continues to remind us that intelligence, ingenuity and problem-solving come in many forms.

You can listen to the whole episode here: https://www.bbc.co.uk/programmes/w3ct977x

The greatest cycle race in the world completed its 113^th edition yesterday in Paris. Even people who do not follow the Tour will be familiar with the iconic and much coveted maillot jaune (yellow jersey), the prize for the cyclist who finishes the three-week race quickest. And yesterday Slovenian Tadej Pogačar wrote his name into the history books by winning his ffith Tour de France to equal the all-time record, wearing yellow from day three all the way to the Champs-Élysées.

But from a mathematician’s point of view, perhaps the most intriguing questions about the Tour have nothing to do with the overall winner and more to do with understanding why riders repeatedly do things that seem irrational. Almost every day a small group of cyclists will attempt to break clear of the peloton despite knowing that they are unlikely to stay away until the finish. Why?

Subscribe now

You would be forgiven for thinking that the most important science for understanding bike racing is biology: aerobic capacity, lactate clearance and muscle fibre composition all help determine how much power a rider can produce and for how long. If not biology then surely physics? Cyclists must contend with gravity on the climbs and inertia whenever the pace changes, making the efficient use of energy a constant challenge.

Yet neither of these disciplines fully explains why races play out as they do. Arguably, the clearest lens through which to see the sport is the science of game theory. Game theory has its roots in economics, but can more generally be applied to understanding the interactions of competing individuals. It can suggest optimal strategies for taking penalty kicks, or explain why sometimes it is better for a team to lose than it is for them to win.

When it comes to bike racing, cyclists act in response to the incentives they face. For a select few this might be personal glory, but this doesn’t explain the actions of the vast majority who will never win a grand tour.

Share

Perhaps the most important motivating factor in the modern-day Tour de France has nothing to do with the competitors, the biology or the physics of the sport at all. Marketing is what keeps the Tour’s wheels spinning around. Running a pro cycling team is not cheap, so teams must rely on their sponsors, whose names the teams often carry, for financial support. In return, sponsors expect to have their brand highlighted to the Tour’s global TV audiences at any available opportunity.

The most obvious way to do this is to go after the famous yellow jersey in the competition referred to as the General Classification (GC). But it’s important to remember that there is not just one prize to contend for. There are other jerseys awarded over the course of the race: green for the points classification, red and white polka dot for the King of the Mountains and white for the best young rider. The current leader of each competition heads to the podium after each race, to bask in that hard-earned publicity.

Indeed the Tour de France is not just one race. It is races within races within races. There are also two other daily prizes, more transient in nature than the jerseys, but which also afford a trip to the podium – one for the stage winner and one for ‘combativity’. The combativity award is generally given to the rider who most aggressively attacks the stage. One of the best ways to go about winning this is by getting in a ‘breakaway’.

Share Kit Yates - Math(s) and the real world

Naively you might think that the best riders would just get into the breakaway every day and win the race that way, providing non-stop coverage for their sponsors in the process. But that would be to neglect the very significant impact of air resistance. Walking around we barely notice its effect, but at 25 miles per hour, professional cyclists can’t help but feel its drag. Riders at the front of a group bear the brunt of the physical effort of cutting through the air, effectively shielding the riders behind them. Riding in a big group – the peloton - the hard work can be rotated between hundreds of riders. There is an incentive for everyone, even across rival teams, to cooperate because everyone benefits from sharing the aerodynamic burden.

So riders trying to get off the front of the peloton in a breakaway are penalised. A single rider alone would have to do all the work, while even a small team of riders has to do much longer energy-sapping turns on the front (while the others draft behind) than they would in the bunch. Perhaps it is the potential for the individual glory of the stage win that motivates them? But the probability of the breakaway staying out is low. Even if they succeed, only one rider in the group can win, further reducing each individual’s chances. With the odds stacked against them, why do most stages see a small bunch of riders forging out alone? Understanding these seemingly irrational actions is where game theory comes into its own.

The primary motivating factor is the TV coverage afforded to the sponsors of the teams with a rider in the group. This provides the incentive the teams’ directors need to compel their riders to attempt to get in the breakaway. According to game theory, incentives dictate ‘players’ actions.

But despite their best intentions, it is typically within the power of the peloton to pull back any breakaway. Indeed breakaways containing any of the major yellow jersey contenders will typically not be allowed to get away in the first place. The risk is too great for the other GC contenders, so their teams will come to the front of the peloton for a short stint, riding hard to pull them back.

But a group of riders who are not deemed to be a threat for the GC race might be allowed to get ‘up the road’. And for a while they might be allowed to stay there. But if it’s a ‘flat stage’, one in which the big sprinting contenders for the green jersey are interested in winning for themselves, then their teams will come to the front towards the end of the stage, different teams working together to absorb the breakaway back into the pack in the last few kilometres, giving their star performers a chance to battle it out for the win.

Indeed, such is the strength of the incentive for the sprinters’ teams and the importance of air resistance that on flat stages, which suit the heavier more powerful sprinters better than the mountainous stages, only 2% of breakaways stay ahead until the end and win the stage. On the more mountainous stages where air resistance is less important, diminishing the peloton’s advantage, the percentage of successful breakaways rises into the 40s.

Even when a breakaway looks likely to have a big enough advantage to hold off the charging peloton, game theory has one more nasty trick to pull. While the incentive at the start was for all the members of the breakaway to pull together to distance the chasing pack, despite being on different teams, as the end approaches the calculus changes. In a sport where second is the first loser, the overwhelming incentive of the stage victory sees this band of former comrades turn back into competing individuals. People will stop taking turns on the front as they conserve their energy for a sprint finish. Those who do find themselves at the front are forced to constantly look around guarding against attacks from riders (especially those who aren’t the strongest sprinters) thinking of making a break from the breakaway.

While finishing ahead of the chasing pack would be a better result for everyone than being overtaken by the peloton, each rider has a personal incentive to save energy for the finish. The premium on winning over placing high serves to slow the group down to the extent that they can be caught. Everyone loses in what game theorists would call a ‘tragedy of the commons’.

The complexities thrown up by the intricate incentive structures that grand tour cycling offers really can be best understood in light of the maths of games, explaining to some extent why this cerebral and calculating aspect of cycling is referred to as chess on wheels.

A version of this article appeared first on the BBC future website.

Photo by Árni Svanur Daníelsson on Unsplash

It has been called the most extraordinary game of football ever played. Yet there was no global audience that comes with the World Cup Finals, there was no league title or famous trophy on the line and there was no fierce local rivalry to add an edge to the game. Instead, this extraordinary match was a round-robin game between Barbados and Grenada in qualifying for 1994’s Caribbean Cup.

Barbados looked to be cruising to a 2-0 until the 83rd minute when Grenada pulled one back. Upon the restart Barbados funnelled the ball to defender Terry Sealey, who passed it back and forth with goalkeeper Horace Stout until Sealey intentionally blasted the ball into his own net to make the score 2-2.

Subscribe now

Then things got even weirder. The Grenadians tried to do the same thing. They attempted to score in their own net only to find the Barbadians blocking the way, so they went up the other end to try to score in the Barbadian’s goal only to find their way blocked again. They continued to try to score in both nets until the full-time whistle. In extra time the normal pattern of play seemed to resume, with the Barbadians eventually scoring the ‘golden goal’ which ended the game and saw them win the match.

So what was going on? In an attempt to make the qualification stage of the cup more exciting, the organisers had decreed that there would be no draws. If a match finished tied then it would go to extra time. A ‘golden goal’ was introduced which, if scored, would immediately end the game. The golden goal would be worth two normal goals - a quirk expected to make little difference. Only it made all the difference.

Share

Having beaten Barbados 1-0, but lost to Grenada 2-0 and with only the top team going through to the cup proper, Puerto Rico were effectively eliminated. Barbados, however, had a slim chance. They would have to beat the table-toppers by two clear goals or more to make up the three goal deficit in goal difference and overtake Grenada.

At 2-0 in the 83rd minute they were sitting pretty, but when they conceded, the 2-1 scoreline would no longer cut it for them. Instead of trying to score in the remaining 7 minutes, the Barbadians reasoned it would make more sense to take the game to extra time by scoring an own goal, giving them 30 minutes to score the golden goal, which would be worth two. So they duly did.

When the Grenadians cottoned on to what their opposition were doing, they tried to score an own-goal themselves which would have seen them defeated, but not by a big enough margin to allow Barbados to overthrow them at the top of the group. Scoring in the Barbadian net would also have given them the victory, which is what led to the Barbadians having to defend both goals. Somehow they managed to achieve the remarkable feat, holding on for extra time, scoring the golden goal and finishing the match 4-2 winners: the two-goal margin they needed to top the group.

This is a stunning example of a perverse incentive: a tweak in the rules which causes behaviour that is the opposite of what was originally intended. Perverse incentives are sometimes known as the cobra effect. The name derives from a story which goes back to the era of the British Raj in India.

Share Kit Yates - Math(s) and the real world

Bureaucrats in Delhi became concerned about the number of venomous cobras in the city. To resolve the problem, they placed a bounty on each cobra’s head. Soon after, thousands of dead cobras began to pour in. But rather than going out and catching cobras, some entrepreneurial individuals set up lucrative cobra-breeding programmes to cash in. As soon as the British got wind of the breeding ruse, they withdrew the bounty. With no viable income stream, the temporary serpent-breeders decided to release the cobras in huge numbers.

In a sporting context the cobra effect has reared its head on multiple occasions. At the 2012 Olympics, badminton players from China, South Korea and Indonesia deliberately attempted to lose matches to secure an easier draw in the knock out stages.

The England national football team had a similar incentive to lose their last group-stage game to Belgium in the 2018 World Cup, to ensure they were placed in the easier side of the knockout draw. England’s manager made eight changes and his side duly lost the game, securing a route which led them all the way to playing the lowest-ranked team in the semi-finals^[1].

In American sports, teams with a worse record one season get earlier picks in the draft (the process by which teams select new players for the next season). As a result, there has often been an incentive for teams who aren’t fighting to make it into the playoffs to ‘tank’ by losing games late in the season.

There is a branch of mathematics which potentially allows tournament organisers to avoid these perverse incentives. Game theory assumes that parties competing with one another are rational and act in their own self-interest, and provides a framework for understanding how rational participants will act under a given incentive structure.

Indeed, in Major League Baseball, the National Basketball Association and the National Hockey League, a mathematical ‘anti-tanking’ strategy was developed to prevent teams from deliberately losing matches towards the end of the season. Instead of the worst team getting the first pick in the draft, the teams that aren’t in the playoffs are entered into a lottery to determine who gets the first pick, dampening the incentive to finish bottom.

Ultimately, if you don’t think carefully about what your rules are incentivising, like the tweaks in the Caribbean Cup qualifying, then even rational actors can end up doing some seemingly strange things.

^[1] There is no suggestion that England deliberately played poorly in order to lose the game.

Title photo by Joakim Nådell on Unsplash

A version of this article was first published on the BBC.

England are playing Argentina at the World Cup. Every four years, the tournament brings back memories of the very first World Cup I was ever truly invested in. The excitement of watching England perform in a World Cup for the first time since I’d been old enough to engage with the sport has never left me. But England vs Argentina always stirs up very specific memories for me. Memories I would rather leave in the past.

As a 13-year-old, France 1998 would set me and the rest of my peers up for a lifetime of disappointment as an England fan. But the thrill and the adrenaline of watching that last 16 game against Argentina, even though we lost, was undeniable. Being only dimly aware of Maradona’s antics in 1986, and the political backdrop of the Falklands war, this game renewed the rivalry for a new generation of football fans. Batistuta’s and Shearer’s penalties, Zanetti’s cleverly worked free kick routine, punctuated by that wonder goal from an 18-year-old Michael Owen, left the tie neatly poised before Beckham got himself sent off. Still, we (and it was ‘we’, not ‘they’, by this point) struggled on through extra time to penalties, where we eventually succumbed – a pattern that would repeat itself over and over throughout the tournaments of my youth.

Subscribe now

I find the arrival of each World Cup cathartic. Each time the tournament rolls around it acts as a marker, a catalyst for me to remember where I was and what was happening in my life as I shared these collective moments with the nation. But those extra-footballing memories, especially those triggered by England vs Argentina, are always tinged with melancholy for me. They take me back vividly to the moments before and after the death of my best friend, Nick, during the World Cup of 2002. And when I reflect on World Cups past, this is inevitably where I end up fixating.

Nick and I were both ardent followers of domestic football. Nick was a huge fan of Manchester United, while I, indoctrinated by my dad against the sway of almost all my friends, supported Manchester City. I had been to Old Trafford with Nick a few times over the years, when his dad’s seat in the South Stand was going spare. More recently, since we got season tickets at Maine Road, Nick had come a few times to sit next to me in the Main Stand to watch as City battered sides in the First Division when my dad couldn’t make it.

Usually the animosity between City and United fans would have made this sort of thing difficult, but Nick and I were tight enough that we never let our support of rival teams get in the way of watching a game. Instead, we learned to live behind enemy lines, standing when the home fans stood and pretending to celebrate our rivals’ goals so as not to stand out too much among them, although neither of us could ever muster the full shouting-arms-in-the-air-fist-clenching celebration that the other would exhibit at home games. It was a kind of truce, and it felt grown-up at the time, like we’d found a way to still be Kit and Nick, even when we weren’t allowed to be exactly who we wanted to be. And besides, with City, until that summer, a whole division below United, we weren’t even their main footballing rivals. Liverpool, Arsenal and by some reckonings even Leeds ranked above City in the pecking order of United’s antagonists; a fact that, in itself, put a huge dent in the pride of City fans of that era. But as we both saw it back then, it wasn’t as if City were ever going to be a threat to United’s seemingly unending dominance of English football, was it?

Share

City had just won promotion back to the Premier League, so we faced the prospect of that fierce rivalry being renewed, with those much-missed Manchester derbies back on the table. But that summer was a time for putting domestic rivalries to the back of our minds and focusing on our joint support of the national team.

In truth, the end of the domestic season and all the international football that followed that summer was overshadowed by Nick’s illness. Just a few months before, Nick had been diagnosed with familial adenomatous polyposis, a genetic condition which made cancer much more likely. And just a few weeks after that, we had been told that the gastrointestinal cancer he had indeed developed was terminal.

It was barely believable. The transformation from the vivacious young lad who had bested me in a kick-up competition in my back garden in April, all the twists and turns of life seemingly in front of him, to the cancer patient who lay dying in June, ferried between his bedroom and hospital, was almost too surreal to wrap my head around.

I went to see Nick both in hospital and at home while he was ill. The last time I went to see him was on the 7^th of June, the afternoon of England’s group game against Argentina. He was so unwell that I felt I was trespassing on his time, forcing him to be social when probably all he wanted to do was sleep. We discussed the upcoming match and he told me how he was looking forward to watching it, but our conversation was stilted and felt forced.

What did we have to talk about: these two seventeen-year-olds who were now on such radically different paths? I tried some reminiscences, but they all seemed to fall flat. What do you talk about with a teenager who is about to die? How pissed off you are that you are going to lose your best friend? How cheated you feel for them that they will never get to live out their dreams? How you will try to honour them in the way you conduct yourself throughout life as your timeline continues while theirs is cut abruptly short? I couldn’t get the words out.

I found the whole experience so upsetting that I am ashamed to say I didn’t see him again before he died. I told myself it was because he wouldn’t have been able to muster the energy to see me and that I didn’t want to intrude on his final moments with his family. And maybe this was true in part, but we had been closer than brothers. He was my family. Secretly, my excuse was just a lie I told to comfort myself because I was too afraid to look my best friend in the eyes as he lay dying.

Share Kit Yates - Math(s) and the real world

Richard, Nick’s dad, told me later that Nick barely managed to watch any of the Argentina match. Nick did manage to catch Beckham’s penalty, Richard told me, and held his fist aloft when the final whistle blew, celebrating an act of revenge which went some way to undo the hurt of that first indoctrinating match in 1998. The victory was ultimately crucial in securing England’s qualification from the group stages, so that we could face Denmark in the round of 16.

He didn’t get to see that game. On the 12^th of June, Nick died in hospital, surrounded by his family. I watched England beat Denmark and then lose to Brazil in the quarter-finals. It didn’t really matter. There wasn’t much that really seemed to matter that summer after Nick had died.

Now, whenever the World Cup kicks off, I think of Nick lying on his sickbed, punching the air, and am reminded of that internal battle I faced as a 17-year-old. The battle that, to my eternal shame, I won and lost.

Whether it’s football stickers in the UK or baseball stickers in the US, kids have been buying, coveting, trading, swapping and sharing collectibles of their favourite sports icons for decades. With the arrival of the World Cup comes the excitement of a new sticker book to complete and the expectation that every new packet of stickers purchased will fill in gaps in the album, taking you one step closer to completion.

But as the tournament progresses and your collection grows, it seems harder and harder to fill those elusive gaps – to find the missing shiny or the obscure defender. Buying a new packet that yields only swaps with no new gap-filling stickers can feel like throwing money down the drain. The despondency can leave many, initially avid collectors, to question how much it will cost them to complete the album and many more to abandon their collection before reaching the end.

Subscribe now

So how much should it cost to complete an album? Well that depends on your strategy. If you are planning to simply buy packet after packet to complete your set then you’re going to have to have deep pockets. The 2026 Panini World Cup sticker album comes complete with 980 different blank spaces to fill. With each pack of seven stickers costing £1.25 you’re looking at a minimum possible outlay of £175. But this would require each sticker you find to be a new one, with no swaps, and while this is a statistical possibility, it’s so astronomically unlikely as to make it practically impossible.

In reality, for each gap we want to fill, we are going to have to buy multiple stickers. The first sticker we get is guaranteed to be completely new to us, so the expected number of stickers we need to buy to fill the first empty spot in the book is just one. To fill another spot in the album our next sticker can be any of the 979 stickers we haven’t already got, which happens with probability 979/980. This means that on average we will need to buy 980/979 (just over one) stickers to fill this next slot. And so the trend continues with the next spot needing 980/978 stickers and the one after that requiring 980/977 stickers on average.

Share

Because these numbers are all quite close to one, it’s likely that we won’t get any repeats for a while. However, as we draw closer to completing the album, the expected number of stickers we need to buy to fill the next gap balloons massively. We find the penultimate sticker, which is only one of two that we don’t yet have, with probability 2/980. This means we need to buy 980/2=490 stickers on average to fill the penultimate gap. The last sticker is even worse. On average we need to buy 980 stickers to find the elusive one we are looking for.

Adding all these up, in total we would expect to have to buy

stickers at a cost of £1307.50 (remembering you have to buy stickers in packs of 7).

In general, if your sticker album contains spaces for N stickers, then you will need to buy

stickers, on average, to complete your set.

Share Kit Yates - Math(s) and the real world

The diminishing set of fractions inside the brackets are a well-known mathematical object called the ‘Harmonic Series’. The name comes from the concept of ‘overtones’ or ‘harmonics’ in music, which give each instrument its distinctive sound. The fundamental frequency of vibration of a string determines its basic pitch. The string’s harmonics have frequencies that are whole number multiples of the string’s fundamental frequency – that is to say they oscillate 2, 3, 4 etc times more often than the fundamental wave does. Correspondingly, the wavelengths of the harmonics are 1/2, 1/3, 1/4, etc of the string’s fundamental wavelength (as illustrated in the image below).

The formula we found for completing the sticker album crops up in a huge variety of different areas because it relates to the cost or effort required to fully explore an unknown set. In market research if you want to sample the full range of different consumer preferences then the harmonic series tells you how many people you will expect to have to survey. For DNA sequencing, where DNA is broken into fragments, sequenced and then reassembled, the harmonic series helps you approximate how many fragments you should expect to sequence before you have coverage of the whole genome.

In ecology, scientists use the harmonic series to estimate the number of species in a particular habitat. The underlying theory is not dissimilar to the sticker album filling problem. Imagine there are a number, N, of species in the ecosystem the scientists are studying. Each time they observe the system they note how many different species they find. If it were a collector’s album of species then the question would be ‘how many observations would the scientists expect to make before they observed all N species?’ Instead, since the number of species in an ecosystem is not known in advance but something we would like to know, the problem is often the other way around: given that I’ve seen a certain number of species from my observations of this ecosystem so far, how many are still out there that I’ve not seen. The harmonic series is key to estimating the distribution of this unseen biodiversity.

Heading back to our sticker collection problem, the harmonic series formula not only tells us how much it will cost to fill a sticker album, but also explains why it is so much more satisfying (not to mention quicker) to fill the first half of the album than the second. To fill the first 490 gaps of a 980-sticker album we would expect to have to buy just 679 stickers at a cost of £121.25. But to fill the second half we would need to buy another 6637 stickers, at a cost of £1186.25.

If we thought it were going to cost us well over £1000 to fill our sticker albums then we’d probably never even buy them in the first place. In reality, once you start to get duplicates then you can begin to swap them with your friends, filling in the gaps in both your albums. There is a mathematical formula relating to the harmonic series which can tell you how many stickers you and your friends would need to buy to complete all your albums, but it’s complicated. Imagine you’ve got 10 friends who are all collecting and you want to build up 11 complete albums (one for each of you). With our 980-sticker album example, you would need to buy roughly 26,230 stickers, which works out at roughly 2385 stickers each at a cost of about £426 – a substantial saving compared to the £1300 you’d spend collecting alone. Even with just four other friends, the cost would come down to £533 each – still a substantial saving.

Even without the help of your friends, once you get close to the end of your collection, you can usually head to the Panini website and order any missing stickers. They typically cost you twice as much as a normal sticker and there’s a strict limit on the number you can purchase. If you’re interested, the most cost effective way to complete the album would be to collect the first half (490) of the stickers by buying packets at a cost of about £121 and then to order the rest from the website for about £175. This would set you back £296, which isn’t too far from the minimum possible outlay of £175. Whether Panini let you order the 490 missing stickers directly is a different matter.

Of course, in this modern age, you can collect your ‘stickers’ in an online album. There’s no need to go to the shops to buy your sticker packets, no call for spending time swapping with your friends and no requirement to painstakingly stick the stickers into the album in the right place. But where’s the fun in that?

A version of this article was originally published on BBC Futures.

The last ten days have been a bit of a busy/surreal for me, so I thought I’d write a summary of everything I’ve been up to.

The main thing that’s been going on is that my new book, You don’t Know What You’re Missing, came out last Thursday (04/06). On Wednesday I went down to Old Broadcasting House in London to record an episode of BBC Radio 4’s All in the Mind (with the excellent Claudia Hammond), about the more psychological aspects of the book. That episode comes out in a couple of weeks, so if you’d like to hear it, then watch this space.

Buy the book

On Thursday morning – publication day – I woke up early, headed over to the ITN studios on Gray’s Inn Road to appear on Jeremy Vine’s Channel 5 TV show. Jeremy has always been super kind to me, ever since he read my second book - How to Expect the Unexpected. It was lovely of him to have me on the show to promote this new book.

I also got a little taste for what his regular guests are letting themselves in for, presenting and commenting on a few of the day’s news stories that had been handed to me just a few minutes earlier. I have to say, I don’t envy them the job of having to talk coherently about a topic they’ve had less than five minutes (on average) to prepare for. Stressful.

If you’d like to see some of the more visual examples from the book, then I’ve included the full clip of the segment here.

I spent a quiet hour in the British Library before heading over to New Broadcasting House to appear live on Jeremy Vine’s Radio 2 show. Where I also bumped into Michael Berkeley from Radio 3’s Private Passions.

On Jeremy’s show we talked about the different sorts of missingness, from the gaps in our perception to the power of expectation, the ways in which information can be biased before it even reaches us and even the ways in which missingness can be useful.

You can catch up here from 1:06:30 (just so you know what you’re missing!).

The publicity drive seems to have had an impact, as You Don’t Know What You’re Missing made its way up into the low 100s across all books (it’s currently still the number one best seller in popular maths!).

After a quick pit stop at home on the Thursday evening, I headed over to Cheltenham on Friday to host a session in the Town Hall called “How to Win the World Cup”. On the panel were Liverpool’s former head of Data, Dr Ian Graham, and sports psychologist at Liverpool John Moores University, Dr Gillian Cook. On the night, their insights into everything from forming a cohesive squad to how to manage penalty shootouts were absolutely fascinating.

Share

Ian also has an excellent book called How to Win the Premier League, which is well worth a read if you’re interested in football (particularly the evolution of English football over the last 15 years). I had a lovely chat with him over dinner afterwards gleaning lots of insight, but also chatting nonsense about football for an hour, which is always high up my list of priorities for a Friday night.

I took the chance later in the evening to catch up with a few science communication colleagues, including Andrea Sella, Mark Miodownik and, a bit later, (too late really – I need early nights) Adam Rutherford, Kevin Fong and Alice Roberts. This is the really nice thing about Cheltenham Science festival: the opportunity to meet up with old friends and to make new connections.

And then came the most surreal part of the evening. As I was sitting down at one table, someone else was just leaving. I introduced myself as “Kit”, and he replied, “Kit Yates? I know you”. It turns out that this was the comedia John Harvey (aka Count Binface – not in costume, otherwise it might have been quite obvious!). He works with another maths communication friend of mine, the comedian Matt Parker, hence the connection. So let me finish by wishing John (Count John?) the best of luck in next week’s Makerfield by-election: you can read his manifesto here.

Subscribe now

So that was it for my surreal week. You can tell how hectic it was because it’s taken me a whole week to recover form it and write this post.

If you think you’d like to join the other people who are reading the book, then you can pick up your own copy:

Amazon: https://www.amazon.co.uk/Dont-Know-What-Youre-Missing/dp/1529438039

Bookshop.org (supports independent bookshops): https://uk.bookshop.org/p/books/you-don-t-know-what-you-re-missing-the-science-of-what-s-lost-and-how-to-find-it-kit-yates/497ab9dcf971763a

*

Ahead of the men’s Football World Cup in a few days, former England captain Kevin Keegan shared the news that he had been diagnosed with cancer. While much of the focus on the reporting of the news has rightly been on how the footballing world has rallied around the former Liverpool and Newcastle great, there is an interesting lesson to be learned from the way in which Keegan’s cancer was discovered.

Keegan was in a car crash just a few weeks before his diagnosis. Multiple studies have shown that people who have the sorts of injuries caused by car crashes are also found to have a higher incidence of cancer. That is to say, there would be a larger number of cancer diagnoses in a group of people who’ve had car crashes than there would be in the same size group of people with similar backgrounds who haven’t had car crashes.

Subscribe now

Most experts believe that there is little evidence to support the hypothesis that traumatic injuries sustained in car crashes can cause cancer. So if it isn’t the effect of traumatic injury sustained in the accident, what could explain this link? Maybe there is an underlying common cause? Perhaps people who drive more are likely to live more sedentary lifestyles which comes with increased cancer rates? Maybe frequent drivers are exposed to more cancer risk factors (such as obesity or UV exposure), while the increased driving time means they are also more likely to be involved in accidents? Maybe sleep deprivation leads both to more accidents and a higher cancer risk?

To correctly resolve the conundrum, we have to think about how we find cancer. Typically, we diagnose cancer in people who undergo some type of medical examination – either because their cancer has caused them to feel unwell or for some other reason. Often, people who are involved in car crashes end up in hospital. If we run a CT scan or an MRI on them to understand if there has been any internal damage as a result of the accident, then we may incidentally find a tumour. It’s not that being in a car crash has increased their chance of having cancer, but rather the chance that it is detected.

Indeed this is exactly how Keegan recently described the detection of his cancer. “I was in a car accident and, through that, I had to have an operation. Whilst having the scan for the operation, they found out I had cancer.” The underlying rate of cancer isn’t any higher in a group of people who’ve been in car crashes than in a similar group of people who haven’t. It’s just that we have a better chance of diagnosing it in car crash victims because of the increased medical scrutiny they undergo.

Share

That’s not to say car-crash victims are the only group of people with higher rates of cancer diagnoses. People who attend accident and emergency for almost any reason are subject to increased medical attention and, as a result, will also have higher rates of incidental diagnosis.

As I outline in my new book, You Don’t Know What You’re Missing, this is a classic example of a detection bias - increased monitoring of particular scenarios in comparison to another can induce the perception that one phenomenon has a higher probability than another. For example, despite shark attacks being incredibly rare (around eighty unprovoked attacks and only a handful of fatalities worldwide each year on average), the public are disproportionately worried about sharks.

Buy the Book

In large part, this is probably due to an availability bias – shark attacks are so graphic and feature prominently in popular culture (think Jaws, the Reef, the Shallows, etc.) that they occupy a disproportionate amount of space in our imaginations. However, in part this may also be due to the misconception that sharks are attracted to popular beaches, where many people like to go to swim.

While there is some evidence that some species of sharks might be attracted by splashing, as it sounds like struggling prey, other species are put off by it. There is no strong evidence to suggest that more humans in the water will lead to a higher probability of attracting a shark.

It is true, however, that there are more shark attacks in places where lots of people swim, but this isn’t primarily because sharks are disproportionately attracted to these popular areas. Gavin Naylor is the director of the Florida Program for Shark Research. His data show that the number of shark attacks in the US is strongly correlated with the numbers of people attending the beach. Naylor argues that ‘The more sharks and people there are in one place, the greater the chance of them bumping into each other’; so popular beaches may see more attacks simply because there are more people in the water, not because sharks have a preference for popular beaches.

Share Kit Yates - Math(s) and the real world

It’s also true that busy beaches are disproportionately more likely to report shark sightings. But again, this is purely a function of the fact that more people are around to spot their tell-tale fins poking out of the water – another detection bias. Indeed, well-frequented beaches are more likely to have lifeguards who may be on the lookout for sharks or even to employ drones to help reassure beach users that it’s safe to go in the water.

Other detection biases can mean that interventions can seem to have the opposite impact to the one we might naively have expected. When policing is increased in an area, you might expect that key crime measures would go down, but instead they often go up. Why? Because more police usually means that more crime is detected and recorded even if the underlying crime rates remain the same or even, in fact, go down. Similarly, workplaces with better safety protocols can counterintuitively appear to have higher rates of workplace safety breaches. This can simply be a result of the fact that more breaches are detected and reported, rather than a result of the safety protocols making things less safe.

Multiple studies suggest that, similarly to Kevin Keegan’s case, many cancers are first spotted incidentally in accident and emergency in people who have attended for completely unrelated reasons. It’s important to remember that just because we see a higher incidence of cancer in people who attend accident and emergency, it doesn’t mean that being acutely unwell for other reasons is linked to having cancer. In fact, there is evidence to suggest that being unwell to such an extent that you require increased medical attention can actually prolong your life expectancy through the incidental finding of other diseases like cancer.

It’s publication day, and this is the final post in this short series summarising the chapters of my new book, You Don’t Know What You’re M ss ng. The aim has been to give you a sense of the shape of the argument, the kinds of stories I use in the book, and the ideas I hope will stay with you after you’ve finished it.

In this post I’m going to summarise Chapter 7, The silence that speaks.

Buy the book

“Shape clay into a vessel. It is the space within that makes it useful.” - Lao Tzu

This final chapter’s opening story is about the Soviet physicist Georgy Flyorov. During the Second World War, he noticed something strange. Papers on nuclear fission, which had previously appeared in major Western journals, had suddenly stopped - a piece of negative data. Flyorov inferred that the work had not dried up, but was instead being hidden as a result of wartime censorship because the Allies were pursuing an atomic bomb. He wrote to Stalin, urging the Soviet Union to act. His reasoning from negative data helped trigger the Soviet nuclear programme.

This chapter is all about that kind of reasoning - what I call constructive missingness. Earlier in the book, missingness often involved our brains filtering things out or smoothing things over (intrinsic missingness), or the world presenting us with incomplete and biased information (extrinsic missingness). In this chapter we consider a third type of missingness. This time the key is not just that something is absent, but that its absence is itself informative: sometimes not finding what you expected to find is a vital piece of information.

Subscribe now

Detectives (especially fictional ones) understand this well. Sherlock Holmes’s famous “curious incident of the dog in the night-time” revolves around exactly this sort of reasoning. A guard dog at a stables doesn’t bark during the night a prize thoroughbred is stolen. That absence of action, Holmes realises, is a key piece of information. If the dog did not bark, the intruder who is linked to the crime must have been known to it. The absence of an expected reaction becomes evidence. The chapter traces that same logic into modern court cases and forensic puzzles, where missing fingerprints, absent noise, or the failure of some expected trace to appear can be almost as telling (if not more) than the clues that are physically present. The challenge, of course, is that negative data are often harder to notice precisely because they are not there to stare us in the face.

But this kind of reasoning comes with an important warning label. The absence of evidence is not always evidence of absence. If you open the fridge and cannot find the butter after a proper rummage, that is pretty good evidence that the butter is not in the fridge. But when the thing you are looking for is more elusive, or when the search has been only partial, or the evidence would be hard to detect anyway, that logic becomes much shakier. Not finding evidence for something does not automatically prove that it is not there. That is why the chapter spends a bit of time deconstructing an oft-used truism: “absence of evidence is not evidence of absence” – sometimes it is and sometimes it isn’t. What the absence of evidence can never be, however, is proof of absence.

Share

The case of the coelacanth is an important example that reinforces this idea. For decades, all the available evidence (no-one had ever found a live one) suggested this fish had gone extinct tens of millions of years ago. Then in 1938, a living specimen turned up. One fish was enough to overturn a century of increasingly confident scientific assumption. Negative evidence can be strong without being definitive. The more extensively we have looked, the stronger the evidence from absence becomes - but proof is another matter altogether.

That tension leads into one of the chapter’s bigger themes: the difference between science and mathematics. In mathematics, if the axioms hold and the logic is sound, a valid proof gives certainty. In science, things work differently. We gather evidence, we develop theories, we test them, and we gain confidence - but we never prove anything in the same absolute sense. Newton’s theory of gravitation worked remarkably well until it didn’t. Einstein’s general relativity repaired some of the gaps, but it too is incomplete. Scientific theories are robust not because they are ever proven beyond all doubt, but because they survive repeated attempts to break them.

Share Kit Yates - Math(s) and the real world

The chapter becomes somewhat philosophical towards the end as we consider the importance of falsifiability. The real power of a scientific theory is not that it can be endlessly confirmed by examples that fit it (although that may give us more confidence), but that it makes risky predictions that could, in principle, prove it wrong. If your theory is “all mammals give birth to live young”, then observing yet another dog having puppies doesn’t really give you much more evidence than you already had. Finding a platypus that lays eggs, however, can completely debunk the theory. A single contradictory example can do what a million confirming examples cannot.

That sounds straightforward, but psychologically it is harder than it should be. We are naturally drawn towards confirmation. We look for examples that fit our current ideas rather than cases that might undermine them. The chapter revisits Peter Wason’s classic selection task, where most people choose options that might confirm a rule rather than the options that could show it to be false. It is a succinct little demonstration that the habits of good scientific reasoning do not come naturally. We prefer examples that reassure us. Contradictions are more informative, but they are also less comfortable.

If we want to know anything new about the world, we have to give up the dream of perfect certainty. Mathematics can build towers of logic, but it cannot tell us by itself what the world is like. Science can tell us about the world, but only by living with uncertainty, by testing ideas against evidence, and by accepting that what survives scrutiny has not been proved true, only not yet been proved false. Constructive missingness is the way in which we use the absence of evidence to reason constructively about the world. Sometimes the thing that matters is what fails to appear. Sometimes the most important clue is the one that isn’t there. But to use that clue well, you have to think carefully about whether the silence really means something, or whether you simply haven’t listened hard enough.

Buy the book

A favour: order the book

If you’ve enjoyed this summary and the other summaries in the series, you’ll find much more detail in the book itself, with the stories and the science explained in more detail.

If the book sounds like your sort of thing, please consider ordering it this week. First-week orders matter far more than most readers realise. They’re one of the strongest early signals that a book has an audience, which influences everything from how many copies are stocked to how widely it’s recommended.

Amazon: https://www.amazon.co.uk/Dont-Know-What-Youre-Missing/dp/1529438039

Bookshop.org (supports independent bookshops): https://uk.bookshop.org/p/books/you-don-t-know-what-you-re-missing-the-science-of-what-s-lost-and-how-to-find-it-kit-yates/497ab9dcf971763a

Thanks,
Kit

Today is the penultimate of the short series of posts I’m writing for this Substack summarising the chapters of my new book, You Don’t Know What You’re M ss ng, ahead of publication later this week (4^th June). If you’ve been enjoying them, then take heart because the book is almost here. If not then take heart because this series of posts is almost over! Either way the aim is to give you a feel for the shape of the argument, the kinds of stories I use in the book to get the message across, and the ideas I hope will stick with you long after you’ve finished reading it.

In this post I’m going to summarise Chapter 6, Where the light is.

Buy the book

“Where the light is brightest, the shadows are deepest” - Johann Wolfgang von Goethe

One of the easiest mistakes to make when thinking about the past is to assume that what survives is somehow representative of what that era or epoch in history was like. We speak casually of “cavemen” as if our prehistoric ancestors mostly lived in caves, when in reality caves are simply the places where the evidence of their existence is most likely to have survived. Tools, bones, art and ornaments are preserved there because caves protect them from the elements. Most of the homes, objects and other traces of the people who lived out in the open have vanished. Caves are where the evidence lasted long enough for us to discover it.

That is the story of Chapter 6: Where the light is. Again and again, we focus on what survives, what is visible, what is memorable, what is measured, what is easiest to observe. And because those things are available to us, we instinctively treat them as representative. But very often they are not. They are simply where the light happens to fall.

Subscribe now

One form this takes is survival bias. Survivors are the ones who remain visible. They are the successful people whose stories get told, the organisations that endured, the species that made it, the products that still exist. The casualties, both literal and metaphorical, are often missing from view. That is why stories of success can be so misleading. We hear from the author who persisted through rejection and became famous, but not from the many equally talented people who gave up on the twelfth rejection letter and never made their mark on the cultural tapestry. The danger is that we miss all these representative failures and build our theories of success from the winners alone.

A close cousin is the spotlight fallacy: the assumption that something highly visible is representative of the whole situation. A high-profile error can make a person seem more generally incompetent. A few widely publicised cases can make an entire group seem dangerous, unruly or untrustworthy. Media coverage can exacerbate the effect. The chapter explores the spotlight fallacy through the treatment of Liverpool and Liverpool supporters in the years before and after the Hillsborough disaster. Because football fans - and Liverpool fans in particular - had already been cast in the media spotlight as hooligans, the disaster was initially understood through that lens. Drunkenness and disorder amongst fans were blamed, even though the underlying causes were failures of crowd control, policing and infrastructure. The spotlight did not merely distort public understanding after the event through biased media coverage, it almost certainly shaped the decisions that made the disaster worse while it was unfolding.

The spotlight fallacy is one of a broader family of biases known as availability biases: what comes to mind comes to matter. If an example is vivid, shocking or easy to recall, we tend to give it disproportionate weight when making a judgement. That is why people often worry more about shark attacks than about far more likely dangers in the sea like rip currents. Shark attacks are graphic, memorable and heavily represented in popular culture. Rip currents are deadlier, but much less cinematic. Availability biases, in this context, can result in a skewed sense of risk.

Share

And because memorable things are often the strange or unusual ones, the bias feeds on itself. The most bizarre and unusual items in a list stand out while the ordinary ones dissolve into background. In the chapter we encounter the story of a bank robber whose brightly coloured orange trainers end up giving him away because they were so memorable to witnesses. Distinctive details are easier to notice, easier to remember and easier to over-interpret. The flip side, also covered in the chapter, is hiding things in plain sight: if you want to make something go missing, one strategy is to hide it among more of the same. The best place to hide a leaf is in a forest.

Share Kit Yates - Math(s) and the real world

Another major theme of the chapter is detection biases. Sometimes a phenomenon appears more common in one place than another not because it really is more common, but because we are looking harder there. Busy beaches report more shark sightings partly because more people are there to spot sharks. Better-policed areas can seem to have more crime because more crime is detected and recorded.

This idea becomes especially important in medicine. The chapter uses the example of incidental cancer diagnosis to show how increased scrutiny can create the appearance of a relationship where none exists. People who have had major accidents or acute illnesses can seem to have higher cancer rates simply because they are subjected to more imaging and investigation. It is not that the accident caused the cancer. It is that the accident caused the cancer to be detected.

In some cases, as with my brother-in-law, Simon, that increased scrutiny is life-saving. But detection bias also complicates how we evaluate screening programmes and treatment success. Catching a disease earlier can make survival “after diagnosis” look longer even if no extra life has actually been added. That is why we should be wary of simplistic uses of five-year survival rates when trying to sell the virtues of screening.

Then there are observer effects: the fact that measurement itself can alter the thing being measured. Sometimes this is psychological, as in the Hawthorne effect or white coat syndrome. Sometimes it is physical, as in quantum measurement, tyre-pressure gauges, code profilers, or even (in very rare circumstances) tumour biopsies that can in some circumstances alter what happens next. The act of looking is not always passive. Sometimes the very process of trying to gather information perturbs the system we want to understand.

One of my favourite ideas in the chapter, though, is the stranger subcategory of observer selection effects. These are cases where what you observe is constrained by the fact that you are only in a position to observe it under particular conditions. That is (almost tautologously) why the thing you lose is always in the last place you look: once you find it, you stop looking. It is why your friends, on average, have more friends than you do: the highly connected people are overrepresented in your experience of the network. Observer selection effects even explain why we should be cautious about marvelling at how finely tuned the universe seems for life. We can only observe a universe in which observers like us are possible.

All of this leads to the chapter’s titular metaphor. Scientists (asking the questions they can easily find the answers to and not necessarily the most relevant questions) and really, more generally, all of us, are often like the drunk searching for his keys under the streetlight. Asked whether he is sure he lost them there, he replies: “No, I lost them in the park, but this is where the light is.” We tend to study the things we can measure, count or recall, and we mistake that convenience for importance. We ask the questions to which we know we can get an answer, rather than the questions that would get us the answer.

That doesn’t mean measurement is futile, or that studying what is available is foolish. It means we have to remember that there are areas on which the light does not shine. The danger is not in looking where the light is, but in forgetting that there is darkness beyond it.

Buy the book

A favour: pre-order the book

If you’ve enjoyed this summary, you’ll find much more detail in the book itself, with the stories, the science, and the slightly uncomfortable implications.

If the book sounds like your sort of thing, please consider pre-ordering it. Pre-orders matter far more than most readers realise. They’re one of the strongest early signals that a book has an audience, which influences everything from how many copies are stocked to how widely it’s recommended.

Amazon: https://www.amazon.co.uk/Dont-Know-What-Youre-Missing/dp/1529438039

Bookshop.org (supports independent bookshops): https://uk.bookshop.org/p/books/you-don-t-know-what-you-re-missing-the-science-of-what-s-lost-and-how-to-find-it-kit-yates/497ab9dcf971763a

In May 1845, two ships, the HMS Erebus and the HMS Terror, set sail from England under the command of Sir John Franklin, carrying 129 officers and crew on a mission to chart the last unknown stretch of the Northwest Passage. Instead, it became one of the great disasters in the history of polar travel. Within little more than a year, the ships were locked fast in Arctic ice in the vicinity of King William Island, where they remained trapped through successive winters. Franklin himself died in 1847, and by April 1848 the remaining survivors (105 men in all) abandoned the vessels and attempted a desperate trek south across the frozen landscape, hauling boats and supplies by hand. None would make it to safety.

For over a hundred years, despite search parties being sent out, little was discovered about the ill-fated expedition. Search parties in the nineteenth century recovered scattered bones, fragments of equipment, and a few written records, while contemporary Inuit testimony and modern forensic analysis suggested the hapless sailors were subject to starvation, and even cannibalism. But although the names of every man on the expedition were known, the identities of the discovered remains largely were not. It was announced yesterday that, modern DNA analysis has begun to identify who some of the recovered human remains belong to. By matching genetic material from skeletal remains with living descendants, researchers have been able to identify several of the sailors, including four announced recently, finally putting names to bones that lay anonymous for over 170 years and shedding new light on the expedition’s final, fatal journey.

Subscribe now

Long before the more recent and successful expeditions to discover the fate of Franklin’s expedition, in 1852, Sir Edmund Belcher set out to find and even potentially rescue Franklin and his colleagues. His mission was ultimately fruitless, abandoning four of the five ships he set out with, but ultimately returning safely to England. His mission is notable today, however, for spawning the most expensive typo in history.

Before Belcher set out on his expedition, the brewery Samuel Allsopp and Sons provisioned him and his crew with a bespoke “Arctic Ale” to take with them on the long journey. Not all the ale was drunk on the mission and some of the unopened bottles made their way back with the crew on the one remaining ship to England.

Fast forward to 2007 and eBay user petere92346 finds himself the recipient of one of these bottles – a little piece of Arctic history passed down through his family for over half a century. In June of that year, he tried to sell the full sealed bottle on the auction website with a reserve price of $299. Sadly, when he had typed in the name of the collectors’ item in his listing, Peter missed off the second ‘p’ in Allsopp’s. It meant that potential collectors simply couldn’t find the item when they searched. When the allotted time for bidding was up, the winning bid (of two that were placed) from ‘CollectorDan’ stood at just $304. When Dan relisted the bottle just a couple of months later, with the correct spelling, it attracted 157 bids with 74,000 users watching the item at one point. When the auction ended, the winning bid stood at over $500,000.

Buy the book

This is just one of the stories about missingness from my forthcoming book: You Don’t Know What You’re Missing. Peter missed his typo when proofreading – not out of carelessness, but simply because our brains have been trained to read what we expect to see. And needless to say, what we expect is not always what is actually there. We have fine-tuned our brains to smooth over typos and efficiently extract the information we need. When proofreading, we often ‘read’ what we meant to write, rather than what we have actually written. Peter’s listing goes down as one of the most expensive missing letters in history. What, in hindsight, would he have given for an insight into his own proofreading blind spots?

Peter’s expensive spelling mistake illustrates a crucial lesson from The Missing – that we see what we expect to see. Our expectations influence our perceptions, making it hard to spot mistakes. We know what we want to see and this is often what we think we have seen, irrespective of what is actually there. Once we are conscious of this, however, we can start to implement fixes and use techniques to override these subconscious shortcuts, so that we take in the finer points of the scene that confronts us or we really do read every word in situations where attention to detail matters.

Share

A favour: pre-order the book

As I mentioned, the latter part of this article (about the typo) is a short, adapted excerpt from You Don’t Know What You’re Missing.

If the book sounds like your sort of thing, please consider pre-ordering it. Pre-orders matter far more than most readers realise. They’re one of the strongest early signals that a book has an audience, which influences everything from how many copies are stocked to how widely it’s recommended.

Amazon: https://www.amazon.co.uk/Dont-Know-What-Youre-Missing/dp/1529438039

Bookshop.org (supports independent bookshops):

https://uk.bookshop.org/p/books/you-don-t-know-what-you-re-missing-the-science-of-what-s-lost-and-how-to-find-it-kit-yates/497ab9dcf971763

Thanks,
Kit

We’re now into the second half of the short series of posts I’m writing for this Substack, summarising the chapters of my new book, You Don’t Know What You’re M ss ng. Publication is under two weeks away on the 4th June. As before with these summaries, the aim is to give you a feel for the shape of the argument, the kinds of stories I use in the book to get the message across, and the ideas I hope will stick with you long after you’ve finished reading it.

In this post I’m going to summarise Chapter 5, Select Company.

Buy the book

“The greatest obstacle to discovery is not ignorance - it is the illusion of knowledge.” Daniel J. Boorstin, American historian.

“A QUARTER of students catch an STI during their first year at university - and half are too drunk to remember who gave it to them.”

So read a headline in the Daily Mail in October 2013. Timed to coincide with the first couple of weeks of a new university year, the article brought a number of questions immediately to my mind; prosaically and perhaps indicative of my mathematically preoccupied mindset, the most pressing one was ‘How had the headline statistics been gathered?’

The answer, it turned out, was not a university-led study, or a medical survey, or anything you would reasonably call research. The figures came from a self-selected poll of the subscribers to a dating website - ShagAtUni.com - whose “sole purpose”, in their own words, “is to help students meet up for sex”.

You don’t need to be a statistician to see the problem. If you only ask the people most likely to have lots of sexual partners, and you exclude people in stable relationships, you are going to get a very skewed picture of student sexual health. Whole groups are missing before you even begin.

Subscribe now

That is the central idea of this chapter: we can end up making mistakes, not because we have missed something in our own reasoning, but because the information we’re reasoning from has already been filtered. Something is missing by the time it reaches us. In the book I call this extrinsic missingness: missingness in which important data is removed from view by some external process before it is even set in front of us, so our view is partially blocked and we are not seeing the full picture.

Surveys are common places where extrinsic missingness is perhaps most readily found, because surveys come with a built-in illusion of objectivity: there is a number; it looks precise; it feels scientific. But surveys go wrong in predictable ways, and this chapter is, in large part, a guide to those predictable failures.

One obvious issue is representativeness. Even if every respondent answered perfectly honestly, you can still be wrong if the group you asked isn’t representative of the population you’re trying to talk about. And you can also be wrong simply because you didn’t ask enough people. If the true preference between two options is 52:48, small samples will often tell you the wrong answer, purely through randomness. Even at sample sizes that feel large, close-run things (like the Brexit referendum) are hard to call.

But the bigger, and arguably more interesting failures of surveying are the ones that don’t go away with bigger samples. Because if the method is biased, taking more data doesn’t pull you towards the truth. It pulls you more confidently towards the wrong answer.

Share

A lot of the chapter is then about different flavours of bias that cause the wrong people to appear in your results, or cause some of the right people to disappear.

One set of problems comes from how people end up in the survey at all. There is a distinction between how you choose the sample and who ends up in the results. Even if you start with a decent sampling plan, you can still end up with skewed outcomes through non-response, drop-out, missing data, and all the other ways people can vanish from the final dataset.

A prominent example of overrepresentation was recently illustrated in a BBC headline about chronic traumatic encephalopathy (CTE) in NFL players: “Brain disease affects 99% of NFL players in study”. The underlying study examined brains in a brain bank, many donated by families who suspected their loved one had neurological problems. That is not a random sample of players. It is a sample heavily enriched for the very thing you’re trying to measure. To their credit, the scientists who wrote the study warned against drawing population-level conclusions from this sample. But that warning is exactly the kind of nuance that is easily stripped away as results travel from paper to press release to headline.

Underrepresentation can be just as damaging. The classic cautionary tale is the 1948 US presidential election, when the Chicago Tribune printed “DEWEY DEFEATS TRUMAN” before the results were in. The call was supported by a Gallup poll that looked reliable, but had a fatal flaw: quota sampling meant interviewers had targets by demographic category (in an attempt to ensure representativeness), yet they were free to fill those quotas with whoever was easiest to find and willing to talk, which tended to miss working-class Truman voters on long shift patterns and over-represent more affluent respondents. Truman won comfortably, and the paper was immortalised in embarrassment.

Share Kit Yates - Math(s) and the real world

One almost surefire way to create a biased sample is to let it pick itself. Open-access surveys are cheap, quick, and often completely unrepresentative. People who respond tend to be those with the strongest views, the most time, or the biggest grievance. This creates a familiar distortion: the moderate majority goes missing and the extremes dominate.

HOPE not hate’s “National Conversation on Immigration” report illustrates the point neatly. The same question (“On a scale of 1–10, with 1 very negative and 10 very positive, do you feel that immigration has had a positive or negative impact on the UK, including your local area?”) was asked in two ways: an open online survey and a nationally representative poll. In the open survey, responses clustered at the extremes. In the representative poll, far fewer people held the most extreme views. Changing the sampling method made people with differing views become more or less visible.

And even when you try to correct for missingness, you can make things worse. Pollsters weight survey results to match demographic proportions, but if turnout varies strongly by demographic group, then “making your sample look like the population” can still misrepresent who will actually vote. The 2015 UK election polling miss is a cautionary tale: pollsters corrected for underrepresented younger voters (who tend to be more left-leaning), but failed to account properly for the fact these same younger demographics tend to have lower turnout. Consequently, many polls ended up with an overly optimistic picture of Labour support.

Quite apart from the question of whether your sample is representative, there is a different, more human set of issues: even if you ask the right people, they might not answer truthfully.

Sometimes people simply misreport. In a closed population over a fixed time period, heterosexual partnerships contribute one partner to the male count and one to the female count, so average reported numbers should not diverge dramatically. Yet in survey after survey, men report far more opposite-sex partners than women do. This is a big red flag that responses are not purely measuring behaviour, but are also affected by the truthfulness (or otherwise) of the respondents.

And beyond misremembering, there are predictable psychological pressures. People say what they think is expected of them. They say what makes them look good. They say what avoids embarrassment. They say what they think people want to hear. Surveys can suffer from acquiescence bias, courtesy bias, compliance or coercion, demand effects, and social desirability bias. In authoritarian settings, “approval ratings” can be meaningless because dissent is dangerous. In less extreme settings, people still tweak their answers towards what feels socially acceptable.

So what can you do when you want to measure something people don’t want to admit to?

You get clever about how you ask. Chapter 5 ends with an illustration of an elegant survey technique known as a randomised response trial. When trying to find out what proportion of the population practice an embarrassing or taboo habit, the idea is to introduce randomness so individual answers carry plausible deniability, while still allowing us to estimate the true prevalence of a behaviour across a whole population. The simplest version gets everyone to flip a coin: heads answer truthfully, tails answer “yes” to the embarrassing behaviour regardless. A “yes” answer becomes deniable - “the coin made me do it”. The cost is that you’ve deliberately contaminated your data with false answers. The benefit is that people can answer honestly without feeling exposed, and because we know how many default “yes” answers we expect due to people flipping a tail, we can recover an estimate of the true prevalence by working back from the overall proportion of “yes” responses.

Extrinsic missingness doesn’t just mislead us. It gives us the confidence of a number while hiding the fact that the number may have been born biased.

So the take-home message from this fifth chapter is about questioning where the statistics that are put in front of us come from. When you see an alarming statistic or a suspiciously neat survey result, your first question should be “how was this measured?” To help you answer, ask: “Who was missing?” “Who was overrepresented?” “What incentives shaped the answers?” “What might have been filtered out before the number reached you?”

Buy the book

A favour: pre-order the book

If you’ve enjoyed this summary, you’ll find much more detail in the book itself, with the stories, the science, and the slightly uncomfortable implications.

If the book sounds like your sort of thing, please consider pre-ordering it. Pre-orders matter far more than most readers realise. They’re one of the strongest early signals that a book has an audience, which influences everything from how many copies are stocked to how widely it’s recommended.

Amazon: https://www.amazon.co.uk/Dont-Know-What-Youre-Missing/dp/1529438039

Bookshop.org (supports independent bookshops):

https://uk.bookshop.org/p/books/you-don-t-know-what-you-re-missing-the-science-of-what-s-lost-and-how-to-find-it-kit-yates/497ab9dcf971763

Thanks,
Kit

We’re about halfway through the short series of posts I’m writing for this Substack in order to summarise the chapters of my new book, You Don’t Know What You’re M ss ng, ahead of publication on 4th June, and things are starting to feel real. I managed to get my hands on a copy of the the new book at a festival I was speaking at on the weekend before I’ve even been sent my own copies. So nice to feel a physical copy of it in my hands. Some lucky people even walked away from the festival owning a copy of the book before I did. You can get your own hands on a copy by clicking the button below.

Buy the book

As a reminder, the aim of these summaries is to give you a feel for the shape of the argument, the kinds of stories I use in the book to get the message across, and the ideas I hope will stick with you long after you’ve finished reading it.

In this post I’m going to summarise Chapter 4, Inside information.

“Information is a measure of one’s freedom of choice” - Warren Weaver

If you can read a sentence with jumbled letters (see last week’s post about Chapter 3 - Reading Between the Lines), or skim past the odd word without losing the meaning, it suggests something slightly strange, perhaps counterintuitive: not every part of a sentence is doing the same amount of work. Some words are carrying the load. Some are almost redundant. The context surrounding a particular word in a sentence is doing a lot of work when we read.

So it’s natural to ask a question that sounds philosophical but turns out to have a huge amount of practical relevance: how much information is actually contained in a sentence, a paragraph or even a whole language? And if we can measure it, can we strip a message down to its essence - a distilled informational concentrate?

Subscribe now

This fourth chapter (Inside Information) is about the art - and the risk - of doing exactly that.

When we compress data, we’re trying to remove what is predictable so we can store or transmit what remains more efficiently. That’s the heart of the information revolution that has transformed daily life in the last few decades: squeezing photos, calls, videos, documents and messages down so they travel quickly and reduce the amount of storage needed.

But there’s a catch. The moment you boil down message down to its bare informational bones, you strip away the helpful redundancy that makes communication robust. Perfect compression gives you no safety net. If something goes wrong, you may have no way to spot the error, let alone correct it. The challenge is finding the balance between economic efficiency and reliable resilience.

Information theory is not concerned with whether a message is true, wise, or worthwhile. In that sense, the information content of a sentence is quite different from its meaning. Instead, information theory is concerned with the mechanics: how messages can be encoded, stored, transmitted, and recovered.

Share

The person who, more than any other, showed us how to think about information properly was Claude Shannon. Shannon’s breakthrough was to quantify information using a concept called entropy.

Thinking about entropy as a measure of surprise is a useful analogy. If something is very predictable, it contains little information. If it’s genuinely uncertain, you gain more information when you discover the outcome. In Scrabble, picking a Z gives you more information (because you don’t expect to see a Z because it’s rare) than drawing an E would (because you expect to see the common letter E). The surprise you get is different because the probabilities of the letters are different.

The maximum information you can gain from a yes-no question is one bit (a portmanteau of ‘binary’ and ‘digit’) - and you only get close to that maximum if “yes” and “no” are equally likely. This is why the best strategy in Guess Who is not to ask something unlikely like “Are they bald?” (high payoff if yes, but usually no), but to ask questions that roughly split the remaining options in half (“Are they female?” is usually a good starter question). The same logic underpins the most efficient way to play Twenty Questions: keep dividing the space of possibilities as evenly as you can. It’s not the most fun way to play, but it’s a good illustration of how you can gain as much information as possible.

Another of Shannon’s brilliant insights takes us back to language. Letters don’t appear independently of each other. They come in patterns - words. They come with context - sentences. That context makes them predictable, which means each letter carries less information than you might naïvely think.

Shannon tried to estimate how much information there really is per character in English, once you include the predictability created by surrounding letters and words. The startling conclusion is that context can reduce the effective information content of English to around a bit per character (whereas 5 binary digits would be needed to encode the 26 letters of the alphabet – 00000 encoding A, 00001 encoding B, 00010 encoding C, 00011 encoding D etc). In other words, a lot of written language is redundancy - usually helpful redundancy.

Share Kit Yates - Math(s) and the real world

This is where compression comes in. Standard text encodings like ASCII use seven bits per character, which is robust but not very efficient. If English can be encoded, in principle, far more economically, then a lot of what ASCII stores is redundancy - space we could make go missing. Compression algorithms are built to exploit exactly this sort of structure: they remove predictable parts while keeping enough to reconstruct the message.

But once you start stripping away redundancy, you start living dangerously. A beautifully compressed message can become brittle. If a single error creeps in, the receiver may not be able to tell.

Morse code is a lovely historical stepping stone here because the choice of encodings for the letters was an early form of compression in action. Morse assigns short codes to common letters and longer codes to rare ones. But Morse comes with a vulnerability: its symbols are not naturally self-delimiting. If spacing is sloppy, messages can become ambiguous. A pause too short can turn clarity into confusion.

The most haunting illustration is the story of the Star Dust, a plane that vanished over the Andes in 1947. Its final transmission ended with the enigmatic word “STENDEC”, repeated three times, and then silence. No one knew what STENDEC meant. What makes the tragedy instructive rather than just intriguing is not the mystery, but the mechanism: in Morse code, missing or misplaced gaps can destroy meaning. When context is stripped away, a small formatting error can convert a message into a riddle with no solution.

It’s similar to the risk we run when we compress information too aggressively. If all the context is removed and a mistake occurs, the message may become ambiguous or unintelligible.

So what do we do about that fragility? We add redundancy back in - but deliberately and in a way that helps us catch and correct errors.

This is where error-detection and error-correction codes come in. The point is not to avoid redundancy entirely, but to choose the right kind. Add a single parity bit to a string of binary digits and you can detect that something has gone wrong. Add more structured redundancy and you can often pinpoint the error and correct it. These techniques sit behind the everyday miracle of modern computing: the fact that we can store and transmit huge volumes of data through noisy, unreliable physical systems and still have it arrive intact.

If you want an example of how small an error can be and how dramatic the consequence, you could do worse than look to the Belgian federal election in 2003, where an electronic voting anomaly turned out to be consistent with a single flipped bit - a tiny physical event in memory leading to a large jump in the number of votes and almost to an incorrect winner being announced.

The lesson is not “computers can’t be trusted”. The lesson is that without redundancy and checking, even a naturally occurring one-bit glitch can change the result.

This information theory chapter ends by returning to the broader theme: the key to efficient, robust communication is not maximal compression, nor is it no compression. It’s a finely struck balance between the two. We need enough context to give messages clarity and the possibility o correcting errors, but not so much that we bury the signal in superfluity. It sounds oxymoronic, but the main take-home from the chapter is that we really need this essential redundancy.

A favour: pre-order the book

If you’ve enjoyed this summary, you’ll find much more detail in the book itself, with the stories, the science and the slightly uncomfortable implications.

If the book sounds like your sort of thing, please consider pre-ordering it. Pre-orders matter far more than most readers realise. They’re one of the strongest early signals that a book has an audience, which influences everything from how many copies are stocked to how widely it’s recommended.

Amazon: https://www.amazon.co.uk/Dont-Know-What-Youre-Missing/dp/1529438039

Bookshop.org (supports independent bookshops): https://uk.bookshop.org/p/books/you-don-t-know-what-you-re-missing-the-science-of-what-s-lost-and-how-to-find-it-kit-yates/497ab9dcf971763a

Buy the book

Thanks,
Kit

Heard melodies are sweet, but those unheard are sweeter - John Keats, ‘Ode on a Grecian Urn’.

First off, I wanted to say a massive thanks to all of you who’ve already gone and pre-ordered a copy of the book. Your support means so much to me. As you’ll probably know by now, I’m writing a series of posts summarising the chapters of my new book, You Don’t Know What You’re M ss ng, ahead of publication on 4th June. The aim is to give you a feel for the shape of the argument, the kinds of stories I use in the book to get the message across, and the ideas I hope will stick with you long after you’ve finished reading it.

In this post I’m going to summarise Chapter 3, Reading between the lines.

Buy the book

It mhigt spusiere you to konw taht dispete the fcat I hvae jbumled up all the ltteres in the mddile of the wdros in tihs scetnene you can sltil raed it woutiht mcuh erffot: as lnog as the fsirt and lsat ltteres are in the rghit pacle you can wrok it out. 51M1L4RLY, 3V3N 1F 50M3 L3773R5 4R3 5UB5717U73D F0R NUMB3R5 Y0U C4N R34D 7H15 53N73NC3 W17H0U7 3V3N BR34K1NG 57R1D3. P#rh#ps #v#n m#r# s#rpr#s#ng #s th#t #f #ll th# v#w#ls #n th# s#nt#nc# g# m#ss#ng #nd #r# r#pl#c#d w#th #n#ther ch#r#ct#r y## c#n st#ll j#st ab##t r##d #t. It was harder, but you could still manage, right?

We are incredibly adept at decoding garbled messages, in which seemingly important linguistic structures are missing. One version of this is disemvoweling: a technique internet moderators sometimes use to tread the fine line between overzealous censorship and an unregulated free-for-all. It renders potentially offensive posts less objectionable, while ensuring readers c#n st#ll r##d th#m #f th#y r##lly try. What I’m really concerned about in this chapter, though, is what this says about how our brains operate when confronted with missing information.

A big part of the answer is context. The words in a message prime us to expect particular words to come next. Even if a word is ambiguous because we haven’t looked at it carefully or it has become corrupted, context combined with out prior experiences can allow us to interpret what it should be. At a finer scale, the letters of a word prime us to expect other letters. If you see a “q” you’re already leaning towards “u” coming next before you even get to it.

Subscribe now

But it isn’t only the context that comes before the missing piece that’s important. Often it’s the context that comes after that helps us out. With heavily jumbled text, you might not truly parse each word as you encounter it. You can feel, in hindsight, as if you understood everything smoothly, when in reality later words have allowed your brain to go back and fill in gaps you didn’t even notice were there. Reading isn’t just a predictive process; it’s also a repair job.

I’ve had first-hand insight into this while helping my son learn to read. A couple of years ago, he reached the stage where he no longer needed to sound out every letter in order to parse a word. He lets context help him out. If he gets stuck, he knows that reading on often resolves the ambiguity. That is a tremendous advantage. It speeds up reading dramatically. But it comes with a familiar trade-off: the more your brain relies on expectation, the more it risks seeing what it expects rather than what is actually on the page. Which is why proof-reading our own work is so difficult.

When re-reading, we tend to automatically interpret many of our own mistakes as correct. We know what we meant to write, so we expect to see the correct form on the page when proofing, and this is often what we end up “reading”. I sometimes type “form” when I mean “from”. Word doesn’t flag it because it isn’t a spelling error. And I often don’t spot it because my brain knows what it is expecting to see and that is what it sees. I’ve resorted to searching for both words and checking each instance manually.

Share

This general phenomenon also explains why certain “spot the mistake” puzzles are so effective. Try this one one.

70% per cent of the public can’t spot the mistake in this text.

Apart from it probably being poor English to start a sentence with a number in digit form, did you manage to spot the mistake? I certainly didn’t the first time I read it. The answer is the repetition of % and per cent. Our brains tend to simply filter this out. Even when you are primed to catch mistakes it isn’t always easy to spot them.

We miss repeated little words. We glide straight past a duplicated “the” or an extra “a”. We can stare at a sentence and confidently feel we have inspected it, when our brains have actually filtered out exactly what we were trying to catch. Even when you are primed to detect errors, your mind is still optimised for extracting meaning, not for conducting forensic audits.

The same is true for sound. In this chapter we spend some time looking at the ways we mishear and misinterpret language, not because we are careless, but because we are trying to make sense of noisy signals in real time. Mondegreens and eggcorns are the harmless, funny end of this spectrum: plausible substitutions that fit the context and make a surprising kind of sense. They are reminders that, while context can help us interpret missing parts of language, it can also lead us astray.

Sometimes the consequences are not funny at all, but deadly serious. Under the wrong conditions, an ambiguous phrase or a misheard command can become fatal. When people are anxious, excited, frightened, or simply primed to expect one thing over another, they can fill in the gaps with the wrong word and not even realise they have done so. This is one of the uncomfortable themes of the chapter, that misunderstanding can feel, from the inside, exactly like understanding. Sometimes we don’t even know we are making a mistake.

All of this pushes us towards a natural question: if the brain is constantly filling gaps, how does it decide what to fill them with?

One possible explanation for how our brains reason under uncertainty is called the Bayesian Brain Hypothesis. Many neuroscientists think it’s useful to treat the brain as a statistical organ: one that combines prior knowledge with new evidence in order to reach the best current guess about what is happening. This is sometimes called Bayesian integration. The basic logic is simple. Your prior experiences give you a set of expectations about what is likely. Incoming information nudges those expectations. The brain updates.

The Bayesian Brain Hypothesis helps explain why we can read meaning into degraded text, why we can often reconstruct what someone said in a noisy room, and why we can predict the next word in a sentence. It also helps explain why we make systematic errors. Sometimes our prior expectations are too strong. Sometimes they are applied in the wrong domain. Sometimes we mistake familiarity for predictive power and end up with the cognitive equivalent of the gambler’s fallacy: feeling that the world “ought” to balance out in the short term, even when it doesn’t work that way.

The brain (Bayesian or otherwise) is a fantastic tool. It often allows us to draw correct inferences from incomplete information. But it is so fluent at filling in gaps that we are sometimes not even aware that anything is missing. And if you don’t realise something is missing, you don’t realise you might have filled it in incorrectly.

Buy the Book

A favour: pre-order the book

If you’ve enjoyed this summary, you’ll find much more detail in the book itself, with the stories, the science and the slightly uncomfortable implications.

If the book sounds like your sort of thing, please consider pre-ordering it. Pre-orders matter far more than most readers realise. They’re one of the strongest early signals that a book has an audience, which influences everything from how many copies are stocked to how widely it’s recommended.

Amazon: https://www.amazon.co.uk/Dont-Know-What-Youre-Missing/dp/1529438039

Bookshop.org (supports independent bookshops): https://uk.bookshop.org/p/books/you-don-t-know-what-you-re-missing-the-science-of-what-s-lost-and-how-to-find-it-kit-yates/497ab9dcf971763a

Thanks,
Kit

If you’ve been reading this substack over the last few weeks you’ll know that I’m writing a series of posts summarising the chapters of my new book, You Don’t Know What You’re M ss ng, ahead of publication on 4th June. The aim is to give you a feel for the shape of the argument, the kinds of stories I use in the book to get the message across, and the ideas I hope will stick with you long after you’ve finished reading it.

In this post I’m going to summarise Chapter 2, Putting things in context.

Buy the Book

On the face of it, the story that opens this chapter feels like just a fun anecdote. Eddie Vedder, frontman of Pearl Jam, steps out on the street after a Chicago Cubs game, hears a band busking, and ends up playing some of his most famous songs with them. A world-famous musician, playing some of his best-known songs, right outside a stadium that had hosted Pearl Jam a year earlier. And hardly anyone notices.

Paul McCartney once did something similar outside Leicester Square tube station, playing Yesterday for spare change as part of a film shoot. A globally recognisable artist singing an iconic song in a busy public place. And again: almost no-one stops to listen.

It’s tempting to think this is about distraction, or maybe the death of our attention spans, or just the sheer pace of modern life. And those things might be part of the story, but the more revealing explanation is simpler and more general. It’s about context. If you’re leaving a stadium after a game, or rushing through a crowded station, you’re not in “spot a global superstar” mode. You’re in “get home” mode. The context of a situation shapes your expectations, and your expectations shape what you experience. Out of context, even something extraordinary can go missing.

Subscribe now

That idea - context as a guiding determinant of perception - runs throughout the rest of the chapter.

A good way to understand the impact of the surrounding context is to look at what happens when something genuinely is missing from your sensory input. You’re probably aware that you have a physiological blind spot: a small region of the retina where there are no light-detecting photoreceptors because that’s where the optic nerve exits the eye. In daily life you don’t notice it, partly because you have two eyes and your brain blends their images together. But when you test it with one eye (see below), the striking thing is not that you get a hole in your vision. You don’t. The object you’re looking at disappears and the brain simply fills in the gap with whatever pattern it thinks should be there. White background produces white. Black background produces black. The missingness is plastered over.

Position the screen about 35cm from your face (or between three and four times the distance between the cross and the dot if that is markedly different on your device. You might have to experiment with this positioning. Cover your left eye so you can’t see out of it at all. Line up your right eye with the cross on the left hand side of the page and focus on it. You may need to adjust the distance at which you hold the book slightly, but you will find a relatively large range of distances for which the black spot on the right hand side disappears. What remains, however, is not a hole in our field of vision. Instead the brain has filled in the gap with white, the colour of the background surrounding the black dot.

Share

The same basic thing happens in time as well as space. Your eyes make rapid saccadic jumps as they scan the world, but the in-between bits are not delivered to your conscious experience as a blur or a flicker. Instead, your brain down-regulates your visual input for the movement and fills the gap with the stable image after the movement has finished. This is known as saccadic masking. The result is that you feel like you saw the scene continuously, even though you didn’t. You can experience this happening in practice as part of the stopped-clock illusion: moving your eyes to focus on a clock face, the first “second” can feel strangely long, as if time has briefly frozen and then restarted – consistent with the visual system filling in the gap during which your eyes moved with the image input at the end of the movement.

Once you are aware of these effects, you start to understand why the phrase “Sorry Mate, I Didn’t See You”, often heard after traffic accidents, is not always a confession of carelessness. It can be a description of a real perceptual mechanism. When we pull out of junctions we often feel as if we’ve checked the whole panorama, but what we may actually have is a small set of snapshots stitched together, with the missing bits smoothed over. The stitching is normally a blessing. In the wrong place, at the wrong time, though, it can be disastrous.

That brings us to one of the most unsettling ideas in the chapter: the illusion of absence. This is not just failing to see something because it was behind an obstruction. It’s the stronger, more dangerous belief that the obstructed region is empty. Not “I couldn’t see what was there”, but “I effectively experienced it as not being there at all”.

Share Kit Yates - Math(s) and the real world

Driving is full of opportunities for this sort of mistake, because cars come with built-in occluders: pillars, headrests, passengers, windscreen stickers, mirrors that don’t quite show everything. The A-pillars of modern cars are particularly pernicious. They don’t need to be large to hide something important, and under certain conditions another road user can remain hidden behind a pillar for long enough that by the time they emerge into view there’s no time left to do anything useful.

We all know that blind spots exist. But perhaps what is harder to understand is that a driver can genuinely feel they have checked, genuinely feel the road is clear, and still have been wrong - because their brain has treated the occluded region as “seen and empty”, rather than “unseen and unknown”.

For a more general psychological handle on why this happens, we explore a phenomenon called amodal completion: our propensity to fill in gaps when objects are partially occluded. We do it all the time. In many cases it’s incredibly useful. A scene full of partial information becomes coherent because the brain completes it. The problem is that the same mechanism can also suppress alternative possibilities. Once the brain has completed a scene in one plausible way, it can be surprisingly hard to hold in mind what else might be behind the occluder.

The chapter then widens from occlusion to a broader class of context effects: situations where the same physical stimulus can be experienced in dramatically different ways depending on what your brain assumes about the surrounding conditions.

The classic visual example is light constancy. In the real world, your brain tries to keep the perceived lightness of surfaces relatively stable even as illumination changes. Normally we would regard that as a triumph of perception. But it also means that ambiguous lighting can produce wildly different interpretations of the same image. The Dress is the famous case: people genuinely saw The Dress in different colours because their brains made different assumptions about the illumination, and then corrected for it. The assumed context of the image was vital to how The Dress was perceived.

And although many of the examples in the book are visual, our other senses employ context to help them out as well. For example, it turns out that, labels matter for smell and names matter for taste. If you tell someone a pine-like odour is “Christmas tree” they don’t just enjoy it more than if you call it “disinfectant” - many people seem to experience it as a different smell altogether. Likewise, taste can be influenced by expectation: what you think you’re about to taste can change the taste you actually experience.

Hearing perhaps even more vulnerable to these context-dependent overwritings. The internet has produced some wonderfully compelling illustrations, like the Brainstorm/green needle clip, where the same ambiguous sound can flip depending on what you’re primed to hear. If you hear the stimulus thinking about one phrase, then this is probably what you’ll hear. But if you prime yourself by thinking about the other, then that’s what you’ll hear. Your expectations don’t merely bias a judgement after the fact. They shape the perception itself.

Although many of the stories in the chapter emphasises the places where context can mislead us to fill in the gaps incorrectly, there’s also a more useful side to context-aided perception. Cochlear implants can initially sound fuzzy and distorted to new users, and yet with time many people report that their perception adjusts. The brain reroutes in order to fill in the gaps using the context it previously learned. The same top-down machinery that can mislead us can also help us adapt, sometimes in ways that feel close to miraculous.

All of these phenomena sit under a broader umbrella of top-down processing: the brain starting with higher-level frameworks - memories, expectations, assumptions - and applying them to make sense of lower-level sensory inputs. It’s a key driver of what I call intrinsic missingness. It helps us cope with ambiguity and incomplete information.

Once you accept that expectations can change perception at this level, it becomes easier to believe that disagreements about the world are not always about one person being irrational or dishonest. Two people can be looking at the same thing and genuinely experiencing it differently, because they are bringing different contexts and different life histories to the scene. That doesn’t mean “everyone’s right”. But it does mean we should sometimes be a little less certain that our own interpretation is the only reasonable one, and a little more willing to ask: is there something missing here?

Buy the Book

A favour: pre-order the book

If you’ve enjoyed this summary, you’ll find much more detail in the book itself, with the stories, the science and the slightly uncomfortable implications.

If the book sounds like your sort of thing, please consider pre-ordering it. Pre-orders matter far more than most readers realise. They’re one of the strongest early signals that a book has an audience, which influences everything from how many copies are stocked to how widely it’s recommended.

Amazon: https://www.amazon.co.uk/Dont-Know-What-Youre-Missing/dp/1529438039

Bookshop.org (supports independent bookshops): https://uk.bookshop.org/p/books/you-don-t-know-what-you-re-missing-the-science-of-what-s-lost-and-how-to-find-it-kit-yates/497ab9dcf971763a

Thanks,
Kit