Readers have probably noticed that my posts have been becoming more focused on AI over the past couple of years. Despite appearances otherwise, this will remain a philosophy newsletter and not turn into yet another tech focused blog. However, I find myself musing more and more on AI as there is something deeply interesting about the topic for a philosopher. As a discipline, philosophy is famously light on empirical evidence and relies on deep thinking with appeals to human concepts and intuition. AI is partly changing that as it provides new real world examples that illuminate many long standing philosophical debates.

In the history of human philosophical reflection, humans have stood out as both a primary object of study and the examplar by which we think through the most difficult and interesting philosophical issues. No other beings (that we are aware of) are universally believed to possess minds or conceptualise the world as an object to think about.1 This has meant that humans were the only beings philosophers could study when considering issues like consciousness, free will, existence, knowledge and ethics - as all of these depend on a being possessing a mind (or equivalent).

Put simply, modern AI systems have changed the game. We suddenly have a different entity that behaves in many ways like a human and possesses (or convinces some people it possesses) core characteristics of a mind. This will provide fertile ground for exploring many philosophical issues so long as we pay close attention to how AI develops and to what extent it behaves like humans, or not. The challenge is to get past the current emotion and hype given many people are fixated on prophecies of doom or utopia. In fact, a more serious and disciplined examination of what sort of thing AI is, and how that is similar and different to humans, will shed light on many important philosophical topics. I do not have time or space in one post to do any of this rigorously, but want to try to set up some issues and questions we should be thinking about.

The primary question

Humans, as persons or entities with a mind, possess a cluster of different but related attributes and abilities. We are intelligent, conscious, and know things about the world. We have a will (or agency), moral responsibility, emotions and strong sense of ethics or values. We can reason, make abstractions, form concepts, and draw pictures that represent things in the world. We speak, read and write in language, make music and care about beauty. So long as humans have been the only entity that we consider to have a mind, it is natural to take all these attributes and abilities as a unified package that naturally belongs together.

However, we now have AI systems that have at least some of these attributes. This raises important questions. Does it follow that they will possess or develop most or all of the others? Or are many of these distinct and separable attributes that humans happen to have together? For example, is there a necessary connection between the fact that humans can reason and the fact that they are conscious? Or is it possible to be conscious and not reason and vice versa?

AI systems can clearly reason, use language and are intelligent (in at least some senses of the word). How many, if any, of the other attributes listed will follow from these? Or are the other attributes distinct in such a way that AI will never possess them?

We routinely anthropomorphise AI systems as an easy way to talk about systems that exhibit many properties that were once reserved for humans. But this can often lead us to assuming AI has attributes that it does not, and may never, have. A good example is that many AI extinction worries turn on a superhuman AI system ‘deciding’ something that runs contrary to human interests.2 This assumes a level of will and agency that is capable of weighing up different options and actively choosing. It is not a given that AI systems can do this, but also not clear that they cannot. However, if high levels of abstract intelligence can exist independently of will and agency, then AI systems will only ever follow the consequences of internal mathematics and reasoning - meaning they will be different types of actors and behave differently to humans.

Or to pick a different example, there is a lot of focus on trying to make sure that AI systems align with human morals and values. Aside from the completely fraught questions about what these actually are, we need to ask whether AI is even the type of thing that can be moral or not. If they are by definition amoral, then alignment attempts can only ever be an attempt to structure the systems to behave in ways that are incompatible with the underlying logic of the system.

Where these questions have been debated, and AI consciousness is the most prominent one, there is no consensus on whether AI will possess the attributes that humans do. However, some of these will will become clearer over time, especially as the trajectory and limits of AI continue to emerge.

Share

New evidence for philosophical questions

For the record, I expect there to remain deep differences between AI and humans across many of these attributes. For example, I don’t expect AI to ever be conscious, have genuine (or free) will, be truly moral (or immoral) agents or to be able to form new abstract concepts and representations like humans can. However, I may well be proven wrong over time - and what turns out to be true will have important implications for important philosophical issues.

One of the many questions is about free will and moral responsibility. Human societies are typically based on the assumption that people can choose between actions and deserve reward / punishment for certain types of actions. Many have, however, argued that humans do not possess free will and so cannot genuinely choose. AI looks like it will provide an interesting case study as it (currently) is a system that behaves with reason and intelligence but doesn’t appear to have genuine free will. Everything follows from the programming, albeit stochastically and not deterministically. How these systems behave over time, and the extent to which they match human behaviour, will give us some more insight into whether human free will exists and what it is like.

Related are ongoing questions about whether morals and values emerge from evidence or our interactions with the world - or whether they come from somewhere else (are either innate or purely unanchored preferences). The extent to which AI develops morals and values will give us some insight on human morality.

The potential for insight from AI extends into the philosophical topics I most care about. I have written about the structure of knowledge and truth, and differences I currently see between humans and AI. For example, I have argued that AI will always be limited as it can only rely on data and language for its inputs. If it turns out that AI functions differently to today, then I will be forced to revise my views.

And, of course, a hot topic at the moment is the future of mathematics given the success of modern AI at providing formal proofs of previously unanswered research questions. Of interest here are the debates about whether AI is doing genuine mathematics or something different. All of this should feed back into questions in the philosophy of mathematics about what mathematics actually is and what it investigates.

These are just some of the philosophical issues for which AI should provide further insights. The challenge is to make sure that we observe AI clearly and think carefully about the consequences for what it is to be human. The evidence I see so far is that AI is something different to a human, more like an alien, and so will be a useful contrast to humans. This should always give us pause when we anthropomorphise AI as it will confuse us as much as clarify.

Subscribe now

There’s been a lot written recently about the existential risks posed by current forms of AI. One striking thing about the debate is the vast distance between the views of leading proponents. Some tech leaders have publicly assessed the chances of AI leading to human extinction as >10% while many others have argued that the idea of human extinction from AI is preposterous. This is yet another chapter in ongoing debates between camps with very different readings of what AI will actually be able to do. The ongoing question of whether AI will render human jobs obsolete, or even lead to higher employment, is another part of the same debate.

One side sees LLMs solving longstanding mathematical research problems, rapidly improving on various benchmarks and achieving a higher standard than humans on many tasks and concludes that LLMs are well on the way to outpacing human intelligence. It follows therefore that LLMs will be more successful than humans across everything we do and so pose huge opportunities or threats. The other side focuses on the the many ways that LLMs hallucinate, struggle with tasks that are simple for humans and produce outputs that lack anything distinctive or special. Many then conclude that LLMs are just fancy auto-complete and are not really thinking. Therefore they cannot threaten or replace humans in any significant way.

Both sides often look at the same data and come to entirely different conclusions. In my view, these come back (in large part) to different understandings of one core concept: intelligence.

One word, a collection of meanings

Humans use words and concepts in a distinctive way that differs from how computers and more formal languages approach them. We often have fuzzy, overlapping meanings for the same word. Ludwig Wittgenstein described the meanings of words as having family resemblances rather than shared definitions. He used the word game as a good example.1 The way we use the concept intelligence matches this pattern perfectly, and causes confusion when we are thinking about LLMs.

To begin with, intelligence, as a concept, is codified and quantified by IQ and represents a certain type of abstract reasoning. However, this is only one of its meanings in the collection of family resemblances. We talk about someone making an intelligent decision, which doesn’t mean they used highly sophisticated reasoning skills but rather it was thought through and had good results. It is also common to describe children (or adults) as intelligent when we see some spark in them that suggests they can think differently or make connections other people don’t make. At the same time, we often see results in school or exams as definitive evidence of intelligence. This list could continue.

While these different senses of the word intelligence are related, they clearly aren’t describing the same thing. There are many people who are intelligent on one measure and not on others. We all know people with really high IQs who make very dumb life decisions. And the children who are intelligent because they think differently are often not the same ones who get really good marks. We often expand our vocabulary to try and capture the idea that there are very different types of intelligence. Practical intelligence or emotional intelligence are phrases we sometimes use. Similarly, “street smarts” are often contrasted with “book smarts” to capture a useful distinction.

As noted already, there are people who hold that LLMs are clearly more intelligent than humans and others who think that the ‘I’ in AI is a case of marketing over substance. Both views have merit based on our different senses of the word intelligence. To pick the four overlapping meanings above, I’d argue that LLMs are intelligent on two out of the four meanings. The are great at exams and easily outscore humans on IQ, but make some really dumb ‘life’ choices2 and are widely viewed as producing safe, dull but polished material - so don’t have that spark. You may disagree with this assessment but the point still stands. There are various meanings of the word intelligent and we need to be clear about these when we discuss whether AI is intelligent.

A good example of this contrast can be found in the recent announcement that OpenAI had provided a solution of the Navier-Stokes equation, one famous open mathematics research problem. Solving a mathematics problem that no human has done is a clear demonstration of intelligence - in some senses. But it required a huge amount of computational work, so much that some estimated it was the equivalent of a human mathematician working on the problem for 10,000 years. I’m sure many human mathematicians would be confident they could solve a problem like this if they had 1,000 years to work on it and try all conceivable options. This kind of brute force approach strikes us as a profoundly unintelligent. AI tends to behave like the students who study hard for an exam and do well without really understanding what they are doing.

Share

AI is not like us

To ask whether LLMs are intelligent is a poorly defined question. They are intelligent on some definitions and not on others. The most sensible answer is to view them as a kind of alien intelligence. They are intelligent on some important measures but they operate very differently to humans. They have already outpaced humans in games like chess and go, and will be smarter at humans at various other types of tasks. However, as I’ve argued before, they operate under some fundamental, structural limitations that mean that won’t match humans in other ways.

Our reading of the threats (and opportunities) posed by AI for humans turns then on our assessment of the role different types of human intelligence play in our successes. If we see the more formal intelligence represented by IQ as most important component of our success, we will likely expect an AI dominated future. Rationalists and Effective Altruists are the backbone of those debating AI doom for this very reason. They prioritise and value the types of intelligence that AI can produce. On the other hand, if you see our success as dependent on more practical intelligence or the wisdom of making good life decisions, then you will have a much lower view of the long term ability and capability of AI systems.

We therefore need to better understand both the alien intelligence that is AI and how human intelligence works out and leads to our successes. As we do that, we can build a better picture of the future of LLMs and other AI systems.

Subscribe now

In my previous post on modal concepts, where I reflected on the h-logic presented by Philosophy Stan, I promised to come back to epistemology. The core point to explore is that, as Stan writes, “h-Logic is in part motivated by epistemic fallibilism”. In brief, that is the view that I can correctly say that I know something even if it turns out that thing isn’t true. Another way of explaining it is to argue that knowledge is simply Justified Belief, rather than Justified True Belief.

I’ve written various articles in defence of similar ideas, focusing on the concept of epistemic humility as a philosophical principle, and I’m not going to repeat those arguments here. What interests me here is the way that epistemic humility or fallibilism is jarring, even unthinkable, to mainstream philosophers and especially analytic philosophers. Their core intuitions about knowledge don’t allow for this as a coherent position.

An intuition leads us in the wrong direction

A short section in Stan’s article illustrates this neatly. He quotes David Lewis - one of the luminaries of analytic philosopher in the second half of the twentieth century - who provides an explicit challenge to epistemic fallibilism:

This is an explicit call to a particular intuition about knowledge, that is both widely held by philosophers and one that Stan concedes as a strong intuition. However, it is one that jars with common human experience and puts us, especially if we are philosophers, in an extraordinary position. The easiest way to see this is to turn Lewis’ phrase into a positive statement about knowledge that captures the intuition: He knows because he has eliminated all possibilities of error.

Think about that commitment for a minute. Is it ever possible to eliminate all possibilities of error? More pertinently, is it ever possible for a philosopher to eliminate all possibilities of error?

Philosophers famously take seriously arguments about whether we are brains in vats, or tricked by Descartes’ demon, or that we are living in an computer simulation. The philosophical literature is full of scenarios like these and, while few people think any of them are true, they have to be possible for anyone to take them seriously. Thus, based on the philosophical literature, it is possible that we are deluded about the nature of the world around us. Therefore, we cannot eliminate all possibilities of error in anything we believe about the world. It follows then from Lewis’ intuition that we cannot actually know anything about the world around us.

The challenges don’t just stop there. Even if we move to mathematics, which is widely considered to be an example of pure analytic, or theoretical, knowledge, we run into similar issues. You may argue that you know certain mathematical truths as you’ve eliminated all possibilities of error in concluding that ‘2+2=4” or that “there is no largest prime number”.

I’d humbly suggest in either case that you are suffering from a lack of imagination. Surely it is possible that our system of mathematics is a collective hallucination - perhaps caused by Descartes’ demon or the wiring to our brains in vats. Alternatively, it is entirely possible that we humans evolved in a way so that our reason and logic is optimised for something other than truth. Our understanding of logic, and therefore mathematics, may be useful but ultimately wrong. There are, after all, serious philosophers who argue that true contradictions are possible.

I may not think any of these scenarios are true, or even likely, but it’s hard to argue that they are not, even in only some minimal sense, possible. And in that case, we cannot eliminate all possibilities of error in our understanding of mathematics. According to Lewis’ intuition, it follows that we do not truly have mathematical knowledge.

This same line of argument holds for any claim to knowledge anyone might have. For any knowledge claim, it is not beyond our imagination to come up with a way our knowledge could be wrong despite everything we have evidence for, which means we cannot eliminate all possibilities of error. Perhaps if I was God, I could do so, but I don’t see how it is possible for any human.1 If we accept Lewis’ intuition, which I have labelled as epistemic certainty elsewhere, we can only conclude that Socrates was right all along: the only thing we can know is that we know nothing.

Share

A widespread, but flawed, intuition

Even though it ends up in a difficult place, the standard intuition, as expressed by Lewis, retains a strong pull on our intuition. I’d argue this is because it reflects what we typically mean, and hope people accept, when we claim that “I know that X”. The content of this claim is that (I’m sure that) X is true and there are no real possibilities of error. What we mean when we say something, and what we are actually justified in saying, are often two different things.

In practice, I can only ever claim to know something as justified against some limited standard of justification. Often this is contextually relevant. My standard for knowing what my friend believes is entirely different for my standard for accepting the existence of a new subatomic particle. We can never eliminate absolutely all possibilities of error and so there is a gap between the claim I am (implicitly or explicitly) making and what I am strictly justified in saying.

This points to what I believe is a flaw in the linguistic turn of analytic philosophy in the twentieth century. While it is often useful to be clear and precise about what the words and concepts we use mean, we humans often mean things in ways that aren’t always fully justified. To pick a simpler example, when we use the word ‘proof’ we mean an argument that absolutely establishes the truth of something with no rational possibility of doubt. However, whenever we claim to prove something, we are often only referring to an argument that we find highly convincing, but for rhetorical reasons we want it to have the aura of being logically undoubtable.

This same dynamic can often be seen in our discourse and societal assumptions. To pick one example, people often assume that they know X, because X is taken to be true in the relevant peer-reviewed literature, and the assumption is therefore that there is no possibility of error in X. The obvious corollary is then, especially on social media, that anyone who disagrees with X is obviously wrong and probably stupid.

You can replace peer-reviewed literature with any number of sources of confidence. For some people it is a religious text, for others it is a particular body of knowledge or tradition, but the logic is always the same. It is one reason why debates these days often end up being so toxic and why, in my view, we need to rediscover epistemic humility, or some kind of fallibilism, if we want to resolve our current societal tensions. After all, the foundations for toleration, free speech, and liberalism formed by people like JS Mill were built on epistemic humility.

Subscribe now

A reader has sent on a provocative short commentary by a scientist, John Tregoning, from Imperial College London that asks “Is the scientific paper still a fraud?” The title riffs off a short article from 1963 by Nobel prize winner Peter Medawar who argued that “The scientific paper in its orthodox form does embody a totally mistaken conception, even a travesty, of the nature of scientific thought.”

Medawar’s core point is that the format and structure of the scientific paper is highly stylised and misrepresents the nature of scientific research. He argued that the “conception underlying this style of scientific writing is that scientific discovery is an inductive process.” In his words, science “starts with simple observation - simple, unbiased, unprejudiced, naive or innocent observation - and out of this sensory evidence ..... generalizations will grow up and take shape, almost as if some process of crystallisation or condensation were taking place.” He then argues that real science proceeds in a very different way. Thus, as Tregoning summarises it, “the scientific paper is still a fraud, because it masks the humans in the process.”

Medawar argues that scientific research more accurately takes a hypothetico-deductive approach, a particular form of an abductive method, in which the act of coming up with hypotheses or theories in a fundamentally creative process that is distinct from the attempt to demonstrate they are true. This is a theme that I’ve covered many times. The argument is that writing scientific papers as though the process is inductive, when in practice it is not, counts as a type of fraud.

Others have recently raised concerns that this ‘fraudulent nature’ of scientific publishing is misleading students and harming those wanting to learn how to do science. There is merit to this argument but more important are similar issues arising with another class of entity trying to learn to do science: LLMs.

In 2024, a multi-organisation team (including Google and OpenAI) published the results of research into the effects of limiting the training data for LLMs in different ways on their performance for different tasks. Most remarkably, one finding was that “removing ArXiv, PhilPapers, and NIH ExPorter from the training corpus improved the LLM’s performance at answering academic questions and its average performance across all benchmarks while also making the model less likely to generate toxic output.”1 At first glance, this result seems highly counter-intuitive and potentially outrageous. Surely scientific publications should rank as among the most reliable forms of literature and so removing them should degrade LLM performance?

This is one (well-resourced) study from a couple of years ago and I haven’t identified any follow-up research. However, if it is a valid result, I strongly suspect that the highly stylised, or fraudulent, nature of scientific publishing is a cause.

If the academic literature reports scientific findings in a form that misrepresents actual scientific reasoning and practice, then using it as training data on reasoning systems, like LLMs, will plausibly negatively affect their reasoning abilities. In other words, papers that misrepresent the genuine reasoning involved in any field will likely pollute training data and degrade performance.2

This throws up some warnings for the ongoing use of LLMs in scientific research. If LLMs are trained on data that misrepresents the reasoning actually involved in scientific discovery, this will severely limit their ability to generate scientific breakthroughs and, as many boosters promise, do things like cure cancer. It also points to my ongoing concern about limitations inherent to the nature of LLMs. On an abductive, or hypothetico-deductive, model of reasoning, the critical breakthrough is always a conceptual or theoretical insight, or leap. I’m not convinced LLMs can make this sort of creative leap as they don’t have the ability to grasp a concept like a human does.

Finally, as an aside, LLMs are shaking up mathematics at the moment as they have solved a number of open research problems. Many people expect this translate into science. However, there are at least two structural reasons why this success won’t translate. For one, mathematics fundamentally follows a deductive logic, rather than abductive reasoning. This is because everything that is true in a mathematical system is defined by the concepts or axioms themselves, not with reference to some external world.

This leads to the second reason, a reasoning machine can (at least in principle) access all the relevant information about a mathematical system just by thinking. Science, by contrast, is about understanding an external reality that defies our expectations and operates differently to what we expect. And LLMs cannot access external reality in the way humans can.

Subscribe now

1

As summarised at:

Reinvent Science

The Scientific Literature is Poisonous to LLMs

We can think of no corpus of text more toxic to the training of useful LLMs than the 21st century scientific literature. Imagining a two-by-two matrix with the axes honest vs. dishonest and right vs. wrong, the scientific literature is splashed haphazardly across all four boxes. Papers often belong in multiple quadrants at once, and not always because o…

Read more

3 months ago · 12 likes · 3 comments · Reinvent Science

This result can be seen in Figure 5 of the original paper.

Earlier this year I argued, with reference to fiction, that human knowledge should be always understood as knowledge about something. We can know things, for example, about Star Wars or Sherlock Holmes, even though all that information is strictly speaking false (in the real world).

Another way of making the same point is to note that all human knowledge claims are referenced or indexed to some topic, world or context. There are many examples in the real world, not just fiction. To decide whether I am tall or not will depend on who I am being compared to. I’m tall when I visit various Asian countries but short if I hang around with pro basketball or volleyball players.

I’ve recently come across a formal approach that represents this idea in a neat way, albeit applied to a different type of concept: modal expressions like necessity and possibility. This is an area where I’ve long thought it made sense to apply a kind of ‘aboutness’ and I’m glad to see that has worked it through with some formal depth and details an approach he calls h-logic (entirely independently of anything I’ve done). While his write-up is highly technical, it highlights some useful ideas for every day life - and offers some reflections for thinking about knowledge.1

Modal concepts depend

Let me explain some background first. There are whole fields of philosophy and mathematics, centred around formal logic, devoted to considering modal concepts and their relationships. There are various motivating questions to this study, including trying to understand and justify the correct logic at play and articulating what makes it true (or false) when we say something like “it’s not possible to travel faster than the speed of light.” Many readers will have heard of the concept of ‘possible worlds’. Within modal logic and semantics, these have a technical definition and role. Most simply, something is necessary if it is true in all possible worlds, and something is possible if it is true in at least one possible world.

Within the study of modal logic, a concept like necessity is generally treated as if it is a property of the world. Something is, or isn’t, necessary (and therefore has to be true) and there is a fact of the matter as to whether this is the case. By contrast, I have long thought that when we ask, for example, if it is possible to travel faster than the speed of light, it depends on what we are thinking about. In my view, it is logically and mathematically possible to travel faster than the speed of light, but it is not physically possible in our universe that is governed by general relativity.

Philosophy Stan frames this dynamic as different constraints on a definition of possibility. In one case, we hold the laws of physics as a constraint, while in another case we only constrain ourselves to logic and mathematics. In his h-logic system, Philosophy Stan offers a nice formal notation to differentiate varying concepts or meanings of possibility (and other modal concepts) and tie them to particular sets of constraints.

I do recommend anyone interested to read the details of his formal, symbolic approach. Like all such formalism, it has limited daily application but the way it works provides a good reminder for all of us, as it pushes us to be clearer about what we are talking about. For example, when we consider counter-factuals (i.e. what would have happened if something turned out differently), his formal approach is designed to force us to think about what we are holding as constraints. Classic debates like “would Germany have won World War II if they didn’t invade Russia” often turn on what historical situations and dynamics we expect to continue and what would vary. Many of our arguments hinge on a lack of clarity about the background constraints and assumptions we are adopting.

I do have one quibble with the way he presents and discusses constraints, namely he envisages a set of constraints as a set of individual statements. This doesn’t take into account the nature of much of our knowledge and therefore the real constraints around what we think is possible or necessary. The constraint that we cannot physically travel faster than the speed of light ultimately derives from a particular theory that we are confident is true. This reflects the way that our knowledge is ultimately based in theories of various sorts, not facts.

Subscribe now

An application of modal thinking

Being clear about the constraints we are holding when thinking about probability and necessity is useful in many contexts. A good example is in the world of foresight and scenario planning, where there is a famous conceptual diagram in that field called the Futures Cone (as developed by Joseph Voros):

The idea is to visually represent the way that the range of potential outcomes or futures expands the further into the future we consider. The further into the future we look, the greater the range of different things that can happen. The diagram highlights some key points: there are the things we expect to happen, the things we see as plausible, then the things we see as possible and those beyond that.

If we adopt the perspective motivated by h-logic, it follows that there isn’t one Futures Cone, but instead there are many different cones indexed by different constraints. For example, there may be one where our current understanding of physics is the defining constraint. On another, we may take current economic and financial models as a binding constraint; or we may take existing political systems as fixed and therefore a constraint.

An approach inspired by this insight, where we try to index many different Futures Cones, is likely to be entirely unhelpful in any practical setting. But it points to a real dynamic when we are thinking about the future. Different people often have a different set of constraints in their minds and so are effectively working with different Futures Cones. They don’t agree on what is probable, plausible and possible. Many discussions about what the future holds tend to run into this problem. People have different assumptions about what can and can’t change and therefore have sets of possible and plausible futures in mind.

In my experience, the most productive discussions about the future often emerge when people intuit this problem and start identifying their constraints: what they think can or can’t change, or the assumptions about the future they are making. What we think is probable, plausible and possible is shaped by these constraints, so addressing them directly and testing our thinking can be the most valuable outcome of foresight or scenario projects.

The same principles often hold more broadly. Some of our discussions that don’t go anywhere happen when we are working with different assumptions or constraints without realising it. It is always helpful to be reminded of the principle behind Philosophy Stan’s h-logic and try to articulate more clearly what we are talking about. Many, although not all, confusions should then disappear.

Share

Over the past decade or two, a global revolt has picked up against the ‘system’ from various angles. Traditional political parties are collapsing, or being taken over from within, ‘populism’ is on the rise on all sides of the political spectra, and there is growing interest in opting out of society. This is playing out all over the place and isn’t restricted to particular political systems. The logic of this revolt is common across the globe. People are angry at various Elites, who are assumed to have stacked the system to suit their vested interests.

However, the Elites themselves are not having the good time you would expect if they’d rigged everything to suit themselves. There is a common frustration amongst this group, from political leaders, to NGOs, to tech billionaires that it is impossible to get much done. It has been vividly expressed in UK politics by many people, including the Prime Minister late last year (when he was still in that position), complaining that when they ‘pull the levers of government’ not much happens.

People across the world feel like things aren’t working as they should. As humans, we want there to be someone to blame. This is the impulse that often drives conspiracy theories across all ideological spectra. However, the situation we find ourselves in is no the result of the nefarious plans of any particular person, or group of people. Instead, it is the system-wide consequences from a particular intellectual and cultural mindset that we have accepted in the modern world, overlaid with various human and system incentives.

I attempted to diagnose and explore this mindset in a previous series of essays, where I argued that the characteristic attitude of the modern world is that the driving force of modern life is the conviction that the world is knowable/controllable by means of applying or adopting efficient algorithmic approaches for all questions and issues. This analysis was drawn from the combined works of Hartmut Rosa and Jacques Ellul, and to give it a name, I dubbed it the Rosa-Ellul Thesis. This same attitude underpins technocratic bureaucracy, Silicon Valley tech solutionism, much scientific policy making and the endless optimisation characteristic of both business processes and modern approaches to health.

While it focuses on efficiency and precisely defined processes, this mindset habitually leads to gridlock and stasis when applied over time in the real world. To borrow an analogy from a recent article, it is like the tangle of cabling, wires and tech that sits behind the scenes of any modern office, or the sluggish performance of any modern computer after years of disconnected installation of new software and updates. Each change and update was sensible and rational, but the combined effects undermine the goals we hope to achieve.

The algorithmic mindset summarised by the Rosa-Ellul Thesis is highly intuitive to us today and has many good justifications. By articulating clear, repeatable rules to follow in every situation, we expect to achieve consistency, fairness and smooth over the biases and idiosyncrasies of human decision making. However, it can also has the common effect of reducing accountability, complicating contextual decision-making and often leads to unexpected and weird outcomes. We’ll unpack this via reference to few different concepts that have been published in the last few years.

Accountability Sinks

In his book The Unaccountability Machine, Dan Davies describes the way that organisations set up rules, processes and procedures that remove the ability for local decision makers to take accountability for decisions or outcomes. The issue arises with the “delegation of the decision to a rule book, removing the human from the process and thereby severing the connection that’s needed in order for the concept of accountability to make sense.”1 As accountability disappears, you then end up with an accountability sink.

The dynamic he identifies follows naturally from the Rosa-Ellul Thesis. The core operating mode of modern organisations is to adopt standardised processes, protocols and (digital) systems to ensure their operation is as efficient, repeatable and (in the case of the private sector) profitable as possible. These are essentially algorithms, even though many are analogue sets of rules. In organisations, they are agreed and adopted at scale, commonly across a whole organisation or significant business units and everyone is required to follow the algorithm.

Davies’ insight was that the real world inevitably throws up situations that were not foreseen when the processes or algorithms were designed. Staff in organisations are still required to follow the process even when it doesn’t clearly apply or leads to some ridiculous outcome. The problem is that when something unexpected pops up, those at the front line have no authority to change the algorithm, and the reward structure of the organisation means they will be penalised for not following the right algorithm. The authority to make a change sits at some higher level (or multiple points across the organisation), where there is no incentive to change the process for one specific case as it might mess with everything else. Thus, when something egregious happens, those directly in charge point to the fact that they did the right thing by following the right process and argue they weren’t allowed to do anything else.

Genuine accountability therefore does not really exist as no-one is empowered to take responsibility for many decisions. This directly arises from the conceptual flaw in mindset articulated by the Ellul-Rosa Thesis (as argued especially by Rosa but also by others): our conviction that the world is knowable and controllable is only valid in some limited circumstances. Our algorithmic mindset works extremely well for those contexts, but erodes accountability when it encounters something that isn’t knowable in advance.

Governed not by people, but by a swarm

A second natural consequence of our algorithmic mindset is neatly captured by term swarm governance, coined by Mary Harrington in the context of online algorithms. She argued that, especially on online platforms but in a dynamic that bleeds over into real life, decisions are made, and we are governed, by a swarm of interconnected algorithms. No one person or machine is in charge. Instead there is a swarm of overlapping and competing algorithms that govern behaviour, and what emerges is the consequences of their interactions.

If we take a more expansive definition of an algorithm as set of rules to follow, then the swarm nature of governance is not limited to digital algorithms. It occurs with the many analogue algorithms we look to for governance in the real world. All our behaviour is regulated by laws, regulations, booking forms, governance processes, IT systems, reporting requirements and so on. All of these are written out as a series of rational steps in their own particular field. However, the combined result is that our behaviour is regulated by a complex set of overlapping rules and decision making processes in which no-one is accountable to anything except their own particular algorithm.

Take, for example, the process of opening a restaurant in Australia right now. Anyone looking to do so needs at least 10 licenses, permits or registrations, at least 8 different types of insurance, and then needs to comply with a varied series of regulations, like employment conditions, food safety, workplace safety and so on. So, in a real sense, the ongoing governance of a restaurant is jointly managed by at least a dozen organisations or agencies, from various jurisdictions and the private sector (insurance, financial support, software providers), plus the restauranteur themself. Each of these is, in turn, is governed by a similarly complex set of institutions, rules and policies. So the governance of the restaurant is, in a real sense, a swarm of people, processes and institutions.

Each of the governing regulations and institutions today operates via a set process, clear rules and/or a checklist. Each of these is often a reasonable, efficient algorithm for its own context. However, we rarely operate in a world governed by only one set of rules or processes and so we end up with a proliferation of overlapping rules and processes that govern what we can and can’t do. The point here isn’t to provide yet another complaint about bureaucratic burdens and red tape, but to think through the consequences for governance and accountability. Who make the decisions about the restaurant and how are they made?

At a minimum, the owner or manager is making decisions in (sometimes forced) collaboration with all the institutions behind the licenses, the insurances, their finance and so forth. It is like making decisions with a swarm of disconnected people always looking over your shoulder. Moreover, the interactions between the different requirements across the different avenues sometimes entirely constrain decisions, or lead to weird outcomes.

These dynamics feed into, and enlarge, the accountability sinks that Davies identified. It is next to impossible to shift the constraints or decisions forced by the swarm as ownership and management is entirely distributed across the swarm. The consequence is that trying to do anything can be difficult and often it feels like doing something new or different is impossible. Prime Ministers pull the levers of government and not much happens. And the problem is often that the thing we want to change is not written down anywhere but it is an indirect consequence of the interactions between a range of different rules, incentives and algorithms. In other words, we have ended up with a tangle of wires and processes that are incredibly hard to sort out, but we can’t disconnect any as things we need will stop working.

Late stage technocratic liberalism

This tangle of regulations, governance and processes is a natural consequence of our commitment to an algorithmic mindset. In fact, we are so committed to the mindset that we commonly try to fix the tangle or tame the swarm by adding in yet more oversight, regulations or processes to fix the problems. We pass more regulations to fix the problems from too many overlapping regulations, or appoint a ‘tsar’ to oversee the problems in an area that are often caused by distributed responsibility and accountability. Adding another voice in the swarm, or cables to the tangle, rarely does much beyond a superficial change to the system.

This tangle is, I would argue, the reason why so many feel like the ‘system’ is broken - including those working inside it. We are living, societally, in the equivalent of a tangled mess of cables behind the computing equipment or, to use a different analogy, inside a desktop computer running software from the last 25 years on a constantly updated operating system where we cannot shut anything down. We get by but it is increasingly hard to get anything done.

It will not have escaped reader’s notice that the mindset encapsulated by the Rosa-Ellul Thesis is quintessentially the technocratic mindset. It’s the deep conviction that we can fix our problems with improved legislation, an organisational reorganisation, better processes and new IT.

This mindset has delivered incredible benefits to our world. The optimisation or manufacturing or logistics allows us to have new goods anywhere in the world for very little money, and underpins vital services like our electricity grids. However, these fields where the technocratic approach has worked are highly knowable. We can write models and algorithms that predict what will happen.

However, much of life isn’t knowable in this same way. Human behaviour is notoriously idiosyncratic and diverse. Social interactions are situation, context, and culturally dependent. The weather and natural world are mathematically chaotic and unpredictable. And everything ends up overlapping with everything else, especially in the modern world of the ‘uni-context’.

As noted at the beginning, the common complaint around the world is that all our problems are due to the ‘elites’. This analysis, if it is correct, validates part of this criticism, only the problem arises far more from their mindset than anything they have tried to achieve. The elite mindset over the past few generations has overwhelmingly been a technocratic one that has bought into the Rosa-Ellul Thesis. In the Western world, it has been a technocratic liberalism, whereas other countries, like China, share the technocracy but not the liberalism.

However, the elites, as people, are just as trapped by their own mindset as everyone else. The only approach they can imagine is to continually try to apply technocratic solutions, new laws, new regulations, new governance mechanisms, new algorithms. The problem is that many of the problems are caused by the swarm of overlapping analogue and digital algorithms and adding more just slows the system down.

What’s next?

The obvious question is to ask what comes next. Societally, the current system is unstable and there is the potential for all sorts of conflicts. Assuming the status quo is unsustainable (beyond the short to medium term), there are only a few broad directions things could go. These are easiest to see if we return to our analogy of the old desktop computer.

If we have an old computer that is struggling, there are a few clear options. One is to upgrade the hardware. Another is to wipe the harddrive and start again. A third is to carefully reconfigure and clean up the computer, including stripping lots of no longer needed programs off it. Societally we are facing similar options.

It is now, at least in principle, possible to upgrade the ‘hardware’ that our societal infrastructure runs on. At least, that is the hope promised by generative AI. The many moves by businesses and governments around the world to harness AI to improve governance and decision making point to many people trying to upgrade the ‘hardware’. All of these are essentially leaving the broader system and mindset in place, but throwing more intelligence, more data, more compute at the problems.

The second option is the promise of radical change, which is currently coming from outside existing centres of power. This is the promise of most populist parties or factions, whether it is the Reform party in the UK or the Democratic Socialists of America (DSA) or many others across the world. Elements have this have been tried already, such as the short-lived Department of Government Efficiency in the US.

The third option is harder to imagine and tends to be discussed in smaller circles. A careful and extensive tidy-up of existing processes and governance to reduce complexity and improve accountability, without adding new processes and programs, is generally abstract and boring. One example is the way that various companies have pursued moves in this broad direction, for example, by cutting layers of management.

If our analysis here is correct, the AI solution can only ever be, even if successful, a stop-gap solution that will postpone change. The whole governing mindset is build on a faulty assumption about the world - that it is knowable and controllable. The chaos and unpredictability of the real world will always throw sand in the gears of the most robust algorithms, laws, processes, governance mechanisms, especially if the swarm of governing algorithms expands with AI.

This suggests we have the choice to carefully and sensibly rewire the building or refresh the computer, or we will get those in who want to tear it all down without knowing what to build in its place. Rewiring or refreshing requires, however, letting go of the technocratic mindset. Unfortunately, I fear that is beyond the imaginations of almost everyone currently in a decision making positions.

Subscribe now

My last article may have left some readers a little puzzled. I argued that evidence is important for decisions but evidence-based policy doesn’t work as intended. This leaves some gaps that need explaining. To start with, if evidence isn’t enough, what should we base our policy decisions on? And, more philosophically, what does good evidence actually look like?

There are a range of views about these questions in the broader world of public policy, but, in my experience, not many of them are consistent with practice. Even fewer of them have a solid basis in epistemology, or the logical structure of human knowledge. For this reason, my aim here is to think things through from first principles, and draw inspiration from other fields with strongly stress-tested knowledge practices.

The components of policy making

The previous article touched lightly on the question of what we base policy decisions on. My views line up neatly with a simple summary provided recently by Sean Innis. He has written a series of articles on the ‘Policy Recipe’, where he argued that policy decisions are made from three ingredients: values, pragmatism and evidence. While I’m not convinced by the metaphor, his three components make sense.

As I noted in my previous article, values are essential to policy decisions (which was why evidence alone is not sufficient). A policy is always designed to make the world better in some way, and the definition of ‘better’ always depends on some set of values. These values may be largely self-evident (say, it is better to live than to die) but there are always groups of people who espouse the opposite value. The challenge is that there is no way of deciding these objectively.

Pragmatism is essential. A policy that is implemented is better than a brilliant policy idea that never goes anywhere. It focuses on the question of what can we actually achieve at this point in time, given all the various constraints. This can include everything from laws, to budgets, to public opinion, to constitutional structures, to the capabilities of people involved.

Evidence covers what we have learnt from experience, what we know about what works and what we should expect to happen.

Sean used the analogy of a recipe, but an alternative is to think of policy as akin to an architectural drawing.1 The architect is always guided by values, whether it is an aesthetic, intentions for use, goals, and similar considerations. They then have to work within all the pragmatic constraints: site, location, weather, budget, time, etc. And it is critical they take into account all the evidence about what does or doesn’t work regarding engineering, structural features, sun and weather and so on. All three are core considerations, and, as an aside, there is a fourth that our modern world adds: all the laws and regulations that need to be complied with. This is equally relevant for policy decisions.

What is policy evidence?

What counts as evidence for a building is fairly clear. We mostly want to know if the building will stand, not have issues and be able to be built for the cost we can afford. Evidence for policy is less clear cut. What counts as valid evidence? On what basis should we support a decision that is pragmatically possible and lines up with our agreed values? How do we know if it is going to work or not?

As noted in the previous article, it is common to cite systematic reviews, Randomised Control Trials (RCTs) or other forms of rigorous academic work as the gold standard for evidence. However, in an important way, this focus on facts and data misses the point of what we need evidence for. It might be scientific in some senses but it typically doesn’t give us the same kinds of products as truly useful science.

To see how this works, it helps to go back to first principles and look at the logic underpinning what we need evidence for when we are making a policy decision. At an abstract level, any policy is designed so that certain people take particular actions to change what is happening for the better. A good policy has to, at a minimum, get the logic that connects all of this together correct. That is, when we put the policy in place in the real world, the relevant people do the sorts of things that lead to the types of positive changes we anticipated. It is similar to building the house. We want evidence to show that the house will actually stand and look like the drawings.

Values tell us what we want those positive changes to be, and pragmatism helps us know we can actually do. But we need the evidence to tell us if our planned policy will actually lead to the changes we want.

So, ideally, evidence will provide us a reliable way of identifying what people (or systems or organisations) will do in response to various policy actions (or the status quo). That is, it should tell us what will happen if we implement policy A versus policy B and so on. It follows that the perfect evidence would be some kind of fully reliable prediction machine or crystal ball to tell us what would happen in various situations. However, such a machine doesn’t exist for us mere mortals (despite what some proponents of generative AI might suggest).

This isn’t a new problem or one specific to policy. We deal with it in all fields and there is a standard scientific approach. What science typically does is work hard to build broadly applicable and reliable theories, or (in some cases) models, that we can use to predict what we expect to happen in the relevant specific circumstance. Strong theories and models allow us to plug in a range of different situations and see what the likely real world effects or outcomes are. For example, the scientific theories that tell us how diseases are spread by bacteria or viruses through various fluids have allowed many improvements to health care and human longevity. We can evaluate the likely effects of different interventions without needing to trial every one of them individually.

Importantly, the benefits of scientific theories or models go beyond their predictive abilities - and beyond what a crystal ball might give us. They often provide us with an understanding of why or how certain things will happen; they explain the mechanisms or dynamics that lead to particular outcomes and not just the outcome itself. The understanding allows us to identify ways of intervening positively or pick where to leave things alone. Down the track, understanding the mechanisms or dynamics also allows us to diagnose why something went wrong. In the absence of this, we can only guess, try and hope.

The logical structure of a policy decision, therefore, requires evidence in the form of something that explains what is likely to happen and why it will play out that way. This allows us to assess different options and anticipate real world changes when we implement something.2

Good evidence should be scientific, in the proper sense of the term

The ideal evidence for a policy decision is, therefore, a robust scientific theory (or perhaps model) that is reliable and directly applicable to the policy issue at hand. In practice, this is rare and we often have to use multiple overlapping theories of various grades of reliability. Importantly, these are logically distinct from the common outputs of systematic reviews or RCTs. Those outputs typically tell us what tends to happen in past situations, within a statistical standard of reliability.3 However, they less commonly tell us anything about why these results hold or the mechanisms at play.

In strong scientific practice, facts, data, RCTs and reviews help us decide between competing theories and identify which are the most reliable or most likely to be true. If our best theory is contradicted by the data, we need to find a better one. Where we don’t have a compelling or strong theory, we may have to rely on the experimental evidence as our guide but this has intrinsic limitations. It tells us what happened in a particular situation in the past. Without the reliable theory, we can’t be sure that it will happen in the same way in our current context as we don’t know why it happened that way.

In the world of public policy, it is common to hear complaints that decision makers, especially politicians, rely on anecdotes they have heard rather than the data that is available. An anecdote is only one data point and therefore data should always outweigh an anecdote. What these complaints miss is that anecdotes match the logical structure of human decision making better than data.

An anecdote is always a data point plus a story. The story provides a critical role as it provides an account of why something happened. The why is logically important for decision makers as it allows them to naturally play forward different scenarios and test them against policy options. Data, on the other hand, might provide us with robust correlations but it is much harder for humans to think it through for different scenarios. Data doesn’t provide the why. It gives us no understandable account of how the world is and what will change.

Strong policy evidence, therefore, should be a rigorously justified theory that explains how the particular part of the world works and what the effects of policy changes will be. Everything we normally talk about as evidence is important here, it should just justify our theory rather than be all there is to evidence. When evidence is logically structured as a theory, explanation or account of the world it is much easier for humans to use it to make good decisions. Facts, data and charts are harder as they don’t fill in the gaps of how or why something happens.

There are other benefits to having evidence in the form of a theory or explanation, rather than statistics and data. One notable one is that it provides a conceptual blueprint for how the policy will play out in practice. That is, it provides us with the ‘theory of change’ for how the policy will succeed to achieve the desired positive outcomes. Having this clear makes it easier to change course if things aren’t working and empowers us to evaluate outcomes more simply at the end. Once we understand how we think something will roll out, we can check to see if we are correct.

To summarise, we need to downplay the role of facts and data in our policy making so that we can make policy evidence more scientific. Science is ultimately about providing theories and models that explain the world, not data and charts about what happened. Of course, science doesn’t land on these theories without significant effort and uncertainty, and some fields never achieve certainty. A more scientific approach to evidence for policy decisions includes greater comfort with uncertainties and unknowns.

Subscribe now

2

In a related point, there are interesting arguments that we also need to explain the why when we are capturing processes and standard operating procedures. If we just describe the what, it is often hard to achieve the same outcomes. See this Substack article for more details:

1517 Fund

Forgetting The Asbestos.

The Problem…

Read more

4 years ago · 23 likes · 9 comments · areoform

In fields like public policy and medicine, there is a strongly expressed norm that all decisions should be ‘evidence-based’.1 While this might seem like common-sense and perfectly reasonable, there are endless complaints that decisions fail to live up to this standard. People’s frustration can be seen in public policy discussions by the repeated line that too often we rely on ‘policy-based evidence’ rather than ‘evidence-based policy’.

There is much to criticise in many public policy decisions, especially as there are always poor actors who make decisions for all sorts of, sometimes self-interested, reasons. However, proponents of decisions being ‘evidence-based’ are often blind to the limitations of their frameworks and ways of thinking. These limitations mean that, in the real world, purely ‘evidence-based policy’ is often impossible and can easily lead to poor decisions.

Before we get to these limitations, it is important to be clear about what is meant by ‘evidence-based policy’, especially as there are different ideas at play within the one phrase. The most basic meaning is that we should identify and take note of all the relevant evidence whenever we make a decision. More particularly, it is normally taken to mean that we should rely on the most scientifically rigorous evidence available. Along these lines, proponents often propose various evidence hierarchies that help us navigate the reliability of different forms of evidence. Systematic reviews and randomised control trials (RCTs) are almost always at the top of this list, as in this example:

On top of these two commitments, proponents typically believe that the evidence (of this scientific type) should be a definitive, or even the only, factor in deciding on a policy direction.

There are many good reasons to call for a greater reliance on evidence in policy decisions. Too many decisions are made based on the vibes, someone’s gut feel, or what seems like a good idea at the time - when there is clear evidence and good reasons why it isn’t a good decision. Greater rigour and a better understanding of what we do and don’t know about a policy issue are critical for good decision making. However, a straight-forward appeal to ‘evidence-based policy’, in the sense articulated here, brings in other flaws - some of which undermine good policy.

The most obvious limitations with an ‘evidence-based’ approach are practical, and may people who work in policy fields have first hand experience of this. Commonly, the evidence that currently exists is typically out-of-date, patchy, contested and designed for something else rather than the decision we are facing now. I have personally worked on policy where the best policy evidence was decades old and based on small cohorts in a country on the other side of the world.

These practical problems arise because there is a fundamental mismatch between the time required to build strong policy evidence and the pace of decision-making. We need to make decisions now, but rigorous evidence takes years or decades to collect. However, there are more serious structural or philosophical reasons why evidence in this sense can only ever be one input to policy decisions. We will focus on two.

Policy challenges are contextual and multi-factorial

Pick any live policy issue at the moment, whether it is housing, AI, provision of health care, immigration or anything else. None of these issues are simple. Each includes intersecting social, economic, political, environmental, legal and technological issues. Moreover, the exact shapes of the issues vary by country and location; and they often have features that are particular to our current time and context.

If evidence is expected to play a decisive role in the policy process, then we need rigorous evidence that takes into account all of the issues that are relevant in our particular time, place and context. But this requirement is impossible to achieve with any consistency. Policy evidence can only be collected from a particular place at a particular time in the past. The findings may, or may not, be applicable to the situation you are looking at now. It depends on whether important factors are the same now as in the context when and where the evidence was collected. To conclude the evidence is applicable requires analysis or judgement that goes beyond the evidence.

Another issue is that there is a fundamental tension between rigour and applicability. The most rigorous policy evidence has to isolate different factors so we can be sure about what causes the outcomes we see. If multiple things are changing at once, including the policies being applied, we cannot be sure why anything happened.

RCTs are the best example of this. They are seen as a gold standard of evidence, but they can only achieve this status by running careful experiments with a control group and varying one factor at a time for the other groups. This narrows down the range of situations where we know they can be applied to ones which match all the relevant factors that have been controlled for.

This limitation with RCTs applies in varying degrees to all forms of rigorous policy evidence. Whether we are looking at qualitative evaluations or various forms of data or statistical analysis, these always rely on past data in a particular situation and invariably focus on a limited range of factors. If you are lucky, all the relevant factors that you care about have been rigorously examined and your context is the same. However, this luck rarely holds as the time required for rigour routinely means relevant factors have to be ignored.

This doesn’t mean that evidence of this sort isn’t useful and we should ignore it. We just need to think carefully and make judgements about how applicable it is in our current situation. In other words, the pure ‘evidence-based’ approach cannot hold as judgement, which inevitably leads to arguments, is required.

Policy decisions depend on values

Done well, the types of evidence we are talking about reveal important information about the world and the way it works. In other words that give us information about what is the case. However, there is always a gap between evidence about what is, and decisions about what we should do. As David Hume famously pointed out, you can’t derive an ought from an is.

The core philosophical point is that policy decisions are always goal oriented. We make a decision because we want to make things better in some way. If we didn’t want some kind of positive change (for someone) there we wouldn’t need a policy or a decision. But how do we decide on what ‘better’ looks like?

Inherently that depends on values questions, which do not have answers we can find objectively in the world around us. For any judgement about what a better world would look like, there is someone out there who takes (or could take) the opposite view and remain consistent with the evidence.

Let us consider one simple example. It is well known that smoking causes cancer and smokers live, on average, shorter lives than non-smokers. What should the policy response to this evidence be? Well, it depends on your values and what you think is a good outcome. It doesn’t have to be the standard government response of trying to reduce smoking rates.

Let’s consider one current ideological position that is popular with some groups: there are more humans on the earth than its ecological capacity allows, so we need fewer humans for the health of the planet. If you take this view, it is logical to argue that we should encourage smoking so more humans die more quickly and the numbers of people shrink faster. This will speed up the healing of the planet.

You may feel highly uncomfortable with this conclusion, or not, but the point is that the evidence doesn’t determine how we act on its basis. Our decisions are based on values that can not be derived from empirical evidence.

This doesn’t mean that evidence like this is not relevant for our policy, or medical, decisions. If our policy objective, as informed by our values, is to try to reduce death rates and help people live longer, then it would be illogical to decide to promote smoking as a way to achieve this. The evidence gives us important information about how to go about our policy goals effectively, but does not tell us what those objectives should be.

Share

Evidence is important, but it isn’t everything

Advocates of ‘evidence based policy’ are often frustrated by the messiness of political decision making, whether that involves the party politics of parliaments or the negotiation of interests across different agencies, factions and interest groups. However, in our democratic systems of government, this messiness is one way that we address the two dynamics outlined here.

For one, we have to make values decisions and the way we negotiate these is through political mechanisms. Political parties represent, in some form, the desires and values of the public that are to be reflected in government policies. Many of the democratic conflicts we see today have arisen because this has broken down. The broader public do not think political decision makers share their values.

The other factor is that our evidence bases can never take into account all the relevant factors we need to consider. To mitigate this risk we, at least when we consider the structure of our systems, require different players with different interests and different knowledge bases to contribute to decisions. Committee based decision making, when it works well, partially alleviates some of these limitations in our evidence as it forces us to take a more holistic view. We consider more of the factors and bring different evidence bases to bear on a problem.

Of course, none of this is clean or simple, and our practices often fall short of any epistemic ideals. But it is worth remembering that purely technocratic decisions based purely on the evidence are philosophically unrealistic in most, if not all, situations. You need incredibly well resourced, up-to-date and deep evidence combined with an environment where are no disputes about our values. Policy-making is messy, but that is inevitable as it reflects the messiness of the real world.

Subscribe now

Critical thinking is often seen as the solution to many problems. It is thought to be the antidote to mis- and dis-information, the key to improving education, and even the solution to capability gaps in public services or businesses. There is a growing demand for critical thinking training and yet, as I know from personal experience, it often doesn’t work as well as we hope.

I was chatting to a young lady recently who had an interesting reaction when I said I run courses in critical thinking. She was initially highly negative, as she’d had a bad experience in a critical thinking class at university. From her perspective, the class just tried to teach her what to think, not how to think.

While I’d like to think this is unusual, I have my doubts. Critical thinking is often taught as a set of reasoning skills, that are assumed to lead people to the right conclusions. However, what that ‘right’ conclusion might be varies and seems to get more rigid as people learn more sophisticated thinking skills: in various countries, there is a positive correlation between education and polarisation. That is, the more educated people are, the more polarised they become. The reason for this appears to be that stronger thinking skills allow people to find better reasons for what they believe already.

You can see this dynamic in any of our deeply polarised societal debates, such as around climate change. All sides of the debate tend to bemoan the critical thinking skills of the other side, and often believe that teaching critical thinking will inevitably lead everyone to agree with them.

The lesson I think we can draw from this is that critical thinking skills are not sufficient for addressing many of the problems we would like solved. In my view, there are at least two factors that need to be in place to get real value out of our critical thinking skills. To start with, we need sufficient subject matter knowledge about the topic in question. And secondly, if you do not think there is a genuine chance that you might be wrong, then critical thinking skills will most often only be used to build more sophisticated arguments about why you are right.

Subject knowledge is necessary

It is often assumed that critical thinking skills are more important and foundational to good reasoning than subject knowledge. Instead of learning or memorising vast amounts of information, the idea goes, we are better off learning how to think as this is a skill that can be applied in any context. The problem is that these two factors, knowing about a subject and thinking critically about it, are not separable.

Any number of examples can illustrate this, but we’ll touch on two. The first is the relatively common case of a colleague who takes their car to a mechanic but doesn’t know anything about how cars work. The second is far less common, but imagine you’ve been asked by a friend to help read through their PhD thesis on quantum physics.

No matter how good your colleague’s thinking skills are, they are not going to help figure out check the mechanic’s explanation that the car won’t start because of a problem with the electronic fuel injection. And I doubt many of my readers would have any chance helping out a friend with that PhD, regardless of your critical thinking skills. Most of us will struggle to know even whether many of the sentences make sense.

Subject knowledge matters in both cases if you want to be able to test whether the claims make sense or might be true. But they also matter for the types of reasoning involved. The practical engineering logic needed to fix a car is not the same as the abstract, mathematical reasoning involved in physics research. The overlaps between the thinking skills needed in both cases are limited as the skills we need are often tied to the subject matter area.

Share

Humility is foundational

Subject knowledge might be necessary for critical thinking but it is clearly not enough. There are many people with deep knowledge of any subject but who are rigid, rusted on to their views and unable to see clear flaws in their arguments. Such people are invariably highly confident in what they already think and are unwilling to consider that there is any real chance they might be wrong. In short, intellectual certainty, or arrogance, is incompatible with genuine critical thinking. Intellectual, or epistemic, humility, on the other hand, is vital for genuine critical thinking. If you are open to being wrong, it is far easier to see strengths in other views and flaws in your own thinking.

In fact, I’d argue that epistemic humility is the foundation for genuine critical thinking, and is more important than a command of logic, the ability to research or skills in synthesising information. Consider a case where we have two people with deep subject knowledge on a subject. The first is a brilliant rational thinker, with highly sophisticated logic and reasoning skills, but is utterly convinced they are right. The second is less sophisticated and often slower at reading or writing strong arguments, but is well aware they might have things wrong.

The first will invariably be able to spot all the flaws in any arguments or cases they don’t agree with, but is unlikely to notice any issues with things they believe and is unlikely to fix any problems with their beliefs over time. The second may not be so good at arguments, but will take new information on board, genuinely consider alternatives and, I would argue, be a much more reliable guide to truth or what we need to know.

In other words, our attitudes towards knowledge and our own abilities matter more than the specific skills we have. This is often implicitly acknowledged in the ways that critical thinking is explained and taught. While there are many different approaches, most emphasise conceptual frameworks and lists of principles, rules or norms. Ideas like ‘consider alternative viewpoints’, ‘check your sources are reliable’, and ‘provide reasons for your conclusions’ are all common components but if you pay attention, they are things to remember to do rather than skills to learn. That is, they capture principles or norms. Importantly, all of these principles reflect a broader commitment to intellectual humility - recognising that I could be wrong.

Improving critical thinking

There is good and bad news in this analysis for how we try to improve critical thinking. The good news is that the important aspects to learn are not technical skills like understanding formal logic, advanced research techniques or statistical analyses - which are often hard to teach. The bad news is that critical thinking depends much more on values, norms, expectations and mindsets. These are all things that are simple to articulate but very difficult to change in practice, both at individual and organisational levels. All our actions are based on our default biases and respond to incentives, few of which encourage genuine epistemic humility.

I have been teaching critical thinking with this in mind, where we always start with a tangible expression of epistemic humility. It is an approach resonates with people and they can see the benefits. Moreover, the value of various thinking tools and techniques then becomes more obvious and people get genuine value out of the training. However, the challenge is that a short period of training for individuals rarely changes the broader norms, expectations and incentives that tend to work against genuine intellectual humility. These are, however, what matters most.

Subscribe now

A while ago I offered my own (fairly short) solution to the foundational question in the philosophy of mathematics: what sort of thing are numbers and other mathematical objects?

The traditional approaches to this question turn on whether numbers (and similar objects) actually exist. Platonists hold that they do (e.g. there is a real thing that is the number 2). Constructivists, by contrast, hold that numbers are simply human creations and are therefore largely arbitrary. A third school, the formalists, typically treat mathematics as a subset of logic, so a number is a just a type of formal logical construct.

My view, as I wrote previously, is that “mathematics is the systematic exploration and elucidation of the inevitable (or logical) consequences of different foundational concepts.” The good news is that I’m not the only person on Substack arguing for a ‘conceptualist’ account. David Bessis, a trained theoretical mathematician, entrepreneur and author, has been arguing for a position like this and has also explored the implications with respect to current AI systems.

Conceptualism

Bessis argues that the objects of mathematics are akin to ‘universals’ in language, which are “abstractions that are detached from any particular individual: beauty, roundness, youth…” The conceptualist position on universals, on his reading, is that “universals are products of human cognition” and he argues that the same holds for mathematics. That is, mathematical objects, map to things in our brains or cognition. They don’t have an external existence (Platonism) but aren’t entirely arbitrary conventions (constructivist) as they reflect something that genuinely exists.

There are some potentially uncomfortable consequences of this approach for mathematicians, that Bessis himself accepts. He argues that mathematics “is an imaginary activity supported by symbolic writing systems”. This means that mathematics isn’t true in the sense it describes anything in the world, but it works as a field because we treat it as if it were true. For Bessis, one huge benefit of mathematics for humans is that it is a large driver of neuroplasticity - it makes our minds and thinking better.

A stronger conceptualism

I agree broadly with his conceptualist approach, that mathematics is about understanding what follows from various concepts. However, I’d argue that Bessis underplays the importance and role of concepts in all human cognition. Concepts are not simply things that exist in our brains, but are the foundational structures of all human knowledge and, if we take Kant seriously, all human experience. Causation, for example, is a concept that we cannot make sense of the world without but also something we cannot define or see in the world.

Even if you don’t accept the Kantian point that concepts structure human experience, it still holds that concepts are the building blocks of all of our knowledge and theories about the world, especially scientific theories. As I’ve noted before, if you examine any scientific theory you’ll find it is built on key concepts. Without concepts, there are no theories.

This foundational importance of concepts explains one astonishing discovery in the history of science - the critical importance of mathematics. There is no a priori reason why the world should turn out to be mathematical. For example, many Greek philosophers saw the world as flawed while mathematics was perfect. For them, there couldn’t be any use for mathematics in science.

Yet not only is mathematics foundational for science, but weird mathematical fields have scientific applications that we would never expect. A good example is the way that Complex Numbers, which are based on what are known as Imaginary Numbers, are essential for electrical engineering.

Let me explain a little for those who don’t know what any of this means. A square number is any number that is the result of one number multiplied by itself. One basic property of square numbers is that they are all positive (as two negatives multiplied together produce a positive number). The concept of an Imaginary Number was discovered when mathematicians asked: “What if we assume that there is a number that you multiply by itself and the result is -1?” This number is called i - for imaginary - and it creates a whole new field of mathematics. The unexpected thing is that this field, formed by creating an imaginary number that can’t really exist, is indispensable for understanding real world phenomena.

Yet the existence of situations like this isn’t so bizarre if we take a conceptualist position seriously. The building blocks of all scientific theories are concepts, which can interact in weird and interesting ways. Mathematics is the study of the consequences of different concepts, so is the rigorous study of how concepts interact and what flows from them. It therefore provides a series of worked out concepts and conceptual structures for scientists to explore and test for fit with the real world.

As we know but sometimes forget, the world often surprises us and scientific theories often go beyond our human concepts. The exploratory nature of mathematics has supplied us with a range of different conceptual structures that we have used in our scientific theories. It is hard to predict in advance which will be useful and which won’t. There is no a priori reason why Complex Numbers are useful, but also no a priori reason why not.

In one sense, mathematics is, to follow Bessis, imaginary as it is investigating and exploring objects and structures that we don’t know whether they exist in the physical world or not. On the other hand, it rigorously explores the core building blocks which we use to build and test scientific theories, so isn’t fanciful or a wild flight of fantasy. To ask whether mathematical objects are real is therefore a bit like asking whether architectural drawings are real. They both describe something that could turn out to exist, but not always. Some architectural drawings are impossible to actually build, just as some mathematical fields cannot be applied. But that may change in the future, and we cannot know without exploring the those fields.

Share

What about AI?

Whether you agree with my stronger account or not, a conceptualist account of mathematics has an important consequence for the practice of theoretical mathematics and research that Bessis also writes about at length. Historically, mathematicians have celebrated proofs as the pinnacle of mathematical achievement: the goal of the mathematicians is to prove difficult and interesting theorems.

However, Bessis argues that the real work was the conceptual work involved in new ideas, new definitions and new insights. As he wrote “Mathematicians created value by introducing new concepts, but the rule was that only theorems could put bread on the table.” If we take a conceptualist account of mathematics seriously, it naturally follows that exploring and elucidating foundational concepts is the work of mathematics. Proofs and theorems are the demonstration of coherence, not the bulk of the productive work. New proofs are often most valuable when they create new concepts or elucidate old ones. The proof itself is less important.

This has significant consequences for the future success of AI systems in mathematics. There has been a lot of buzz over the past couple of years about generative AI systems solving various theoretical mathematical problems, including research level questions. AI systems, or more precisely teams of researchers using AI systems, have solved various research level questions. However, the strengths and weaknesses of these systems is significant and it lines up nicely with the conceptualist account I am exploring here.

Looking at one competition, the First Proof project, various AI systems were put to work and, as a useful commentary noted, it “seems likely that somewhere between 6 and 8 [of the 10] problems were solved correctly if one combines all attempts.” However, “Even the best models/scaffolds seem not to be able to reliably detect when they are producing nonsense.” That is, the AI systems could generate correct proofs but would also generate nonsense - and couldn’t tell the difference between the two.

This strongly suggests that there is a foundational limit to the ability of generative AI systems. They can manipulate words, logic and work with formal systems, but do not have the ability to grasp or comprehend the concepts that sit behind them. This explains whey they cannot tell whether they are providing valid proofs or nonsense. They are in the same boat as all of us who are not research level mathematicians: we cannot grasp the concepts at play and so have no idea whether any proof is correct or not.

So AI systems may be able to generate new and interesting mathematical proofs, even if humans will have to decide if they are correct or not.1 However, they will not be able to do the most valuable work human mathematicians do: elucidate new concepts and definitions. It comes back to the problem I have been writing about this year: AI systems cannot grasp concepts like humans can.

The act of grasping a concept, the “sudden flash of meaning”, is something we are all personally familiar with but unable to explain in any real sense. We don’t know how it works and what goes into it, and we definitely cannot teach it to a digital system. It is something foreign to any current AI system, both in their experience and in the logical structure of their training or code base. AI systems do statistical analysis, logical deduction, correlations and various other sophisticated operations, but they don’t grasp concepts.

Subscribe now

I was interested recently to find an argument that the success of LLMs in some contexts “can only be explained satisfactorily by the supposition that they have built a world model of some kind.” Given my previous articles about the inherent difficulties of AI systems creating world models, the argument and the intuitions behind it are world examining in some more detail.

The author, who calls himself Philosophy Bear, expands on his intuition with the following claim:

I suspect that not only is text adequate for inferring how the world works, but that it is the single most data-efficient way to do so. Why? Because, like the regurgitated food of a bird to its chick, it comes pre-chewed. It is designed to convey the relevant sorts of information needed for understanding the world in a certain way as efficiently as possible.

While this may sound plausible, there are a range of significant issues with the intuitions behind this argument. To my mind, they cluster into two different groups. The first set are the more philosophical, and perhaps speculative, arguments from my previous two posts. In short, LLMs don’t have the right mental architecture to experience the world or think as we do, and human produced text (or pictures) presupposes this architecture in the way it communicates information about the world.

The second set of issues are more practical, but also reveal a structural challenge for LLMs to reliably infer how the world works. These issues bite even if we assume that the issues I have just raised are not valid, and that human-produced text (or pictures) is the right sort of thing for communicating how the world works to a machine. The problem is that the ways LLMs are trained, and any way we could train an LLM, are structurally ill-suited to enable the LLM to build a genuine picture of how the world works. That is, they are not well suited for building reliable world models.

Not all information is relevant

To see this, it is useful to note there is a common, implicit assumption about the concept ‘information’ that I have written about previously. Within the foundations of Information Theory, which underpins all digital technology, is the idea that all information is informative, that is that all information tells us something useful or meaningful about the world.1

This view of information conceptually underpins the standard approach for training an LLM, which is feeding it all the text, pictures and information we possibly can. Training an LLM (or a human) on more and more information makes sense if we accept this assumption as all new information helps us understand the world better.

However, this assumption about the equal value of all information is flawed - as can easily be seen if, for example, we consider the information available online. Some things on the internet are true. Many are somewhat true. Many are false. But, just as importantly, much of what is on the internet isn’t even trying to be a true description of anything in the real world.

Fiction, hoaxes, jokes and games all make up large chunks of information online and none of these is an attempt to be any sort of true description of the world. Any person or machine who takes these as training data and assumes (in any way) that they tell us something of use about the world is ingesting a whole lot of misleading information.

The text online might be pre-chewed to help us make sense of the world, but it isn’t clear which world it would help us make sense of. Often, it isn’t our reality.

Processed information is no guarantee of reliability

The differences in the utility or truthfulness of different texts online as guide to inferring how the world works wouldn’t create problems if there was some fairly reliable way of distinguishing between what is true and what is not from the text itself. However, there is no universal textual (or pictoral) marker of truth or falsity. We have to either rely on coherence with other texts or on something external to judge against.2

We can see why relying purely on the information contained within texts is insufficient if we consider the well known problems in dealing with different witnesses to any sort of event. Witnesses inevitably end up giving somewhat contradictory versions of what happened, even when they are all acting in good faith. How do we decide conclusively which versions are accurate and which aren’t if we only have the statements from the witnesses? This same structural dynamic exists for LLMs, just imagine that we have an internet sized collection of witnesses.

The mathematics at the heart of LLMs solves this in a particular way. They are, at heart, statistical machines that decide the most likely response to some input. In other words, they are structurally similar to a (highly sophisticated) popularity contest as they return whatever it is the most common take or connection in the relevant training data. Many companies are now taking advantage of this mathematical structure at the heart of LLMs as the basis of new public relations strategies such as Generative Engine Optimisation. Put simply, they are making sure there is enough positive (or ‘reliable’) information about their company available online in sources considered authoritative so that LLMs will produce the story the company wants customers to hear when prompted.

As we all know, popularity is often a poor guide to truth. The more reliable way to decide between a long list of competing statements is to refer to a trustworthy external authority, ideally reality itself. So we might want some other evidence like video or audio to corroborate witness statements if there is significant disagreement. More broadly, we judge many claims against encyclopedias or peer-reviewed literature. We judge these in turn with reference to experiments, deep research and ultimately to reality.

No incentives for reliability or truth

This is the fundamental structural challenge that LLMs, and all forms of AI, face. They have no access to reliable external reference points to test their statements or pictures against. Humans can look up and deliberately test things in the real world as an external reference point. Even when dealing with statements they have no way of testing, humans can rely on someone who can do the testing, or see how the statements line up with other statements they can potentially test. This doesn’t mean our statements are always reliable but it anchors our truth claims (or pictures) in something external to the text or information. It is on the basis of this experience that we construct and validate our world models.

None of this invalidates the claim above that LLMs can build a world model of some kind. LLMs can and do build models of language, words and pictures and how they fit together. To the extent that the relationships between these reflect the real world, we can accept that an LLM has a model that reflects some features of the world and that could be counted as a world model in a minimal, derivative way. However, there are no guarantees of the reliability of these models as representations of the world.3 The structures they are built with include no metrics or measures that will drive these models towards reliability or truth. To put it in more economic terms, there are no incentives within the LLM training systems that reward truth or genuine understanding of how the world works.

The success of LLMs has demonstrated that these sorts of derivative world models are highly useful and provide us with relevant and often reliable information about the world. However, it is a very different claim to argue on this basis that they can reliably infer how the world works.

Subscribe now

We all know that learning and knowledge requires us to understand and grasp concepts. Philosophers spend their time trying to clarify and define concepts. Researchers often try to develop new concepts to underpin new scientific theories. Teachers put great effort into teaching and scaffolding concepts. And yet, no-one seems to stop and ask: what actually is a concept?

Perhaps stop and think about it for a minute. Can you explain what a concept is, where it comes from or how to make one? More importantly, can you explain any of this without using essentially similar terms like an ‘abstract idea’ or a ‘thought’ or perhaps a ‘framework’.

I’ll admit this is an unfair question to ask of you. There are professionals who spend their lives on questions like this. One of them wrote a 12,000 word entry on Concepts for the Stanford Encyclopedia of Philosophy and simply provided a summary of a range of diametrically opposed ideas. For a start, there is no consensus on whether they are mental representations, abilities, or abstract objects.

Yet, despite this definitional confusion, in every day situations we know what concepts are well enough to be able to grasp, learn, teach, build on, scaffold or deconstruct them. We talk about them as quite tangible things that we know well, yet they clearly aren’t something we can point at, touch or smell.

So, if we are honest with ourselves, concepts are something quite mysterious. We intuitively recognise them, use them all the time, and talk about them very tangibly. But, I would argue, we don’t really know what they are and definitely could not build a concept from scratch.

These mysterious things - concepts - are at the heart of human intelligence and human knowledge. All our scientific, and not-so-scientific, theories about the world are built out of concepts. Gravity, force, infection, reaction, power, jealousy, illness and many others are the building blocks for our theories. Without them we cannot build substantive theories or test them for correctness.

If AI systems are to deliver on the hype and achieve human-like intelligence, it follows that they will need to be able to build, grasp and understand concepts. However, there are core features of human concepts that appear entirely beyond the structural capability of AI systems, and beyond our ability to train them into AI.

Share

Concepts and language

Generative AI, or LLMs, are very proficient with language and their ability to produce coherent and structured text is beyond the ability of many humans. However, using language and understanding concepts are two different things. For a start, we all know people who can put together entirely coherent sentences yet entirely misunderstand the concepts behind the words they are using.

This may seem counter-intuitive. Concepts typically appear to be words with specific definitions (or are made up out of these words). So the distinction I’m making between language and concepts may seem misplaced. For example, gravity is a concept and it is also a word with a specific definition. However, there are a couple of notable issues with this line of thought.

For one, it doesn’t capture our everyday experience of concepts. One word can stand for a number of different concepts, especially when it is used in different contexts. Some of these line up with different dictionary definitions of words. Gravity is a good example of a word that stands for different concepts. Sometimes a word may technically have the same meaning but represent a different concept due to different theoretic assumptions at play. Gravity in Newtonian physical and in General Relativity are distinctly different concepts. Or a word like ‘alienation’ stands for a different concept in Marxist theory compared to psychological literature.

Another example of the difference between concept and language is the way we often have a fixed concept that lines up with a specific word but argue over the definition of it. It is common in academic literature, for example, to provide an alternative definition of the same concept and word. Importantly, the definitional arguments don’t really change the concept, just the words we use to explain it. In fact, we often find a new definition can helps us better understand and existing concept.

Also, as I’ve pointed out previously, there are many simple concepts that are fiendishly difficult to define in a precise, universal way. I used the example of trying to capture the definition of a chair versus a stool. It is essentially impossible to provide a coherent explanation of the difference between these two concepts that holds in all situations. But we all use them easily with very rare confusion.

So concepts are different from language and words, and it follows that proficiency with concepts is different to proficiency with language. Just because generative AI is good at one, it doesn’t mean it is good at the other.

The very human nature of concepts

The way humans grasp, formulate and reason with concepts has a number of distinct features. To keep this post short, here is a list of observations about human concepts without justification. Let me know if you disagree with any of them.

• We can judge clearly whether someone gets a concept or not. And once you get it, suddenly lots of other things often make sense or fall into place.

• Different people take different paths to grasp a concept. Teachers, for example, often have to explain something multiple different ways for different students. Yet it is the same concept.

• It often only takes a couple of examples or definitions for a person to grasp a concept, which they can then widely apply. This is in contrast to AI image recognition, for example, that takes thousands of points of training data.

• Our understanding of concepts is often based in past experiences, or the ways our minds structure experience. Forces, jealousy or brightness are all concepts we know less from language and more from experience.

• We can invent new concepts. There isn’t a fixed set of concepts somewhere that we draw on.

• We can work through the interaction of different concepts in our minds, often in a dynamic way.

• We accept that there are objectively right and wrong consequences that follow from concepts or combinations of concepts. All our academic work and teaching relies on this.

Many of these features are distinctively human, for better or worse, and don’t hold for computing or AI systems.

One good example is the difference between how humans and computers approach mathematics. Children get taught basic operations like addition and multiplication and it often takes them a while until they get it. But once they do, something clicks, they immediately understand a whole lot more and can do significantly more mathematical calculations. This pattern holds more broadly. Our mathematical knowledge and ability improves via significant jumps that occur when concepts click.

Computers and AI display different patterns in how they learn mathematics. The classic way is to teach it definitionally, based on the core logic on which computing operates. In this mode, mathematics for a computer is just a matter of following set rules in response to inputs. It’s the same way a calculator works and there is nothing for the computer to ‘get’.

Generative AI systems by contrast have typically ‘learnt’ mathematics via their normal statistical methods over very large datasets of mathematical information. In this mode, improvements are incremental and more fragmented. Different generations of LLMs have improved with scaling and improved training, but not in the stepwise pattern that matches the way a human grasps a concept. Humans develop in a fragmented way across different mathematical skills or fields, while LLMs tend to be more uniform. There are no signs that LLMs get some mathematical concepts and not others in the ways that humans do.

Artificial, not human, intelligence

This structural difference illustrates many of the differences between AI systems and humans. There is a mysterious magic to the way a human will grasp a concept and run with it. AI by contrast remains at heart a set of statistical calculations that does not have flashes of insight or brilliance. This isn’t to say that concepts are not encoded by AI systems in some way, but it is fundamentally different to the way humans understand concepts.

This, in my view, puts some interesting structural limits on AI systems. They will never be like humans and use concepts in the ways that humans do. This means that AI systems will never achieve the equivalent of human intelligence - or what many people refer to as Artificial General Intelligence. This doesn’t mean they are not intelligent on important definitions or can surpass humans at many tasks. Theirs is just an alien intelligence, fundamentally different to ours.

It also provides another wrinkle that will get in the way of implementing world models in AI systems. World models, as humans understand them, are built out of concepts.1 But the concepts we use are have a form or structure that seems out of reach for AI systems. Building world models requires us to provide AI systems with some sort of conceptual structure or apparatus that maps onto human concepts (if we want them to understand human theories).

However, human concepts don’t properly map onto human language and we don’t know what concepts really are well enough to code or produce them. All we can do is teach them to other humans who already work with and get concepts. So, as far as I can currently tell, we don’t really know where to start on this project. The best we can do is hope some kind of conceptual apparatus within AI systems emerges via current approaches. But hope is not a rigorous support for an entire line of research and development.

Subscribe now

In a previous essay, I illustrated some differences between humans and AI by contrasting the empiricism of David Hume with the ‘transcendental idealism’ of Immanuel Kant. Put simply, Hume believed that all human knowledge was derived from human experiences while Kant argued there are certain concepts or structures hardwired into the way we experience the world.1

My argument was that we train generative AI in a Humean way. We feed it lots of data or experiences and let it derive all its knowledge from the data. However, we humans have essential concepts or structures hardwired into us, which means our cognition and our knowledge will be structurally different to that of any generative AI system.

Importantly, it is structurally different in ways that make it easier for us to have accurate knowledge of the world. These essential structures (or, to use Kant’s term, ‘categories’) form neat building blocks for the theories, stories or world models that make up our knowledge. We don’t have to start from scratch. There is important information about the structure of the world hardwired into our minds that we don’t need to learn, and this gives us an advantage over other potential minds or knowers.

Our experiences are inherently structured

In the previous essay I focused on causation as an example of one of these essential concepts or structures. However, I have found another philosophy essay on Substack that provides a clearer example, even though the essay is on a different topic. The essay starts from a famous article called “What is it like to be a bat?” in which Thomas Nagel argues that humans can never acquire certain types of knowledge, such as the subjective experience of what is like being a bat (or any other creature). Nagel picked this example to make a clear point. Humans see with their eyes while bats echo-locate with their ears, so we have fundamentally different ways of experiencing the world.

In the essay, Barnes critiques this core point of Nagel’s argument on the basis that there are blind humans who have taught themselves to echo-locate. Therefore, there cannot be as big a chasm between our subjective experiences and those of bats as Nagel assumed. The substance of that argument is not relevant here, but Barnes made a telling observation along the way.

We tend to assume that we see space. That is, our understanding of spatial dimensions is derived from the way we see with our eyes. However, as Barnes observes:

Congenitally blind individuals report structured, oriented, navigable space. Not a stream of tactile contacts but rooms with corners and centers, hallways with length, environments with layout.

Barnes’ point is that we tend to assume our understanding of the world is tied to the sensory input. Instead, we have an underlying structure that we use to structure this input.

spatial consciousness does not require vision. The felt sense of inhabiting a structured world (surfaces at distances, paths between them, a body moving through) can be achieved through non-visual means. The maps exist without pictures.

In my words, this shows that humans are hardwired to think of or imagine the world in three dimensional space. We cannot help but make sense of the world in this way, regardless of what our sensory experiences are. This is exactly the point that Kant’s ‘transcendental idealism’ was based on. There are structures in our minds that structure our experiences. And, interestingly, space is one of the transcendental categories, or hardwired structures, that Kant included in his list.2

Share

AI lacks these innate structures

The fact that we humans are hardwired to experience the world in three spatial dimensions is incredibly useful. Given the world (as far as we know) has three spatial dimensions, our minds are structured in a way that matches the world we live in. It’s a bit like cheating. We don’t have to work to understand space and figure it out as our minds already work in the right way. In turn, this innate structure provides a foundation or logic for us to use as we build theories, world models and our understanding of the world. We don’t have to develop a mental model of three dimensional space, but rather we can just put things or objects into the hardwired model we have and play around with them.

The Kantian argument is that space is only one of many such structures or concepts that are hardwired into us. 3 And all of them serve the same function. They provide ready made structures that enable us to understand the world that we don’t have to figure out or derive. In turn, these structures provide powerful building blocks for our theories and understanding of the world.

As noted, generative AI systems are not trained in this way, but in a highly Humean or empiricist manner. They don’t have existing structures to work with but have to derive everything from scratch. This is, for example, one reason why it takes vastly more training data to teach AI a concept than it does for humans.

Building blocks for world models

In my most recent article, I looked at the push to add world models into generative AI systems, but noted there are a few wrinkles that might get in our way. The first was that we don’t know how to meaningfully translate between human language and the world (or world models). A second one was that we don’t know how we humans build our world models.

We have started to get some clues on what this might look like here, namely that we could use our hardwired concepts as building blocks. If we want to take this approach, we need to find some way to code these into generative AI systems, or to train AI systems so they develop them. The wrinkle is that we don’t know how to define many of these or train someone in them who doesn’t already think in the right way. Causation is a classic example. We all know what it means in practice, but the definition and real meaning of it is a live and heated debate.4

Space and time seem like obvious exceptions to this skeptical observation. Computers calculate spatial dimensions and render spatial objects easily, all the time. Computer games and self-driving cars run up-to-date spatial and temporal models of the world they care about. However, they may be the exceptions that prove the rule.

Computers reason spatially and temporally in a very different way to humans. Computers run sophisticated mathematical calculations on Cartesian (or other) coordinate systems - they reduce space and time to numbers and then reverse the process. By contrast, humans think spatially in a highly analogue, non-exact way that is only occasionally quantifiable. And we all know that for humans, time never feels like it progresses at a consistent, mathematical rate. Readers can disagree about whether this means computers can genuinely reason spatially or temporally, but if they do it is in a different way to humans.

This all means that AI systems face a huge challenge if we want to include useful world models in them that connect to our human knowledge. We humans have in-built, hardwired concepts and structures we use to make sense of and think about the world - and we use these to build our models. We would need to build these from scratch for AI systems but we cannot define many of them and so have no clear idea of how to even start coding them. The critical point is that these concepts aren’t equations or definitions, but inherent ways we experience, relate to and think about the world.

Subscribe now

While different AI systems continue to surprise and shock people, there is a growing sense that the current approaches to improving the underlying technology are capping out. The buzz is in new applications of the same underlying LLM (large language model) systems, rather than fundamentally new AI capabilities. An increasingly common view among leaders in the industry is that scaling to bigger data and training sets is no longer leading to major improvements and that, instead, AI systems need to start building what are known in the field as ‘world models’. This has been publicly stated amongst others, by Elon Musk and the CEO of DeepMind, Demis Hassabis.

From my philosophical point of view, this makes perfect sense. It is an explicit move to address a fundamental issue I have already written about: LLMs have no access to or understanding of a real world behind the language, pixels or data. Many of the ongoing issues with generative AI systems, especially ongoing problems with hallucinations, can be traced back to this limitation. LLMs may be able to understand how words or images fit together, but have no understanding of how the world behind them works.

As I’ve written before, humans (and likely animals) encode their understanding of the world in various theories, pictures, mental models and abstract representations. This isn’t how current AI systems work and the clue is in the name. LLMs are ‘Large Language Models’, rather than models of the world, and copying the knowledge structure humans use makes intuitive sense.

The important question isn’t, however, whether adding ‘world models’ is a good idea. It is whether it will work or is even possible. My prediction is that it is much more difficult than people expect, so much so that I expect it to be impossible. There is the challenge of building accurate and useful world models (science has spent hundreds of years on this and only got so far). But connecting the world models to the language (and pictures) that is native to LLMs is just as challenging.

What is a ‘world model’?

To be able to understand, or at least make reliable predictions that go beyond current data, any AI system (or person) needs some kind of reliable representation of the world they are focused on. In AI research, this type of representation is called a ‘world model’ and has various definitions.

One definition from a prominent proponent is that a world model is a “persistent, stable, updatable (and ideally up-to-date) internal representation of some set of entities within some slice of the world.” Another definition, this time from DeepMind, is that a world model for a particular agent (could be human or AI) is a “predictive model of its environment”. The important caveat here is that the environment we care about is the real world.

You might assume that this is the sort of thing that large AI systems already build from their training data, given they have access to all the data we can possibly throw at them. However, it has been demonstrated in various ways that they don’t, which is unsurprising from a structural philosophical point of view.

A research paper published last year provides a nice example.1 The researchers trained an LLM on the orbital mechanics of solar systems so it could predict the trajectory of planets orbiting a sun. The training data was (from a human perspective) huge - it was given data from 10 million simulated solar systems. The encouraging part of the results was that the system was able to make highly accurate predictions for any given solar system. However, the underlying physics it used looked nothing like the correct Newtonian mechanics (that both describes the real world and was used to generate the simulated data). Moreover, the physics it used was highly instance specific and varied across different solar systems. In short, it could predict specific instances accurately but had no coherent understanding of the underlying physics and could not generalise.

The authors describe the problem as follows:

A foundation model [their term for an LLM or similar system] uses datasets to output predictions given inputs, whereas a world model describes state structure implicit in that data.

To explain the underlying dynamics, it is helpful to take a simpler example and consider words like ‘left’ and ‘right’. These terms describe specific spatial relationships that are tied to a speaker’s location and perspective. Intuitively, you can learn as many uses of the words as possible, but unless you grasp the relevant spatial concepts, we would never say you really understand the words. That is, there is a gap between what you can learn directly from our language that uses the words ‘left’ and ‘right’ and the spatial meaning, as encoded in a world model, of the words.

We can see this difference play out in two very different ways of learning and applying the words ‘left’ and ‘right’. One is to see how they are used together in sentences and learn various rules about what follows as a consequence when people use these words in combination with others. The second is to grasp the spatial meaning and translate sentences with the word into spatial representations (either in our head or by drawing a picture) and then predict based on these. Think of the difference between a student who crams for a maths or physics exam by learning all the rules and what to do when versus a student who understands the underlying theories or concepts.

LLMs and gen AI are like the student cramming without understanding. Which is one reason why they require so much data. However, those students don’t really understand what they are studying, as they don’t really have a world model to work with.

Are world models feasible?

The research about Newtonian physics explains why people are excited about adding ‘world models’ to existing gen AI systems. The underling LLM (or transformer) methodology does not build coherent world models but without these world models any knowledge of the real world will be limited. Given we do know how to build computerised models of Newtonian physics it makes sense to combine the two and add world models into broader LLM architecture. The obvious results should be to harness the capabilities of LLMs while improving their accuracy about the real world.

Intuitively it sounds easy, but there are many deep challenges. I do not have any insight into the potential and likely technical side so will approach it from a philosophical perspective. To boil it down to the simplest structure, this approach requires us to be able to build or train accurate world models and to connect these in a coherent and consistent way to the linguistic and pictorial functions of LLMs. The first is difficult enough - scientists have been working at this for centuries with amazing, but still limited, results. But the second is far more difficult than people generally assume.

Connecting language to entities or relationships in the world is easy for humans to do but incredibly difficult to explain precisely. There is a long history within philosophy of people trying to do this, with limited success. If we cannot explain how humans manage this, then it is unlikely we will be able to program computers to do the same thing.

Share

Meaning - a vexed problem

Many of the hardest problems to solve are those phenomena that we find so obvious and intuitive that we don’t realise there is anything mysterious about them. A good example is the problem of consciousness. It is so normal, intimate and familiar that it can be hard to realise it is something that we don’t know where it comes from. A similar example is the idea of a ‘meaning’ of a word in the philosophy of language. What exactly is the meaning of words? How are they determined or decided? How do we learn them?

To abridge a very long story (the entry on Theories of Meaning in the Stanford Encyclopedia of Philosophy is over 23,000 words and ends with a list of questions rather than answers), we don’t really know. There are lots of things going on and we don’t really know how they fit together - and this particularly matters as understanding the meaning of words and sentences is crucial for figuring out if they are true. We can illustrate some of the core issues with what should be a simple case of meaning: proper names. And let’s use the example of everyone’s favourite person today, Donald Trump.

Clearly, the meaning of the name ‘Donald Trump’ is the person who is currently the US President. In philosophical language, the real person is the reference of the name. As such, it seems quite simple. However, lots of complications quickly arise.

1. For one, there isn’t just one person with the name ‘Donald Trump’. How do we determine which person is the reference in a given situation?

2. There are many other names that refer to the same person. POTUS, Leader of the Free World, Orange Man, Drumpf, 45, and many other names are used to refer to the same person. If the meaning of these words is the person they refer to, it should follow that all these names mean the same thing. Yet to say that POTUS and ‘Orange Man’ mean the same thing simply seems wrong, even if they currently refer to the same person.

3. We often use names of people in hypotheticals or counter-factuals. Perhaps: “If it hadn’t been for the covid pandemic, Donald Trump would have been re-elected in 2020.” How can the meaning of a word be a real person today when the sentence is about a counterfactual that never happened?

I could go on but you should be getting the picture. Connecting a simple word like a proper name to the real world referent is difficult. It gets harder with more complex instances like the word ‘chair’. As I’ve written before, it’s essentially impossible to come up with a coherent, universal definition of the word chair (especially as distinct to a similar word like stool). And then we can get to really hard words like power, or love, or the ‘global order’.

Two things should stand out from this extremely brief introduction. We all manage to understand meanings of words, use them and decide if sentences are true or not without any effort every day. But we don’t understand how it works in any precise way. Ordinarily this wouldn’t matter, except that we aren’t dealing with humans who do all of this instinctively.

We are trying to teach artificial computer systems how to connect words with entities within world models, that is, what the words mean. We need to understand it precisely to be able to train a different type of system to do the same thing. I may be proven wrong, but the fact we don’t know how to do this for humans means we don’t even know how to start with AI systems.

Moreover, we have already faced this problem with computers before and given up. The original dream of computer language translation was to teach it the meanings of words and sentences. That approach didn’t get anywhere and instead all language translation relies on statistical analyses of words, sentences and phrases. The underlying models are increasingly sophisticated and accurate, but they use the same approach and have the same limitation as LLMs. They are like students who have crammed and practised to be able to answer the maths or physics questions without actually understanding the theories.

World models - a dead end?

So I find myself in an interesting position. I strongly believe that adding world models into the architecture of gen AI systems is essential for them to improve and become more accurate and intelligent. This is how humans (and likely animals) operate. The first wrinkle is that, as far as I can tell, we have no clue about how to connect world models to the language that is native to LLMs. We don’t know how it works for humans and so don’t know how to code it for machines.

A second wrinkle is that we don’t know how we humans build coherent world models outside a limited set of fields where we already have good mathematical theories (like Newtonian physics). But that is a story for another post.

Subscribe now

1

If you want to read an explanation rather than the original paper, I’d recommend

The Algorithmic Bridge

Harvard and MIT Study: AI Models Are Not Ready to Make Scientific Discoveries

A study by researchers at Harvard and MIT sheds light on one of the key questions about large language models (LLMs) and their potential as a path to artificial general intelligence (AGI): Can foundation AI models encode world models, or are they just good at predicting the next token in a sequence…

Read more

a year ago · 211 likes · 43 comments · Alberto Romero

There has been a lot of buzz around the latest set of autonomous AI coding tools, most commonly Claude Code. These tools are significantly better at autonomously producing coherent, quality code. For many, we are reaching the tipping point where serious coders will use these tools rather than write code themselves.

What is most intriguing though is how these improvements have been achieved. As I understand it, the companies behind these tools have changed the ways the AI tools work to better reflect their inherent limitations. In other words, they have forced the tools to be more humble, and they have improved significantly as a result.

Humbler Machines

To avoid a technical explanation, I’m going to rely on Ethan Mollick’s explanations. On his account there are two particular things that the new systems do differently. The first is to work around their limited memories, and so they use new processes to save what they have done and not lose track.1 The second is to employ a number of plug-ins, or ‘skills’, to accomplish different tasks. In human terms, these AI systems have learnt to take summary notes so they don’t forget things and outsource knowledge or skills they don’t have.

From a philosophical perspective, each of these is a clear recognition of the limitations of the AI tools. The second change in particular is an acknowledgement that there are specialised skills that a general AI system will struggle with. Instead of having one AI try to do everything, it is now programmed to turn to specialist help (via plug-ins).

This might sound sensible and unremarkable, except that we are talking about AI systems that are trained on all the data and information we have available. It is noteworthy that, despite this, there are limitations to what a single AI tool or model can do at one time, and it is better to turn to outside expertise for particular tasks.

Instead of one super-intelligence that can do everything, we are getting better results from a humbler AI tool that recognises what it isn’t good at and works within its limitations.

Subscribe now

Humbler Humans

As someone who has written many words arguing for human epistemic and intellectual humility, it is significant that AI tools get better when they are also more humble. This fits the pattern we observe with humans. For example, I’ve noted previously that the genius of the scientific method was to be explicit about our epistemic limitations and build methods to be humble in a serious way. It also lines up with my argument that no one person or agent (short of God) can know everything and that different pictures or perspectives can provide valuable information on the same thing.

This points at a structural feature in how much we (or machines) can know. Short of infinite knowledge and information, any single person or agent will always be limited and will build more knowledge if they work within the limitations. Humility is important not because we aren’t smart enough, but because we cannot know everything.

Finally, and more as an aside, I doubt that the rapid progress AI tools are making in coding will lead to equivalent improvements in other areas. Computer code is a field where there is no real world outside or behind the code. This means that the entire ‘world’ of code, and all the ways it will play out, are accessible to an AI system. In this way, they are analogous to games like Chess and Go. Most of the other knowledge we care about relates to a real world that is independent from the information we have about it, and is not accessible to AI systems in the way it is to humans. Success in understanding the real world is a very different thing.

Share

I have long had a fascination with different philosophical paradoxes, not just because of inherent puzzles involved, but also for what they tell us about our assumptions, theories and mindsets. The peak of that interest was writing my PhD thesis on the Liar Paradox, with the very short (2,000 word) version available on this Substack. It turns out that I’m not the only author on Substack who has written a PhD thesis on this topic and who has published a summary of his argument (currently, but not always, paywalled). This approach from Ben Burgis has some similarities to my views but there are notable differences that, in my view, reflect a flawed way of thinking about the relationships between language, truth and world.

While it may be philosophically flawed, this way of thinking is common amongst logicians and philosophers, and widespread in our broader culture and online. To understand why it is flawed we first need to focus in on the key assumptions. As a result, the first part of this post is fairly technical and may not interest all readers, but the final section is more broadly applicable.

Does “is true” really mean anything?

For the sake of brevity, I want to hone in on one particular view, summarised in this quote from Burgis:

I subscribe to a view called “disquotationalism,” according to which the effect of saying, e.g. that the sentence “Snow is white” is true is simply to assert that snow is white. In other words, the effect of adding “is true” to a quoted sentence is just to remove the quotation marks.

Some technical context will be useful for this to make sense. For logicians and philosophers interested in these topics, the canonical use of the concept of truth is as the predicate “is true” and so they focus on sentences like:

“Snow is white” is true.

or

P is true. (where P is the name of some other sentence)

The disquotationalist position is therefore that that the predicate “is true” adds nothing meaningful to the original statement. It may have a different function in language, perhaps a marker of emphasis, like using an exclamation mark, but doesn’t add any additional information. As Burgis puts it (using more technical terms):

sentences of the form “‘P’ is true” inherit 100% of their propositional content from whatever proposition is expressed by P. Sentences of the form “‘P’ is not true” inherit their propositional content from the negation of P.

For context, the propositional content of a sentence is the claim it makes about reality and therefore the content against which we judge it to be true or false.

The motivation for disquotationalism is the intuition that a sentence like “snow is white” doesn’t say anything more or less about what the world is like than “It is true that snow is white” does.1 Both are either correct or incorrect together. However, this intuition only holds if we assume that the primary or only function of language is to make descriptive statements about the world (or reality), and that the only meaningful sentences are those that do so.

A more complex relationship

There are various ways to critique this view,2 but one important one is that this view of language assumes there is a relatively neat and straightforward mapping or correspondence between our language (and the statements we make) and the reality we are trying to describe. This assumption misses the messiness and partial nature of the relationship between language and reality. The words and statements we use normally only partially describe or specify what we mean. Context, gestures, background assumptions and shared communal norms are all important for determining exact meanings but aren’t explicit parts of our language.

Given this messiness, our languages have inbuilt concepts and apparatus for clarifying some of these relationships between language, context and reality. One of these is the concept of truth.

To explain, let’s consider an example I’ve used elsewhere from fiction. Take the sentence “Sherlock Holmes lived on Baker St.” If someone asked you “Did Sherlock Holmes live on Baker St?”, the correct response would normally be “Yes”. However, if someone asked “Is it true that Sherlock Holmes lived on Baker St?”, the question suddenly gets much more difficult.

For a start, the fact that our answers to these two questions are not immediately the same is evidence that disquotationalism can’t be correct. But the reason why this question is difficult is that, as I wrote previously, our knowledge that Sherlock Holmes lived on Baker St is knowledge about the fictional world created by Arthur Conan Doyle. Given the context of the statement or question, we naturally assume we are talking about this fictional world, and so would answer ‘yes’ to the original question. However, if we are challenged about the truth of the statement, that typically brings the question back into the real world, in which case the answer turns into various forms of ‘No’.3

This is a good illustration of the complexity of the relationship between a sentence and reality in the language we use in every day situations. There are at least three different components we need to understand or grasp about a sentence before it makes sense to decide on whether it is correct or incorrect, true or false. These are:

1) The immediate meaning of the sentence or statement.

2) The meaning of (or information being expressed by) the sentence within the particular time, place, and context, it is uttered or written in.

3) What the information expressed is about - it could be the real world, a particular discipline, a scientific theory, a fictional world.

The typical assumption in many logic and philosophical discussions, and often shared by people more broadly today (especially on social media), is that we only need the first of these to decide on the truth of a statement (perhaps with rare exceptions). From the meaning of a sentence we can decide whether it is true or false.

The Sherlock Holmes example demonstrates that it isn’t that simple, and given our human affinity for story telling, jokes, irony and wordplay, I’d argue that the straightforward, straight-line examples favoured by logicians and philosophers are the exception rather than the rule.

If we accept this analysis, contra disquotationalism, the assertion that a sentence is true does have an important role and meaning in our language. It is a slightly complex assertion that multiple things hold, namely that a sentence is (i) meaningful; (ii) legitimately about the real world and (iii) is a correct description of the world.4

Sentences can fail to be true when they fail against any of these criteria. Obviously, a sentence that isn’t meaningful cannot be true. And if it doesn’t provide a correct description of the world it clearly isn’t true. But there are also different ways a meaningful sentence can fail the second criterion and not be legitimately about the real world.

Fictional statements are one example. Others are statements with false assumptions built in, such as the common philosophical example that “George stopped beating his wife last year”. The statement is not true if George never beat his wife or isn’t married. Importantly in both of these cases we also can’t say that “George didn’t stop beating his wife last year”. Instead, the original statement fails to be about the real world as it assumes something that isn’t the case. I have argued that the Liar Paradox falls into this category - it doesn’t succeed at saying anything about the real world - which prevents the paradoxical logic biting.5

Looking at it this way, there is something correct about the disquotational view: the statements “snow is white” and “it is true that snow is white” don’t give us any different information about the world. But the latter statement gives us important information about which world we are talking about and specifies that the statement holds for the real world. This means that we cannot simply dismiss “is true” as not adding any additional meaning. Instead the meaning is about the relationship between language and the world, rather than the world itself.

Subscribe now

Not just an esoteric argument about language

This article so far might read like classic academic nit-picking, but the assumptions it reveals matter in daily life. I’ll illustrate with respect to social media as it is an extreme case, but the point holds more generally.

It is common for people to make the same assumption that the logicians and philosophers do, namely that all we need to be able to evaluate the truth of some statement is to know what it means. This drives a huge amount of outrage online as people read a tweet, post or article and react to the simple statements based on what the reader takes it to mean.

The problem is often that people ignore context, background, and implicit assumptions (i.e. everything covered by point (2) above). The reader therefore takes it to mean something different from what the original person intended. This is particularly an issue when people are making jokes, playing with ideas or riffing off each other as what is often being said is either (a) not about the real world and/or (b) means something different from what is literally being said.

Human communication and relationships thrive on these gaps between language, meaning and reality and we play around with these in all sorts of interesting and sophisticated ways. Assuming a literal correspondence between language and reality misses what it means to be human, and so often provokes outrage and confuses people over things that should be easy to understand.

Philosophers end up puzzling over things like how fiction works. Fact checkers end up tying themselves in knots trying to fact check memes and jokes. Social media users can’t help but fall for rage bait. And builders of generative AI systems end of using everything online as a single training data set based on the assumption that everything there is equivalently about the real world and is all useful for determining what is true.

Share

The organising insight of this Substack - Humble Knowledge - is epistemic humility: the idea that we need to be humble about what we know and recognise that we can easily be wrong. This principle, closely aligned with the broader concept of intellectual humility, is often discussed across range of fields. It is often cited as an intellectual virtue, often in recognition of the biases or flaws in human thinking processes. While these analyses are justified, they miss the full force of the argument for epistemic humility.

Framing humility as a virtue suggests it has value but is not essential: you will do better if you have humility but can get away without it. Similarly, focusing on humility as a response to human biases suggests that as we reduce our biases we can decrease our humility. Neither approach recognises a foundational fact about human knowledge: our information sources and ways of building knowledge are necessarily and inherently limited.

Accepting epistemic humility simply recognises this truth about the nature and structure of human experience. This does not mean that we can never know anything with a high degree of confidence or certainty, but that these situations are the exception rather than the norm. They therefore require significant justification.1

I recognise that this is a strong and uncomfortable claim, especially as our broader intellectual culture is built on an assumption that certainty is common. But I would argue that epistemic humility cannot be avoided when we think seriously about human knowledge. It should be the starting point from which we understand knowledge, information, decision processes and action, rather than something we consider later.

This means, for example, that we should not set up decision processes on the assumption that we can have certainty about our informational inputs. For this assumption is not justified and says more about human hopes (or delusions) than anything about the real world.

Share

Evidence for humility

To say that epistemic humility simply recognises the truth of human experience is a strong claim, but there is a lot of evidence behind it. I have already covered lots of strands of evidence in previous articles, so this article will mostly aggregate previous information.

Nevertheless, we can start with an academic estimate of the amount of information contained in the visible, physical universe. A few years ago, physicist Melvin Vopson estimated that “there are ∼6 × 10⁸⁰ bits of information stored in all the matter particles of the observable universe.” We should not take this as definitive, but a reasonable ballpark estimate.

By comparison, if we approximate the total of human knowledge by the amount of information stored on the internet (which clearly is not accurate, especially as the evidence for the Dead Internet Theory keeps growing), then human knowledge (or at least the internet) contains about 1 x 10²⁷ bits.2 For context, this means that if the information stored on the internet doubled every year, it would still take almost 200 years to match the amount of information stored in the physical, visible universe. However you look at these numbers, the volume of human knowledge is insignificant and rounds to 0 under all approximations compared to the volume of things to know in the universe.

Of course, it can be objected to this that human knowledge is coded more in theories, especially scientific theories, about how the world works rather than disaggregated data or information. But the present state of science only adds weight to the arguments that we are fundamentally limited. Some of the most foundational and highly successful scientific theories developed in the twentieth century, like quantum mechanics and chaos theory, explicitly build in limits to our knowledge to the core frameworks of the theories. Even more seriously, there are modern discoveries of physical problems that are logically unsolvable and so can never have an answer.

We can also look at the many discoveries in psychology about limitations to human reasoning and knowledge, of which cognitive biases and heuristics are the most well-known. Or, more generally, the very structure of the scientific method, the most consistently successful approach for building knowledge, is founded on an explicit epistemic humility.

This same pattern shows up if we look at the history of philosophy and structured thought, which is a long story of repeated failures to establish any basis for certainty about our knowledge, including via formal logic or mathematics.

This isn’t just a historical perspective, but if we pay close philosophical attention to the structure and nature of personal human experience, we find the same results. Human knowledge, whether individual or group, is always tied to perspective and is theory-bound. This doesn’t mean it isn’t reliable or generalisable, but that it comes with inherent limitations. Instinctual human behaviours about knowledge recognise this and actively track a nuanced scale of trust for knowledge claims that doesn’t reduce to the simple true/false dichotomy that logicians focus on.

Given it is a fact of human existence, humans behave in many ways that recognise the limitations of their knowledge. Longstanding principles like requiring multiple witnesses, or checking your sources, reflect this understanding. However, the way we theorise about knowledge and have built it into formal processes often misses these insights. We try to avoid the fact of our limitations and often build neater and easier processes that leave out the messiness required to deal with human epistemic limitations. However, if we want long term success in the real world, we can’t avoid facing the truth of our human limitations and need to accept epistemic humility.

Subscribe now

Recently, Sean offered an interesting hypothesis in response to a post. He wondered whether “The thinking limit of an AI reflects the limit of statistics as a way of seeing the world.” I’m not sure if the idea is right, but it seems like a significant insight. For there are a range of reasons why statistics are inherently limited as a method for understanding the world. For example, in a post on scientific research, I summarised the argument that the reliance on statistics has slowed down scientific discovery. The key worry was that “statistics encourage researchers to play with numbers and data to look for different effects, rather than focusing on attempts to genuinely understand what the real dynamics, systems or factors are.”

However, there is a particular, more philosophical, worry that I want to highlight. A core engine of modern statistics is the calculation of probabilities, but statistical probabilities are often highly imprecise and conceptually vague when we apply them to the real world. The mathematical precision of our calculations often disguises significant conceptual fluffiness which means that probabilities and statistics can never be highly precise when applied to the real world. This will necessarily impose limits on what statistics, and possibly therefore AI, can tell us.

This fuzziness exists across a few categories, and we’ll look at three in turn.

What does a probability mean in the real world?

We all encounter and consider probabilities every day but we rarely have (or need) a precise idea of what they mean. Let’s take a weather forecast as a typical case. Imagine I’ve consulted the forecast and have read that there is a 50% chance of rain in Canberra tomorrow. But what does this actually mean for the real world? There are a wide range of possible interpretations.

For a start, it is unclear what it means when we consider geography. It could be any one of these:

• There is a 50% chance that it will rain somewhere across Canberra tomorrow.

• There is a 50% chance that it will rain at the designated weather station that provides the official record.

• For every place within the Canberra region, there is a 50% chance it will rain tomorrow.

• Within the Canberra region, it will rain in 50% of places and not in the other 50%.

It is also unclear what it means about time and timing, as these are both equally valid interpretations:

• At any point in time tomorrow, there is a 50% chance that it will rain.

• There is a 50% chance it will rain at some point in time tomorrow.

The weather forecasters can no doubt tell us exactly which of these definitions is the one they use. But as we ordinarily use these concepts (and weather is just one example), the concepts and phrasing are highly imprecise. While this bothers some people, for almost all practical purposes this imprecision doesn’t matter. The weather forecast gives us enough information to make decisions about what to do, what clothes to wear and whether to have a picnic regardless of the precise meaning.

However, this imprecision matters if we are trying to use probabilities and statistics as part of a research project or as a method of uncovering truth. For example, how we test and validate probabilities depends crucially on the exact interpretation we have in mind. But this becomes very difficult when we only have access to the openly available statements, which rarely include the precise definitions needed for accurate research.

Share

What actually is a probability?

While the imprecisions around the real world interpretation of a probability are worth noting, there are deeper issues. There exists significant conceptual fuzziness and deep philosophical debate about what a probability actually is or actually measures.

For a start, some people read probabilities as objective statements about the world - referred to in the academic literature as chances. Using our weather forecast example above, on this view, based on the current state of the world, it isn’t set whether it will rain tomorrow. If we could create many universes and start them in exactly the same state as our current one, in 50% of them it would rain in Canberra tomorrow, and in 50% it wouldn’t.1 This seems like a robust way of thinking about probabilities in certain cases, like flipping a coin or rolling a dice.

A different view is to see probabilities primarily as statements about human knowledge rather than as objective statements about the world - in the literature often referred to as credences. On this view, the laws of physics and weather have already locked in whether it will rain in Canberra tomorrow, but based on our best knowledge and information, the best we can do is judge that there is a 50% chance of rain. This way of thinking clearly makes sense if we are assessing the probability, say, that a certain suspect committed a murder.

You might think that these different interpretations are only relevant for philosophical debates, but they underpin different mathematical theories. Roughly speaking, the world of statistics and probabilities is split into two camps based on these different interpretations: frequentists and Bayesians.2 These two camps have different foundational formulae for the calculation of probabilities, and therefore sometimes provide different answers to statistical questions.

So we don’t agree on what a probability actually means about the real world, nor on the correct way to calculate them. Any methods that rely on probabilities for research or to uncover truth therefore need to step carefully, and this includes generative AI.

What is the right way to calculate a probability?

The question of how to calculate a probability, or run a correct statistical analysis, quickly gets very complicated, technical and philosophical if we take it seriously. There are a range of statistical tests we can use on data sets, but these all make assumptions about the nature of the data that we often cannot know in advance. Moreover, we very often use probabilities to help us make predictions - and predictions open up more philosophical and mathematical questions.

Some methods base future probabilities on properties of existing datasets but, as the financial advice tells us, past performance is not a reliable predictor of future performance.

In other cases, we rely on expert judgement to assess the probability, often by relying on the ‘wisdom of the crowd’ and asking multiple experts. Exactly when expert judgement is reliable, and why we can trust it in certain circumstances, are usually unanswered questions.

More mathematically sophisticated approaches build models of the situation under consideration and use these to assess future probabilities. But in this case the reliability of the numbers depends entirely on the accuracy of the model to capture the relevant real world phenomena.

For information, this is the approach weather forecasters use. They have highly sophisticated weather models that they then run many, many times with slight variations on the data. Technically speaking, the statement that there is a 50% chance of rain tomorrow means that the forecasters have run their model many times (at least hundreds) and in 50% of the model runs it rains tomorrow. What this precisely means about the real world depends on your views about the match between models and reality, plus all the questions above.

Vague and imprecise, but still very useful

From my sketch of the issues here, we can see there are a range of reasons why statistics and probabilities are not immediately reliable as guides to the real world - and can often cover over gaps in knowledge or information. In one sense this is a typical philosopher’s complaint, as it doesn’t reduce their usefulness in many day to day situations. We don’t need to know exactly what the 50% chance of rain means, as the fuzzy statement gives us enough information to make better decisions. A more conceptually precise formulation would often be less useful if it isn’t as easy to calculate, understand or apply.

However, it does mean there are limits to what statistics can ever teach us about the real world. As we have seen, probabilities are highly sophisticated and detailed mathematics based on fuzzy and sometimes sloppy philosophical foundations. If we are looking to do serious work to discover truth, the fuzziness becomes a problem.

Given that generative AI is fundamentally built on statistical analysis, it seems likely that these same limitations will be inherited. Despite many claims that AI will help us uncover deep truth in a neutral, objective way - the fundamentally statistical approach may be forever limited by the fuzziness of the underlying probabilities and statistical theories.

Subscribe now

It is common these days to read about the importance of navigating, or dealing with, uncertainty - often accompanied by the claim that we are living in times of unprecedented or extraordinary uncertainty. What is curious is that the focus is almost always on leadership or psychology or emotional resilience. What people don’t notice is that uncertainty is, at heart, an epistemic issue. We are uncertain because we do not know what is happening or what to do. This means that understanding epistemology, and the connection between knowledge and action, is critical for navigating uncertainty.

This has been a consistent theme of my writing and readers can look at previous articles for some of my thoughts on the topic. There is, however, a concept from the philosophical literature that is useful when we are dealing with uncertainty: epistemic risk.

Put simply, we have epistemic risks when we are relying on knowledge for a decision or a course of action but it is possible that this knowledge is wrong. Many readers will groan at the thought of yet another category of risk management, but it provides a useful framework to think about uncertainty. In particular, it helps identify how we can apply standard, well-known risk management techniques.

What are the chances we’re wrong?

In making decisions or taking action, we always rely on information or judgements about the world - perhaps what is happening now or what will happen if we do certain things. Sometimes it is something from the news or a report. Sometimes it is a research finding or the outcome of a model. Often it is personal or institutional experience and insights.

To suggest that any of this information might be wrong is not a new insight. It is the starting point for all research practices and techniques - verify sources, run the data, design strong experiments, test your conclusions. Moreover, as I’ve noted before, it is an instinctive human practice. We implicitly keep track of how much we trust, or how strongly we believe, different information. In the philosophical literature, this idea is sometimes quantified through the notion of credences. I might have a credence of 20% for some statements but 95% for others - denoting how much I trust that it is true.

Less familiar, however, are clear guidelines or frameworks for what to do when we don’t fully trust our information (or it is uncertain). This is a significant oversight as in life it is rare that we can sit and wait until we have fully trustworthy information before we act. Instead, we often have to make decisions and act when there is uncertainty about what we know.

As acknowledged by the organisational literature, this tension - between wanting to make good decisions based on trustworthy information and needing to make decisions and act now - is often psychologically, organisationally and emotionally difficult. Common responses are either to create apparent certainty or to delay until certainty is possible. A better approach is often to take the concept of epistemic risk seriously and adopt a risk management mindset about what we know.

Mapping epistemic risks

The standard approach to risk management defines risk as a function of both the likelihood of a risk occurring and the consequences if it does. I’m sure many readers have spent significant time with risk matrices like this one:

We can apply this framework to thinking about epistemic risk. In this case, the first question becomes:

1) What is the likelihood that the information we are relying on (i.e. what we think we know) is wrong?

In many contexts, this is a familiar question as we live in a world of incomplete and patchy knowledge, where the exact datapoint we want is rarely available. However, the literature on cognitive biases reminds us that we routinely overestimate the reliability of our beliefs and what we think we know. This isn’t just a personal issue, but affects organisations and other groups. Not only is groupthink a very common issue, but organisational processes often presume that a reliable source is sufficient reason not to question any further.

Business literature, political science and military histories are full of examples where people presumed they knew something which turned out to be wrong. If they had though about their epistemic risks and seriously considered the likelihood that their core information and judgements might be wrong, we would have fewer cautionary tales.

The second standard risk question in our context is then:

2) What are the consequences for our decisions/actions if the information we are relying on is wrong?

Considered systematically, this question forces us to think about what would happen to our plans or actions if critical information turns out to be incorrect. It might be considering what would happen if we act on the rumour we heard about a friend. Or the outcomes if the medical diagnosis is wrong. Or what will happen if the science a company is selling is flawed. Or where things will fail if the theory of societal change a government policy is relying on is incorrect.

Readers will have their own assessments, but my observation is that asking this question is far less common than wondering whether our information is wrong.

Share

Epistemic risk management

If we think about uncertainty as epistemic risk, then navigating uncertainty becomes an exercise in epistemic risk management. Risk based approaches to uncertainty are common, but we also need to think about how we manage the epistemic risks. The standard risk framework, which means we try to treat risks by reducing their likelihood or by mitigating their consequences, can help us think this through.

For epistemic risks, there are many methods for trying to reduce their likelihood, that is, to ensure our knowledge is more reliable. Surveys and polls have sophisticated methods to sample populations and correct for errors to reduce these chances. Scientific experiments are carefully designed and calibrated. Individuals seek input from outsiders to minimise biases and get different perspectives.

Adopting a risk mindset helps provide some context to guide the way we approach this. The priority for treating risks is always on those risks with high likelihood and/or severe consequences. This means we should focus our efforts for improving the reliability of our knowledge on the information that is either particularly shaky, or which is crucial for the success of a project or some work. If getting some information wrong means that things will fall over, we should focus there. Getting all the footnotes properly sourced matters less if the information isn’t highly consequential.

The nature of the world and human knowledge means that there will always be epistemic risks as we cannot guarantee knowledge with certainty. As a result, we need to be alive to ways of mitigating the consequences of epistemic risks. This overlaps with typical risk management approaches - most risk strategies will also help mitigate the consequences of epistemic risks. However, they can be strengthened if we focus closely on epistemology.

For example, a classic approach to managing uncertainties is to hedge. Rather than putting all our eggs in one basket, we take precautions in case our approach is wrong (e.g. take out insurance) or try multiple strategies at once that cover off different likely situations. If we don’t know what is going to happen, that is if we have epistemic risks, then hedging to ensure we can succeed regardless of which information turns out to be correct is often a sensible strategy.

However, systematically hedging can be difficult or expensive. An alternate approach, which has turned into something of a buzzword, is adaptive or flexible planning. If we genuinely aren’t sure whether our information is reliable, we should still take actions based on what we currently know, but build in options to shift approach or pivot if it turns out the information is wrong.

Both of these can be supported by particular types of analysis. One good example is the practice, common in intelligence analysis, of identifying indicators or signposts of particular scenarios playing out ahead of time. So, for example, the current assessment might be that a particular government is stable and will continue. However, if this was going to change, then the analysis would identify things that we would start seeing before it happens. This allows us to update our information and be ready to shift strategy.

This method of analysis is directly applicable to epistemic risks. If we are worried that some important information isn’t reliable, we should try to identify what the alternatives are and what would happen if the information is wrong. Being aware of these possibilities means we will notice if things are playing out as we expect, or if things are going wrong consistent with our information being unreliable. In turn, this allows us to hedge, adapt or pivot.

Put differently, we can better mitigate epistemic risks if we take a more scientific approach to decision-making by being explicit about our expectations (or predictions) and updating our facts or theories when things don’t turn out as we expect.

Not too many risks

Many people’s eyes start glazing over when someone starts talking about risk management, so I hope readers have made it this far. The systematic approach that is often taken to dealing with risks can be dreary or overwhelming, so in general I wouldn’t recommend adding a whole new epistemic risk section to existing risk registers. However, being aware that there are epistemic risks and building practices to minimise both their likelihood and consequences can be very valuable. These can range from regularly asking questions about what would happen if our information is wrong, through ensuring the rationale and expectations for a course of action are articulated and known, to more explicit scenario, contingency or adaptive thinking and planning.

A key point about epistemic risks is that, like all risks, they can never be eliminated. All human knowledge is somewhat fragile and tenuous, but the consequences are greater for the critical information we are using for important decisions. Taking epistemology seriously and consciously treating epistemic risks gives us some strategies to make navigating uncertainty easier.

Subscribe now

I recently spoke at a school conference about AI. While there, I was reminded of the ongoing debate within the theory and practice of education between proponents of ‘inquiry based learning’ and ‘explicit direct instruction’. Unfortunately, it often has the same subtlety and calmness as every other current internet-fuelled culture war and it is beyond my scope to go into it in any detail here.1

However, it got me thinking about how we train AI and LLMs, and especially the parallels there are to how we teach humans. One of the philosophical benefits of modern AI is that it provides us with an interesting test case of something that thinks (in at least some senses of the term) but isn’t human. This gives us the opportunity to think about the structural features of different teaching and learning approaches somewhat independently of human psychology.

To look at this, we will consider whether we apply an explicit instruction or inquiry approach to training LLMs to see what we can learn about AI, and ourselves.

What is the debate about?

For those who are unfamiliar with the theory of education, these two approaches to education are roughly distinguished by whether they put the teacher or the student in charge of learning. Let’s begin with a definition.

Explicit instruction is a highly structured, teacher-led approach where skills and concepts are broken down into clear, manageable steps. Teachers model skills, provide guided practice, and gradually encourage independence.2

The identified strength of this approach is that it ensures that students master important concepts and skills and are therefore able to build more complex or sophisticated skills more easily. They learn the basics and, once they have those, they can move on. It also ensures that their knowledge is properly grounded on what we already know.

To its critics, however, explicit direct instruction is old-fashioned, dreary rote learning that drills students in facts and figures but doesn’t teach them how to think or provoke any joy in learning.

Inquiry-based learning, on the other hand, encourages students to explore concepts through questioning, problem-solving, and collaboration. Teachers guide students but do not provide direct instruction in the same structured way.

The strength here is in taking advantage of human curiosity and self-awareness. Students using the inquiry-based approach thrive when they know that they don’t know the answers and are excited to find out more. For critics, however, this only describes some students and generally those who already have a strong foundation. It asks too much of students, overloads their cognitive abilities, and ensures that many children do not effectively learn.

What about LLMs?

At first glance, it appears that direct instruction describes the way we train LLMs better than an inquiry-based approach. We feed them huge amounts of facts and data and focus on ensuring they are appropriately responding to direct queries. Questioning and problem-solving are not traditionally a focus of either direct instruction or LLM training.

However, this misses the core principle of explicit instruction: students are taught skills and concepts in clear steps - and only move on once they have mastered each step. LLMs are not trained in a carefully structured, or scaffolded, way. If they were, they would be initially trained on a limited set of information and only be given more once they have mastered that.

Instead LLMs are trained on huge data sets predominantly containing unstructured and uncurated data. To use a human analogy, they aren’t given a textbook to follow but let loose on the whole library and asked to figure it out themselves. Importantly, this involves figuring out how to deal with contradictory information, which LLMs navigate by applying statistical techniques.

This means that LLM training is more like a ‘student led’ approach where the student (in this case an LLM) has the responsibility for putting the knowledge together. There is some guidance in the form of human reinforcement but, importantly, this is not a structured training program but a test of outputs at the end. It is more like feedback on a final report than an explicit, structured instruction. In these aspects, LLM training is more like an inquiry-based approach.

Nevertheless, the training of LLMs lacks many of the essential ingredients for a genuine ‘inquiry-based’ approach. LLMs do not possess the curiosity, agency and ability to collaborate that inquiry approaches rely on.

So the best answer is that LLM training uses neither of these approaches and likely is not built on any typical human pedagogical framework. This is likely appropriate as LLMs do not function like humans, and so we shouldn’t train them in the same way. However it does raise some questions.

Share

Can we properly educate an AI?

The ongoing concerns about AI hallucinations and reliability suggests that we need better ways to train LLMs. Intuitively, for example, adopting more of an explicit instruction approach, by structuring the training so that LLMs master trusted information first, would improve reliability.

However, in working through this, I’ve noticed that there are structural limitations that mean we can’t easily use human pedagogy to train LLMs. Explicit instruction relies on humans being able to master skills and concepts, and demonstrate their mastery. This is entirely normal, except that we don’t really know what concepts are and they aren’t the sort of thing that AI can grasp.

Inquiry-based approaches rely on curiosity and doubt, plus the conviction that any fair minded person will come to a strong and correct conclusion after studying enough evidence. AI, however, lacks the ability to be unsure or partially sure about information - which are essential human skills. And it regularly demonstrates that it can come to different conclusions about the same topic when asked different questions (or by different people).

This doesn’t mean that approaches inspired by human education can’t be applied, just that the differences between humans and machines are significant and need to be understood. In any case, there is growing evidence that the hallucination problem isn’t something that can be solved, just managed, with fundamental mathematical constraints. Of course, humans also get information wrong and come to faulty conclusions all the time. It is a good reminder that we always need to remain humble about what we know, whether it comes from AI or a human.

LLMs training match neither of the two approaches as they don’t have the capabilities.

Interestingly, the shallowness of LLMs and their common hallucinations match one of the criticisms of inquiry-based approaches, reflecting the structural challenge I noted earlier. If someone is left to sort it out themselves amidst a huge sea of information, it is often impossible to decide what is true or false.

However, inquiry based proponents could also point out that LLMs lack the fundamental sense that their knowledge is inadequate to describe the world which is essential to make inquiry approaches work.

Subscribe now