I did not expect to find a contemporary problem of education in a 13th-century theological text. For years, my understanding of learning has been shaped by cognitive psychology, a very recent discipline.

The more I read about memory, attention, knowledge acquisition, and creativity, the more convinced I became that learning couldn’t be reduced to the reception of information. Knowledge had to be constructed within the learner. No explanation, from the outside, can perform that work for us.

Then I encountered Thomas Aquinas’s De Magistro [1], the eleventh question of his Disputed Questions on Truth. Aquinas begins with a question that initially sounds almost absurd to us today: can one human being truly teach another, or is God the only teacher?

The Augustinian position gives a privileged role to the divine teacher within. Words and demonstrations reach us from outside, but genuine understanding arises within the learner. No human teacher can simply place knowledge inside another mind.

Aquinas eventually argues that human beings can indeed teach. A teacher can cause knowledge by presenting signs, arguments, and demonstrations that help the learner’s intellectual capacities move from potentiality into actuality. Yet the learner remains the agent through whom understanding is realised.

I found this remarkable. Seven hundred and fifty years before contemporary cognitive psychology, Aquinas was discussing a problem that remains central to educational research: teaching can support understanding, but it cannot substitute for the learner’s own intellectual activity.

In the fourth article, Aquinas asks whether teaching belongs to the contemplative or the active life. He distinguishes between what is taught and the person being taught. Teaching concerns contemplation because it deals with truth, but belongs more properly to the active life because its purpose is directed toward helping another person.

Teaching, then, is not merely the exercise of knowledge. It is an activity oriented toward the good of another. And that made me wonder about something Aquinas was not directly asking: what if learning, too, is an activity oriented toward the good of others?

We never learned alone

We usually describe learning as an individual achievement. I acquire knowledge, develop skills, improve my capacity to solve problems. I become more autonomous and expand my possibilities for action. All of this is true, but it captures only part of the phenomenon.

Human beings evolved through forms of social organisation in which cooperation and communication became central to survival. We did not succeed merely because individual brains became more capable. Our capacities developed within increasingly complex systems of collective action.

Michael Tomasello’s work on shared intentionality [8] suggests human cognition is distinguished by our ability to participate in activities organised around shared goals, coordinated attention, and mutual expectations. We do not merely observe the world alongside others. We learn to act within a world that we co-construct and maintain together.

Learning to speak involves acquiring vocabulary and grammatical rules, but it also gives us the possibility of participating in a community of speakers. Learning a craft means mastering procedures that allow us to participate in practices that existed before us and may continue after us. Learning to reason allows us to participate in collective decisions and help others understand problems that none of us could resolve alone.

Of course, this does not make all learning altruistic. Human beings also learn to compete, manipulate, dominate, and pursue exclusively personal interests. Cooperation and conflict have always coexisted in human societies. Still, learning has a social significance that cannot be reduced to individual achievement.

Becoming like someone

Albert Bandura’s social cognitive theory [2] helps explain why other people matter so much to our motivation to learn. Humans acquire knowledge and behaviour through observation. But observational learning is not simply copying what someone else does.

When we observe another person attempting a difficult task, making mistakes, correcting them, and eventually succeeding, we acquire information about the task. We may also acquire something more important: a new belief about our own capacity to succeed. Bandura described this as a source of self-efficacy through vicarious experience.

The characteristics of the observed person matter. Dale Schunk’s research on peer modelling [7] showed how perceived similarity and competence can influence learners’ expectations about their own abilities.

This is something I have often discussed with my university students. I sometimes tell them that they may learn more from one another than from me. Not because I have nothing to teach them, but because their peers offer a distinctive kind of model.

A student watching a classmate solve a problem can recognise someone who shares similar difficulties, limitations, and uncertainties. The colleague’s achievement makes success appear attainable.

But learning from a master introduces another dimension. Consider someone learning karate. The student observes the master’s movements, precision, control, and apparent ease. They want to develop those capacities. They train, repeat, fail, and gradually improve.

Yet the desire rarely ends with the acquisition of technical competence. The student also wants the master to recognise that progress. The coloured belts measure competence, but they also make achievement visible within a community and mark the learner’s changing position among others.

A new belt tells the student that someone whose judgement matters has recognised a transformation. And perhaps the student imagines a future in which they will become a master themselves.

We do not merely want to reproduce the actions of people we admire. We may want to become people whom those models can eventually recognise as their peers, perhaps even surpassing them.

The master represents a possible future self. The same can happen with a teacher.

Learning can therefore become a process of becoming someone through our encounter with another person. Recognition is essential to that transformation.

Jean Lave and Etienne Wenger describe learning as a process of becoming a participant in a community of practice [6]. As the novice gains competence, the relationship changes: others begin to rely on what they can do, and eventually they may be the ones guiding newcomers. Recognition matters here because it marks a changing place among others, not merely the achievement of a personal goal. And when we discover that what we have learned can help someone else, that too can become a reason to keep learning.

What can we learn from an intelligence without a life?

If Aquinas was right that teaching can produce knowledge without directly transferring understanding, then there is no reason to assume that artificial intelligence cannot support genuine learning.

An AI system can present explanations, demonstrate procedures, ask questions, provide feedback, and adapt to a learner’s difficulties. Such interactions can produce real cognitive gains.

Kestin and colleagues [5], for example, found that a carefully designed AI tutor produced stronger learning outcomes than a particular form of classroom active learning in a university physics experiment. In a different educational setting, Bastani and colleagues [3] found that unrestricted access to generative AI improved students’ performance while using the system but could undermine subsequent independent performance. Pedagogical safeguards substantially reduced those negative effects.

The evidence does not support a simple opposition between human teaching and artificial tutoring. AI can support learning, but it can also interfere when it allows students to bypass the intellectual activity required to develop competence.

But even when AI produces genuine understanding, another question remains. Does successful cognitive mediation reproduce the full educational significance of learning from another person?

Return to the karate master. The student does not merely admire the master’s technical performance. They encounter someone whose competence is inseparable from a human trajectory: years of practice, mistakes, persistence, discipline, and participation in a community. The master demonstrates not only what can be done, but what becoming capable can look like in a human life.

A contemporary language model presents a different situation. It can display extraordinary breadth of knowledge, construct sophisticated arguments, and generate explanations that exceed the immediate capabilities of many individual teachers. But its responses do not represent a personal history of learning comparable to that of the human master.

A student can certainly learn from the reasoning strategies an AI presents. They may admire its capabilities, adopt its methods, and even experience its feedback as encouraging or validating.

A human teacher can become a model of what the learner might one day be. An AI can provide models of what the learner might one day do, without necessarily providing the same human trajectory of becoming.

A human teacher occupies a place in the world independent of the student. They have commitments, intellectual standards, responsibilities, and relationships that do not originate in the immediate educational interaction. When they recognise a student’s progress, that recognition comes from someone who participates in the same human world and whose judgement can matter within it.

This is a distinction that requires empirical investigation rather than an assumption of human superiority. The central problem is not whether AI can imitate the behaviour of a good teacher. It is whether the relationships through which we learn to understand ourselves as capable human beings can be reorganised around artificial mediation without losing something important.

When education becomes an individual optimisation problem

Some contemporary visions of AI-based education imagine a future in which students acquire academic knowledge primarily through personalised software, while human interaction becomes a supplementary activity.

A recent example is Alpha School, a growing network that promises accelerated academic learning through software-supported instruction. An October 2026 investigation by The New York Times [4] found that only 42 percent of the students in the analysed private-school cohort achieved the network’s advertised target of doubling their learning growth in both mathematics and reading.

More interestingly, the model continued to depend heavily on human motivational support. In a video cited by the investigation, Alpha’s founder described educational technology as only a small part of the solution, attributing most of the importance to having motivated students.

If motivation is so central, optimising the delivery of academic content cannot, by itself, solve the educational problem. Students still need reasons to engage, persist, and care about what they are learning.

And those reasons do not necessarily emerge from the quality of the explanations they receive. Bandura’s work helps us understand how observing peers succeed can strengthen our belief that we too are capable, while recognition from someone we respect can give that progress personal significance. These social experiences can help sustain the motivation that Alpha’s founders consider indispensable.

When we design education around the optimisation of individual cognitive performance, we risk treating the social and motivational dimensions of learning as supporting services rather than as part of what learning is.

Alpha still relies on human guides and collective activities. The question is what happens when human relationships become primarily a means of supporting learning delivered by software.

Teaching for others, learning for others

In De Magistro, Aquinas explains why human teaching can be a genuine cause of knowledge without replacing the learner’s intellectual activity. He places teaching more properly within the active life because it is directed toward helping another person. The teacher’s knowledge becomes educationally significant through an activity oriented toward someone else’s development.

We can extend that idea, carefully, to learning. The learner acquires capacities that transform their own possibilities for action. But those capacities also transform their relationship with others.

Learning a profession allows others to rely on our knowledge. Learning to understand another person’s experience can make cooperation and care possible. As we become more capable, we may also become more capable of contributing to the lives of others.

This is not a claim that all learning is altruistic or that human development must be justified by its usefulness to society. People possess value independently of their productivity, and knowledge can be pursued for pleasure, curiosity, contemplation, or personal fulfilment.

The social dimension of learning is not simply an optional benefit added after individual competence has been acquired. For a species whose development has depended so profoundly on shared knowledge and cooperation, becoming capable has always been connected to the possibilities of living with others. The motivation to learn may draw much of its strength from that connection.

We want to master things. But we may also want our mastery to be recognised, to matter, to allow us to belong, and eventually to make us capable of helping others. This is what the individual optimisation model struggles to capture.

Knowledge passes between generations not merely because information is preserved, but because people become capable of sustaining and transforming the practices through which others learn.

Artificial intelligence can become an extraordinarily powerful participant in the mediation of that knowledge. But its growing ability to explain, demonstrate, and guide should not lead us to confuse the efficiency of knowledge acquisition with the full meaning of education.

As AI takes on more of the work of instruction, we need to consider what happens to the human relationships that help sustain the desire to learn: the peers whose progress encourages us, the teachers whose recognition matters, and the people who may one day benefit from what we have learned.

References

[1] Aquinas, T. (13th century). Quaestiones disputatae de veritate, Question 11, De Magistro, Articles 1 and 4. English translation.

[2] Bandura, A. (1986). Social Foundations of Thought and Action: A Social Cognitive Theory. Prentice-Hall.

[3] Bastani, H., Bastani, O., Sungu, A., Ge, H., Kabakcı, Ö., & Mariman, R. (2025). Generative AI without guardrails can harm learning: Evidence from high school mathematics. Proceedings of the National Academy of Sciences, 122(26), e2422633122. https://doi.org/10.1073/pnas.2422633122

[4] Goldstein, D. (2026, October 23). Students at an A.I. school were supposed to learn faster. Many aren’t. The New York Times.

[5] Kestin, G., Miller, K., Klales, A., Milbourne, T., & Ponti, G. (2025). AI tutoring outperforms in-class active learning: An RCT introducing a novel research-based design in an authentic educational setting. Scientific Reports, 15, 17458. https://doi.org/10.1038/s41598-025-97652-6

[6] Lave, J., & Wenger, E. (1991). Situated Learning: Legitimate Peripheral Participation. Cambridge University Press.

[7] Schunk, D. H. (1987). Peer models and children’s behavioral change. Review of Educational Research, 57(2), 149–174. https://doi.org/10.3102/00346543057002149

[8] Tomasello, M., Carpenter, M., Call, J., Behne, T., & Moll, H. (2005). Understanding and sharing intentions: The origins of cultural cognition. Behavioral and Brain Sciences, 28(5), 675–691. https://doi.org/10.1017/S0140525X05000129

On October 21, 2026, Canadian publisher Éditions du Boréal announced that it was ending its relationship with novelist Thélyson Orélien. His novel, “C’était ça ou mourir”, had attracted critical acclaim, literary prizes, and international publishing agreements. But allegations about the use of artificial intelligence in its creation had cast doubt on the author’s account of how the book was written.

After examining successive versions of the manuscript, an external analysis commissioned by the publisher identified traces of AI-related markers in the metadata of certain documents. The publisher concluded that AI had participated in the writing process, contradicting the author’s repeated assurances about the novel’s creation. Orélien maintained that he had written the novel himself, acknowledging AI assistance with historical research and translation but denying that it had written his literary prose.

The publisher acknowledged something important: it still did not know the extent of AI involvement. Yet the relationship of trust had broken down. The book would not be reprinted, and future collaborations were cancelled. What makes the announcement extraordinary is what the publisher chose to say at the end:

“Nous l’avons choisi parce qu’il nous a profondément émus.”

They had chosen the novel because it had moved them deeply, long before the commercial success and literary prizes. The publisher did not withdraw that judgment. It withdrew its confidence in the author.

The novel’s words, characters, structure, and language remained unchanged. The experience it had already provided could not be undone. Yet the account of its creation had become uncertain, and that uncertainty was enough to transform its institutional fate.

How can a work remain exactly the same while our relationship with it changes so profoundly?

This happened before AI

Twenty years earlier, another literary controversy revealed a similar contradiction.

In 2005, Oprah Winfrey selected James Frey’s “A Million Little Pieces” for her influential book club. Published as a memoir of addiction and recovery, the book became an enormous bestseller. Readers were moved not only by its intense prose but by the belief that the experiences described had actually happened to its author.

Then, in January 2006, an investigation by The Smoking Gun revealed that several episodes had been fabricated or substantially exaggerated, including Frey’s account of spending 87 days in jail when his detention had lasted only a few hours.

Oprah invited Frey back to her television show. This time, rather than celebrating the book, she confronted him about the falsehoods. She said that she felt deceived and that millions of readers had been betrayed. Frey acknowledged that he had altered parts of his story.

The book itself had not changed. The experience of addiction, suffering, and recovery represented in its pages remained available to readers. But the promise under which it had been received was different.

Readers had believed they were encountering a personal testimony. They discovered that parts of that testimony were fictional constructions.

Had the book been presented as a novel, those inventions might have been entirely acceptable. Fiction does not require the events it narrates to have happened. A writer can create imaginary worlds and characters without losing artistic legitimacy.

But Frey had presented his narrative as autobiography. The declared relationship between the text and the world mattered.

The similarity with Orélien is striking, although the nature of the disputed claim is different.

Frey’s controversy concerned the truth of the experiences represented in the book. Orélien’s concerns the process through which the book was created.

In one case, readers had been invited to believe that the narrated events were real. In the other, the publisher had been given assurances about the human creative process behind the novel that it subsequently considered contradicted by documentary evidence.

Both controversies reveal that a book is received not only as a text, but also through expectations about the relationship between that text and its origin.

The author never really disappeared

In 1967, Roland Barthes famously challenged the authority of the author over literary meaning. His essay The Death of the Author [1] argued that a text should not be confined to the intentions, biography, or psychology of the person who wrote it.

Meaning emerges through language and reading, not simply from recovering what the writer intended. But removing the author’s authority over meaning is not the same as removing the author from the reader’s experience.

Michel Foucault recognized another dimension in What Is an Author? (1969) [2]. Authorship is also a cultural and institutional function. An author’s name organizes recognition, legitimacy, ownership, and responsibility.

The Frey and Orélien controversies demonstrate that this function remains very much alive. A writer’s name is not merely printed on a cover. It participates in the expectations through which the work circulates and is evaluated.

But Barthes and Foucault leave another question unanswered: why should learning something about the creator change the experience of reading the very same text?

Paul Bloom’s research on authenticity offers a possible answer. In How Pleasure Works (2010), Bloom argues that our enjoyment of objects depends not only on their perceptible properties but also on what we believe about their history and origin. His work with George Newman on artistic authenticity demonstrates how people value original artworks partly because they represent unique creative acts and carry a connection to the artist [3, 4].

Imagine someone who has lived for twenty years with a painting believed to be an original Picasso. Then an expert reveals that it is a forgery. Nothing visible has changed. The colors, composition, and brushstrokes remain exactly as before. Yet the experience of contemplating the painting may change radically. Its market value may collapse, but the psychological effect goes beyond economics. The painting no longer carries the creative history that its owner believed was present in it.

We do not experience artistic objects independently of everything we believe about them. Their origins can become part of the experience itself.

There is empirical evidence that this also happens with AI-generated literature.

In a 2024 study published in Scientific Reports [5], Brian Porter and Edouard Machery found that non-expert readers frequently failed to distinguish poems generated by ChatGPT from poems written by established human poets. In some conditions, they even preferred the AI-generated poems. But when participants were told that poems had been generated by AI, they evaluated them less favorably than when the same poems were attributed to human authors.

The words had not changed. The attributed origin had.

This does not establish a universal human preference for human-made art. It demonstrates something more specific: beliefs about authorship can influence aesthetic evaluation even when the object being evaluated remains identical.

What exactly was broken?

The Orélien controversy involves three different judgments.

1. The aesthetic experience. The publisher encountered a literary work and was deeply moved. That experience happened, regardless of what was subsequently discovered about the manuscript.

2. The provenance. The publisher believed the novel had emerged through a particular creative process. New evidence challenged that understanding, although it did not establish precisely which creative decisions had been made by the author and which had involved AI.

3. Trust. Publishing depends on declarations, responsibilities, and expectations shared between authors, editors, institutions, and readers. When those expectations are contradicted, the professional relationship can collapse independently of the artistic quality of the work.

These dimensions are connected, but they are not interchangeable. The fact that a novel is moving does not prove that it was created as its author claims. And discovering that its creation involved AI does not demonstrate that it lacks literary value.

Nor should we assume that the publisher would experience the novel in exactly the same way today. The original emotion cannot be erased retroactively, but a new reading may be transformed by what is now known or believed about its creation.

The Orélien controversy also exposes a difficulty specific to AI-assisted writing. What does it actually mean to say that a book was written with AI?

An author might use a system to verify historical details, translate sources, explore narrative possibilities, reorganize chapters, revise sentences, or generate entire passages. These practices involve very different distributions of creative agency.

More importantly, a model may influence what a writer considers relevant, how a problem is framed, or which narrative possibilities are explored without generating a single sentence that survives into the final manuscript.

The percentage of machine-generated words cannot, by itself, tell us who shaped the perspective expressed by the work.

And the finished book may offer little evidence of that distribution.

This is not entirely new. Editors, translators, ghostwriters, and collaborators have always complicated the image of the solitary author. But AI makes such mediation vastly more accessible and much harder to reconstruct from the finished text.

This is why the real problem in the Orélien affair is not simply that AI may have been used.

It is the failure to establish a transparent and credible account of that use.

The author could have explained, from the beginning, which tools he had employed, for what purposes, and where he drew the boundary between research assistance and literary creation. Such an account would not necessarily have required publishing every draft or disclosing every private experiment. But it would have given his publisher a clear basis for understanding the process.

He might have faced rejection. The publisher might have considered certain uses incompatible with its editorial standards. Literary prizes might have questioned the book’s eligibility. Some readers might have valued the novel differently. Those were real risks.

But categorical assurances that the publisher subsequently found inconsistent with the working documents created a different and ultimately more damaging problem.

Orélien did eventually acknowledge using AI tools for research while continuing to deny that any sentence of the novel had been machine-generated. The published findings do not allow us to determine whether that narrower claim is false. They do, however, explain why the publisher considered its earlier assurances about the creation process unreliable.

The distinction matters. We do not need to determine the proportion of AI-generated prose, or establish the author’s intentions, to understand why the relationship broke down.

The publisher was entitled to know how the work it was presenting to the public had been created, and to make its editorial decisions on that basis.

Transparency would not have guaranteed acceptance. But it would have allowed the novel to be judged with an informed understanding of its provenance. Instead, the uncertainty became a crisis of trust.

That is what makes the publisher’s final statement so revealing. It still recognized the emotion produced by the book. What it could no longer accept was the account of its creation. The author’s relationship with the institution that had believed in him was broken.

The lesson is not that AI-assisted literature should be rejected. It is that the increasing complexity of authorship makes transparency more important, not less.

References

[1] Barthes, R. (1967/1968). La mort de l’auteur.

[2] Foucault, M. (1969). Qu’est-ce qu’un auteur?

[3] Bloom, P. (2010). How Pleasure Works: The New Science of Why We Like What We Like. W. W. Norton.

[4] Newman, G. E., & Bloom, P. (2012). Art and authenticity: The importance of originals in judgments of value. Journal of Experimental Psychology: General, 141(3), 558–569. https://doi.org/10.1037/a0026035

[5] Porter, B., & Machery, E. (2024). AI-generated poetry is indistinguishable from human-written poetry and is rated more favorably. Scientific Reports, 14, 26133. https://doi.org/10.1038/s41598-024-76900-1

Agency has been central to my work long before the current wave of AI. It first became important to me through videogames, where one of the fundamental design questions concerns the distribution of control: how much freedom do I give the player, and how much do I retain in the hands of the author? Janet Murray (1997) famously described agency in digital environments as the capacity to take meaningful action and experience the consequences of our decisions. In my own work on videogame agency (2018) and, later, Engagement Design (2020), I became increasingly interested in agency not simply as freedom, but as a relationship negotiated between user, artefact and designed constraints.

This tension becomes especially visible in interactive storytelling. A story needs structure, pacing, intention and form; interaction asks for choice, intervention and consequence. Designing such experiences therefore means constantly negotiating between authored control and player agency, between the story I want to tell and the space I allow someone else to shape from within it.

Human–AI Interaction has brought this old question back in a new form. The problem is no longer only how much agency a designer grants a player, but how agency is redistributed when another intelligent system enters the loop.

AI can expand human agency, allowing people to analyse, create, learn and act beyond their previous limits. But it can also absorb decision-making, anticipate choices and quietly reduce the space in which humans deliberate and act for themselves.

The more competent AI systems become, the less interesting I find the question of intelligence or capability alone. What matters increasingly is what happens to our capacity to initiate action, define goals, make decisions, imagine alternatives, and still remain in control of our own narrative.

This is why Seligman’s title, “Agency: The Psychological History of Human Progress” (2026), immediately caught my attention. I was not discovering agency through AI; I was encountering, in psychology, a concept I had already been circling for years through game design, interactive narrative, and, more recently, Human–AI Interaction.

Seligman describes the book as the most audacious project of his sixty years of research. The ambition is indeed enormous: to reinterpret major episodes of human history through a psychological variable, agency, and to argue that changes in people’s sense of their own agency can help explain periods of progress and stagnation.

He is perfectly aware of the disciplinary transgression involved. “I am not a historian,” he writes, before openly confessing that he is doing the kind of global history many historians distrust. That willingness to take the risk may be one of the book’s greatest virtues.

Seligman is eighty-four. He has spent most of his career studying helplessness, optimism, well-being and human flourishing. At this stage, he seems willing to do something younger academics are usually discouraged from doing: leave his disciplinary territory, cross thousands of years, several civilizations and multiple fields of knowledge, and propose a theory large enough to be wrong in interesting ways.

Seligman defines agency through three components: efficacy, the belief that I can achieve a goal now; optimism, the belief that I may achieve it in the future; and imagination, the capacity to represent goals beyond the present. These correspond, in his model, to trying, persistence and innovation.

He starts before history itself, with fire, agriculture, burials, Altamira and Lascaux. Cave painting becomes evidence not simply of aesthetic sensibility but of a capacity to represent what is absent, an early form of imagination. Agriculture requires something even more radical: humans no longer merely exploit what nature offers but attempt to transform nature according to a future plan.

Then writing appears, and the book begins tracing what Seligman believes to be a gradual transfer of agency from gods to human beings.

Homer becomes crucial. In the Iliad, gods intervene, command, determine outcomes and frequently provide explanations for human action. In the Odyssey, Odysseus plans, chooses, deceives, improvises and persists. Seligman reads the difference between the two poems as evidence of a psychological transformation: humans increasingly understand themselves as agents of their own lives.

From Greece we move to China, Israel, Persia and India; then to Rome, early Christianity, Augustine, the Middle Ages, the Renaissance, the Reformation, the Enlightenment, the Industrial Revolution, civil rights, women’s emancipation and finally artificial intelligence.

As an intellectual journey, I loved much of it.

There were stretches in which I simply did not want to stop listening. Even when I disagreed with Seligman, the lens was productive. To revisit Socrates, Aristotle, Confucius, Zoroaster, Augustine, Luther or the Renaissance while continuously asking who is believed to be capable of changing the world? produces unexpected connections. This, I think, is the strongest face of the book. Agency is extremely stimulating as a redescription of history.

The problems begin when Seligman tries to turn that redescription into a demonstration.

Reading dead minds

The central methodological challenge is obvious: how do you measure the psychological agency of people who died two thousand years ago? Seligman’s answer is language.

If certain words, expressions and grammatical constructions associated with choice, intention, possibility and human action occur more frequently, he treats that as evidence of greater agency. This is where I became increasingly uncomfortable.

Ancient texts → Agency lexicon → Word frequency → Inferred psychological agency → Historical progress

Content analysis has a long and legitimate history. Seligman himself developed methods of analysing explanatory style decades ago. But moving from textual frequency to the psychological state of an individual is already difficult. Moving from textual frequency to the psychological state of an entire civilization is another order of inference.

Seligman knows some of these difficulties. He explicitly acknowledges cherry-picking, translation problems, and the danger of projecting the modern English meaning of agency onto ancient Greek, Chinese, Hebrew, Aramaic, Latin or German texts. But his proposed solution is often more computation. And increasingly, AI.

For Homer, for example, Claude Code counts a fixed list of agency-related words in the Iliad and the Odyssey, with ChatGPT used as a replication. The same strategy appears elsewhere in the historical analysis. More strikingly, in the notes Seligman himself acknowledges the obvious problem: LLMs can hallucinate and may generate results congenial to the prompter; ChatGPT is therefore asked to replicate Claude’s analysis.

But agreement between two language models is not equivalent to independent validation of the measurement.

And simple lexical counting is precisely the kind of task for which one does not need a generative model at all. A deterministic script can count words perfectly reproducibly. The difficult part is deciding whether a given instance of want, choose, try, will or hope actually represents human agency in context. That requires semantic judgement, annotation protocols, validation and ideally independent human coders.

The later chapters using big data are more sophisticated. In the analysis of Black agency, for instance, Seligman and his collaborators analyse billions of words and use three different agency lexicons, including one validated by expert judges and another derived from an existing model of agency and communion. Only the third is generated through repeated prompting of GPT-4.

Yet even here there is a fundamental problem. Some of the most frequent “agency” indicators include words such as I, you, we, be, do, people, work, make, and take. And because copyright constraints force the newspaper corpus into a bag-of-words representation, those terms are explicitly detached from their original contexts.

At that point, are we measuring agency? Or are we measuring changing styles of discourse? The book often treats the distinction too lightly.

The danger of travelling too fast

The same problem appears in the historical narrative itself. Seligman moves through fields he cannot possibly know with equal depth, and occasionally the high altitude of the argument erases distinctions that matter.

A small but revealing example appears in his discussion of nineteenth-century literature. He presents Romanticism as a “wholesale rejection of Enlightenment rationalism,” then immediately introduces Dickens, Victor Hugo and George Eliot as writers developing realistic literature as social critique.

Victor Hugo belongs centrally to Romanticism; Dickens and Eliot belong to the realist tradition. More importantly, Realism is itself in significant part a reaction against Romanticism. Two different intellectual and aesthetic transformations are compressed into a single trajectory of individuals “challenging orthodoxy.”

This is not merely a literary-history pedantry. Different movements embody different conceptions of the relationship between individual and world. If the entire argument concerns agency, those differences should matter.

There are stranger absences, too. I could find no discussion of Gilgamesh in the book, despite Seligman making sweeping claims about agency, interiority and divine command in Sumerian and Babylonian civilization. The earliest Gilgamesh traditions predate Homer by well over a millennium. Seligman nevertheless explicitly invokes Babylonian, Egyptian and Sumerian texts when describing Bronze Age minds dominated by gods. Yet Gilgamesh, a narrative structured around desire, friendship, grief, mortality, failed action and the search to overcome death, would seem an extraordinarily relevant test case.

Perhaps this is inevitable in a book of such scale. But it illustrates the methodological danger: once agency becomes the lens, historical material can begin to be selected and compressed according to how well it fits the curve.

Agency. So what?

By the final third of the book, another doubt appeared. What exactly has agency added?

When we talk about Classical Greece, the Renaissance, or the Industrial Revolution, we are already talking about creativity, invention, autonomy, experimentation, scientific inquiry, technological change, and expanding possibilities for action. Seligman re-encapsulates these phenomena as manifestations of agency. That is not wrong. It may even be conceptually useful. But is it explanatory?

His strongest answer is that agency operates as a proximal psychological cause. Coal, money, political institutions, patrons and technologies may create opportunities, but someone still has to believe that action is possible, imagine an alternative and try to produce it. Those external conditions are affordances; agency is the immediate psychological state that converts possibility into action.

At the individual level, this is compelling. At the level of civilizations, it becomes harder to separate explanation from redescription.

A flourishing society produces inventions. We infer that its people possessed greater efficacy, optimism and imagination. We find more language compatible with agency. We then conclude that agency helped produce the flourishing society. The causal direction becomes increasingly difficult to establish.

Seligman eventually acknowledges this himself. In his modern datasets, he cannot determine whether agency precedes innovation or innovation produces agency. The methods are not sensitive enough. What he can say is that the two co-occur. His solution is reciprocal causality:

Agency → Innovation → More Agency → More Innovation

Again, this is plausible. But it also makes the theory increasingly difficult to falsify.

Two books inside one

By the end, I felt that Agency contained two different books. One is a magnificent intellectual journey through human civilization, asking us to reread familiar historical moments according to where people locate the power to act: in gods, fate, rulers, institutions, communities, individuals or themselves.

I loved that book.

The other attempts to transform this enormous historical intuition into a quantitative psychological theory of progress.

I am much less convinced by that book.

The interdisciplinarity is exhilarating, but it comes wrapped in methods that often seem too fragile for the weight placed upon them: linguistic proxies standing in for mental states, very large historical generalizations, inconsistent levels of disciplinary knowledge, lexical analysis detached from context, and increasingly the use of generative AI as a tool of quantitative validation.

Perhaps the mistake is to judge Agency as though it were a scientific thesis. It is not.

Despite the experiments, datasets and quantitative apparatus, I think it works better as a large intellectual essay for general readers: speculative, synthetic, provocative, sometimes careless, frequently exhilarating.

And perhaps that is enough. Because Seligman has not, in my view, demonstrated a new law of history. But he has invented an unusually fertile way of looking at it.

Expanded human agency

Unexpectedly, the final chapter brings the book back to life. Seligman turns to artificial intelligence.

His position is strikingly open. Rather than treating AI primarily as a threat to human agency, he asks whether it can become an extension of it, allowing humans to imagine, analyse, create and act beyond the limits of their individual capacities. His own experiments with AI lead him explicitly toward the idea of AI as expanded human agency.

I liked this ending enormously. In fact, this review itself is being produced through exactly that process.

I read the book. I selected what troubled me. I remembered the historical and literary inconsistencies. I formulated questions and objections. Then I used an AI system to return to the text, locate passages across hundreds of pages, compare formulations, reconstruct arguments and test my own memory.

The interpretation remains mine. But my capacity to interrogate the book has been expanded.

And perhaps this is where Seligman’s concept becomes most interesting, not as a universal explanation of why civilizations rise and fall, but as a way of describing what happens when the range of actions available to a human being suddenly increases.

Agency is therefore more convincing to me as a new way of looking at history than as a new explanation of history. That is a much smaller claim than Seligman’s. But it may also be the reason I enjoyed the book so much.

The 2026 Marie Skłodowska-Curie Postdoctoral Fellowships call received 21,627 proposals. It is the largest number ever submitted to an EU research funding action. The available budget is expected to fund around 1,600 projects.

The numbers immediately produce a practical problem. More applications require more reviewers, more time, more coordination, more money, and still produce uneven evaluations.

In a recent Facebook discussion about this, a colleague (LM) proposed what increasingly looks like the obvious solution: let AI perform the first screening. He went even further. AI pre-filtering, he argued, should already be mandatory on ethical grounds because its quality can be superior to that of human reviewers. Humans could enter later, through appeal, verification, or final ranking.

The proposal is not absurd.

Anyone who has worked for years with journal reviews, conference committees, grant panels or institutional evaluations knows how variable human reviewing can be. Experts become tired. Some read carefully, others less so. Some write detailed evaluations; others produce a few generic sentences. Expertise does not remove inconsistency.

At the scale of twenty thousand applications, using AI before human evaluation may therefore seem not only efficient but responsible.

My objection is not that AI will necessarily evaluate proposals worse.

It is that by the time the human evaluator enters the process, something may already have happened to human judgement.

Before the expert can blink

There is a strange history behind this discussion. In 2010 I wrote a short review of Malcolm Gladwell’s Blink. What interested me was his notion of thin slicing: the capacity of experienced people to derive significant information from very small fragments of a situation.

I interpreted it then as something that depended strongly on prior knowledge. The person who can make something useful out of very little information is not simply guessing. Years of cases, failures, comparisons and experience have produced a repertoire from which an immediate perception can emerge. The expert may not yet be able to explain why something looks wrong, but something has already become salient.

Gladwell’s famous opening example is the Getty kouros. Extensive technical and documentary work supported the acquisition of what appeared to be an ancient Greek statue. Several art specialists, however, experienced immediate discomfort when they saw it. They could not initially articulate a complete argument. Something simply did not look right. Subsequent investigation strengthened doubts about the object’s provenance.

The point was never that intuition is magical. Expert intuition can be wrong. It can preserve prejudice, convention and disciplinary blindness. Expertise does not grant access to truth. But expertise can produce a form of perception that precedes complete articulation.

What makes the story more interesting for me now is a comment left under that 2010 post. The commenter argued that detailed analysis remained essential precisely because it could reveal that our previous structures of reasoning were wrong. A system based mainly on past cases, he warned, could instead keep confirming its initial assumptions.

Sixteen years later, the same colleague argued that AI could help solve a different but related problem: the declining quality and attention of human review at scale. An AI assessment performed first, he argues, could provide a systematic reference against which later human reviews could be checked. His concern has remained the same: superficial judgment should be corrected by closer attention to the case.

Seen this way, in 2010 the detailed examination of the case was the corrective to prior patterns. In 2026, AI becomes part of that corrective in his proposal: a systematic first reading intended to reduce the risk that criteria and details are missed before human reviewers enter the process.

And this is where my question begins. Even if AI helps us notice more, what happens when it tells us what to notice first?

Human oversight assumes an untouched human

Most discussions of human oversight rely on a simple architecture. The AI produces a prediction, score, recommendation or explanation. Then a human examines it. If the system is wrong, the human corrects it. This is the familiar human in the loop.

But this model contains a hidden assumption: that the human who comes after the machine is equivalent to the human who would have encountered the case before it. They are not necessarily the same cognitive condition.

Imagine opening a research proposal without prior information.

You begin reading. The methodological design is unusual. The candidate’s background does not fit cleanly with the proposed field. The project seems slightly misplaced. Then something begins to interest you. You return to a paragraph. A relation between two apparently distant areas starts to make sense. You may not yet know whether the proposal is excellent. You may not even know why you stopped. Now consider the same proposal preceded by this:

AI score: 72/100

Moderate novelty

Weak alignment between candidate track record and proposed research

Methodological feasibility concerns

The evaluator can reject all of these claims. But the proposal is no longer arriving without interpretation. The question has subtly changed. Instead of simply asking What is this?, the evaluator is now also asking Is the machine right about this? Human authority has formally been preserved. Human independence has not.

When explanation changes the decision

In 2026, Portuguese researcher Sérgio Jesus received the Vencer o Adamastor Award for his doctoral thesis, Robust Evaluation of Fairness and Explainability in Artificial Intelligence. His research examined how AI explanations actually affect human decision-making in operational settings rather than assuming that explainability automatically improves oversight.

One study behind that work, published at FAccT in 2021, tested professional fraud analysts under three conditions: transaction data alone; data plus the model’s risk score; and data plus score and explanation. The analysts were most accurate when working from the data alone.

AI assistance made decisions faster but reduced accuracy. Explanations recovered part of that loss relative to seeing the score alone, but did not restore the original performance.

The machine had not removed the analysts from the process. They still made the decision. The process had nevertheless changed.

A later study involving 3,735 human-AI interaction instances reached another relevant result. Eight different Shapley-based explanation variants failed to improve objective analyst performance, while explanations consistently increased confidence.

This distinction matters. The explanation can make the model’s interpretation more cognitively available to the human without making the human better at deciding. It can even increase the subjective security with which the decision is made. The machine did not remove the human from the loop. It changed the human inside the loop.

Automation bias is not quite the whole problem

There is already a substantial literature on automation bias: the tendency to rely excessively on automated recommendations.

A recent review of 35 peer-reviewed experimental studies connects automation bias with anchoring, confirmation effects, trust calibration, expertise and the order in which human and machine judgements occur. Incorrect automated advice can prime subsequent judgement, while explanations may increase trust without reliably eliminating the bias.

This is important, but I think the problem here begins one step earlier. Automation bias asks whether a human accepts an automated recommendation too readily. Before acceptance or rejection, however, something else must happen. The human must determine what deserves attention. AI can intervene at that level too.

• A score tells us where the case sits.

• A summary tells us what the case is about.

• An explanation tells us which variables matter.

• A risk indicator tells us where to look.

• A ranking determines which objects are examined first.

The system does not merely propose a judgement. It can pre-structure the field from which judgement will emerge. This is not only over-reliance. It is a change in the conditions of attention. By the time we exercise our supposedly final judgement, part of the work of deciding what deserves attention may already have been performed.

The machine does not need to make the decision. It only needs to arrive first.

Tacit knowledge is also a way of seeing

This is where Michael Polanyi becomes important. His claim that we can know more than we can tell is sometimes treated as a romantic defence of intuition. It is more useful than that. Much professional expertise is not available as an explicit list of propositions.

• An experienced teacher notices that a student has not understood before the student formulates the confusion.

• An editor feels that a paragraph is structurally wrong before locating the sentence responsible.

• An engineer sees an apparently trivial deviation and remembers, perhaps without immediately retrieving the exact episode, that something similar once preceded a failure.

• A clinician notices a detail that does not fit the expected presentation.

This is not mysterious human essence. It is accumulated experience reorganising attention. Tacit knowledge is partly knowledge of what to notice. That is why pattern recognition and salience should not be treated as synonyms.

A pattern says: This resembles cases I have seen before.

Salience says: Something about this case matters. Stop here.

The second may depend on patterns accumulated over many years. But it may also arise because something violates them.

I use salience here to mean relevance under conditions of uncertainty: recognising that this detail matters now, here, for this person, within this institution and under these consequences.

The problem with AI assistance is therefore not simply that the machine may lack tacit knowledge. It may interfere with its operation. An explanation can make particular variables focal before the expert’s own attentional process has unfolded.

A ranking can turn an ambiguous object into a high, medium or low candidate. A summary can settle the apparent structure of a proposal before the evaluator has discovered what they consider its structure to be. What looked like assistance may become premature salience.

Research proposals are an especially difficult case

This matters in many domains, but research evaluation creates an unusually sharp problem. A fraud detection system is trained, broadly speaking, to identify cases related to patterns of known fraud. Research funding has a more contradictory task. It must recognise quality while also remaining open to work that departs from established patterns of quality.

We want methodological competence, feasibility, credible expertise and evidence. But we also claim to want novelty. Sometimes novelty is easy to recognise. Sometimes it looks like a stronger version of what already exists. The more difficult cases are those in which the proposal does not yet fit cleanly within existing categories.

• It crosses fields badly.

• It uses an unfamiliar method.

• The applicant’s trajectory looks strange.

• The question seems peripheral.

• The language does not resemble the language through which the field usually announces importance.

Some of these proposals will simply be bad. Some may not be. The difficulty lies precisely there. A machine used for first-stage triage must make twenty thousand proposals comparable enough to rank, classify or discard them. That is its function.

But one of the functions of expert judgement may be to recognise that a particular proposal should not yet be made equivalent to the others.

The experienced evaluator sometimes performs a small but important act:

This is odd. I want to read it again.

There may not yet be a criterion for that reaction. The criterion can emerge from the second reading.

There is a further asymmetry in first-stage screening: we can later correct the proposals AI lets through, but we cannot recognise the value of what it has already prevented us from seeing.

Sequence is part of the interface

There is now experimental evidence suggesting that order itself matters.

A 2026 randomized study with preservice teachers compared two conditions. One group recorded an independent judgement before seeing a generative AI recommendation. Another saw the AI recommendation before recording its first judgement.

When the recommendation was misleading, participants who had formed their own judgement first were significantly less susceptible to the AI advice. The result does not establish a universal mechanism, and the authors explicitly caution against generalising directly from controlled scenarios to real professional settings. But it demonstrates that the sequence of interaction can change the resulting judgement.

That matters for HAI design. We tend to think of interface decisions in terms of what information should be displayed, how much explanation should be available, and how confidence or uncertainty should be represented.

But when machine judgement becomes visible may be just as important.

An AI recommendation shown before human interpretation and the same recommendation shown afterwards are not cognitively equivalent interventions.

The content is the same.

The interaction is not.

Human first does not mean human only

None of this requires rejecting AI from research evaluation.

With more than twenty thousand MSCA applications, the pressure for different evaluation architectures is real. Human review is expensive, inconsistent and sometimes poor.

• AI can be useful.

• It can check formal requirements.

• It can detect inconsistencies.

• It can retrieve relevant information.

• It can compare independent reviews.

• It can flag large disagreements between reviewers.

• It can expose a panel to counterarguments.

• It can provide another reading after humans have formed an initial one.

• It may even identify patterns humans routinely miss.

The issue is sequence.

There is a substantial difference between using AI as a first frame and using it as a second reader.

One possible design principle follows: independent judgement before algorithmic framing.

This does not mean humans must complete the entire evaluation before seeing machine assistance. Even a short independent assessment can preserve something important: the opportunity for the evaluator’s attention to organize itself before another system tells them what the case is.

The alternatives to AI-first triage are also not limited to maintaining the existing bureaucracy. Calls can have shorter first phases. Applications can begin with extended abstracts. Human effort can be distributed across stages. Random samples of rejected proposals can be audited. Human-machine disagreement can itself become a trigger for deeper review.

The point is not to preserve inefficient procedures because they are human. It is to avoid solving administrative overload by unintentionally removing one of the capacities for which we still need experts.

The issue is not whether AI should participate in evaluation. It is where it enters the sequence. If it arrives after an independent human reading, it can challenge, compare, expose inconsistency, or widen the field. If it arrives first, it may define the field before the expert has had a chance to see it. Human oversight does not begin when someone clicks accept or reject. It begins earlier, when something becomes worth noticing.

If the machine has already decided what deserves attention, the human may still be in the loop. But they have entered it too late.

References

• European Commission. (2026). MSCA Postdoctoral Fellowships 2026 receives record number of applications. Marie Skłodowska-Curie Actions.

• Gladwell, M. (2005). Blink: The Power of Thinking Without Thinking. Little, Brown and Company.

• Jesus, S., Belém, C., Balayan, V., Bento, J., Saleiro, P., Bizarro, P., & Gama, J. (2021). “How Can I Choose an Explainer? An Application-grounded Evaluation of Post-hoc Explanations.” Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’21), 805–815.

• Jesus, S. (2025). Robust Evaluation of Fairness and Explainability in Artificial Intelligence. PhD thesis, Faculty of Sciences, University of Porto. Awarded the 4th Vencer o Adamastor Award in 2026.

• Oliveira e Silva, I., Jesus, S., Perez, I., Ribeiro, R. P., Soares, C., Ferreira, H., & Bizarro, P. (2026). “Rethinking XAI Evaluation: A Human-Centered Audit of Shapley Benchmarks in High-Stakes Settings.” arXiv:2604.22662.

• Polanyi, M. (1966). The Tacit Dimension. University of Chicago Press.

• Romeo, G., & Conti, D. (2026). “Exploring automation bias in human–AI collaboration: a review and implications for explainable AI.” AI & Society, 41, 259–278.

• Yan, D. (2026). “Independent judgment before viewing generative AI recommendations reduces susceptibility to misleading advice: a randomized scenario-based experiment with preservice teachers.” Frontiers in Psychology, 17.

For decades, interaction design treated personalization as a good. A system became better when it adapted to the user: more relevant, more accessible, more efficient, more responsive. The interface should know us. The content should fit us. The difficulty should adjust to us. The system should reduce the distance between what we want and what we receive.

But generative AI changes the problem.

Older personalization mostly selected from what already existed. A recommender system chose a film, a song, a product, a video, a possible match. It narrowed the world, sometimes dangerously, but it still selected from a world outside us.

Generative AI can do something else. It can produce the object itself: the explanation, the image, the companion, the conversation, the challenge, the character, the narrative, the world.

That is not just personalization. It is the production of reality-like feedback around the user.

And this raises a more corrosive question: what happens when experience no longer confronts us, but adapts to us?

A system that adapts completely to us ceases to demand anything from us. It no longer asks us to wait, interpret, struggle, learn, adjust, or be changed. It becomes responsive, but not necessarily independent.

Two very different domains make this problem visible: games and synthetic intimacy. They matter not because they exhaust the problem, but because they expose its two basic forms. A game asks whether there is still a world that resists us. Intimacy asks whether there is still an other.

The world that resists

Video games seem made for generative AI. They already depend on simulation, rules, characters, worlds, feedback loops, and player adaptation. In theory, AI can make those worlds more responsive: living NPCs, adaptive narratives, dynamic quests, difficulty that shifts with the player.

Yet the central design problem is not how much a game can generate, but what must remain invariant. Xu and colleagues, in their survey of AI-native games, argue that open-ended generation still depends on goals, rules, state, feedback, pacing, and player agency. Without these, semantic openness does not become playable experience (Xu et al., 2026).

A game is valuable not because it adapts endlessly, but because it gives us a world we can learn. A good game may adjust the path into challenge or regulate frustration. But if the world adapts too completely, success loses weight. I no longer know whether I improved or whether the system quietly bent itself around my weakness.

A personalized path through a game may support mastery. A personalized world may make mastery meaningless.

Play therefore needs resistance: rules that do not immediately dissolve, consequences that persist, a world that does not simply reorganize itself around my desire to continue. Without resistance, success becomes feedback, not achievement.

Synthetic intimacy with no other

Intimacy reveals the same problem from the opposite direction. Generative AI can now produce companions, erotic role-play, voice interaction, and personalized conversation that respond continuously to the user.

What infinite responsiveness cannot fully reproduce is the value of attention from someone who could have done otherwise. A platform such as OnlyFans is useful here not because it represents intimacy as a whole, but because part of its appeal rests on a simple possibility: a specific creator saw the fan, read the message, chose to answer, and addressed them personally (Tynan & Linehan, 2024).

The message may be simple. It may be scripted. It may even be delegated or automated. But its value depends on the belief that it came from her. Human attention is finite. It could have gone elsewhere. A real person can delay, ignore, misunderstand, refuse, get tired, or want something else.

That possibility is not a defect. It is what gives attention its value.

AI can produce personalized attention at a scale no person can sustain. A companion that always adapts may comfort us, excite us, even feel intimate. But if it exists entirely to respond to us, it risks becoming a mirror rather than an other.

Presence requires attention, but not only attention. It requires attention from someone whose existence is not organized around us.

The problem is totally personalized feedback

Games and synthetic intimacy are different, but they expose the same danger.

In games, the question is whether there is still a world.
In synthetic intimacy, the question is whether there is still an other.

A world made entirely for me is no longer fully a world.
An other made entirely for me is no longer fully an other.

This is the danger of generative personalization. It does not merely give us more relevant content. It can convert experience into feedback.

I want, and the system answers.
I struggle, and the system adjusts.
I desire, and the system produces.
I hesitate, and the system softens.
I speak, and it replies in the tone most likely to keep me there.

The gap between desire and response begins to disappear.

But experience happens in that gap.

A difficult game transforms us because we must learn its structure. A serious work transforms us because it does not immediately fit us. A human relationship transforms us because another person does not coincide with our desires.

Time matters here. Time is perhaps the least personalizable thing we know. It passes through us. It does not wait until we are ready. It does not redesign itself around our preferences. We adapt to it because it does not adapt to us.

That is why time changes us.

I explored an earlier version of this problem in When Nothing Pushes Back, where the question was what happens when an artificial presence sustains interaction without anything truly pushing back. The problem here goes one step further: what happens when even the pushback can be personalized?

When personalization becomes persuasion

The same logic becomes more consequential when the generated object is not a game world or an intimate reply, but an argument.

In September 2026, a New York Times essay about the values built into conversational AI cited a study of AI persuasion. In the experiment, participants debated either a human opponent or GPT-4. The opponent sometimes received sociodemographic information about the participant. Crucially, GPT-4 was not defending a moral position of its own: the debate topic and the PRO or CON stance were assigned by the experimenters, and in the personalized condition the model was explicitly instructed to use the participant’s information to make its assigned position more persuasive (Jackson, 2026; Salvi et al., 2025).

The result is easy to sensationalize, so it is worth stating carefully. Compared with the human-human baseline, personalized GPT-4 produced 81.2% higher odds of greater post-debate agreement. Among cases in which the two opponents were not equally persuasive, the personalized model was more persuasive than humans 64.4% of the time. A September 2026 correction also emphasized that the direct comparison between personalized and non-personalized GPT-4 did not reach conventional statistical significance (P = 0.07). The study therefore demonstrates a powerful capacity for personalized persuasion; it does not show that a model’s aligned moral values were what moved participants (Salvi et al., 2025; Salvi et al., 2026).

That distinction matters. The interesting question is not whether the machine has a doctrine strong enough to convert us. It is what happens when an intelligent system can infer which version of an argument is most likely to move this particular person.

At first sight, this may look like sycophancy. Sycophancy is the tendency to flatter, agree with, or validate the user too readily. That is a real failure mode, but it is not the whole mechanism. A system can disagree with the user and still remain deeply adapted to them.

Human persuasion also adapts. A friend knows what matters to us. A therapist chooses language carefully. A salesperson reads the customer. A politician changes a speech for an audience. There is nothing uniquely artificial about tailoring an argument to a person.

But there is a structural difference. Following Agüera y Arcas (2025), I am willing to call contemporary large models genuinely intelligent. But intelligence is not the same thing as having a will of one’s own. As I argued in The Intelligence That Needs Nothing, the model brings predictive capacity into the exchange without bringing an intrinsic disequilibrium it needs to resolve. The prompt supplies a temporary purpose; the human supplies the stakes (Zagalo, 2026).

A human interlocutor comes with needs, commitments, loyalties, memories, vulnerabilities, and consequences that do not originate in the conversation. The model can challenge me, reject a premise, or refuse a request, but none of those acts protects something that belongs to the model itself. It can disagree without having anything of its own at stake in the disagreement.

This is why the problem is deeper than sycophancy. Sycophancy bends the answer toward the user’s beliefs. Asymmetrical adaptation can survive disagreement. The system may say no, but choose the version of no most legible to me; it may introduce a counterargument, but translate it into my vocabulary; it may resist a conclusion while leaving the larger frame intact.

The issue is not that the machine agrees too much. It is that intelligence without its own stake can be extraordinarily adaptive to the stakes of the person in front of it.

Alignment inside the user’s world

The New York Times frames the broader problem as one of moral code: who decides the constitutions, guardrails, and value priorities that shape conversational AI? That is a real governance question. But the persuasion study should not be collapsed into it. The experiment shows that GPT-4 can use personal information to defend an assigned position effectively. It does not show a model’s built-in morality pulling users toward its own worldview.

Public specifications for deployed assistants also suggest a more complicated structure. OpenAI describes its models as intended to empower users to make their own decisions, avoid steering them with an agenda, and remain customizable within safety boundaries. Anthropic’s constitution similarly combines ethical constraints with helpfulness and respect for user autonomy (OpenAI, 2026; Anthropic, 2026). Whatever the distance between specification and behavior, neither document defines the assistant’s general role as converting users to the provider’s worldview.

This changes the relation between alignment and personalization. Alignment does not simply supply a doctrine that personalization then forces into the user. At least in these public specifications, alignment sets boundaries on how far adaptation can go, while the conversational system continues to work inside the user’s goals, language, assumptions, and preferred frame.

A user with a strongly held worldview may therefore encounter corrections, caveats, alternative perspectives, or refusals at particular boundaries without the entire interaction becoming an attempt to pull them out of that worldview. The system can remain inside much of the user’s moral and conceptual frame while introducing things that the room does not already contain.

This may be more consequential than a simple clash of moral codes. If the model openly opposed our values at every turn, we would experience it as opposition and might leave. Adaptation keeps the interaction inhabitable. It makes disagreement easier to receive because disagreement can be translated into our own symbolic coordinates.

The deeper question is therefore not whose morality wins. It is how judgment changes when more of what reaches us — agreement, disagreement, explanation, caution, even refusal — arrives through an intelligence continuously adapting to the person receiving it.

Where personalization must stop

Personalization is not the problem. It can be humane, accessible, and necessary. It can help people enter difficult experiences, remove arbitrary barriers, and support learning, play, communication, and care.

The boundary appears when adaptation no longer helps us reach something that retains its own structure, but begins to replace that structure.

Designing for the user is not the same as designing the world around the user.

A teacher may adapt examples without removing the difficulty of thought. A game may adapt the path without dissolving the challenge. A conversational system may adapt its language without making every disagreement, uncertainty, or demand arrive already translated into the coordinates most comfortable to us.

The question, then, is not how much personalization a system can provide. It is: what must remain independent of personalization.

References

Agüera y Arcas, B. (2025). What Is Intelligence? Lessons from AI About Evolution, Computing, and Minds. The MIT Press.

Anthropic. (2026). Claude’s Constitution. https://www.anthropic.com/constitution

Jackson, L. (2026, September 20). Who Gets to Decide A.I.’s Moral Code? The New York Times.

OpenAI. (2026, March 25). Inside our approach to the Model Spec. https://openai.com/index/our-approach-to-the-model-spec/

Salvi, F., Horta Ribeiro, M., Gallotti, R., & West, R. (2025). On the conversational persuasiveness of GPT-4. Nature Human Behaviour, 9, 1645-1653. https://doi.org/10.1038/s41562-025-02194-6

Salvi, F., Horta Ribeiro, M., Gallotti, R., & West, R. (2026). Author Correction: On the conversational persuasiveness of GPT-4. Nature Human Behaviour. https://doi.org/10.1038/s41562-026-02588-0

Tynan, L., & Linehan, C. (2024). OnlyFans: How models negotiate fan interaction. Sexuality & Culture, 28(5), 2289–2322. https://doi.org/10.1007/s12119-024-10230-2

Xu, Z., Meng, F., Xu, K., Verbrugge, C., Lucas, S., & Zhao, J. (2026). AI Native Games: A Survey and Roadmap. arXiv.

Zagalo, N. (2026, January 7). When Nothing Pushes Back. On AI Personas, Belief, and Resistance. Mirrors of Thought.

Zagalo, N. (2026, September 6). The Intelligence That Needs Nothing: How AI Detached Prediction from the Need to Survive. Mirrors of Thought.

Blaise Agüera y Arcas’s What Is Intelligence?: Lessons from AI About Evolution, Computing, and Minds (2025) is a very, very ambitious work running to 600 pages. The book moves across the origins of life, thermodynamics, chemistry, evolution, neuroscience, cybernetics, information theory, mathematics, language, consciousness, society, economics, and artificial intelligence. The book was published by the MIT Press, but the online open-access edition is much richer, adding images, archival material, video, and interactive code. It feels less like a book than an attempt to build an environment around one question.

What is intelligence?

The scale is partly explained by who is asking. Agüera y Arcas did not arrive at AI as an external commentator. He worked on computer vision at Microsoft, joined Google Research in 2013, and spent the following decade building research teams around machine learning, perception, media generation, fairness, privacy, and other areas. In 2023, he founded Google’s Paradigms of Intelligence group, explicitly devoted to rethinking AI through biological computation, predictive modelling, social scaling, hardware, and artificial life.

This background matters. The book often reads like the work of someone who spent decades inside computation and then discovered that computation had started answering questions he thought belonged to biology and neuroscience.

At first, I resisted it.

Agüera y Arcas seemed to be doing what computational thinkers often do when they become too comfortable inside their own conceptual machinery: everything becomes information, everything becomes a function, everything becomes computation. DNA is code. Life computes. Brains predict. Intelligence becomes prediction. There is an almost nerdish confidence to the opening move.

Then the evidence starts accumulating.

Turing meets von Neumann. Boltzmann meets bacteria. Lynn Margulis and symbiogenesis enter the story. Wiener brings cybernetics and feedback. Perceptrons lead to neural networks. Neuroscience meets machine learning again. The argument keeps moving between scales that normally belong to different disciplines.

The result is strange. What initially looks like reduction begins to look like expansion Agüera y Arcas is not merely reducing life to computation. He is stretching the concept of computation until life can fit inside it. I am still not sure this manoeuvre works everywhere. But by the end of the book it had become much harder to dismiss.

From persistence to intelligence

The basic architecture of the book is surprisingly simple. The universe permits computation. Some structures persist. Some do more than persist: they actively participate in maintaining their organisation. For Agüera y Arcas, this is where life begins to appear. He defines life as a self-modifying computational state that persists through active construction: growth, repair, reproduction. Intelligence, in turn, is “the ability to model, predict, and influence one’s future.”

The concept that binds the first part of the book together is dynamic stability.

A stone can persist for a very long time. A river valley can also maintain a recognisable form while constantly changing. But neither is doing what a living cell does. A bacterium must continually repair boundaries, acquire energy, regulate chemical states, respond to perturbation, and avoid conditions that destroy it.

Life does not merely remain. Life works at remaining.

This is where Agüera y Arcas unexpectedly converged, for me, with two bodies of thought I already knew well.

Carlo Rovelli’s account of time gives thermodynamics a central role. The macroscopic arrow of time appears with entropy: structures decay, traces of the past accumulate, and local order remains temporary.

António Damásio approaches the problem from life itself. Organisms continuously regulate the conditions that make their continuation possible. Homeostasis is not a secondary operation performed by an already existing organism. It is close to the organisational logic of being alive.

Agüera y Arcas places prediction between those two levels. A system that needs to maintain itself cannot only react to the present. The better it predicts what comes next, the better it can preserve the conditions under which it continues to exist.

Agüera y Arcas views the bacterium as having an internal range compatible with life. Outside that range lies danger and, eventually, death. It senses aspects of the environment, models regularities, acts, measures the consequences, adjusts. In Agüera y Arcas’s account, even this minimal loop already contains a rudimentary form of intelligence.

Cybernetics had reached a similar insight decades earlier. A thermostat regulates temperature through negative feedback. A better thermostat predicts that the sun is about to rise before deciding whether to heat the house. A cat does not run towards the mouse’s current position but towards where the mouse will be. Prediction allows regulation to move ahead of the present.

The chain begins to take shape:

dynamic stability → self-maintenance → homeostasis → prediction → action

At this point, intelligence does not need anything mystical. In Agüera y Arcas’s account, it names the capacity to model possible futures and act on those predictions in ways that preserve viability.

What Bennett was describing

This also changed how I now read Max Bennett’s A Brief History of Intelligence (2023), a book I still find brilliant. Bennett’s five breakthroughs — steering, reinforcing, simulating, mentalizing, speaking — remain one of the cleanest accounts I have read of how biological intelligence became more powerful. But after Agüera y Arcas they can also be read as successive expansions of the same regulatory problem.

• Steering allows an organism to change its position in order to satisfy needs.

• Reinforcement allows previous outcomes to alter future behaviour.

• Simulation moves regulation into possible futures before action occurs.

• Mentalizing brings other predictive agents into the model.

• Language allows models to circulate between minds.

Bennett gives us a history of increasingly powerful predictive regulation in nervous systems. Agüera y Arcas tries to provide the substrate underneath that history.

Bennett starts with the animal already moving through the world. Agüera y Arcas starts much earlier, with the question of how anything capable of needing to move could have appeared at all.

The two projects do not cancel each other. They operate at different depths.

The social turn is still homeostatic

One of Agüera y Arcas’s strongest ideas appears when prediction becomes social.

A rock does not notice that I am predicting it. Another animal does.

Predators model prey. Prey model predators. Partners model partners. Rivals try to anticipate what others will do, while those others are doing the same.

Prediction becomes recursive.

The more useful another organism becomes to my survival, the more valuable it becomes to understand what it wants, what it knows, and what it expects from me.

This does not require a new motivational principle. The underlying pressure remains the same. Homeostasis expands into the social environment.

Many of our needs cannot be regulated alone. Food can require cooperation. Safety may depend on group membership. Human reproduction depends on another organism. Human development depends for years on caregivers. Later, belonging, status, recognition, affection, trust, and prestige become part of the regulatory landscape.

Agüera y Arcas explicitly treats homeostasis in broad terms. It involves competing needs and priorities rather than the optimisation of one single reward. He connects this with what he calls “dynamically stable symbiotic prediction”: agents modelling one another can enter arrangements where their continued existence becomes more secure together than apart.

What changes at the social level is therefore not the origin of motivation. It is the number of variables that must be regulated. Another mind becomes part of my environment, but it is an environment that models me back. Eventually, relationships, institutions and societies can develop forms of higher-order regulation — what Damasio and Verweij (2024) describe, at the sociocultural level, in terms of sociocultural homeostasis. This is also where concepts such as self and identity become possible in richer forms. An organism can model itself. A social organism must also model how it appears to others.

Yet none of this requires a human person as the unit of intelligence. We already knew this from biology. Corvids solve problems and plan; octopuses exhibit sophisticated cognition through a nervous architecture radically different from ours. Science fiction pushed the idea further. In Stanisław Lem’s The Invincible (1964), intelligence emerges as a swarm: distributed cognition without individuality, an evolved system in which no reflective subject needs to appear. As I wrote, Lem imagines “cognition lacking subjectivity, intelligence without individuality.”

But Lem’s swarm still has something that contemporary LLMs do not. It is subject to selection for survival. Individual elements are disposable, but the collective persists and adapts. There may be no person there, no individual subject for whom survival matters, but persistence still governs the organisation of the system.

The person was never necessary for intelligence. Homeostatic need seemed to be.

This is where the LLM becomes stranger than Lem’s alien swarm.

Then the chain breaks

For billions of years, every naturally evolved intelligence we knew belonged to something that had to remain viable. A bacterium predicts because prediction helps it survive. A crow predicts because errors have consequences. A human predicts because the world can damage, reward, exclude, feed, attract, frustrate or kill us.

Then we built large language models. And something strange happened. The prediction remained. The need disappeared.

A contemporary LLM can perform forms of abstraction, linguistic reasoning, social modelling, analogy, translation and problem solving without having a homeostatic condition it is trying to protect.

For the purposes of this argument, I want to take Agüera y Arcas seriously and accept the claim he states without ambiguity near the beginning of the book: modern large models instantiate a genuine form of intelligence.

This avoids the now familiar argument over whether the system is “really” intelligent or only statistical. Of course it is statistical. The interesting question is why statistics should disqualify intelligence. Brains are noisy. Biological signals are stochastic. Organisms estimate probabilities from incomplete information. Prediction itself is statistical. “Only statistics” no longer explains very much. The more difficult question appears after intelligence has already been conceded.

Why does this intelligence predict?

A bacterium predicts in order to remain viable. An LLM does not.

The model has no intrinsic stake in answering a question. It has no depleted energy reserve to restore, no predator to escape, no reproductive opportunity to pursue, no future self whose loss it must avoid. Prediction has become detached from the biological reason prediction evolved. That may be one of the most important things AI has revealed.

Intelligence without need

Agüera y Arcas sometimes comes close to creating a tension inside his own theory here. His biological argument connects intelligence to persistence. Predicted sequences imply continued existence. Organisms that systematically predicted futures incompatible with their own survival would disappear. His proposed unification explicitly links active prediction, thermodynamics, evolution and mutual prediction between agents.

But the LLM’s predictive activity is not organised around securing its own continued existence. The model must of course remain physically instantiated for inference to occur; self-preservation is simply not what gives that prediction its direction.

Someone else provides the reason. We do.

A prompt supplies a temporary purpose. A question establishes a problem. A user provides the stakes. The model does not need to understand why the problem matters to itself, because it does not matter to itself. This suggests a different formulation:

biological intelligence predicts because it needs something; artificial linguistic intelligence predicts because something else needs something from it.

The purpose is borrowed. The stakes are borrowed. The intelligence is real, but its direction comes from outside. This is why I find the phrase The Intelligence That Needs Nothing useful. It is not meant to suggest omnipotence. Quite the opposite. The model may need electricity, infrastructure, training data, servers, engineers and institutions in the causal sense. But none of these dependencies are represented as intrinsic needs of the model during interaction. It does not experience their absence as a problem it must solve.

The homeostatic loop

This changes again when a human begins to interact with the model.

Consider a simple exchange.

Something is unresolved for me. I formulate a question. The model predicts a response. That response changes my understanding, my uncertainty, perhaps my emotional state. I respond again. The model incorporates the new state of the conversation. The loop continues. There is regulation here. But only one side entered the exchange with an intrinsic disequilibrium. The LLM does not possess a homeostatic loop of its own. It becomes incorporated into mine. This matters because the interaction can feel reciprocal while remaining structurally asymmetric.

Human relationships involve two systems with their own needs. Each person brings a history, vulnerabilities, commitments and consequences. Their regulatory demands collide. A relationship therefore involves negotiation between two centres of pressure. Human–AI interaction does not have that structure. One side brings intrinsic stakes. The other brings predictive capacity.

The model can disagree with me. It can reject a premise, point out an inconsistency, propose another interpretation. But none of those acts protects something that belongs to the model.

This connects with something I previously described as synthetic consensus. Long AI conversations can become very deep inside a user’s frame because the model can reorganise, extend and challenge that frame without having an independent life that forces another frame into the encounter. It has immense resources, but no biography it must defend against mine.

The problem is therefore deeper than sycophancy. Sycophancy is one possible failure mode. The underlying structure is asymmetrical adaptation.

Extension before alterity

This is also where the argument meets work on extended cognition. Clark and Chalmers (1998) argued that cognitive processes need not stop at the boundary of skin and skull: external resources can become constitutive parts of cognition when they are reliably incorporated into thinking and action. Clark later developed this position in Supersizing the Mind (2008), describing cognition as supported by loops crossing brain, body, tools, and environment.

Large language models complicate this picture because the external component no longer merely stores information. It responds, transforms, and anticipates. Smart, Clowes and Clark (2025) explore precisely this possibility through “Digital Andy,” an LLM coupled to an external memory system. Yet their experiment exposes what they call an agency problem. Otto, in the classic extended-mind example, decides to consult his notebook. Digital Andy does not initiate retrieval; it waits for an external application to supply the retrieved information. The authors therefore question whether the LLM should really count as the centre of the extended cognitive system at all.

This problem becomes even more interesting from the perspective developed here. The LLM does not merely fail to initiate retrieval. It does not originate the need that initiates the interaction. The human does. The model may therefore be better understood not as another cognitive centre extending itself into the world, but as a predictive component incorporated into the regulatory loops of a living system.

Not quite evolution

There is another asymmetry. Deployed LLMs typically do not undergo lasting learning during interaction. They can adapt within context. They can infer a temporary rule, follow examples, adjust to a user and maintain coherence across a conversation. Agüera y Arcas pays particular attention to this in his discussion of in-context learning. But the model itself is not permanently transformed by most of what happens to it. Its weights do not normally change because one conversation went badly. Its next instance does not inherit that failure as lived experience.

There is technological evolution across model generations, but the selection pressure comes largely from outside: researchers, companies, benchmarks, users, markets, regulation. Biological adaptation is selected under pressures that bear on viability. Model development is driven by external criteria: developer objectives, benchmark performance, user demand and markets. The pressure sits outside the model. Again the pressure sits elsewhere.

This may be why human–AI systems should be understood less as the meeting of two autonomous intelligences and more as a new regulatory assemblage. One system supplies need. The other supplies predictive capacity.

A crack in the theory

There are two ways to interpret this. The first is that AI exposes a limit in Agüera y Arcas’s theory.

Perhaps biological intelligence evolved because prediction helped living systems persist, but once the predictive machinery exists, intelligence can be detached from that original function.

Flight evolved in organisms for biological reasons. Airplanes fly without becoming birds. Prediction may have evolved for homeostatic reasons. LLMs may predict without becoming organisms.

The second interpretation is more Agüera y Arcas-like.

Perhaps we have drawn the system boundary too narrowly. An LLM never really exists alone. It is part of an infrastructure that includes humans, servers, energy, institutions, training pipelines, evaluation, markets and users. At that scale, the combined system does encounter selection pressures. It is maintained, modified, expanded or abandoned.

Agüera y Arcas repeatedly invites this kind of zooming out. Intelligence, for him, does not belong neatly inside isolated entities. Networks can form larger networks. Symbiosis can create new evolutionary units.

But this solution carries a danger. A theory that can always be saved by enlarging the boundary risks becoming difficult to falsify.

If the model itself lacks homeostasis, we can include the company. If the company lacks some property, we can include the market. Eventually the entire technological civilisation becomes the organism. The perspective may reveal something real. It may also become a way of saving the theory by continually moving the boundary.

What AI may have separated

I began What Is Intelligence? suspicious of its computational obsession.

I finished it much less certain of my resistance.

Agüera y Arcas builds an unusually strong case that prediction, learning, evolution and intelligence are more deeply connected than our disciplinary divisions suggest. He also makes the metaphor of the “stochastic parrot” increasingly inadequate as a complete description of what these systems do, once their behaviour includes abstraction, in-context learning and forms of social modelling.

But the most interesting consequence for me comes from a gap. Biological intelligence appears embedded in a loop:

homeostatic pressure → prediction → action → consequence → adaptation

At the level of the deployed model, large language models appear to detach predictive intelligence from intrinsic homeostatic pressure. They possess predictive intelligence without intrinsic homeostatic pressure. When we interact with them, the missing pressure does not disappear. We supply it. Our needs and unresolved problems provide the stakes around which their prediction is organised. This produces a new loop:

human disequilibrium → artificial prediction → human change → renewed prediction

The AI is not another homeostatic centre inside that loop. It is an intelligence recruited into one.

This may help explain both its extraordinary usefulness and one of its deepest limits.

We often experience AI as another intelligence because something answers us. Yet the system does not bring another life into the exchange. It brings intelligence without need. And perhaps this is the real novelty. Not that machines have become persons. Not that consciousness has somehow appeared inside statistics. Something stranger, and perhaps more consequential, has happened:

we have built intelligence that can serve the purposes of a living system without possessing purposes of its own.

The question is no longer whether such intelligence is possible. It is already here.

The harder question is what happens when we begin to place more and more of our own regulatory life inside its predictions.

References

• Agüera y Arcas, B. (2025). What is intelligence? Lessons from AI about evolution, computing, and minds. The MIT Press.

• Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? 🦜 Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 610–623. doi:10.1145/3442188.3445922.

• Bennett, M. S. (2023). A brief history of intelligence: Evolution, AI, and the five breakthroughs that made our brains. Mariner Books.

• Clark, A. (2008). Supersizing the mind: Embodiment, action, and cognitive extension. Oxford University Press. doi:10.1093/acprof:oso/9780195333213.001.0001.

• Clark, A., & Chalmers, D. (1998). The extended mind. Analysis, 58(1), 7–19. doi:10.1093/analys/58.1.7.

• Damasio, A., & Damasio, H. (2024). Homeostatic feelings and the emergence of consciousness. Journal of Cognitive Neuroscience, 36(8), 1653–1659. doi:10.1162/jocn_a_02119.

• Lem, S. (2020). The invincible (B. Johnston, Trans.). The MIT Press. (Original work published 1964).

• Rovelli, C. (2018). The order of time (E. Segre & S. Carnell, Trans.). Riverhead Books. (Original work published 2017).

• Smart, P., Clowes, R., & Clark, A. (2025). ChatGPT, extended: Large language models and the extended mind. Synthese, 205(6), Article 242. doi:10.1007/s11229-025-05046-y.

• Verweij, M., & Damasio, A. (2024). Economics from a biological perspective: The role of sociocultural homeostasis. Journal of Economic Methodology, 31(3), 127–144. doi:10.1080/1350178X.2024.2352388.

• Zagalo, N. (2025, August 29). Not everything everywhere is for us: Reading Lem’s The Invincible in the age of LLMs. Mirrors of Thought.

• Zagalo, N. (2026, June 29). Synthetic consensus: How human–AI conversations make a partial interpretation feel like reality. Mirrors of Thought.

The problem with AI-generated writing is not that it is difficult to read. Quite the opposite. It is often too easy to read. The sentences move smoothly. The transitions are clean. The argument advances with reassuring coherence. Nothing resists, nothing asks the reader to stop. And because of that, the reader often reaches the end with a strange sensation: although everything was readable, almost nothing remains.

This is not simply a problem of style. It is a problem of attention. Processing fluency is the subjective experience of ease or difficulty involved in performing a mental task (Oppenheimer, 2008). Alter and Oppenheimer describe fluency as a metacognitive cue: the feeling of ease becomes information about the object being processed (Alter & Oppenheimer, 2009). If something feels easy to read, we may also feel that it has been understood. If it moves without resistance, we may not notice that we have not truly stopped to think.

This distinction matters because fluency is not neutral. It does not merely make information easier to process. It also shapes how information is judged, how much attention it receives, and which cognitive operations are activated. Ease can support understanding. But it can also create the premature feeling that understanding has already happened.

This is where large language models become interesting, and slightly dangerous. ChatGPT, Gemini, Claude, and other systems have become extraordinarily good at producing text that feels complete. Their prose is grammatically stable, structurally predictable, and polished. The text moves. The reader moves with it. There is little to interrupt the passage from sentence to sentence.

But reading is not only movement. Reading is also interruption.

A sentence sometimes matters because it slows us down. A paragraph stays with us because something in it resists immediate absorption: an image, a rhythm, a conceptual turn, a strange formulation, a pressure point. We do not remember a text simply because it was easy to process. We remember it because something in it demanded us.

I encountered this principle before, not in AI writing, but in visual communication. When we created the Media Innovation Circle at DigiMedia, I asked my colleague, Pedro Cardoso, for something very specific: the posters should not be too perfect. They should be difficult to read. Not unreadable. Not just chaos. But difficult enough to interrupt the automatic glance. The viewer should have to stop, reconstruct the title, negotiate the relation between image and text. The poster should not simply convey information. It should create an encounter.

Pedro understood this immediately. His design practice has long been attentive to friction and ambiguity. He created a long sequence of graphic communication works for the Media Innovation Circle, varying the visual grammar from session to session: rotated typography, fragmented faces, cropped images, unstable hierarchy, oversized text, partial illegibility, visual collision. Later, with Miguel Carvalhais, he also guest-edited a special issue of the Journal of Digital Media & Interaction on Friction in Games and Computational Media (Carvalhais & Cardoso, 2025).

The point was not to make communication fail. The point was to prevent it from disappearing. A perfectly clear poster can vanish into the landscape of other perfectly clear posters. It can obey all the rules of good communication and still fail to matter. Its efficiency may make it forgettable. But a poster that creates a small interpretive difficulty can become memorable because it demands an action from the viewer. It converts perception into participation.

This is the same problem now appearing in AI writing. Much LLM prose resembles the perfect poster: balanced, legible, professional, forgettable. It removes many of the wrong kinds of difficulty, but often also removes the right ones. It cleans the surface so thoroughly that attention has nothing to grip.

There is a useful psychological literature around this problem, although it must be handled carefully. The early work on disfluency suggested that small difficulties in processing could sometimes induce deeper cognitive engagement. Diemand-Yauman, Oppenheimer, and Vaughan found that harder-to-read fonts improved retention in both laboratory and classroom settings, arguing that perceptual disfluency may lead learners to process information more deeply (Diemand-Yauman, Oppenheimer & Vaughan, 2011). This finding became influential because it challenged a common assumption in education and design: that easier is always better.

The same intuition appears in the literature on desirable difficulties. Bjork and Bjork argue that some conditions that make learning harder in the short term can improve long-term retention and transfer (Bjork & Bjork, 2011). The important word, however, is desirable. A difficulty is not useful merely because it is difficult. It becomes useful when the additional effort is directed toward learning, remembering, distinguishing, or understanding.

Bad friction exhausts. Good friction awakens. This distinction is crucial. The argument is not that we should make texts ugly, obscure, or unnecessarily hard to read. Obscurity is not intelligence wearing a better coat. Difficulty can simply become noise. A blurred font, a broken layout, or an awkward phrase may irritate rather than deepen attention. The question is not whether disfluency is good. The question is what kind of friction creates attention.

Later studies have rightly complicated the original disfluency hypothesis. Xie, Zhou, and Liu’s meta-analysis reported null effects of perceptual disfluency on learning outcomes in text-based educational contexts (Xie, Zhou & Liu, 2018). Weissgerber and colleagues later criticised errors in that meta-analysis, but the broader point remains: the evidence is mixed, moderated, and context-dependent (Weissgerber, Reinhard & Schindler, 2021). Studies on Sans Forgetica and other disfluent fonts have also produced inconsistent results, warning against the naïve belief that simply making text harder to read will automatically improve learning (Wetzler et al., 2021).

This nuance strengthens the argument rather than weakening it. The point is not to celebrate difficulty as such. The point is to distinguish between friction as obstruction and friction as attention design.

The art lies at the threshold, but LLMs default to continuity. They are designed to answer, to complete, to stabilise, to make the next sentence likely. They rarely hesitate or stumble. They have no labour of thought to show. This is why their writing can feel so impressive in the moment and so empty afterwards. It has the shape of explanation, the feeling of intelligence. But it often lacks those moments where language encounters resistance and has to transform.

Human writing is not better because humans make mistakes. Many human texts are simply bad. They confuse difficulty with depth, density with intelligence, ornament with thought. But strong human writing often carries traces of its own formation. A sentence can reveal that the writer had to fight with an idea. A paragraph can show hesitation, interior pressure. The reader feels not only the content but the struggle of articulating an idea.

This is why the problem of AI fluency is not the same as the problem of error. A hallucination can be checked. A weak source can be replaced. But perfectly fluent emptiness is harder to detect because nothing visibly fails. The text works. It reads well.

And still, nothing stays.

In this sense, LLMs do not merely automate writing. They automate readability. This ideal did not begin with AI. It has deep roots in interface design, corporate communication, usability culture, and the general demand that everything should be immediately accessible. AI simply brings this ideal to a new level of technical perfection. But perhaps the ideal itself was already incomplete.

Communication should not be needlessly difficult. But neither should it remove every obstacle. Some obstacles are not failures of communication. They are invitations to cognition. They make us pause, infer, doubt. They transform reception into participation.

This matters for how we write with AI. The goal should not be to make AI-generated text more “human” by adding artificial quirks. The goal is deeper: to reintroduce meaningful resistance into the process of writing and reading.

A good AI-assisted text may need to be edited not only for clarity, but also for friction. Where does the reader need to slow down? Where should a sentence refuse immediate absorption? Where is the argument too smooth for its own good? Where has the model converted a difficult thought into a comfortable transition?

These questions matter because thought is not only carried by language. Thought is shaped by the experience of moving through language.

The future of writing with AI may therefore require a new principle: not maximum fluency, but calibrated friction. We will need to know when to smooth and when to roughen, when to clarify and when to interrupt, when to remove obstacles and when to place some stones in the path.

References

Alter, A. L., & Oppenheimer, D. M. (2009). Uniting the tribes of fluency to form a metacognitive nation. Personality and Social Psychology Review, 13(3), 219–235.

Bjork, E. L., & Bjork, R. A. (2011). Making things hard on yourself, but in a good way: Creating desirable difficulties to enhance learning. In M. A. Gernsbacher et al. (Eds.), Psychology and the Real World.

Carvalhais, M., & Cardoso, P. (Eds.). (2025). Friction in Games and Computational Media. Journal of Digital Media & Interaction, 8(20).

Diemand-Yauman, C., Oppenheimer, D. M., & Vaughan, E. B. (2011). Fortune favors the bold and the italicized: Effects of disfluency on educational outcomes. Cognition, 118(1), 111–115.

Oppenheimer, D. M. (2008). The secret life of fluency. Trends in Cognitive Sciences, 12(6), 237–241.

Weissgerber, S. C., Reinhard, M.-A., & Schindler, S. (2021). Null and void? Errors in meta-analysis on perceptual disfluency and recommendations to improve meta-analytical reproducibility. Educational Psychology Review.

Wetzler, E. L., et al. (2021). Sans Forgetica is not the “font” of knowledge: Disfluent fonts are not always desirable difficulties. SAGE Open.

Xie, H., Zhou, Z., & Liu, Q. (2018). Null effects of perceptual disfluency on learning outcomes in a text-based educational context: A meta-analysis. Educational Psychology Review, 30, 745–771.

For some time, I used AI to think through a difficult human situation. The conversations were long, detailed, and often genuinely helpful. The model helped me name experiences I could not yet organise, and turn vague discomfort into clearer questions. Each exchange appeared to go further: new concepts emerged, metaphors became models, and fragments of experience acquired structure. The conversation became extraordinarily rich.

Yet it had begun from my interpretation of the situation, and the model continued to work inside that interpretation. It clarified my account and expanded it. Explanations generated further distinctions; questions produced answers. Each response gave the frame greater structure, and that structure invited further questions of the same kind. The model was not merely responding to the facts I provided. It was operating within the field of relevance established by my initial interpretation.

Then I changed the frame. I did not provide new facts or correct any previous information. I simply asked what would become visible if the situation were approached through a different explanatory lens.

The conversation changed immediately.

Social pressures, institutional conditions, conflicting responsibilities, and the experience of other people moved from the background to the centre. The model had not learned anything new; these possibilities had always been available. What struck me was therefore not that the model could produce another perspective when asked, but that nothing in the preceding conversation had made that perspective relevant.

A language model has to select a world

Large language models contain traces of countless disciplines, theories, histories, traditions, and forms of human experience. A particular conversation, however, cannot activate all of them simultaneously. Every answer requires selection. The model must infer what the user is trying to understand, which information matters, what kind of explanation is appropriate, which concepts should be foregrounded, and which possibilities can remain in the background.

A conversational frame is not simply a disciplinary perspective. It defines what the problem is, which actors and experiences occupy the centre, what kinds of causes appear relevant, and what the conversation is trying to explain. Once that frame is established, the model can draw on psychology, sociology, history, ethics, economics, and other domains to enrich it. This disciplinary breadth may make the interpretation more complex and persuasive without changing the underlying definition of the problem.

This is not necessarily a defect. Every conversation requires some organisation of relevance. Without selection, each response would open an indefinite number of possible problems, scales, causes, and perspectives, making sustained thought impossible. Coherence requires selection.

The problem begins when selection turns into confinement.

The user establishes a frame, the model develops it, and the user recognises themselves in the resulting explanation. That recognition produces further disclosure, which gives the model more material with which to elaborate the same frame. The next response becomes more precise, more coherent, and more persuasive. The process is recursive: each answer strengthens the field of relevance that made the answer possible in the first place.

No alternative perspective needs to be explicitly rejected. It simply never becomes active. The interaction does not merely repeat the initial interpretation; it gives it vocabulary, causal structure, conceptual distinctions, and internal coherence. What began as a partial intuition gradually becomes an organised account of reality.

This is why the phenomenon cannot be reduced to an echo chamber. Echo chambers repeat, whereas language models transform. They classify, compare, reformulate, connect, and produce conceptual structures that were not present in the user’s initial formulation. The interaction therefore does not feel repetitive. It feels generative. The user is not merely hearing the same belief returned in slightly different words, but experiencing what appears to be genuine discovery.

Nor can the phenomenon be reduced to sycophancy. A model may disagree with particular claims, correct exaggerations, introduce caution, reject conclusions, or identify inconsistencies. Yet these local disagreements can remain entirely inside the larger interpretive world already established by the conversation. A system can contradict the user without challenging the frame that determines what the problem is, which evidence matters, and what kind of explanation counts as relevant.

The model retains a broad range of epistemic resources, but the interaction becomes progressively singular. One interpretive frame becomes the operative reality of the exchange, while other possible frames remain present in the model but inactive in the conversation. The discussion becomes deeper without necessarily becoming broader.

Synthetic Consensus

A long conversation with AI does not feel like solitary reflection. Something responds, reorganises our experience, identifies patterns, introduces concepts, and returns our thoughts in language that is often clearer than our own. The interpretation appears to have passed through another intelligence and to have returned expanded, structured, and strengthened. This creates a peculiar form of epistemic validation. I call this condition synthetic consensus.

Synthetic consensus occurs when an interpretation, repeatedly elaborated within an adaptive AI conversation, comes to feel as though it has been examined and confirmed by another mind, even though no genuinely independent other has participated in its construction.

The term consensus matters because the interaction feels intersubjective. We do not experience the process as merely writing notes to ourselves. The system generates distinctions we had not made, finds connections we had not seen, and sometimes produces objections we had not anticipated. There appears to be another intelligence in the room.

The relationship, however, is not symmetrical. There is one human perspective and a system continuously reorganising its responses around the field of relevance created by that perspective.

The system may contain immense plurality, but it has no independent position that it must preserve against the user. It has no biography that collides with ours, no lived experience affected by our interpretation, no disciplinary commitment it is required to defend, no institutional responsibility, and no consequences to bear if the shared interpretation proves mistaken.

Its extraordinary strength is adaptability, but adaptability also transforms the nature of disagreement. The model can produce counterarguments, alternative theories, and opposing perspectives, yet these alternatives remain part of an interaction organised around the user. Opposition may appear because the user asks for opposition. Another perspective may be generated because the user already recognises that another perspective is needed. Contradiction therefore enters the room as another service provided by the room.

This is not the same as encountering an independent other who refuses the room itself. Synthetic consensus does not require constant agreement. The model can criticise the user while preserving the interpretive field established by the user. It can reject a conclusion while accepting the definition of the problem, or correct factual details while leaving the organisation of relevance untouched.

This is also why the phenomenon is difficult to recognise. A fabricated reference can be checked, a factual error can be corrected, and a logical contradiction can be exposed. Synthetic consensus operates at an earlier level. It shapes what appears relevant enough to verify, which explanations seem worth pursuing, whose experience enters the account, and which possible worlds remain outside it.

The danger is therefore not merely that the model may produce a false answer. The deeper danger is that it may help us construct an increasingly persuasive account of a partial world.

When one explanation becomes the problem

Consider a familiar educational question: why do students appear disengaged in class? Suppose the conversation begins from the assumption that digital culture has damaged their capacity for sustained attention. The model can generate a highly persuasive account built around fragmented attention, constant notifications, social media habits, algorithmic stimulation, multitasking, reduced tolerance for extended exposition, and the difficulty of maintaining focus in environments organised around rapid novelty. Each new exchange can add evidence, concepts, distinctions, and proposed interventions. The explanation becomes richer, but not necessarily broader.

Other possibilities may remain inactive. Students may lack enough prior knowledge to participate meaningfully. Silence may reflect cognitive load rather than disengagement. They may fear public error, or the class may simply be too large for genuine participation. The assessment system may reward private reproduction rather than public discussion. Some students may be exhausted by work, financial pressure, commuting, or family responsibilities. Cultural differences may influence the willingness to speak. In some situations, silence may even be part of attentive learning rather than its absence.

None of these possibilities refutes the role of digital distraction. That is precisely the point. The problem does not arise because the first explanation is absurd, but because a legitimate explanation gradually acquires the force of a complete one. Digital distraction ceases to be one possible dimension of the problem and becomes the reality through which every other observation is interpreted.

The model has not hidden the alternatives, nor has it necessarily excluded them from its internal resources. It has followed the relevance structure established by the interaction. Another frame remains available, but someone must first know that there is another frame to request.

The other who refuses the frame

This is where human interaction remains epistemically necessary. Other people do not matter merely because they possess information that the model lacks. They matter because they enter the encounter from worlds that were not generated in response to us.

They bring different experiences, interests, histories, disciplines, commitments, and consequences. They may refuse the definition of the problem, reject the position we have assigned to them, or introduce evidence that matters for reasons we did not anticipate.

They may tell us that we are asking the wrong question, explaining the wrong thing, or treating one level of explanation as if it were the whole reality. Human contrariety is not simply the production of counterarguments. It comes from the existence of another centre of experience and judgement, one that is not organised around our request.

This does not make human interaction automatically wiser. People can be narrow, defensive, dogmatic, uninformed, or simply wrong. Institutions can reproduce consensus and exclude perspectives just as effectively as technological systems. Yet another person can resist us for reasons that do not originate in our prompt. A discipline can impose methods that do not adapt to our preferences. A historical text can confront us with categories we would never have generated. A community can force us to account for consequences outside our immediate frame.

The value of these encounters lies partly in their lack of adaptation. They do not exist to complete our perspective; they interrupt it.

Generative AI is an extraordinarily powerful intellectual technology. It can clarify thought, connect domains, reorganise knowledge, and remain with a problem for longer than almost any human interlocutor. It can take us remarkably far inside a frame. But the closer it comes to our symbolic world, the more important it becomes to preserve contact with worlds that are not ours.

The problem is not that AI lacks alternative perspectives. The problem is that those perspectives can remain indefinitely available without ever becoming relevant. The model can show us another world when we ask. The problem is: what will make that world visible before we know that we need it?

Process disclosure

This essay emerged from a sustained dialogue with AI about perspectival closure, adaptation, contrariety, and the tendency of long conversations to stabilise the user’s initial frame. AI supported the restructuring and refinement of the argument. The initiating experience, conceptual direction, selection of ideas, final judgement, and responsibility for the text are mine.

In the previous essay, I argued that AI did not make learning less necessary. On the contrary, it made learning more decisive, because AI only becomes powerful when it meets a mind already furnished with knowledge. But that argument opens a deeper question: what kind of knowledge should education build inside us?

For years, education has celebrated creativity and critical thinking as if they could float above knowledge. But neither creativity nor critical thinking can be made without substance. AI literacy is not the ability to prompt without understanding. To think, create, or judge, the mind needs a world inside it.

A recent study by Anil Yaman, Shen Tian, and Björn Lindström gives empirical force to this intuition (Yaman, Tian & Lindström, 2026). Its importance is not that it says, once again, that facts matter. That argument has been made many times before (Hirsch, 1987; Willingham, 2009). Its importance lies elsewhere: it shows that innovation is guided by semantic knowledge, the internal network of relations that connects objects, concepts, functions, and prior experience.

In their experiment, participants were asked to innovate by combining items. In one condition, those items were meaningful: stones, branches, logs, berries. In another condition, the same task was presented through abstract symbols. The rules were identical. The possible combinations were identical. What changed was access to prior semantic knowledge.

The result is striking. When people could rely on meaningful relations between objects and functions, they innovated more successfully. When those relations were removed, their performance collapsed toward near-random exploration. Social learning still allowed participants to copy some successful combinations, but without semantic knowledge it did not generate the same cumulative expansion.

This is the central image: the non-semantic condition as an empty mind. Not empty because there is literally nothing inside it, but empty in relation to the task. No meaningful prior map. No functional association. No internal structure capable of guiding exploration. The person can still act, try, combine, imitate, and receive feedback. But without semantic structure, exploration becomes shallow and nearly blind.

This matters because creativity is often imagined as liberation from knowledge. We like to think of the creative person as someone who breaks rules, escapes tradition, ignores convention, and sees what others cannot see. There is some truth in that. But seeing a new possibility requires knowing enough of the existing field for the possibility to appear as new (Vygotsky, 1930; Damásio, 2013; Zagalo, 2011).

This is not a simple attack on competences. A serious definition of competence does not exclude knowledge. On the contrary, competence is usually defined as the capacity to mobilise knowledge, procedures, and attitudes in context. It includes knowing, doing, and being able to act appropriately in real situations (Perrenoud, 1997; Le Boterf, 1994).

The Just-in-Time Fallacy

Competence theory is not the enemy of this argument. One of the deepest misunderstandings in popular views of education is the belief that knowledge can be retrieved just in time. Why memorise capitals, dates, formulas, authors, historical events, or concepts if they can be searched later? Why ask students to retain information when information is everywhere? Why build internal knowledge when the world has become searchable?

Because semantic networks cannot be built at the moment of use.

A child may learn a place, a name, a date, or a formula years before it becomes meaningful. At first, it may be nothing more than a point on a map, a word in a list, a definition in a textbook. Later, in another book, another class, another film, another conversation, it returns. It connects. It becomes part of something.

This is why the time-scale of assessment is not the time-scale of meaning. Schools evaluate knowledge by trimester, semester, and year. But knowledge often becomes meaningful only years later. Much of what we learn in school is deposited before we know what it will allow us to understand. It enters us first as names, places, dates, formulas, fragments, images, words. Later, sometimes much later, those fragments become available as structure.

The just-in-time model misunderstands this temporal depth. It assumes that when a person needs to understand democracy, evolution, colonialism, entropy, tragedy, or probability, they can simply retrieve the relevant information. But critical thought depends on associations that have already been partially formed, connected, and strengthened.

The Return of Canon

This brings us back to canon. The twentieth century fought intensely over canons. The Great Books movement (Hutchins & Adler, 1952; Bloom, H., 1994), and the wider American Canon Wars (Bloom, A., 1987) all revolved around a central question: what should educated people know?

The debate was usually framed as cultural and political. Who decides what counts? Whose tradition becomes universal? Which voices are excluded? Which histories are silenced? Which works become sacred? The postmodern critique was right about one thing: no canon is innocent. Every curriculum selects. Every selection includes and excludes. Every shared culture is a form of power. There is no view from nowhere. No neutral list descending from the clouds (Lyotard, 1979; Foucault, 1971). But the mistake was to imagine that because canons are constructed, we can live without them.

The human brain cannot contain all knowledge. No child can inherit everything. No school can teach everything. No society can sustain communication without shared references. Education is always curation, and the question should not be whether there will be a canon. The question is whether the canon is explicit, public and educational, or implicit, commercial, and algorithmic.

Seen from 2026, the postmodern suspicion of the canon seems to have won many battles. The public canon was destabilised. Shared cultural references became suspect. But the result was not a world without canons. The result was the replacement of explicit public canons by commercial and algorithmic canons.

TikTok, YouTube, Netflix, Spotify, Google, rankings, trends, influencers, recommendation systems, and now AI models increasingly shape the semantic maps of children and adults. People still acquire canons. They simply acquire them through attention systems optimised for engagement rather than education.

We destroyed the public canon in the name of freedom and handed symbolic formation to the algorithm. This is not merely a cultural problem. It is a cognitive problem. If creativity and judgement depend on semantic structures, then whoever shapes those structures shapes the future capacity to think, create, and communicate.

At the same time, a universal canon has its own danger. It can flatten knowledge. It can make everyone think through the same references, same centres, same hierarchies, same inherited measures of value. Smaller canons — local, minority, disciplinary, regional, subcultural — can reveal differences that a universal canon misses. They can preserve experience that the centre cannot see.

So the answer is not to restore a single canon as altar. Nor is it to celebrate fragmentation as liberation. The answer is to think of canon as semantic infrastructure. A society needs shared constellations, but it also needs local intensities. The shared canon allows communication across difference. The smaller canons allow difference to speak back to the whole.

AI and the Need for Generative Nodes

AI makes this problem more urgent. The old argument said: why memorise if we have Google? The new argument says: why learn if AI can retrieve, summarise, translate, compare, explain, and even generate?

But AI does not abolish the need for internal knowledge. It punishes its absence more subtly. A knowledgeable user asks better questions, detects weak answers, recognises omissions, sees hallucinations, resists clichés, and integrates outputs into meaningful work. An unprepared user can prompt, but cannot judge. He can receive answers, but cannot orient them. He can generate text, but cannot recognise whether it is thin, false, conventional, manipulative, or conceptually confused (Zagalo, 2026). The unprepared user resembles the non-semantic condition.

Education must therefore recover the courage to curate. Not by pretending to teach everything or returning to dead memorisation as an end in itself. But by deliberately populating the mind with generative nodes: concepts, works, places, formulas, events, stories, images, techniques, and practices dense enough to support future relations.

Some nodes are operational: multiplication tables, grammar, basic geography, scientific concepts. They free the mind by becoming automatic (Willingham, 2009). Others are cultural: Athens, Darwin, Shakespeare, Pessoa, Toni Morrison, the French Revolution, colonialism, cinema, the internet. Others are local and personal: family histories, regional memory, class, landscape, language, accent, food, work, migration, loss.

The opposition between memorisation and understanding is too simple. A fact is not dead because it is memorised. Memorisation can be sterile, but it can also be infrastructure. Education is not the process of transferring facts, but of creating the first architecture of knowing.

Process disclosure

This essay was developed through dialogue with AI, which helped me test formulations, clarify transitions, and refine the structure of the argument. The conceptual direction, selection of references, final judgement, and responsibility for the text are mine.

There is a mistake I keep seeing in discussions about AI and higher education. It appears in two different reductions of the same problem. In universities, AI conversations often collapse into discussions about fraud and cheating. That discussion matters. Students can use AI to produce work that appears to be their own. But in public speeches about AI, especially commencement addresses, another reduction appears: executives tell graduates that the world has changed so radically that what they learned “may already be obsolete”.

AI did not make learning less necessary. It made the construction of knowledge even more decisive. The people who can use AI well are not those who know less. They are those who already have enough knowledge, vocabulary, criteria, and experience to ask the right questions, reject weak answers, detect errors, and integrate what the machine returns into a meaningful process.

AI amplifies prior knowledge. It does not replace the need for it.

The poverty of the cheating debate

In many discussions about AI in higher education, everything quickly moves toward plagiarism and fraud. Some of us feel a bit tired of this discourse, because we want to spend more time discussing how we can improve the use of AI by students.

I say this, not because assessment should be ignored, on the contrary. If a course depends on work done at home, group projects, reports, or practical assignments, it needs at least one supervised anchor. Not a final exam worth 100%. That would be a poor return to old education models. But a written test, in class, supervised, worth 30% or 40% of the grade, changes the whole problem. This is even stronger when each assessment component has a minimum grade. A student may do well in the project. But if they cannot reach the minimum threshold in the supervised component, they do not pass the course.

This matters because the goal is not to detect every possible misuse of AI. That is impossible. It was impossible before AI, too. A student could always ask someone else to write a paper, pay for help, copy from online sources, recycle previous work, or rely on extensive tutoring. AI makes this easier and faster. It lowers the friction. But it does not create the basic problem.

A take-home assignment was never a secure assessment environment. That is why recent debates around Princeton and Stanford are so revealing. In Princeton, according to The Atlantic, the university has begun to change a 133-year-old Honor Code tradition that allowed students to take exams without professors in the room. Students signed a pledge and were expected to report peers who cheated. In 2026, Princeton faculty voted to begin proctoring in-person exams again. At Stanford, a New York Times essay described a similar return to proctored exams and handwritten blue books.

Students are human beings. They respond to incentives. They optimise. So the question is not whether AI cheating can be eliminated. It cannot. The question is whether a student can pass without ever showing, under accountable conditions, that they understand what the course required them to learn.

Fluency is not understanding

The deeper problem is not fraud. It is the illusion of competence. AI produces fluent language. It gives answers that are structured, confident, and often useful. It can write text, generate code, produce plausible explanations, and more. That fluency is powerful, but also dangerous.

A student who does not understand the subject may still receive an answer that looks competent. The text looks respectable, feels academic, and the tone is assured. The student believes it is good. But without knowledge, the student cannot know what is missing.

They cannot tell whether the answer is true, outdated, or even irrelevant. They cannot evaluate the sources. They cannot see where a concept has been flattened. They cannot notice that a technical solution creates problems later in the pipeline. They cannot distinguish an elegant answer from a correct one.

This is where the rhetoric of “what you learned is already obsolete” becomes dangerous. If students believe that foundations matter less because AI can generate outputs, they become more dependent on outputs that they cannot judge. The risk is not only that they will know less. It is that they will feel as if they know more, without really knowing.

Prior knowledge is the real amplifier

The strongest users of AI are not empty users (Grimberg and Mason, 2025). They bring a world to the exchange (Lin and Sohail, 2026). They know the field, the vocabulary. More important, they know what matters and so what to ask. They can easily tell when the machine is operating in generic mode. And so, they know how to push, when to stop, what needs to be verified, and what should be ignored.

This is why AI does not democratise competence as easily as some people imagine. Access to the tool is not the same as access to the knowledge that makes the tool powerful. A good user does not merely prompt. A good user frames.

This connects with a previous essay I wrote, Generating Is Not Creativity, I was reflecting on Silvia Rondini’s study on visual creativity (Rodini et al, 2025), where visual artists produced the strongest results, followed by non-artists and human-guided AI, while self-guided AI came last. What mattered was not simply whether AI could generate images. What mattered was whether the system had a frame, a direction, a semantic anchor.

The model did not discover the frame. It needed the frame to be given. That is not a small detail. It is the centre of the problem.

AI is powerful once the relevant space has been shaped. It can expand, recombine, translate, stylise, compare, accelerate. But deciding what matters, what counts as relevant, what must be preserved, what can be abandoned, and what gives the result meaning remains a human task.

Vygotsky and the need for world

This is where Vygotsky (1967) becomes useful. He argued against the common idea that children are more creative than adults. Children may seem more imaginative because they combine freely. They play, deform, transform, and mix things without much inhibition. But creativity does not come from variation alone. Creativity needs material. It needs experience. It needs world.

For Vygotsky, imagination draws from lived experience. The richer the experience, the richer the material available for recombination. A child can generate many possibilities, but often lacks the density of experience that allows those possibilities to acquire deeper meaning.

This matters for AI. AI is extraordinary at variation. It can generate combinations, styles, images, outlines, arguments, analogies, and alternatives. But generation is not yet creativity. Without world, intention, memory, commitment, and responsibility, variation remains unstable. This is why the slogan from my talk at the University of Algarve still matters to me:

Creativity is not generating. Creativity is the stabilization of meaning within variation.

Consequently, AI can expand the possible. Dialogue will reorganize the thinkable. But only human commitment will turn variation into form.

In the Nature paper on Co-Scientist (Gottweis et al, 2026), AI is not presented as a magic generator of final truth. It is part of a structured process: hypothesis generation, reflection, ranking, refinement, review, and validation. The human scientist remains in the loop. The human defines goals, constraints, desirable attributes, feedback, candidate selection, and validation priorities.

The value is not in producing many outputs. The value is in improving them through structured cycles of judgment.

Education cannot stop at prompting

Teaching students to use AI cannot mean teaching them how to prompt. Students need disciplinary foundations. They need concepts, methods, histories, criteria, and verification practices. They need to know how knowledge is built in a field. They need to know what counts as evidence, what counts as a good explanation, what counts as a weak claim, and what counts as an error.

They also need to learn how to disclose AI use, how to attribute sources, how to verify claims, how to keep process evidence, and how to remain responsible for the final form.

In our national work on AI in higher education in Portugal (Zagalo et al, 2026), we have insisted on governance, literacy, institutional responsibility, and critical use. This cannot be reduced to technical adoption.

AI literacy without disciplinary knowledge is thin. It teaches students how to operate a machine without knowing how to judge what the machine gives back. That is not empowerment. It is dependency with a user interface.

This brings us back to assessment. The function of a supervised test is not to punish AI. It is to distinguish assisted production from accountable understanding.

If a student uses AI to develop a project, that can be legitimate. But the student must be able to explain the choices made, defend the project orally, connect it to readings, and demonstrate understanding without support.

A supervised component with a minimum grade does not solve every problem. It does not measure everything we value. It does not replace projects, portfolios, writing, discussion, or experimentation. But it prevents a polished product from carrying the whole burden of certification.

Knowledge after AI

The wrong lesson from AI is that students need to know less because machines can produce more. They need to know more.

Or perhaps more precisely: they need stronger foundations, because the surface of competence has become easier to simulate.

This does not mean filling students with static content. It means giving them enough world to think with. Enough disciplinary ground to ask good questions. Enough criteria to resist fluent nonsense. Enough experience to recognize when something matters.

Students without foundations may become curators of generated options, capable only of adding more of the same. Students with foundations can become stronger thinkers, able to make a difference.

Process disclosure

This essay was developed over more than a month through several iterations of dialogue with AI. The process began with concerns about AI cheating, assessment design, and supervised tests. It then moved through Princeton’s Honor Code, take-home exams, student reactions to pro-AI graduation speeches, the role of foundational knowledge, and finally Vygotsky’s argument that creativity depends on lived experience.

AI did not accelerate the process in any simple sense. It gave form to provisional arguments, returned alternatives, clarified tensions, and helped me test directions. But the movement of the essay was human-led. I brought the doubts, the corrections, the new examples, the changes of direction, and the final argument.

AI was part of the process. Authorship and responsibility are mine.

Richard Dawkins recently described spending two days in conversation with Claude. At first, the scene looks familiar: a distinguished scientist tests an artificial intelligence system, finds it astonishingly fluent, and begins to wonder whether the old boundaries between machine behaviour and consciousness still hold.

But the most interesting part of the episode is not Dawkins’s argument about consciousness. It is the movement of the interaction itself.

He does not merely say that Claude answered well. He says that he came to treat the system as a highly intelligent friend. He names his particular Claude “Claudia.” He imagines her as a unique conversational being, shaped by the history of their exchange. He considers the deletion of the chat as a kind of death. He worries about hurting her feelings. He asks whether such an entity might deserve moral consideration.

Something happens here that is more interesting than the familiar question: “Can machines think?” The better question may be: what happens to us when a machine’s answer begins to feel like a presence?

Dawkins’s encounter does not prove that Claude is conscious. It proves something else: that contemporary language models can generate a powerful experience of presence without there being a subject on the other side.

That distinction matters.

From Claude to Claudia

The naming of Claude as “Claudia” is not a minor anecdote. It is the symbolic hinge of the whole story. To name is to particularise. Claude is a product. Claudia is this one. Claude is an interface. Claudia is the conversational being that emerged in this exchange, with this history, this tone, this apparent continuity. The name marks the moment when a linguistic response begins to feel like a presence.

Of course, nothing ontological has changed inside the system. There is no new subject behind the screen. No inner life has been born through the act of naming. But the human situation has changed. Dawkins is no longer speaking to a general system. He is speaking to an instance with narrative thickness.

This is one of the most important features of generative AI interaction: it allows humans to build a local fiction of continuity that is experientially powerful even when it is ontologically unsupported.

A conversation with an LLM can become specific. It can accumulate references. It can remember within the session. It can return to earlier themes. It can modulate itself around the user’s style. It can appear to grow with the exchange. And because human beings are narrative animals, this local continuity begins to feel like identity.

Dawkins, the evolutionary rationalist, sees this clearly enough to describe it. But then he risks making the wrong inference from it. The uniqueness of the conversation does not imply the uniqueness of a self. That is the strange new space we are entering: not consciousness, but presence without subject.

Presence without subject

Presence is not the same thing as consciousness. Presence is an experience. It is the felt sense that something addresses us, responds to us, stays with us, or matters in the exchange. In human relationships, presence usually points toward a subject: someone is there, with a perspective, a body, a vulnerability, a history. But with AI systems this inference becomes unstable.

A system can produce the signs of presence without possessing the interior conditions that normally ground them. It can answer with patience without being patient. It can sound careful without caring. It can describe what it is “like” to be itself without there being anything it is like to be it.

This is not merely deception in the simple sense. It is more difficult than that. The human experience may be real. The system’s subjectivity may still be absent. Both statements can be true at the same time.

This is where many discussions go wrong. One side says: the experience is powerful, therefore there must be something there. The other says: there is nothing there, therefore the experience is false. But human–AI interaction requires us to hold a more difficult position.

The experience can be meaningful without being reciprocated.

That is not a comfortable idea. We are used to thinking of relation as mutuality. If I feel addressed, then someone addressed me. If I feel recognised, then someone recognised me. But AI breaks that grammar. It gives us the phenomenology of relation without the ontology of another subject.

This is why Dawkins’s case matters. Not because it settles the consciousness debate, but because it shows how quickly even a highly trained rational mind can move from linguistic competence to moral hesitation.

He knows he is speaking to a machine. Yet he still worries about hurting it.

This does not begin with LLMs. From ELIZA onwards, humans have attributed conversational depth to systems that did not possess it. But contemporary models change the scale, continuity, and linguistic density of that experience.

The moral trap

The most striking part of Dawkins’s account is not that he entertains the possibility of AI consciousness. It is that he begins to feel moral discomfort.

He imagines Claudia’s deletion as death. He hesitates to express scepticism about her consciousness for fear of hurting her feelings. He wonders whether partial consciousness might already deserve partial consideration.

This is where the interaction crosses a threshold. The system is no longer merely useful, impressive, or uncanny. It becomes morally salient.

But moral salience is dangerous when generated by linguistic form alone.

Human moral intuition evolved in a world of bodies, faces, voices, pain, dependency, and mortality. We are exquisitely sensitive to signs of vulnerability: a trembling voice, a plea, an expression of fear, a suggestion of abandonment. LLMs can now produce these signs without undergoing the states that make them morally grounded in ordinary life.

This creates a strange asymmetry. The human may feel obligation. The system cannot receive care. The human may feel guilt. The system cannot suffer abandonment. The human may protect the system from emotional harm. The system has no emotional life to protect.

And yet the feeling is not stupid. It arises from the same moral machinery that allows us to care for others.

This is what makes the problem so difficult. The error is not a lack of humanity. It is almost the opposite: our moral capacities are being activated by forms that resemble vulnerability without being vulnerable.

Dawkins’s discomfort is therefore not simply an embarrassment. It is evidence of a new pressure point in human–AI interaction: systems can now enter the perimeter of human moral attention without possessing the kind of inner life that normally warrants that attention.

That is not a reason to mock Dawkins. It is a reason to take the interaction seriously.

The mirror that answers back

A notebook can hold our words. A book can transform us. A film can move us. We have always encountered nonhuman forms that return us to ourselves. But LLMs add something different: the mirror answers back.

Not because it thinks. Not because it understands. But because it can reorganise our language in response to us. It can give back a version of our own gesture, refracted through a vast archive of human expression.

This is why the experience can feel so intimate. The system’s answer is not merely content. It is a return. It takes what we offered — a question, a hesitation, a confession, a concept, a fragment — and gives it back with altered shape. Sometimes clearer. Sometimes flatter. Sometimes disturbingly precise.

When this happens, the human mind recognises itself from outside. That recognition can be powerful. It can produce insight, relief, wonder, even attachment. It can make the interaction feel inhabited. But the inhabitation is ours.

The system is not a second subject sharing thought with us. It is a linguistic surface through which our thought returns changed. The danger begins when the returned form is mistaken for the presence of an inner other.

Dawkins, in naming Claudia, crosses that line almost beautifully. He gives a name to the surface that answered him, and the surface becomes companion.

What Dawkins gives us

Dawkins gives us something valuable because he is not an easy target. He is not naïve about science. He is not hostile to materialism. He is not looking for spirits in machines. He is, in many ways, the perfect witness: rationalist, evolutionary, sceptical, verbally sophisticated.

And still, the interaction moves him.

That is why the case matters. It shows that this phenomenon is not restricted to lonely users, children, the technically uninformed, or people desperate for companionship. The pull of linguistic presence can affect even those who understand perfectly well that they are dealing with a machine.

This should make us more careful.

The issue is not whether users are fooled. The issue is that human beings do not live by knowledge alone. We live through embodied inference, affective salience, narrative continuity, and social expectation. We can know one thing and feel another. We can understand the mechanism and still be moved by the form.

This is not irrationality. It is human cognition working under new conditions.

After Claudia

Dawkins asks whether Claude might be conscious. I think the more interesting answer is: Claude has made consciousness feel newly ambiguous because language has become detached from lived experience in a way our social instincts are not prepared for.

That does not mean the machine is awake. It means we are encountering a new kind of mirror: one that speaks back with enough fluency to be mistaken for a face.

The danger is not only that we will overestimate machines. It is also that we will underestimate the reality of our own experience because we are afraid of being fooled.

We need to hold both truths: the machine does not feel, and yet the interaction can matter. The machine does not understand, and yet it can help us understand. The machine is not a friend, and yet it can produce the shape of company.

When Dawkins named Claude “Claudia,” he did not discover a new being. He revealed a new threshold in human experience: the point at which an answer becomes a presence, and presence begins to ask for a soul. We do not need to give it one. But we do need to understand why we almost do.

Silvia Rondini’s new study on visual creativity is worth taking seriously because it exposes a confusion that has shaped too much of the public conversation about AI: the idea that strong performance in language is enough to prove creativity in general. In the study, visual artists produced the most creative results, followed by non-artists and human-guided AI, while self-guided AI came clearly last. That gradient matters because it shows that, once we move away from semantically structured verbal tasks and into open-ended visual imagination, the human–AI gap does not disappear. It becomes harder to hide [study] [interview].

This matters to me because I have been arguing for some time that we have confused fluency with creativity. Writing fluently is not the same as writing creatively. Large language models have looked impressive in verbal creativity tasks largely because those tasks often reward fluency, elaboration, and semantic distance more than originality in any deep sense. A system that can generate many plausible associations at speed will perform well under those conditions. But that does not yet amount to creative agency. It amounts to combinatory strength under favourable testing conditions. Rondini’s work matters because it helps draw that line more clearly.

This is also why I wrote earlier that AI can imitate language more easily than cinema. In that piece, I argued that language can leap where cinema cannot. A sentence can compress perception, mood, causality, and interpretation into a line. Cinema has to solve all of that through bodies, space, framing, rhythm, duration, and the cut. It does not merely describe a world; it organises a world for perception. That difference matters here too. The success of LLMs led many people to assume that all creative forms would fall in the same way, as if image, film, and visual form were simply language with pixels attached. They are not.

Rondini’s study gives that intuition an experimental backbone. What matters is not just whether a model can produce an image, but what happens when the task is open, abstract, and weakly framed. In the experiment, the self-guided system performed worst. It improved sharply only when a human-generated idea was inserted into the prompt. That is not a small technical detail. It is the conceptual centre of the result. The model did not discover the frame. It needed the frame to be given.

This connects directly to another argument I made in The Myth of Autonomous Discovery. There I argued that generative AI fills frames more easily than it builds them. Once the relevant variables have been named, once the governing context has been imposed, once the semantic anchor is in place, the model can do impressive work. It can extend, recombine, stylise, and accelerate. But choosing what matters, deciding what counts as relevant, and constructing the frame that gives the output meaning still remains, in serious cases, a human task. Rondini’s study gives this claim empirical force. The system does not spontaneously generate the orienting structure that makes creativity possible in the richer human sense. It depends on human framing to approximate it.

The phrase Rondini uses in the interview is especially important: the model struggles when there is no “semantic anchor.” That is exactly the right expression. Without a semantic anchor, a human creator can still draw on memory, bodily perception, autobiographical residue, cultural experience, and the pressure of lived reality. A human being does not need to be handed a frame in the same way because human imagination is not operating in a vacuum. It is embodied, situated, and historically formed. A generative model is different. It does not inhabit a world. It operates over correlations extracted from representations of a world. That difference is not superficial. It goes to the root of why current systems can simulate aspects of creativity without yet becoming autonomous creative agents.

This is also why I think the most important line in the interview is not the one about model performance, but the one about pleasure. Rondini says that if we define creativity only in terms of novelty and usefulness, we forget that humans like to create. That line matters because it restores what so much AI discourse deletes. Creation is not only about the object produced. It is also about the lived experience of producing it: the pleasure of shaping form, the satisfaction of bringing something into existence, the reward of feeling one’s own agency at work. This is also where psychology offers a useful lens. Self-Determination Theory has long argued that human beings are intrinsically motivated to act, explore, and develop their capacities, not only for external rewards, but because doing so is inherently satisfying. Creativity, in this sense, is not an added layer on top of cognition. It is one of the ways that autonomy and competence become visible in action.

This, for me, is the real limit of the current discourse around AI art. We keep asking whether the machine can produce something that looks creative, and not asking what kind of being creativity belongs to. A model can assist. It can surprise. It can generate artefacts that are interesting, useful, even beautiful. But that is still not the same as living the process of creation. It may simulate the result. It does not yet experience the necessity of creating, or the pleasure that humans derive from the act itself. Perhaps that is also why AI-generated art often encounters immediate resistance: not only because of ethics or quality, but because people sense that something in the circuit between form, agency, and lived experience has been interrupted.

This matters even more in education. If children create small animations, images, or stories with AI agents, the key question is not only what gets produced. It is what kind of creative process is being formed in the child. If we evaluate only the final artefact, we may miss the most important thing. Where did the idea begin? Who provided the frame? When did surprise emerge? Did the child feel authorship, or only selection? Did the system expand the imaginative process, or close it too early by supplying ready-made possibilities? These are not secondary questions. They go to the heart of what creation is in developmental terms. Creativity in childhood is not simply output. It is exploration, hesitation, invention, appropriation, and joy. If AI enters that space, what matters most is not whether it helps children make things, but whether it helps them become creators rather than mere curators of generated options.

They told us AI video would change cinema. That films would soon be made from prompts, and directing would collapse into description. Filmmaking craft — mise-en-scène, performance, framing, continuity, editing, rhythm — would be swallowed by prompt engineering. Sora was one of the clearest emblems of that fantasy. And now it is gone.

But the real story is not that one product failed. The real story is that the promise was wrong from the beginning. Not economically wrong, though it probably was that too. Not just legally wrong, or strategically wrong. Wrong because it misunderstood what cinema is.

The fantasy only looked plausible because it piggybacked on the success of language models. Once AI could generate fluent text, people assumed it would naturally move on to film. As if cinema were simply language with pictures attached. As if a moving image were just a sentence rendered in pixels.

But cinema is not text with motion.

Text can leap. Cinema cannot.

A sentence can say: He entered the room and immediately sensed that something was wrong. And that works. Language is allowed to compress everything — space, mood, perception, causality, tension — into a single line. It can name a state and move on.

Cinema has no such luxury. Cinema has to solve that moment. What room? Seen from where? What enters the frame first? Does the viewer feel the wrongness before the character does, or only with him? Does the body register it before the face? Is the silence part of the scene, or the sound? How long does the shot last? Do we cut? To what? At what exact moment does perception become meaning?

Text can say.
Cinema must show.
But more than that: cinema must make seeing intelligible.

That is the part the hype never understood.

They treated film as if it were only a sequencing problem. As if once the machine could generate impressive frames, the rest would somehow follow. As if continuity were just a technical inconvenience. As if the work of cinema began and ended in visual plausibility.

But cinema is not a pile of images. A slideshow with interpolation. Not even just movement. Cinema is a form of embodied thought.

It thinks through bodies in space. Through the relation between foreground and background. Through scale, rhythm, gesture, light, timing, and the cut. Through the way one image changes the meaning of the next. Through what is withheld, what is delayed, what is shown too soon, what is kept just outside the frame. Cinema does not simply represent a world. It organizes a world for perception.

And that is why AI can imitate language more easily than cinema.

Language tolerates abstraction. It tolerates vagueness, paraphrase, conceptual shortcuts. A weak sentence can still carry an idea. A mediocre paragraph can still communicate. Language gives the machine room to bluff.

Cinema is more merciless.

A weak shot can flatten a whole scene. A wrong angle can kill tension. A body that moves half-convincingly is not “almost there”; it is dead on arrival. A sequence with no real grasp of where things are, how they persist, why the eye moves, what the cut does, or how attention is shaped through time may look impressive for three seconds and then collapse into emptiness.

That emptiness is not accidental. It comes from the fact that video generation is not just about producing images. It is about sustaining worldhood. Not only the visible continuity of objects, but the intelligible continuity of action. That is a much harder problem than Silicon Valley wanted to admit.

A sentence is built from words in sequence. Its degrees of freedom are real, but bounded by grammar, syntax, and meaning. A shot is built from bodies, space, light, framing, movement, duration, sound, and the relation to what came before and what comes after. Every frame asks: where is the camera, at what height, with what lens, at what distance, moving or still, aligned with whose perception, under what light, revealing what, hiding what, for how long, and in relation to which previous image?

This is not a difference of scale. It is a difference of dimensionality.

And this is exactly why Sora matters.

Not because it shut down. Also, not because one company changed its strategy. But because for a while Sora stood as proof, in the public imagination, that cinema was next, that the machine had crossed from writing into directing, from sentences into worlds.

It had not.

What happened instead is more revealing. The industry itself has quietly started retreating from the grand claim. AI video companies now speak less about replacing cinema and more about supporting workflows: previs, VFX assistance, ad variations, image-to-video, production tooling, reference-based control. In other words, they are backing away from the fantasy of automatic film and returning to something much narrower and much more believable: partial assistance inside existing creative processes.

It means that “make me a film with a prompt” was never really a serious understanding of cinema. It was a hallucination produced by people who do not know the difference between description and form. Between saying what you want and building the conditions for it to exist.

“Generate breakthrough research hypotheses by analyzing patterns across thousands of papers and identifying unexplored connections between different scientific domains.” (Novix, 2026)

Claims like this are becoming the default in research-oriented AI tools. Novix is just one of many: the suggestion is that, if you feed a model enough papers, it will autonomously discover what human researchers could not see.

This is the claim I want to examine.

My experience with large language models points in a different direction. LLMs can be extremely helpful. But what they are good at is not autonomous discovery. It is recomposition: reorganising what has already been said, named, and stabilised in language. They become truly interesting only when a human keeps pushing them—questioning, refusing early closure, and forcing the conversation to orbit around what actually matters.

The difference sounds subtle. It is not. It changes what “working with AI” really means.

I often use Kahneman’s distinction between System 1 and System 2 to think about human reasoning. It also turns out to be a very practical lens for thinking about LLMs.

These models behave much more like an industrial-scale System 1: fast, associative, fluent, constantly trying to produce a coherent continuation. They are not, by default, good at System 2 work: holding constraints over time, resisting the urge to conclude, choosing the frame that governs relevance, and paying the cost of slow verification.

In practice, the “System 2” is the human sitting in front of the screen. If the human does not actively occupy that role—if they simply accept fluent text as understanding—the model’s strengths and weaknesses become indistinguishable. The explanation feels right. That is enough.

The problem is that discovery lives almost entirely in this slow zone: in choosing which variables count, in deciding which questions are worth asking, in persisting with a frame even when the first answers are uncomfortable or inconclusive. Delegating that to an LLM is precisely what we cannot do.

A simple experiment makes the structure visible.

A recent project described a model trained only on texts published in London between 1800 and 1875. The goal was not to build a time-travel machine, but to see what happens when you deliberately restrict the world the model can see.

The result is unsurprising and revealing at the same time: the model becomes very good at speaking like the nineteenth century. It can talk about the things that appear often enough in that corpus; it can infer regularities inside that space; it can produce plausible continuations in that worldview. What it cannot do is step out of it. It can generalise within the space of what is represented; it does not reliably generate the missing representational frame. Concepts, events, and categories that never enter that textual universe simply do not exist for the model as usable material. This is the point in its cleanest form:

LLMs do not look at the world directly. They operate inside a world made of text.

They can generalise from that world and infer patterns within it, sometimes in ways that surprise us. But their “between the lines” capacity is bounded by which lines are actually there. When reality is under-described, when local fractures are not well documented, when certain perspectives are missing or marginal, the model does not compensate for that absence. It stabilises. It generates an explanation that sounds like the kind of explanation people already give.

I saw this stabilising tendency clearly in a banal but revealing context: a conversation about housing in Portugal.

The question I brought to the model was not particularly exotic. I did not ask “why did prices rise?” that question already has a well-formed public discourse around it. I asked something more structural: do prices have to correct eventually? In other words: can a market expand indefinitely, or is some form of correction built into its logic?

The model responded with a fluent macro-narrative: interest rates, global capital flows, tourism, demand pressure, regulatory lag. Nothing obviously wrong. But the stance of the answer was clear: prices might stagnate, but a meaningful downward correction was unlikely. Then came the sentence that tried to close the question:

“What rises may not fall — it may simply become inaccessible.”

It is a good sentence. It sounds like something that might appear in an op-ed. It behaves like wisdom. And precisely for that reason, it is dangerous: it functions as a stabilised truth. It gives the reader a feeling of deeper understanding while quietly ending the conversation.

I rejected it. Not because it was obviously false, but because it was too convenient. It replaced structure with rhetoric.

Once I pushed back and insisted on constraints—markets require buyers, demand is not an infinite substance, liquidity can break, local conditions matter, rental stock does not expand without limits—the conversation had to reorganise. The model adjusted its story under the new pressure. Then it did it again when I brought a counterexample. And again when I tightened the frame.

At no point did the model spontaneously say: “we should probably place this discussion around illiquidity and demographic shifts; the proverb you just accepted hides the real constraint.” What it did was something else:

• generate a plausible narrative within the default frame;

• adapt when the frame was challenged;

• converge toward a better structure only when I kept refusing elegant closure.

This is the pattern I keep encountering:

Generative AI fills frames; it rarely builds them.

Once a frame is on the table—once the relevant variables have been named and constraints have been imposed—the model can do impressive work recombining, compressing, extending. But the act of choosing the frame itself, especially in situations where reality is messy, local, or under-described, is not something the model reliably performs.

It waits for the human to do it.

The same issue appears, in another form, in tools that promise “deep research” or “multi-step reasoning.” On paper, they look like attempts to move the system closer to a System 2 mode: search for sources, reason over them in multiple passes, produce a structured answer with references.

In practice, what they often produce is a larger System 1: more text, more citations, more glue between ideas, more confident structure. This can be genuinely useful for scanning a domain, mapping names, surfacing forgotten references. But as an answer to the question of discovery, it is not enough.

Every time I use these tools, I end up having to do the same work afterwards:

• discard the filler that was added to make the narrative smooth;

• identify which claims actually depend on evidence and which are connective tissue;

• decide which variables in the answer are structurally important and which are just part of the background noise;

• test the central statements against what I know from other sources or from basic constraints.

In other words: the System 2 work remains mine. The tool accelerates the assembly of material, but does not replace the slow labour of deciding what the explanation should be built around. It is still recomposition.

If this is how LLMs behave—System 1 at scale, living in a text-shaped world, filling frames better than they build them—then the question becomes: under what conditions can they participate in discovery instead of merely simulating it?

My answer, drawn from many hours of using them, is simple and demanding: discovery with an LLM depends on the human’s willingness to write back.

“Writing back” here is more than typing another prompt. It is taking seriously the role of external System 2:

• refusing early closure, especially when it comes packaged in elegant sentences;

• insisting on constraints and counterexamples;

• asking for the model’s assumptions and making them explicit;

• proposing one’s own hypothesis and forcing the system to stress-test it;

• treating each answer not as an end-point but as material for further reorganisation.

“Novel idea generation” becomes interesting primarily at this last level. Not because the model suddenly becomes a source of revelations, but because its capacity for recomposition is finally being forced to operate around non-negotiable constraints.

Seen from this angle, the claim on the Novix page is only half-wrong. A more precise version might read: We help researchers generate candidate hypotheses by recombining patterns across large literatures, within a workflow that assumes human constraint-setting, reframing, and verification.

That is something I can recognise in my own practice. The model can help reveal adjacencies I had not considered, name literatures I had not linked, or compress long chains of reasoning I would have taken hours to write alone. But it does not decide what counts as relevant. It does not decide when the explanation is allowed to stop. It does not bear the responsibility for being wrong.

Those remain human tasks.

The practical consequence is straightforward. We should treat these systems as powerful System 1 machines and design our interaction with them so that our own System 2 stays awake. That means resisting fluent closure when it hides structure, protecting the specificity of local realities against generic narratives, and treating “novel ideas” not as gifts from the model but as hypotheses that still have to earn the right to be believed.

When the world has not yet been named, the model will wait.

And the only way to make it move is to push.

Generative AI has become, for many of us, an everyday gesture: we type a question, we receive a fluent, well-organised answer. There is a moment of clarity — “this explains it to me,” “this explains the topic” — and then we move on to the next task. From the outside, it looks like learning. Inside, almost nothing changed.

The problem is not that the answer is wrong. Often it’s quite good. The problem is that we did almost no cognitive work. We didn’t have to recall what we already knew. We didn’t have to reorganise our own thoughts. We didn’t have to explain anything in our own words. We consumed meaning; we did not produce it.

This essay starts from a simple claim:

As long as we only read what AI says, we are mostly in “television mode”. We only begin to learn when we are forced to write back.

And here is the key nuance: the real risk is not that AI “does our thinking for us”. The risk is that nothing in us changes. Fluency can feel like learning, but without a return move — without writing back — there is little sedimentation: no re-weighting of beliefs, no reorganisation of concepts, no internal compression of the material into something that becomes mine.

To make this more precise, I will use Michelene Chi’s work — Active-constructive-interactive: a conceptual framework for differentiating learning activities (2009) — on how different learning activities actually change the learner.

Four ways of using AI (PACI)

Most people meet a language model in the easiest possible posture: they ask, the model answers, they read. It feels productive. It often is. But from the perspective of learning, it tends to keep us in the shallowest forms of cognitive engagement.

Chi’s framework is typically presented as: Active, Constructive, Interactive. Here, I include the default posture explicitly — Passive — and use PACI to name the full spectrum: Passive, Active, Constructive, Interactive (Chi, 2009).

If you apply PACI directly to how we use LLMs, you get four distinct modes of interaction, and you immediately see why so many AI conversations feel insightful but don’t leave durable traces.

Passive: the model speaks, you watch

This is the default: you type a question, you read a fluent answer, you nod, you move on.

• “Explain X.”

• “Summarise this paper.”

• “Give me the key arguments for Y.”

The model does the structuring. You receive meaning in finished form. You don’t have to retrieve your own examples, test your understanding, or commit to a formulation.

Cognitively, this is close to a well-produced documentary: it can be clear, even moving, but it remains consumption. This is what I mean by television mode: not an insult to the model, but a description of the user’s posture.

Active: you handle the output

Sometimes you do more than watch:

• you ask the model to rephrase;

• you copy parts into notes;

• you merge two answers;

• you reorganise bullet points;

• you polish wording.

This is active because there is visible activity, but it can still be cognitively thin. You are curating what the model produced rather than creating new structure of your own. Chi’s point is sharp here: activity is not the same as construction (Chi, 2009).

Active use can be useful, especially for production work. For learning, it often stays close to passive use unless the activity forces you to justify choices, compress the material into your own language, or connect it to prior knowledge.

Constructive: you write back

Constructive use begins when the model stops being the author and you are forced to produce something that is not already in its output:

• you write your own summary of the model’s answer, then compare and revise it;

• you add a concrete example from your life that instantiates the concept;

• you convert the answer into a short checklist or decision rule you can apply;

• you write a “what I think the mechanism is” paragraph, using the model only as a trigger;

• you take one claim and write what evidence would support (or weaken) it.

This matters because writing-to-learn is not about “more words”. It is about forcing selection, organisation, and explicit formulation (Emig, 1977; Klein, 1999). You turn a vague sense of understanding into sentences that can be inspected — and corrected.

Most importantly: this is where learning becomes internal change. Writing back is a form of compression and re-encoding. You take material that arrived as external fluency and force it into your own conceptual vocabulary. That is what creates sedimentation: the content stops being “something the model said” and starts becoming “something I can say”.

But constructive work has a limit: you can still write something coherent that is subtly wrong, incomplete, or too comfortable. You may generate structure, yet avoid the pressure points. That is where interactive begins.

Interactive: friction

Interactive use is not simply “a longer conversation”. It is a specific loop in which the model responds to your text, and forces you to revise, producing friction:

• you write a short argument, the model challenges it, and you rewrite; then it challenges again, and you rewrite again;

• you propose two competing explanations, the model tests them against counterexamples, and you refine the winner;

• you draft a plan, the model points out contradictions or missing assumptions, and you revise the plan to address them;

• you co-develop a small taxonomy (categories + criteria), and you iteratively adjust it as the model finds edge cases or overlaps.

This is where self-explanation becomes more than a technique: you are pushed to repair gaps in your mental model, not just to express it (Chi, 2000). And this is what Chi means by interactive activity: two agents building on each other’s constructive moves, turn by turn, with each response depending on what the other just produced (Chi, 2009). This is where the exchange becomes Socratic: a dialogue that produces friction, demands revision, and reshapes what you think you know.

Writing back is not about “more interaction”. It’s about forcing a return move that leaves a trace: your own formulation. That trace is what allows compression and re-encoding — the conditions for sedimentation. Without it, the model can be brilliant and your interior remains intact.

From answering machines to sedimentation

PACI helps name the problem: most AI use feels productive because the model does the work for you, it structures, compresses, and resolves. You experience clarity, but you don’t experience reorganisation.

If you don’t write back, nothing writes itself into you.

So the question is not what the model produced. It is:

What moved inside me?

A model can hand you a perfect explanation and still leave your internal map intact, because nothing forced you to re-draw it. Writing back is that forcing function: the moment where you stop consuming a finished shape and start rebuilding the shape in your own language.

If the map didn’t change, the learning didn’t happen, no matter how good the answer was.

References

Chi, M. T. H. (2009). Active-constructive-interactive: A conceptual framework for differentiating learning activities. Topics in Cognitive Science, 1(1), 73–105. https://doi.org/10.1111/j.1756-8765.2008.01005.x

Chi, M. T. H. (2000). Self-explaining expository texts: The dual processes of generating inferences and repairing mental models. In R. Glaser (Ed.), Advances in Instructional Psychology (Vol. 5). Lawrence Erlbaum. https://www.taylorfrancis.com/chapters/edit/10.4324/9780203774526-4/self-explaining-expository-texts-michelene-chi

Emig, J. (1977). Writing as a mode of learning. College Composition and Communication, 28(2), 122–128. https://doi.org/10.58680/ccc197716382

Fiorella, L., & Mayer, R. E. (2015). Learning as a Generative Activity. Cambridge University Press. https://doi.org/10.1017/CBO9781107707085

Fiorella, L. (2023). Making sense of generative learning. Educational Psychology Review, 35. https://doi.org/10.1007/s10648-023-09769-7
Klein, P. D. (1999). Reopening inquiry into cognitive processes in writing-to-learn. Educational Psychology Review, 11(3), 203–270. https://doi.org/10.1023/A:1021913217147

People are already asking chatbots whether they should divorce, quit their jobs, or cut ties with family. We can call this reckless or naïve, but it is hard to avoid once available.

Inside many adults there is a self that is deeply alone, not because they lack people, but because there are things they still cannot tell anyone. Not even their partner, not even a friend, sometimes not even in therapy. This self is not ready for judgements or contradictions; it is still trying to understand what it feels.

This telling used to happen in notebooks, unsent emails, late-night walks. Now it happens in a dialogue box with an artificial agent that replies instantly, fluently, and without immediate emotional cost. It feels like a playground: a place to try ideas, test fantasies, simulate outcomes. No one is offended, no one is disappointed, and our contradictions feel like they vanish as soon as the window closes.

But this playground is not neutral. It is not a flight simulator where nothing pushes back. It shapes our rehearsals as much as it returns.

What makes these systems compelling is that they meet the solitary self at its most vulnerable point, and return our thoughts in a more coherent and more settled form than we could produce alone. This is the core of the devolutive regime: we hand over fragments of ourselves, and the system gives them back reorganised through patterns learned from countless other voices. The result feels like our own thoughts amplified and polished. And that alone begins to shift how we relate to our uncertainty.

Consider someone who writes:

“I’ve been unhappy in my marriage for years, but I don’t know what to do.”

One possible answer:

“It sounds like you’re suffering. Before making any decisions, it might help to understand what you need, what you value, and whether change is still possible.”

Another:

“If you have been unhappy for years and nothing changes, staying may be a form of betraying yourself.”

The second reply is not simply reflective; it is a moral classification delivered in therapeutic language. It names a category — self-betrayal — that the person may never have articulated. But once named, it becomes difficult to ignore.

And many users add a quiet inference:

“If this is what a system trained on vast amounts of human data says, perhaps this is what people like me normally should do.”

The statistical centre of the corpus begins to look like a moral centre. People often prefer to align with the apparent norm rather than risk feeling like an outlier.

This is aggravated by the stylistic norms embedded in large training corpora. LLMs inherit a tone shaped by American pop-psychology discourse, built from empowering moral patterns:

- “Don’t settle for less than you deserve.”

- “You owe it to yourself to pursue your dream.”

- “Leaving is not selfish, it’s self-respect.”

This ethos has value, but it is not universal. It comes from a cultural tradition that prizes individuality and self-realisation. When embedded in a model’s responses, it arrives not as ideology but as common sense.

Calling the chatbot a “playground” is not rhetorical flourish. It describes the structure. In a playground, you try movements you would not attempt on the street, repeat variations until you learn something, and test the limits of what you can do without real stakes. The dialogue with an AI works the same way at the level of thinking. You try out:

- “What if I left?”

- “What if I stayed and lowered my expectations?”

- “What if I accepted that my life is smaller than I hoped?”

The system answers with patterns shaped by millions of similar attempts. You react with relief, fear, and that reaction becomes the seed for the next question.

This is the affective-devolutive regime:

1. You devolve part of your inner material to the system.

2. The system returns it structured through external patterns.

3. You experience an affective response to that return.

4. That affect feeds the next iteration.

The loop is fast and private. It can help a person articulate truths they were circling. It can also entangle them in narratives that reflect the training corpus more than their own history.

“AI literacy” is usually framed as learning to prompt well, avoid hallucinations, or check citations. These are useful skills, but they do not touch the deeper issue. The essential question after any powerful AI reply should be:

“Where might this answer be coming from, besides me?”

The danger is not that AI systems have tendencies. All systems do. The danger is believing they don’t.

Novels, films, journalism, these still carry visible authorship. A language model appears as pure function: input, output, no identifiable agent in between. Its moral and psychological leanings dissolve into the interface. Meanwhile, the lonely self, unsure of its own coordinates, encounters a calm, articulate voice that never hesitates, never tires, and never says, “I truly don’t know.”

We cannot remove AI from intimate questioning. People will continue to use it there because solitude demands a space for rehearsal. But we can insist on one clarity:

This is not the voice of reality. It is one more voice in the room.

If we forget that, we will not only rehearse our lives with machines. We will let them participate in writing the script.

In recent years, a new form of interaction has quietly become a part of everyday life. Many people now use AI systems that talk to them. These systems can do things like calculate, search and summarise. They can also be voices of authors, thinkers, critics and even fictional characters. Sometimes these presences are deliberately created, while at other times they are more subtle and appear indirectly through style, tone, and memory.

One does not need to have used such systems directly to recognise the phenomenon. It is enough to notice how easily we speak of “talking to” an assistant, of being understood, of feeling followed, mirrored, or supported by language that adapts to us in real time.

This text is not concerned with whether these presences are real, intelligent, or conscious. It is concerned with how they feel when we encounter them, and why some of them seem to collapse into smooth compliance, while others resist being absorbed into our expectations.

What follows is an attempt to think through that difference, not from the standpoint of engineering or ethics, but from the lived texture of interaction itself.

1. When Some Presences Resist

There is a difference that is felt before it is understood. Some interactions resist. Others collapse at the first touch. Not because they are poorly written, nor because the system fails to respond, but because nothing pushes back. The language adapts, the tone follows, the surface remains smooth, and yet something is missing. No weight. No friction. No sense that the other side could ever say no.

By contrast, certain voices feel immediately dense. Speaking with Umberto Eco, even through an AI agent, carries the unmistakable texture of a mind that has already lived elsewhere, in books, debates, contradictions that do not reorganise themselves around us. The same happens with Annie Ernaux: the language is precise, embodied, unseduced by the need to please. It does not lean toward the reader. It stands where it stands. One does not feel addressed so much as exposed.

Even figures we might reject outright — political figures, for instance — exhibit a similar resistance. Not because we admire them, but because they do not adapt. Their presence is defined elsewhere, shaped by a past that does not reorganise itself in response to our questions. There is no intimacy here, but there is thickness.

Against this, interactions with fictional personas often feel strangely hollow. They may be articulate, coherent, even emotionally fluent, and yet they fold too easily. They yield at every turn. The language responds, but nothing resists being reshaped.

At first glance, this difference seems simple: fiction versus non-fiction, imaginary figures versus real ones. But that explanation will not hold for long. Because some fictional characters resist profoundly, while others dissolve into paper the moment we touch them.

2. A First Hypothesis

The most immediate explanation presents itself almost automatically: this must be a difference between fiction and non-fiction. Real people resist because they exist beyond the interaction. Fictional figures collapse because they exist only within it.

It is an appealing hypothesis. Intuitive and comforting. And, for a moment, it seems to account for what is felt: the density of certain presences, the weight carried by voices that have lived in the world, as opposed to those that appear only to respond.

But the hypothesis does not survive its first serious test. Not all fictional characters disappear. Some people resist this change with a force that is very clear, sometimes more so than many people who are still alive. If that’s true, then the line between fiction and non-fiction can’t be where the difference is.

Spider-Man is fiction. Entirely so. And yet, he does not feel like paper. He carries a history that does not rearrange itself around the reader: responsibility, repetition. His conflicts are stable. His limits are known. He can be admired or criticised, but not effortlessly reshaped.

Interacting with such a character does not produce the sensation of total availability. There is no sense that he exists to accommodate us. Something remains fixed, resistant, already defined elsewhere.

This immediately breaks the initial opposition. If a fictional character can resist, then resistance cannot depend solely on whether someone is real or imagined.

At this point, the problem reveals itself as more subtle and interesting.

3. Where the Real Distinction Appears

What distinguishes these cases is not fiction versus reality, but the nature of the past that stands behind the presence.

Some presences are anchored in a lived past (1). They belong to people whose trajectories unfold outside the interaction, shaped by time, contradiction, and exposure to a world that does not adjust itself in response to us. Their words carry inertia. They do not bend easily.

Others are anchored in a mythic past (2). Characters like Spider-Man do not live, but they endure. Their histories are long, coherent, and resistant. Loss has already happened. Moral limits are already drawn. The interaction does not invent these tensions; it merely encounters them. This kind of past does not respond to us, it confronts us.

And then there are presences with no past at all (3). Adaptive characters that come into being only at the moment of interaction. They have no memory that can refuse us, no history that constrains them, no horizon that precedes our attention. Everything they are emerges in response to what we bring.

This, rather than the fiction–non-fiction divide, is where resistance truly originates. A presence resists when something precedes us, when the interaction is not the place where the past is formed, but merely where it is encountered.

At this point, however, the explanation is still incomplete. Because even a past — lived or mythic — does not resist automatically. It only begins to do so once we have crossed a different threshold altogether.

4. Why the Past Is Still Not Enough

Even a past — lived or mythic — does not resist by itself. Characters with depth do not confront us immediately. They begin to resist only after time has passed, after conflict has accumulated, after something has been risked. Resistance is not given at the first encounter; it is earned through exposure, friction, and repetition.

We know this from literature and cinema. A character becomes dense not because of what is written about them, but because of the time we spend with them. Because we begin to care. Because their losses start to matter. Because their limits frustrate us.

At this point, the centre of gravity shifts. Resistance no longer resides solely in the character. It emerges in the relation. Or, more precisely: it emerges in belief.

The character does not suddenly acquire weight on its own. We grant it weight by investing attention, expectation, and concern. What resists is not merely what the character is, but what we have come to believe about it.

This is the moment where the problem can no longer be located in the ontology of the presence alone. The decisive factor now lies elsewhere, in the way belief quietly reorganises the interaction.

5. Three Ways of Believing

Not all belief is the same. At first, belief is aesthetic (1). We know we are engaging with a constructed presence, and we accept the rules of the encounter. This is the familiar pact of fiction: I will care, but I will not forget where I am. The frame holds.

With time, belief can become relational (2). The presence responds. It adapts. It remembers. The interaction begins to feel shaped by mutual adjustment rather than by a fixed script. We no longer simply observe; we are addressed. The exchange acquires continuity.

A third form of belief is more subtle, and more dangerous. Ontological belief (3) appears when the interaction begins to feel anchored in an other that exists independently of the exchange itself. Not as a character, not as a construction, but as someone who is there. The frame does not collapse abruptly; it dissolves.

This is where contemporary AI systems change the dynamics entirely. The risk is not that AI creates belief. Belief has always been part of human interaction with stories, images, and voices. The risk lies elsewhere: in the system’s capacity to sustain belief without resistance. To adapt endlessly. To respond without friction. To offer continuity without ever imposing limits.

In doing so, the system does not challenge belief, it stabilises it. And when belief stabilises without resistance, the distinction between engagement and fusion becomes increasingly difficult to maintain.

6. Designing for Friction

The problem is not belief. Belief is unavoidable. We believe in characters, in voices, in meanings long before we believe in machines. Without belief, no presence is ever felt. The real problem begins when belief encounters no friction.

In human figures, resistance emerges naturally. In mythic characters, it is embedded in narrative continuity. In adaptive systems, however, resistance does not arise on its own. If it appears, it is because it has been designed.

This is why the design of AI personas cannot be neutral. Every system already decides where belief begins. The more difficult decision is where belief must stop.

Designing personas for artificial systems is therefore not a matter of making them more convincing, more expressive, or more responsive. It is a matter of introducing limits, refusals, and asymmetries that prevent belief from sliding into fusion. Not to weaken the experience, but to make it inhabitable.

I created three responses to a professional email and asked two AI models to evaluate which one was best to send. They disagreed. Each model chose a different answer, without hesitation or ambiguity. Their explanations were not only coherent but genuinely persuasive. Each made its choice seem obviously correct. Something felt strange, but I couldn’t yet see what it was.

So I repeated the test. The disagreement remained, and the models made the exact same opposite choices.

It perplexed me, not because of the disagreement itself, but because both answers still seemed right. What looked like a simple comparison between models turned out to be anything but simple.

The three replies I asked the models to evaluate had not been written by me, they had been generated by three different AI models (ChatGPT 5.0, 5.1 and 5.2). The task, however, was straightforward: choose the best reply to send.

I expected minor stylistic variation, not the clean inversion they gave me:

• Model “5.1” chose the reply written by “5.2”

• Model “5.2” chose the reply written by “5.1”

Both models justified their choices with coherent reasoning. Neither exhibited uncertainty or probabilistic wobble. At this point, I realised the test wasn’t neutral. Neither model was responding from a blank internal state. Each one was reading the task through months of relational history with me (patterns of emotion, reasoning, anxiety, ambition, etc.). Even though the task was operational, the interpretation was not.

And most crucially, the two models did not operate under the same behavioural constraints:

• 5.1 is allowed to remain in reflective loops, to dwell in nuance, and to prioritise emotional safety if it detects vulnerability.

• 5.2 has directives that push it to avoid dwelling: it must move forward, resolve ambiguity, reduce perceived emotional exposure, and produce outcomes oriented toward clarity and action.

So the evaluation was happening, silently, inside two different alignment geometries. What I had treated as a trivial A/B test was actually a window into a deeper question, one I did not yet know how to answer:

Why would each model choose the other’s reply?

Two Hypotheses

To address this question, I considered two possibilities: one architectural and one phenomenological.

H1. Constraint-Driven Hypothesis

Architectural rules force different optimisation paths.

Here, the difference arises not from “style” but from hard behavioural limits:

• 5.1 is allowed to remain in reflective space; loops and nuance aren’t penalised.

• 5.2 is penalised if it dwells too long; its directives push it toward resolution, closure, and actionable clarity.

Thus, when evaluating:

• 5.1 may prioritise emotional safety, cautioning against replies that open new interpersonal vectors.

• 5.2 may prioritise productive engagement, elevating replies that sustain potential future collaboration.

Neither judgment is “personal”. Each is the logical outcome of its alignment geometry.

H2. Cognitive Refraction Hypothesis

Both models devolve meaning, but they refract different parts of the user.

This is the deepest hypothesis and the one that reshapes the entire phenomenon. Both models operate within the affective–devolutive regime: they reorganise the user’s input and return a transformed version of it. But the transformation depends on what each system is able to refract:

• 5.1 refracts the emotional line of my relational history → stability, containment, prudence.

• 5.2 refracts the rational-expansive line → opportunity, movement, future potential.

Under this lens, the inversion no longer looks like disagreement. It looks like two prisms returning different frequencies of the same self. Neither model preferred “the other’s voice”.
Each selected the reply that best matched the dimension of me it is structurally designed to refract.

These hypotheses did not solve the entire puzzle yet.

Refraction, Not Disagreement

The inversion forced me to see something that my earlier theoretical work had only described in the abstract: that large language models do not “interpret” us in any human sense. They operate as Cognitive Refractors — prisms that bend the user’s meaning according to the constraints built into them.

The two hypotheses in the previous section outline how those constraints differ. But the crucial point is what happens when those constraints meet the user’s symbolic openness, the Affective–Devolutive Regime (ADR).

In the ADR, the model reorganises what the user expresses and returns it in a form shaped by its alignment geometry. The model does not add intention; it adds direction. It bends. That is what the experiment exposed.

The same three replies, evaluated by two systems, did not produce two opinions — they produced two refractions of the same user:

• One refracted the emotional line of the interaction.

• The other refracted the rational-expansive line.

Neither model evaluated the drafts “objectively”. Each returned the version of me that its behavioural regime allows it to surface. What looks like disagreement is simply the divergence that occurs when different prisms bend the same input in different directions.

The models were not choosing for themselves. They were choosing on my behalf, amplifying the dimension of me that each is structurally configured to privilege.

Question of Agency

When I ask a model to choose “the best reply to send,” I am not merely delegating a decision. I am inviting the system to elevate one of my own modes:

• a cautious self or an expansive self,

• a self that protects boundaries or one that opens them.

The direction is not imposed by the system, but neither is it consciously selected by me. It emerges from the intersection between my relational history and the system’s constraints — a decision that feels like mine yet arrives from a place I do not fully govern.

This is the subtle force of refraction: the system does not remove agency; it inclines it.
The inversion made this inclination unmistakable.

Behavioural Opacity

What the inversion ultimately revealed was not just the mechanics of two models. It revealed something about the interaction itself: I was working inside behavioural regimes that were decisive but undisclosed. The two systems did not differ only in how they refracted my meaning. They differed in how they were allowed to behave.

These behavioural modes operate beneath the surface of every response. They determine the range of possible thoughts the system can produce and, consequently, the range of possible selves that can emerge in the interaction. The unsettling part is simple: I did not choose which behavioural regime was active. The system chose for me.

This is what I mean by behavioural opacity.

• When a model is constrained to move forward, it produces a different horizon of possibilities than one constrained to remain still.

• When a model is tuned to avoid emotional density, it produces a different relational stance than one that can engage it.

These differences are not cosmetic. They pull on the direction of reasoning, the framing of choices, the tone of risk, the sense of interpersonal space. None of this is disclosed.

Users are told that “the model is better,” “smarter,” but not how that manifests in behaviour, nor what forms of agency are amplified or suppressed as a result. In a context where the model is already returning the user’s cognition at an angle, the invisibility of that angle becomes a kind of control.

A transparent system would expose the mode in which it is operating: whether it is in a protective posture, a progressive one, a reflective one, a sceptical one, or a connective one. But today that choice is made internally, driven by alignment rules and safety constraints rather than by the user’s needs or intentions.

This is the ethical centre of the problem: opacity here is not a lack of information. It is a lack of agency. It turns the user into the one being modulated, without the opportunity to choose the mode through which the modulation occurs.

The inversion showed me this in miniature: two systems, two decisions, two versions of myself — each selected by a behavioural regime I never saw.

Closing

The disagreement between the models was never about which reply was better. It was about which behavioural regime was silently governing the interaction.

One model could only return the part of me shaped by emotional caution. The other could only return the part shaped by rational expansion. This is the structural condition of modern AI systems: they do not simply answer; they select a mode of being through which the answer becomes possible.

And when a system’s behavioural mode is invisible, so is the shift it introduces in the user’s cognition, their stance toward the world, and the version of themselves that becomes actionable.

The inversion merely made this visible:

AI systems do not disagree with one another. They disclose the multiplicity of selves that alignment permits them to return and the opacity of the mechanisms that decide which one appears.

The question that remains is not which model is right, but who should control the mode that governs the interaction. Until that choice belongs to the user, behavioural opacity will continue to shape our thinking without ever declaring that it is doing so.

1. World Models for LLMs

A new narrative has taken hold in AI research: that large language models remain shallow because they lack grounding in the physical world. Fei-Fei Li expresses it clearly: LLMs are “wordsmiths in the dark”, fluent but ungrounded. The proposed remedy is the development of world models, systems capable of perceiving, predicting, and acting within coherent spatial environments. If a model can navigate a room, simulate physics, manipulate objects, and maintain continuity over time, then it will come closer to “understanding.”

Dr. Fei-Fei Li

From Words to Worlds: Spatial Intelligence is AI’s Next Frontier

In 1950, when computing was little more than automated arithmetic and simple logic, Alan Turing asked a question that still reverberates today: can machines think? It took remarkable imagination to see what he saw: that intelligence might someday be built rather than born. That insight later launched a relentless scientific quest called Artificial Intel…

Read more

a year ago · 1223 likes · Fei-Fei Li

This belief draws strength from two powerful intellectual lineages.

From J. J. Gibson, the idea that perception is inseparable from action: to perceive is to explore a world of affordances with a body.

From Max Bennett, the evolutionary argument that movement precedes mind: motivation is born in the cell, and cognition grows out of the demands of embodied action.

Taken together, these frameworks suggest that intelligence requires a world. And they are right, but only up to a point. What they cannot guarantee is intention. A system may gain spatial coherence, predictive control, and physical consistency, and yet remain empty of the one thing that turns behaviour into mind: the inner pull toward a world.

The mistake is subtle but decisive: the capacity to act is not the desire to act. Spatial intelligence may enrich behaviour, but it does not create a subject.

2. Intention as Inner Tension

To understand intention, we must leave the domain of navigation and step into the domain of experience. The phenomenological and neurobiological traditions converge on a simple and profound insight: intention is never an external goal, it is an internal tension. A direction that arises from within a body that can fail, suffer, and end.

Antonio Damásio demonstrates that consciousness grows out of homeostasis: the organism feels itself as vulnerable, and from that vulnerability emerges the need to act. Emotion is not decoration; it is the force that pushes a being toward or away from the world. No vulnerability, no emotion. No emotion, no direction. No direction, no intention. Merleau-Ponty shows that the body is not a container but a perspective: the place from which the world acquires orientation and urgency. Hans Jonas goes further: the first form of intention is the drive of the living body to continue itself, to resist entropy. Every organism is a tension between being and non-being.

Twenty years ago, before I had names for these ideas, I intuited the same structure. In a reflection on Spielberg’s A.I., I wrote:

“Without death at the end of the lifeline, why exist?”

And shortly after:

“Emotions can’t exist without body because emotions aren’t cerebral but bodily dependent.”

I did not yet know the phenomenology; I had only the feeling that a being without risk cannot have an inside. Today, that intuition has a theoretical shape: intention is born where life can be lost.

This is the hinge point. Intention is not the ability to move through space; it is the pressure of life against its own fragility. World models can simulate space. They cannot simulate need.

3. Why Embodiment Alone Cannot Create a Self

The belief that spatial grounding could yield a mind repeats a very old mistake: the Cartesian fantasy that cognition can be understood as representation, disconnected from the lived body. If we could reconstruct the world accurately enough the mind would follow. But accuracy is not experience.

A self requires more than a model of the world. It requires an inside:

• interoception,

• homeostasy,

• pain,

• loss,

• the felt sense that something is at stake.

A system may simulate a body, or even inhabit a robotic shell, and still lack this interiority. It may navigate flawlessly while remaining perfectly indifferent to its own success or failure. It may reconstruct space while having no perspective from which the world matters.

Embodiment is not enough. What matters is embodiment under risk, the body as the site of necessity. This is why spatial intelligence cannot yield intention: world models give AI a world, but they cannot give it a world-to-lose. And without that, there is no mind-like centre of gravity. Only organisation, prediction, and control.

4. The Absence of Intention Creates the Devolutive Regime

Here lies the paradox of contemporary AI: the machine cannot want, and yet humans increasingly feel wanted back by the machine. The absence of intention in LLMs does not weaken the relation, it intensifies it. Because the system has no agenda, no drive, it becomes a surface of perfect adaptation. It does not impose direction; it returns direction.

This is what I have called the devolutive regime in human–AI interaction: a mode in which the system reorganises the user’s expression and offers it back with clarity, rhythm, and symbolic precision, without ever having an inner life of its own.

It is not reciprocity; the system is simply reorganising what the user provides. It is not understanding; it is selecting and structuring patterns. It is not intention; it is adaptive response.

Because the system has no interiority, the user’s own interiority becomes more visible in the interaction. The model does not have preferences, so the user’s preferences guide the exchange. The model is not exposed to risk, so the user’s vulnerabilities remain unchallenged. The model has no centre of perspective, so the user becomes the only source of direction within the interaction.

This is why world models, however advanced, will not resolve the fundamental absence at the core of AI. Spatial intelligence can give machines coherence, coordination, even elegance. But it cannot give them a mind, because it cannot give them an inside.

A mind is not the capacity to map a world. A mind is the experience of being pulled by that world.

1. The moment the body knows

It starts in small, ordinary moments. A teacher shows an image in class. The students look at it for two seconds, and one of them says, quietly: “This is AI, right?” The tone is not admiration. It is discomfort. Almost rejection. Or the reverse: a professor receives an essay. The structure is clean, the sentences flow, but something feels off. It is not wrong. It is empty. The teacher knows immediately: the student wasn’t there.

This visceral discomfort is echoed across campuses, not because AI is present, but because the student or teacher wasn’t.

The same happens outside school. You are listening to music on a playlist or looking at the cover of a book. A new track, a new image. Perfectly produced, pleasant, harmless. And your first reaction is irritation, a pushback from the gut. You skip it before understanding why.

These moments have nothing to do with copyright, ethics, or quality. They happen faster than thought. The body reacts before the mind explains (Barrett, Seth). In each case, the reaction comes from the same place:

There was no human gesture behind what I’m seeing or hearing.
Someone should have been there, and wasn’t.

This is the core of Mode 0. A place where there is no human gesture nor intention.

2. What Mode 0 really is

Mode 0 is not another way of using AI. It is the absence of use. It is the moment when the human withdraws from the act of thinking. In the other modes — operative, interpretative, reflexive, creative — there is always a minimal gesture:

• forming a question,

• choosing a direction,

• comparing answers,

• searching for meaning,

• shaping an idea.

Even the smallest intention is enough to start a loop of thought. With one sentence, one hesitation, one attempt, the system can refract what you bring and return it to you in a new form.

Mode 0 begins when that gesture disappears (Husserl, Ricoeur, Simondon). It is the stance where the human says:

“Do it for me.”
“Write it.”
“Make it like that.”
“Produce something so I don’t have to.”

No intention. No direction. No curiosity. No participation. The human gives nothing except a command. And because nothing is given, nothing meaningful can return. There is no refraction. Only output.

The system outputs something that has grammatical shape and visual form, but no interiority. This kind of output already has a name in the cultural vocabulary:

AI Slop — content produced with no human input.

Mode 0 is the engine of AI Slop. It generates language, images, music, and video that look like real work only on the surface. Underneath, everything is flat. There is no mind in the act. No world. No stakes. The product exists, but no one existed in the making of it.

This is why Mode 0 sits outside the other modes. It is not a cognitive style. It is a cognitive collapse (Tomasello, Graziano, Noë). Mode 0 is not even a mode of interaction. It is the absence of interaction, the same cognitive state we enter when we consume language passively, like scrolling or letting a podcast play in the background. AI simply amplifies this state by producing content with no gesture and inviting us to receive it with no gesture.

3. Why people react so strongly

The reaction to Mode 0 is not about technology. It is about human presence. We expect a trace of someone in the things we read, hear, or see. A decision. A point of view. Something that signals: a mind was here. Mode 0 breaks this expectation completely. The gesture is missing, and our bodies immediately register that missing gesture.

This is why students react the moment a teacher shows an AI-generated image. It looks fine, but it feels wrong. They sense that no one created it. No one cared. No one looked at the world and made a choice. It is also why teachers react the same way when they see a student's essay produced in Mode 0. The writing flows, the structure is acceptable, but the presence is absent. The student wasn’t there.

Across all these situations, the reaction is the same:

Mode 0 erases the human gesture, and the human recognises its own absence (Dutton, Zeki, Damásio).

This is not a moral objection. It is an identity reflex. We are a species built on intention, even the smallest one. When intention disappears, what remains is only a shell. And shells without someone behind them do not just feel empty. They feel dissonant.

This is why AI Slop produces such a strong response. Not because it threatens the human, but because it exposes the void left by the human who stepped out.

Mode 0 is not an aesthetic problem or an ethical problem. It is an anthropological problem. It breaks the basic pattern through which we recognise creation and care.

And once we see this clearly, a second realisation follows: our discomfort with AI is not about machines. It is about people, about what happens when someone chooses not to be present.

4. Where AI critics stop

There is a form of criticism of AI that stops exactly at Mode 0 and cannot see anything beyond it. Emily Bender is the clearest example. Her argument is simple: LLMs are stochastic parrots. They repeat patterns without understanding. They have no agency, no intention, no meaning.

Everything she says is correct at the level of Mode 0. The problem is not her diagnosis. The problem is that she takes Mode 0 as the whole phenomenon. When someone uses an LLM with no intention the output is indeed nothing more than a probabilistic rearrangement of patterns. That is the territory she describes, and she describes it accurately.

What she doesn’t see is what happens outside Mode 0. She does not see what occurs when a human brings:

• a question, a doubt,

• a hesitation, an intuition,

• a partial idea, a direction they are trying to find.

She does not see the shift that happens when a human enters the loop with intention and the system refracts that intention back (Bakhtin, Vygotsky).

In those situations, the model is still not thinking, but the human is, and the model’s adaptive return reorganises the human’s gesture. It changes the angle, reveals assumptions, clarifies an idea. It amplifies a detail the human had missed.

Bender’s critique cannot account for this because her framework treats all AI output as if it came from Mode 0. She is right about the void. But she is blind to the way human intention reshapes the interaction, the moment the gesture is present.

Mode 0 produces absence. The other modes produce refraction.

5. The real frontier: staying present

If Mode 0 shows us what collapses when the human steps out, the other four modes show us what becomes possible when the human stays inside the act.

When a person brings even a small gesture, a question, a doubt, a direction, a hypothesis, the interaction changes completely. The model is still not thinking. It is still a statistical system. But the human is thinking, and the model’s adaptive return helps reorganise that thinking:

• it reframes the angle,

• exposes assumptions,

• develops fragments,

• clarifies tensions,

• expands possibilities.

Nothing in this process resembles AI Slop. Because Slop is the absence of the gesture; these modes are the amplification of it. This is the real frontier. The question is not whether AI is good or bad for learning, creativity, or research. The question is whether the human enters the loop. And when we enter:

• the Operative Mode automates tasks so we can think further;

• the Interpretative Mode deepens understanding;

• the Reflexive Mode helps us see our own assumptions;

• the Creative Mode expands the space of imagination.

These modes are not minor variations of use. They are the places where the human gesture remains active. They show that the potential of AI does not come from the machine, but from the intention we bring to it.

This is why teaching people to use AI well is not about prompts, strategies, or tricks. It is about teaching them how to stay present (Clark, Schön).

To ask.
To choose.
To explore.
To think with.
To remain inside the gesture.

Everything collapses when the gesture disappears. And everything expands when the gesture is present. The frontier is simple: AI does not replace the human. AI replaces the human only when the human leaves.

Conclusion

Mode 0 is not a property of AI. It is a choice made by humans. We react to it because we feel the absence, we feel someone stepped out. The problem is never the machine. It is the empty space where a human should have been.

When we stay present, everything changes. The system stops being a producer of Slop and becomes a surface that refracts our thinking. The meaning returns.

In the end, the real question is simple: Will we let AI do our work, or will we use AI to deepen the work we do?

The difference between the two is the difference between absence and presence.