Somewhere along the way, we’ve developed a habit of assuming that making something more efficient automatically makes it better. Efficiency tells us how effectively we’re using resources to accomplish something. It doesn’t tell us whether that something is worth accomplishing in the first place.

We’ve become very good at measuring things

Efficiency is relatively easy to quantify. We can count how many customers an employee serves, how many applications a department processes, how many products a factory produces or how many tasks a software system completes. Managers can establish targets, compare performance and identify areas where resources might be used more effectively. The problem starts when measurable activity becomes a substitute for results.

Let’s look at customer service. Management decides that customers are waiting too long for assistance, so it introduces a target to reduce the average length of each call. Employees are encouraged to resolve enquiries more quickly. Supervisors monitor call times, software tracks performance, and bonuses may be tied to meeting the new targets.

After a few months, the department reports a significant improvement. Average call times have fallen. Except there’s a question nobody may have asked. Were the customers’ problems actually resolved?

If shorter conversations result in more repeat calls, unresolved complaints or customers giving up entirely, the department may have improved its efficiency while making the service worse. And that deterioration might not even appear in the report.

The difference between the measurement and the objective

Economists, social scientists and management researchers have spent decades examining what happens when organisations overly rely on performance indicators.

One principle is Goodhart’s law, expressed as the idea that when a measure becomes a target, it stops being a good measure. A measurement may initially provide useful information about how well something is working. But once people are rewarded or penalised according to that measurement, they begin changing their behaviour to improve the score.

For example a school wants students to develop knowledge, reasoning abilities, curiosity and the capacity to apply what they’ve learnt. But they’re difficult to capture in a single number. Standardised tests are one way of measuring aspects of academic performance. Schools begin paying attention to test results.

But if those results become the primary measure of success, schools may increasingly organise instruction around improving them. Students become better prepared for the assessments. Teachers become more familiar with the format. Administrators can demonstrate measurable progress. None of that tells us whether students have become better thinkers. Have they? Or did they just better at taking the tests.

Faster isn’t always better

Modern computing has allowed us to perform calculations that once needed enormous amounts of time and effort. Medical imaging technologies can produce detailed information that would have been unimaginable a century ago. Communication systems allow people on opposite sides of the planet to collaborate almost instantly. These are genuine advances. But the relationship between speed and quality isn’t universal.

A healthcare organisation might introduce a system that allows physicians to see more patients during the day. From an administrative perspective, the improvement is obvious. More appointments. Shorter waiting lists. Better utilisation of available resources. Those could be valuable outcomes, provided the quality of care is maintained. The organisation may be treating more patients without necessarily providing better care.

Appointment numbers are easy to count. The consequences of a missed detail, an incomplete explanation or a patient who doesn’t feel comfortable raising a concern can be much harder to measure. Efficiency can create additional capacity. But it doesn’t tell us how that capacity should be used.

The economics of saving time

We’re told that technology will give us more free time. Tasks that took hours can now be done in minutes. Information that once demanded a trip to the library is available almost instantly. Software automates calculations, scheduling, communications and countless administrative activities.

So why is it people don’t seem to feel particularly liberated? Expectations.

An email that once might have taken several days to arrive now creates an expectation of a response within hours, sometimes minutes. Documents can be produced more quickly, so more documents are expected. Reports that once required substantial preparation can now be generated automatically, so organisations request more reports. Technology reduces the effort needed to perform individual tasks, but the number of tasks has increased.

We developed tools to reduce the amount of work required, then reorganised our expectations around the assumption that everyone should accomplish more.

AI is making that harder to ignore

We’re seeing systems capable of generating documents, analysing information, writing software, assisting with research and performing tasks that previously required considerable human effort. Some of these capabilities are impressive.

I’ve used AI extensively, and I see enormous value in its ability to help people examine ideas, explore unfamiliar subjects and approach problems from different directions. But I also think we’re asking the wrong questions about its adoption.

Suppose an organisation introduces AI to generate internal reports. If, in the past, employees spent several hours gathering information, examining discrepancies, considering what the numbers meant and preparing a summary for management. AI system can produce the report in minutes. Impressive.

But what happened to the analysis? If employees previously spent time thinking about the information while preparing the report, that process may have contributed to their understanding of the business. Automating the document doesn’t mean understanding.

AI can help people think. It can also make it easier to avoid thinking.

Sometimes inefficiency has a purpose

This may sound contradictory, but not every activity that looks inefficient is wasteful. Some things take time because the time itself contributes to the result. Learning is an obvious example.

A student struggling with a difficult mathematical problem isn’t necessarily wasting time by working through it. The struggle may be part of developing the understanding needed to solve similar problems independently. Give the student an immediate answer, and the task gets completed more efficiently.

But has the student learnt anything? Possibly, if the answer includes an explanation they work through and understand. Not if they simply copy it and move on. The value isn’t solely in obtaining the correct answer. It’s also in developing the ability to arrive at one.

The same principle applies to scientific research, creative work and complicated decision-making.

Exploring an idea that ultimately proves incorrect still reveals useful information. A discussion that takes longer than expected may uncover a problem nobody previously recognised. A designer might throw away several approaches before discovering one that works.

Some of that effort could look wasted. But if we remove every activity that doesn’t produce an immediate measurable result, and we risk removing much of the experimentation, reflection and learning that make progress possible.

When organisations optimise the wrong thing

Maybe the larger problem is that efficiency encourages us to concentrate on individual components rather than the systems they belong to.

• A department is instructed to reduce costs.

• It succeeds.

• Another department is asked to increase output.

• It succeeds too.

Both report improvements against their objectives.

Cost reductions in one area might create additional work somewhere else. Faster production might increase defects. Reduced staffing might shift responsibilities onto employees who are already overloaded. Each department can demonstrate that it met its targets. The combined result may be different. Improving the performance of individual parts doesn’t necessarily improve the performance of the whole. The way we’ve divided responsibilities makes those wider consequences difficult to see.

An organisation can become extremely efficient at executing its existing strategy while failing to recognise that the strategy itself needs to change. Which brings us back to the assumptions underlying the system. Before optimising a process, we should look at whether it’s still the right process after the fact? Shouldn’t we agree on what progress actually means?

Progress needs a destination

Movement is relatively easy to show. We can produce more, process faster, reduce costs and increase output. Progress is something else. It needs a sense of direction.

• If a school improves its test scores, we need to ask what that tells us about the education students are receiving.

• If a hospital increases the number of patients it treats, we need to consider the outcomes of that treatment.

• If a business improves productivity, we need to understand whether it’s producing greater value or simply increasing the workload.

And if AI allows us to accomplish in minutes what once took hours, we should probably consider what we’ve gained beyond those saved minutes.

None of this means efficiency is unimportant.

Quite the opposite.

When we understand what we’re trying to accomplish, efficiency can help us accomplish it with fewer resources, less waste and greater accessibility.

But efficiency is a means.

It isn’t the objective.

We can become extraordinarily good at doing something that doesn’t need to be done, or at least doesn’t need to be done in quite the way we’ve inherited.

And because the improvements are measurable, we may not even notice.

Maybe we should measure what happens afterwards

I wonder whether one of the more useful changes we could make is to spend less time celebrating improvements in activity and more time examining their consequences.

• A new system processes applications twice as quickly.

  • What happens to the people submitting those applications?

• An AI tool allows employees to produce reports in a fraction of the time.

  • Are the decisions based on those reports any better?

• A school introduces technology that improves measurable academic performance.

  • What do students understand that they didn’t understand before?

These questions aren’t always easy to answer.

They may require longer periods of observation, more complicated evidence and an acceptance that some important outcomes can’t be captured neatly in quarterly performance reports. Difficulty measuring something doesn’t make it unimportant. Nor does the availability of a convenient metric make that metric meaningful.

In the previous article, I examined how rules can survive long after the circumstances that justified them have changed.

Efficiency allows us to improve those inherited systems without necessarily examining whether they still serve their intended purpose. We can make an outdated procedure faster, an inadequate measurement more precise or an unnecessary task almost effortless. And congratulate ourselves on the improvement.

Subscribe now

One by one, the researchers supposedly replaced the original monkeys with new ones. Each newcomer tried to climb the ladder, only to be attacked by the others. Eventually, none of the monkeys in the room had ever experienced the cold water, but they continued preventing anyone from climbing. Nobody knew why. They just knew you weren’t supposed to climb the ladder.

It’s a great example of institutional behaviour, although there’s one rather important problem with it. There’s no credible evidence that the experiment actually happened.

Which, when you think about it, makes the story even more appropriate. We’ve been repeating a story about blindly following established behaviour without questioning its origins of the story itself. There are variations of the story, of course, water, electric shock or some other punishment.

Rules usually make sense at the beginning

Most organisational rules aren’t created because somebody woke up one morning and decided to make everyone else’s life difficult. Although anyone who’s worked in a large organisation might think otherwise. Usually, something happened.

A mistake was made. A customer complained. Someone exploited a loophole. A regulator introduced a requirement. An employee made a decision that management didn’t particularly appreciate. So the organisation responded. A new approval was needed. An extra form was introduced. Another person needed to sign off. A reporting procedure was established to prevent the same problem from happening again. Reasonable? More than likely. Are they a problem? Usually down the road.

The problem gets resolved. The technology changes. New employees arrive, responsibilities shift, and the circumstances that originally justified the rule may no longer exist. But the rule is still there. The rule stops being understood as a response to a particular problem and starts being treated as an essential part of how the organisation operates. Following the rule is just easier.

Safeguard versus ritual

Aviation depends on checklists, medical professionals follow clinical protocols, and financial institutions operate under regulatory requirements. These exist because people learnt, sometimes through terrible experiences, what can happen without them. But not every rule deserves the same level of protection.

There’s a difference between a safeguard that prevents a known failure and a procedure that survives because nobody wants to take responsibility for changing it. Imagine an organisation that needs three levels of approval before purchasing relatively inexpensive equipment. Maybe the procedure was introduced because a department once exceeded its budget. Maybe there were issues with unauthorised purchases. Fair.

But years later, that same organisation might have sophisticated accounting software, departmental spending limits and automated audit trails. The original risk has changed considerably. The three approvals? Still there.

Rather than controlling expenditure, the procedure may be using more in employee time than the financial risk it was designed to manage. But removing it means someone has to accept responsibility if something goes wrong. Keeping it means nobody has to make that decision.

When does process become the objective

Large organisations can’t depend entirely on individual judgement, particularly when operating across multiple locations, jurisdictions and departments. Processes help establish expectations, distribute responsibilities and ensure that important requirements aren’t overlooked.

But there’s a point where the relationship between the process and outcome becomes reversed. An organisation will ask if the procedure was followed. A project might satisfy every reporting requirement and deliver something nobody needs. A hiring process might follow an elaborate series of assessments and interviews and still overlook the most capable candidate. A public service might meet its administrative targets while leaving the people using that service increasingly frustrated.

And because everyone followed the established procedures, the system worked as intended. The question really is what, exactly, was it intended to accomplish? If the answer is basically, the process needs to be followed, we’ve turned a tool into a ritual.

The institutional memory problem

One of the more useful functions of a procedure is that it preserves knowledge. People leave organisations. They retire, move to other positions or take their experience somewhere else. Without documented processes, valuable knowledge can disappear with them.

Organisations write things down. They create policies, operating procedures, guidelines and training materials. But documentation tends to preserve what people decided to do not the full reasoning behind it.

A procedure might say that a particular report must be produced every Friday. What it may not explain is that the requirement began because a former executive needed information for a meeting they had every Friday, and that hasn’t existed in fifteen years. The report survives because it’s in the operating manual.

Everyone assumes that somebody, somewhere, must still need the information, but no one really knows why.

Technology can make the problem worse

We like to think that modernising a process means improving it. Does it really?

Organisations like to take an existing procedure, automate it, and declare the organisation more efficient. We see it with digital forms, approval systems, customer service workflows and with AI.

An organisation might replace a paper-based application with an online portal. The forms are now digital, documents can be uploaded, and information moves between departments electronically. That’s more convenient in some ways.

But what if the underlying process still needs unnecessary information, redundant approvals or repeated verification of things the organisation already knows? The technology hasn’t necessarily solved those problems. It made the old process electronic. Because the system is now automated, changing the procedure becomes even more difficult. It’s now embedded in the software.

Changing it may need a development project, a vendor contract, security reviews, testing and several rounds of approval. We’ve made an outdated assumption and turned it into a technical dependency. AI adds another dimension.

If we use AI to automate decisions based on existing policies, we need to consider whether those policies still make sense. Faster decisions aren’t necessarily better decisions. Automating a questionable process doesn’t make it less questionable. It just allows us to repeat it more efficiently.

Nobody wants to be the person who changes things

In many organisations, preserving an existing procedure carries considerably less personal risk than changing it. Suppose an employee identifies a redundant approval process. Removing it could save hundreds of hours annually, improve service delivery and reduce administrative costs. What happens if the organisation eliminates the requirement and, six months later, somebody makes a costly mistake? The person who approved the change may have to explain why they removed a control.

Meanwhile, if the unnecessary procedure remains in place, its costs are distributed across dozens or hundreds of employees. Nobody sees the full impact. There’s no dramatic failure. No obvious incident. No single person responsible. Just a little wasted time, repeated endlessly.

The benefits of changing the procedure are spread across the organisation. The perceived risk of changing it may fall on one person. Maintaining the status quo can become a perfectly rational personal decision, under those conditions, even when it’s a poor organisational decision.

Regulation isn’t the same as bureaucracy

Rules imposed by law or regulation aren’t interchangeable with internal administrative procedures. An organisation can’t ignore a legal obligation because it considers compliance inconvenient. Consumer protection, workplace safety, financial accountability and countless other safeguards exist for reasons.

Some rules are burdensome because the consequences of getting things wrong can be serious.

The issue isn’t that every requirement should be eliminated. It’s whether the requirement is proportionate to the problem. Regulations can become outdated, just as internal policies can. Technologies change, risks evolve and unintended consequences emerge.

Systems are easier to defend than to improve

An established system has an enormous advantage over a proposed alternative. It already exists.

People understand it, budgets accommodate it, employees have been trained to use it, and management has learnt how to report on its performance. An alternative needs evidence, projections, approvals, implementation plans and assurances that the change won’t introduce unacceptable risks.

My question? Why don’t we apply the same scrutiny to what’s already being done?

We rarely demand that every longstanding procedure show, periodically, that it’s still relevant to the needs. We don’t consistently calculate the cumulative cost of decisions delayed, opportunities missed or hours spent complying with requirements whose original purpose may have disappeared. In other words:

• The existing system gets the benefit of familiarity.

• The proposed change gets the burden of proof.

What happens if rules had to justify their existence?

• What problem was this rule originally designed to solve?

• Does that problem still exist?

• What evidence tells us the rule is effective?

• What does complying with it cost?

• And what would happen if we changed or removed it?

I suspect quite a few organisations would struggle to answer them for a surprisingly large proportion of their existing procedures. There’s a potential irony here too. We’d potentially create another administrative process requiring people to review existing processes. Complete with forms, committees, quarterly reports and approval procedures.

And twenty years later, somebody would probably be writing an article asking why we’re still doing it.

The purpose was always the point

None of this means organisations should operate without structure. The ability to coordinate large numbers of people, preserve knowledge and make consistent decisions is one of the reasons institutions exist in the first place.

Rules make much of that possible. But rules aren’t ends in themselves. They’re instruments to achieve particular outcomes under particular circumstances. When those circumstances change, the rules should at least be open to examination. And when an organisation can’t explain why a procedure exists, that should prompt questions, not friction.

• Maybe the procedure remains essential.

• Maybe removing it would expose risks nobody has considered.

• Or maybe everyone has just become accustomed to doing something that no longer serves a useful purpose.

We won’t know unless we’re prepared to ask.

Subscribe now

Ideas that were once considered obvious have been overturned by new evidence, changing circumstances or by someone asking a question nobody thought to ask. It isn’t that people were wrong. Being wrong is an unavoidable part of learning.

How long an assumption can survive after the circumstances that originally justified it have changed. And why are we so reluctant to question it.

When an assumption becomes a fact

For most of us, the five-day working week is simply part of life. We organise employment, education, transportation and other activities around it. There’s nothing particularly natural about five days of work followed by two days off. It’s the product of historical, economic and social developments, including decades of labour movements and industrial reform.

The five-day week became the standard in a world where industrial production, physical presence and relatively predictable working hours shaped much of the economy. Today, we have occupations where productivity depends more on concentration, creativity and problem-solving than on the number of hours someone occupies a chair. Organisations still measure commitment through attendance and availability.

That doesn’t mean a four-day week works for every business, just like a five-day work week doesn’t work for every business, or that physical presence has no value. Some work requires people to be available at specific times, and many services need continuous coverage.

The problem with things that work

The problem with questioning established systems is that they often do work. At least to some extent.

Schools educate millions of children. Healthcare systems treat enormous numbers of patients. Financial institutions facilitate economic activity. Governments administer complicated societies. Businesses coordinate people, resources and services across the world.

But there’s a difference between recognising that a system is useful and assuming that its existing structure is optimal. An organisation might have a procedure that was introduced twenty years ago to deal with a real problem. The procedure worked, people followed it, and eventually it became part of the organisation’s normal operations.

Over time, the original problem may have disappeared. Technology may have changed. The people involved may have moved on. But the procedure is still there. Nobody wants to remove it because nobody is entirely sure what might happen if they do.

The more established a system becomes, the harder it is to distinguish what’s effective versus what’s familiar.

We’ve confused consistency with correctness

Consistency is important. We don’t want important decisions made arbitrarily, and we don’t want the outcome to depend entirely on which individual happens to be responsible. Standardisation helps address that problem.

It creates common procedures, measurable criteria and expectations that can be communicated across large organisations. But consistency doesn’t mean correctness.

An organisation can apply the same flawed reasoning perfectly every time. A school can consistently assess students using criteria that don’t to capture important aspects of their abilities. A business can consistently evaluate employees using metrics that have little to do with their actual contribution. A government department can consistently follow a procedure that it no longer really needs.

This connects to something I explored in my recent series on education and assessment. We develop measurements to help us understand something. Then, gradually, the measurement begins to stand in for the thing we’re trying to understand. The same can happen with procedures. We establish rules to improve judgement, then start treating compliance with those rules as a substitute for judgement.

Technology doesn’t necessarily change our thinking

New technology can make an existing problem faster, cheaper and harder to recognise.

Let’s look at what happens when an organisation introduces artificial intelligence into a decision-making process. The immediate attraction is understandable. AI can examine large amounts of information, identify patterns, automate repetitive tasks and help people work through problems that might otherwise take a lot of time. That’s useful.

But what happens when the decision-making process the AI is using is flawed? If an institution has spent decades relying on poor measurements, introducing AI doesn’t fix those measurements. It just makes the institution much more efficient at applying them.

A questionable assumption embedded in a manual procedure might affect hundreds of decisions. The same assumption embedded in an automated system could influence millions. Because the results are produced through sophisticated technology, they may seem more objective than they actually are.

This isn’t an argument against AI. I’ve written before about its enormous potential as a learning and research tool. It’s an argument about confusing technological sophistication with improved reasoning. But before we do that we should probably ask whether the task still makes sense.

Institutions have memories, but not always understanding

Organisations accumulate knowledge, but they also accumulate habits. A decision made years earlier becomes a policy. A policy becomes a procedure. A procedure becomes a requirement. Eventually, nobody remembers the original reasoning, but everyone understands that the requirement must be followed.

Large organisations need continuity. People retire, move between departments or change careers. Procedures provide a way to preserve institutional knowledge without requiring every new employee to rediscover everything from the beginning.

But preserving a decision isn’t the same as preserving the reasoning behind it. And when the reasoning disappears, questioning it becomes harder. Someone proposing a change may be asked to demonstrate why the existing approach should be abandoned.

That’s reasonable. But shouldn’t the existing approach also be expected to justify itself? We tend to place the burden of proof on change while allowing the status quo to survive largely because it’s already there.

The comfort of familiar answers

Of course, this isn’t limited to institutions. We do much the same thing as individuals. We develop opinions based on our experiences, the information available to us and the people whose judgement we’ve learnt to respect.

Over time, those opinions become part of how we understand the world. And once we’ve built enough of our understanding around a particular idea, questioning it becomes problematic.

Changing one thing means considering several others. It can mean admitting that an argument we’ve defended for years wasn’t sound, or that someone we dismissed may have had a point. That’s not always easy.

There’s also a practical reason we rely on assumptions. Nobody has the time or expertise to independently investigate every claim, every institution and every decision encountered in daily life. We have to rely on accumulated knowledge, specialists and established conventions. We need to remember that these shortcuts to understanding, don’t mean correctness.

An expert can be wrong. A consensus can change. An established procedure can become obsolete. A technology can be impressive without being appropriate for every problem. We need to remember to leave room for the possibility that what we currently accept could be reconsidered.

Asking questions isn’t the same as rejecting answers

Questioning a scientific conclusion doesn’t automatically make an alternative explanation credible. Challenging an institution doesn’t mean every criticism of that institution is justified. And recognising uncertainty doesn’t mean all interpretations deserve equal weight.

Evidence matters. That said, so does expertise, methodology and the ability to distinguish a plausible argument from an unsupported assertion. But those things are strengthened by examination, not weakened by it. A solid conclusion should be able to withstand reasonable questions.

And when new evidence requires us to change that conclusion, that should be considered progress not defeat. Questioning an assumption isn’t necessarily to prove it wrong.

Question the question

Over the years, I’ve become increasingly interested in the relationship between the questions we ask and the answers we’re capable of finding. It’s something that comes up in science, philosophy, literature and technology.

A question establishes boundaries. It tells us what we’re investigating, what information seems relevant and what kinds of answers we’re prepared to consider. But those boundaries can stop us from seeing alternatives.

• If a business asks how to make employees spend more time in the office, it’s already assuming that increased attendance is desirable.

• If a school asks how to improve standardised test scores, it’s already treating those scores as an appropriate measure of improvement.

• If a technology company asks how to collect and process more personal information securely, it’s already assuming that collecting the information is necessary.

Things we inherit

We live surrounded by systems we didn’t design, assumptions we didn’t establish and conventions that existed long before we encountered them. Many are there as the products of human decisions, made under particular circumstances, with particular limitations.

• Circumstances change.

• Knowledge changes.

• Technology changes.

Institutions don’t always recognise when change is needed. Neither do individuals.

Subscribe now

But there’s a reason I wanted to explore them. The more I thought about how we build systems, whether they’re educational, organisational or technological, the more I noticed how often we confuse understanding something with trusting the people or institutions responsible for it. Sometimes we don’t even get as far as understanding it. We just accept that the system works because we’ve been told it does.

We can’t personally investigate every institution, test every piece of software or verify every claim we’re presented with. There aren’t enough hours in the day, and most of us have better things to do.

But there’s a difference between relying on other people’s expertise and assuming that expertise makes their conclusions unquestionable.

We’ve become comfortable with being told how things work

Think about how many systems we deal with every day without really knowing what happens behind the scenes. We trust banks with our money, hospitals with our medical information, governments with our personal records and technology companies with an large amounts of information about our lives. Sure the last one is grudgingly, especially around US big tech.

Most of the time, those arrangements work reasonably well. I’m certainly not saying that everyone involved is incompetent or that every organisation is deliberately trying to mislead us. But we do seem to have developed a habit of accepting assurances without asking very much about what they actually mean.

• A company tells us our information is secure. We assume it’s been protected.

• An institution tells us an assessment is standardised. We assume the result must be meaningful.

• An AI produces an answer with an impressive explanation. We’re inclined to believe it knows what it’s talking about.

We’re placing considerably more confidence in the system than the evidence actually supports. And that’s where the trouble starts.

The difference between what a system tells us and what it knows

In the first article, Measurement Shouldn’t Be Judgement, I looked at how assessments can become conclusions about someone’s abilities rather than tools for understanding their needs. It’s an interesting problem because the test itself doesn’t necessarily have to be wrong. A student might genuinely perform poorly on a particular assessment. The score might be perfectly accurate, according to the rules of the test.

But what does it actually tell us? Does the student lack understanding? Are they struggling with the language? Were the questions presented in a way that didn’t suit how they’d learnt the material? Or is there something else happening that the test wasn’t designed to identify? The measurement doesn’t really explain the result.

We see something similar with AI. In the second article, The Problem With Knowing the Answer, I looked at the difference between producing the right answer and understanding why it’s correct. AI can generate answers, explanations and entire essays almost instantly. But, does it understand the information?

But the fact that a system can produce a convincing explanation doesn’t mean the explanation is right, and it certainly doesn’t mean the person receiving it has learnt anything.

has explored this in his Slow AI work, particularly the importance of slowing down, questioning what the technology produces and recognising when it’s helping us think rather than doing the thinking for us. I find that approach useful.

AI can be an excellent tool for exploring unfamiliar subjects, challenging ideas and examining different explanations. But it can also become an efficient way of avoiding the effort involved in understanding something.

We don’t all arrive at understanding the same way

The third article, The Books That Helped Me Understand the World, was probably the most personal of the series, although not in the conventional sense. It wasn’t about my life. It was about the ideas I’ve encountered through reading and the connections I’ve found between subjects that don’t appear to have much in common.

Mathematics, history, philosophy, science fiction and fantasy all approach questions about how the world works from different directions. Some authors explain established knowledge. Others challenge familiar assumptions. Fiction writers can explore ideas that would be difficult to examine in quite the same way through nonfiction.

An author can be wrong about a particular conclusion while still raising a question worth considering. I don’t read to find someone whose opinions I can adopt wholesale. I’m interested in understanding how they arrived at those opinions, whether their reasoning makes sense and what I might have overlooked. It’s also why I don’t think reading should be confined to subjects we’re already familiar with, or agree with.

If we only encounter ideas that reinforce what we already believe, we’re not giving ourselves many opportunities to discover where we might be wrong. And that applies just as much to technology. One of the difficulties with highly specialised fields is that people can become so familiar with established methods that they stop questioning the assumptions those methods depend on.

Even the words can lead us in the wrong direction

In the fourth article, The Words We Use Change the Way We Think, I looked at how language influences the way we understand information. It’s something we encounter constantly.

A government introduces a security measure. A company announces a transformation. An organisation describes employees as human capital. The terminology can make something sound reassuring, progressive or entirely reasonable before we’ve examined what’s actually happening.

Words aren’t magic. They don’t automatically determine what people believe, and specialised language often exists for perfectly good reasons. But the way an issue is presented can influence which questions occur to us in the first place. That’s particularly relevant to AI.

The way we phrase a question can influence the answer we receive. Ask a system why something is beneficial, and you may get a convincing explanation of its advantages. Ask why it’s harmful, and you may get an equally convincing explanation of its disadvantages.

That doesn’t make either response wrong. It means we need to pay attention to the assumptions built into our questions. Sometimes the problem isn’t the answer. It’s the way we’ve framed what we’re asking. Which brings me back to something that’s been bothering me for years about technology and cybersecurity.

Why do we need to trust quite so much?

We’ve spent decades developing increasingly sophisticated ways to protect information. Encryption, access controls, authentication, monitoring, intrusion detection and a seemingly endless collection of security products and procedures. Modern computing would be more dangerous without it.

But underneath all that, we’ve kept an assumption that needs more attention. We collect information, store it somewhere, give authorised systems access to it and then build layers of security intended to prevent anyone else from getting hold of it.

A doctor needs access to relevant medical information to make informed decisions. Financial institutions have legal obligations to identify customers and monitor certain activities. But there are situations where an organisation doesn’t actually need to see all the information it collects. It only needs to establish that something is true.

Look at age verification. If a service needs to know whether someone is over eighteen, why should it necessarily need their full date of birth, address or other personal details?

Or consider a company that needs to establish whether someone meets a particular financial requirement. Does it always need to examine their entire financial history, or could the relevant condition be verified without revealing everything behind it? These aren’t questions with universal answers. Legal requirements, fraud prevention and the practical details of a particular service all matter. But they’re questions we should be asking more often.

Instead, we start by collecting the information and then figuring out how to protect it. There’s something rather backwards about that. If the information doesn’t need to be collected or exposed in the first place, avoiding that exposure is worth considering.

What if the computer didn’t need to see the information?

This is where my interest in cryptography comes into the picture. And no, I’m not going to turn the final article into a technical paper. But there are developments worth mentioning because they challenge assumptions we’ve lived with for decades.

ZKPs, for example, allow someone to demonstrate that a particular statement is true without revealing the information used to establish it. Under the right circumstances, that means a system can verify that someone meets a requirement without receiving all the personal information behind the claim.

Then there’s FHE. The interesting thing about FHE is that it allows calculations to be performed on encrypted information without decrypting it during the computation. Think about what that means.

If a computer needs to process sensitive information, that information has to become readable somewhere in the computing environment. We can protect that environment, restrict access to it and monitor what happens inside it. But the underlying information is still exposed to the part of the system performing the work. FHE changes that assumption for the computations it supports.

The system can perform useful work without seeing the underlying plaintext. It isn’t magic, and there are important limitations involving performance, supported operations, implementation and security. Nor does encrypted computation automatically guarantee that the result is correct or that the wider system can’t leak information in other ways. Those are real engineering problems.

But the underlying idea is interesting because it changes the relationship between the organisation performing the work and the information being processed. Instead of asking the organisation to promise that it won’t misuse information it can see, we can sometimes design the system so that it doesn’t need to see that information at all.

That won’t solve every privacy problem. But it does reduce opportunities for things to go wrong. And I think that’s a more useful direction.

Trust isn’t the same as evidence

We’re constantly being asked to trust the systems around us.

• Trust that the assessment is accurate.

• Trust that the institution knows what it’s doing.

• Trust that the information is secure.

• Trust that the AI has produced a reliable answer.

And sometimes that trust is entirely justified. But the fact that an organisation has established procedures, qualified people or impressive technology doesn’t mean every conclusion it reaches deserves the same level of confidence.

A security audit tells us something about the systems and controls examined during that audit. It doesn’t guarantee that an organisation can never be compromised. A (standardised) test tells us something about performance under particular conditions. It doesn’t necessarily establish someone’s future potential. A cryptographic proof can demonstrate that a defined mathematical claim is valid under particular assumptions. It doesn’t automatically establish that every component of the surrounding system is secure.

Evidence has boundaries. When we understand those boundaries, we’re in a much better position to decide what confidence is justified. When we don’t, it’s easy to take a limited assurance and turn it into something much broader. We’ve done that with educational assessments, organisational classifications and increasingly with technology.

Have we been asking the wrong question?

I’ve argued before that cybersecurity has spent a great deal of time improving approaches built around assumptions that haven’t changed in forty years. We identify trusted users, grant access, monitor activity and try to prevent unauthorised behaviour. When something goes wrong, we strengthen the controls, add more monitoring and introduce another layer of protection.

I think we should also be asking whether the architecture itself creates exposures that could be avoided. It’s the difference between asking how we can better protect sensitive information and asking whether the system needs access to that information at all.

• Instead of asking how accurately we can classify students, maybe we should ask whether those classifications are helping them learn.

• Instead of asking how efficiently AI can produce answers, maybe we should ask whether it’s helping people understand the subjects they’re exploring.

• Instead of accepting an institution’s description of a problem, maybe we should see whether the terminology is influencing how we think about it.

None of this requires rejecting established knowledge, dismissing expertise or assuming every institution is acting in bad faith. It means being willing to reconsider the assumptions behind systems we’ve become accustomed to using.

A final thought

When I started this series, I wasn’t trying to make an argument against education, institutions, technology or trust. I’m not suggesting that questioning everything automatically makes someone better informed. Scepticism can become just as unhelpful as unquestioning belief if it turns into a refusal to accept evidence that doesn’t fit our preferences.

What interests me is understanding how things work, why we’ve chosen to organise them the way we have and whether those choices still make sense. Sometimes the answers are straightforward. Other times, they take us into subjects we didn’t expect to explore. That’s why I read so widely.

A book about probability might change how I interpret a technical benchmark. A novel about political power might make me think differently about institutional authority. An examination of language might help explain why two people can look at the same information and reach completely different conclusions.

And an idea from cryptography might suggest that a problem we’ve spent decades trying to manage could sometimes be approached in a fundamentally different way. None of those connections guarantees that I’ve reached the right conclusion.

I don’t think the goal should be to eliminate trust from our lives. That’s not practical or desirable. But there are situations where we can reduce how much trust is needed, especially when we’re designing systems that handle sensitive information or make decisions affecting other people.

• If we can verify a claim without collecting unnecessary personal data, why wouldn’t we consider it?

• If we can perform a computation without exposing the information being processed, isn’t that worth investigating?

• If a system makes a decision about someone, shouldn’t we understand what evidence supports that decision and what the system might have missed?

I keep coming back to those questions because they have implications beyond cybersecurity. They affect how we educate people, how organisations make decisions and how we develop technologies that are becoming increasingly difficult for most of us to examine independently.

We’re never going to understand everything, and we’re not going to eliminate uncertainty. But we can become more thoughtful about what we accept, what we question and what we’re prepared to trust.

We can also design systems that give people better reasons to trust them, rather than simply asking them to take someone’s word for it. Maybe that’s the connection I’ve been looking for across these five articles.

It’s not that measurement, education, reading, language and technology are all the same problem. It’s that each of them reminds us how important it is to understand what we’re being told, how that conclusion was reached and whether the evidence actually supports it.

Subscribe now

The underlying policy hasn’t necessarily changed. Neither have the facts. We’ve simply changed the language used to describe it. And that can make an enormous difference to how people understand what’s being proposed.

I’ve always found this fascinating, partly because we tend to think of language as a neutral way of communicating information. We have an idea, choose some words and explain it to someone else.

But language does considerably more than that. The words we choose influence which aspects of an idea receive attention, what associations come to mind and, occasionally, which questions people never think to ask.

Politicians understand this. So do advertisers, corporate communications departments and anyone who’s ever tried to persuade someone to buy something they didn’t know they needed. The interesting part is how often it happens without us noticing.

What’s in a name?

The USA PATRIOT Act, for example, passed in October 2001 following the September 11 attacks. Its full name was the Uniting and Strengthening America by Providing Appropriate Tools Required to Intercept and Obstruct Terrorism Act.

That is a true mouthful (word salad) of an acronym.. But think about the choice of the word patriot. It immediately connects the legislation to loyalty, national security and protecting the country. Those are powerful associations, particularly in the aftermath of a national tragedy.

Now look at the same legislation under a title emphasising the expansion of government surveillance powers. The provisions might be identical, but the public conversation would probably begin from a very different place. The point is that the name of the act encouraged people to approach it through a particular set of associations before looking at what it actually says.

The questions we ask are often influenced by how an issue is presented. If something is described as patriotic, disagreement can become entangled with questions of loyalty. If something is called a safety measure, objections may sound like opposition to safety itself. Neither conclusion follows logically.

You can support national security and still question particular surveillance powers. You can support public safety while disagreeing with the methods proposed to achieve it. But the language can make those points harder to communicate. It’s a useful trick, and governments don’t have a monopoly on it.

Corporate language is particularly good at this

Businesses have developed an impressive vocabulary for describing perfectly ordinary activities in ways that make them sound more sophisticated. Employees become human capital. Problems become challenges. Layoffs become rightsizing, restructuring or workforce optimisation. We’re told organisations are transforming, realigning, leveraging synergies and developing strategic capabilities.

Sometimes there’s a sensible idea hiding underneath all that word salad, and that really is what it is. Take human capital. From an economic perspective, the term has a legitimate meaning. It refers to the skills, knowledge and capabilities people contribute to productive activity. But when people are capital, it changes how they’re perceived and discussed.

Instead of talking about employees with different abilities, circumstances and motivations, we’re talking about resources that can be allocated, optimised and measured. And ultimately, discarded.

That may be useful for a spreadsheet, but it doesn’t necessarily tell us much about the people involved. The terminology can begin influencing the decisions organisations make. If someone is described as a resource, it’s easier to think about utilisation and allocation. If they’re described as a person with particular skills and experience, different questions might come to mind. Neither description changes who the employee is. It does change how the organisation thinks about them:

• A student becomes a test score.

• A job applicant becomes a personality profile.

• An employee becomes a collection of performance indicators.

The classifications may provide useful information, but they’re still representations of people, not the people themselves. And once the terminology becomes familiar, it’s really easy to forget that differentiation.

McLuhan was looking at something bigger

Marshall McLuhan approached language and communication from a rather different direction. He’s probably best remembered for the phrase the medium is the message, which is repeated without much consideration of what he meant.

His argument wasn’t simply that different media deliver information in different ways. He was interested in how the characteristics of a communication medium influence the way people experience information, organise their lives and interact with society. A newspaper, a TV broadcast and a conversation don’t affect us in the same way, even when they’re discussing the same subject.

A complicated political issue might take months to develop and involve thousands of pages of evidence, competing interests and legitimate disagreements. But a broadcast news segment has limited time. It needs a clear explanation, compelling images and something that will hold the audience’s attention. That creates pressure to simplify.

The story needs a beginning, a conflict and preferably a conclusion. People need to understand what’s happening quickly, often without having much previous knowledge of the subject. That doesn’t immediately mean the reporting is inaccurate. It does, however, influence how the issue is presented and what viewers are likely to remember.

A lengthy newspaper, or magazine, investigation can provide more detail, but it has its own constraints. An academic paper approaches the subject differently again. And social media adds another set of incentives, rewarding material that attracts immediate attention and encourages people to react.

The underlying information might be the same, the experience of receiving it can be very different. That’s the part of McLuhan’s argument I find particularly interesting. We’re so focused on whether the information is correct that we overlook how the medium itself influences our relationship with it.

Rushkoff takes the argument a little further

Douglas Rushkoff has spent much of his career examining the relationship between media, technology and human behaviour. In Coercion: Why We Listen to What “They” Say, he explores how advertisers, salespeople, political organisations and others use psychological techniques to influence decisions.

What makes the book interesting isn’t simply the existence of persuasion. People have been trying to influence one another for as long as they’ve been communicating. It’s the way those techniques can work without us consciously recognising what’s happening. We tend to imagine manipulation as someone telling us something false and hoping we’ll believe it. Persuasion is more subtle.

You don’t necessarily have to lie about a product to influence how someone feels about it. You might associate it with independence, status, security or belonging.

The product is the same, but the meaning attached to it changes. And once that association becomes familiar, people may start responding to the association not the product itself.

The same thing happens outside advertising. Think about how frequently we encounter expressions such as innovation, disruption, digital transformation and AI-powered. Those terms can describe real developments. But they can also create an impression of progress without explaining what has actually improved.

An organisation might announce that it’s using AI to transform customer service. But has the service become faster, more accurate or more useful? Can customers resolve problems more easily? Are errors being reduced? Or has the company just installed a chatbot and given the project an expensive-sounding name?

This isn’t just an opinion about language

Psychologists Amos Tversky and Daniel Kahneman explored a related phenomenon in their research on decision-making. In a well-known 1981 paper, they demonstrated how people’s preferences could change depending on how equivalent choices were presented.

In one experiment involving a hypothetical disease outbreak, participants were asked to choose between programmes described in terms of lives saved or lives lost. The outcomes were mathematically equivalent, but people’s preferences shifted depending on the wording. It’s an example of what’s known as the framing effect.

We don’t always evaluate a decision by examining its underlying consequences independently of how those consequences are described. The presentation itself can influence our response. That doesn’t mean everyone is equally susceptible, that framing always works or that language completely determines what we think.

People can recognise misleading framing, challenge the assumptions behind it and change their minds when presented with better information. But the research does suggest that we shouldn’t assume we’re immune just because we consider ourselves rational. We like to believe we examine information objectively, weigh the evidence and reach a conclusion. Sometimes. Other times, the language has already influenced what we consider important before we’ve properly begun thinking about it.

The problem with labels

Labels can be useful. We need ways to describe things, organise information and communicate ideas without explaining everything from scratch.

Medicine relies on classifications. So does education. Businesses use job titles, professional qualifications and performance categories. Without some common terminology, coordinating large organisations would become extremely difficult. The problem starts when we confuse a label with a complete understanding of what it describes.

Suppose a student is classified as having a particular learning difficulty. That classification may be entirely appropriate and may help ensure they receive the support they need. But it doesn’t tell us everything about the student. It doesn’t establish their interests, their ambitions, their other abilities or what they might accomplish with the right opportunities. Once a classification enters an institutional record, it can influence how other people interpret the student’s behaviour. Someone sees the label and begins forming expectations.

The same thing happens in workplaces, where people are described as high-potential employees, underperformers, introverts, natural leaders or poor cultural fits. Some of those descriptions may reflect genuine observations. Others may be based on questionable assessments or limited experience. Terminology can take on a life of its own.

Once a person has been placed in a particular category, any future information will always be interpreted through that category. That label. And because the terminology sounds professional, we’re usually less inclined to question it. That’s what bother me, the tendency to treat them as though they explain more than they actually do.

Now AI is learning to speak for us

In some respects, AI makes the problem of language and framing considerably more interesting, because these systems don’t just deliver information. They generate explanations, summaries, arguments and recommendations. And they can do it with remarkable fluency.

Ask an AI to explain a complicated subject and it can produce an answer that sounds organised, confident and authoritative. It can also describe the same issue from different perspectives, selecting different evidence, examples and terminology depending on how the question is asked. That flexibility can help people understand unfamiliar subjects, compare arguments and explore ideas they might not otherwise encounter.

But it also creates a problem we need to pay attention to. If an AI produces a convincing explanation, how much of our confidence comes from the evidence, and how much comes from how well the explanation is written? A poorly explained argument might be correct. A beautifully written explanation might be completely wrong.

And as AI becomes more deeply involved in search, education, business and public administration, we’re going to encounter more polished explanations of subjects we may not understand well enough to evaluate independently. There’s another issue, too. The way we phrase a question influences the answer we receive. has explored this in his Slow AI work, encouraging people to slow down, reconsider how they frame questions and ask AI to challenge their assumptions rather than simply produce convincing answers. It's an approach I find useful because it reminds us that the conversation doesn't begin with the answer. It begins with the question, and sometimes that's where the problem lies.

• Ask an AI why a particular policy is harmful and you may receive a detailed explanation of its disadvantages.

• Ask why the same policy is beneficial and the system may provide an equally convincing discussion of its advantages.

That doesn’t necessarily mean either response is false. Complicated questions often have legitimate arguments on both sides. But the wording of the original question can influence which arguments get attention.

And if we’re not careful, the system can become a very efficient way of reinforcing assumptions we already hold. Instead of helping us understand something, it may simply help us construct a more convincing explanation of why we were right all along. It will just become an echo chamber. That’s hardly the technology’s most useful application.

The words aren’t necessarily the problem

I don’t think the answer is to become suspicious of every unfamiliar term or assume that anyone using professional language is trying to manipulate us. Specialised vocabulary exists for good reasons. Different disciplines need precise terminology, and simplifying everything into everyday language can sometimes remove important distinctions.

Nor do I think changing a word automatically changes what someone believes. People aren’t that predictable, and language isn’t some magical instrument that eliminates independent thought. Words influence what we notice, the associations we make and the questions we’re likely to ask.

That’s worth remembering whenever an issue is presented in language that seems designed to produce an immediate reaction.

• If a government calls something a security measure, we should still ask what powers it creates and what safeguards exist.

• If a company announces a transformation, we should ask what has actually changed.

• If an educational institution describes a student as low-performing, we should ask what was measured, under which circumstances and what the result really tells us.

• If an AI provides an impressive explanation, we should ask whether the evidence supports it.

None of those questions means we should reject the original description. They stop the description from doing all the thinking for us.

Learning to recognise the frame

We measure people and sometimes mistake the result for an understanding of their abilities. We teach students to produce the expected answers, even when that doesn’t necessarily demonstrate understanding. We encounter ideas through books and discover that different disciplines can describe similar problems in different ways.

And now we arrive at language itself. The words used to describe a problem can influence which solutions appear reasonable, who seems credible and what we consider worth questioning. That doesn’t mean we can escape language or examine the world from some perfectly neutral position. Every explanation involves choices about what to include, what to exclude and how to organise the information.

Even this article does. The important thing is recognising that those choices exist. When someone presents an argument, it’s worth considering not just whether the facts are correct, but how they’ve been selected and described.

What assumptions are hidden in the terminology? What alternative descriptions might change our understanding? And what questions would we ask if the subject had been introduced under a different name?

Sometimes the original framing will hold up perfectly well. Other times, you’ll see that a lot of the argument depended on getting you to accept the terminology before you’d examined the idea.

Subscribe now

When you’ve spent years reading across different subjects, something interesting starts to happen. You begin noticing connections between ideas that don’t seem to have much in common. The connections aren’t always obvious, and they aren’t always intentional.

A historian looking at how civilisations develop might raise questions similar to those explored by a science-fiction writer imagining a future society. A mathematician explaining probability might change how you think about certainty, while a fantasy novel about gods, power and competing realities might have more to say about human behaviour than a book specifically written about psychology.

Fiction isn’t the opposite of serious thinking

There’s a tendency to treat nonfiction as something we read to learn about the world, while fiction is what we read for entertainment.

Take Neal Stephenson. His books are generally classified as science fiction, although that description doesn’t do them justice.

Cryptonomicon moves between cryptography, mathematics, computing, finance and the Second World War. The Baroque Cycle takes us back to the scientific revolution, when ideas about mathematics, money, science and political power were developing alongside one another.

Then there’s Anathem, which explores mathematics, philosophy, the nature of reality and what happens when intellectual communities separate themselves from the societies around them.

They’re novels, of course. There are characters, plots, conflicts and all the other things you’d expect from fiction. But they’re also an invitation to think about subjects that are usually taught separately.

What’s interesting about The Baroque Cycle is how difficult it becomes to separate the development of modern science from everything else happening at the time. Money, politics, religion and the people running institutions all played a part. Scientific progress wasn’t a series of discoveries made by brilliant individuals working in isolation. Those discoveries happened within societies, and those societies had their own interests, prejudices and power structures. It’s not much different today.

We talk about artificial intelligence, cybersecurity and computing as though their development is driven primarily by what the technology can do. But economics, politics, institutional priorities and commercial interests have an enormous influence on which technologies get developed, who controls them and how they’re used.

Stephenson doesn’t have to write an essay about modern technology companies to make you think about those relationships. The historical setting does much of the work for him. That’s one of the things fiction does particularly well. It lets an author explore the consequences of an idea without having to pretend they’re predicting the future.

What fantasy can teach us about power

Science fiction gets plenty of attention whenever people discuss books that explore society, technology or the future. Fantasy rarely gets the same consideration, which I’ve always found rather strange given how much of it deals with power, institutions, morality and the limits of human understanding.

Roger Zelazny’s Chronicles of Amber is a good example.

On the surface, it’s a story about a powerful family whose members can move between different ‘earth's’. There’s intrigue, rivalry, betrayal and a great deal of fighting over who gets to rule. Underneath that, Zelazny is playing with some interesting ideas about reality, authority and human nature.

The characters inhabit a universe where what most people consider reality is only one possibility among many. They have extraordinary abilities, yet they’re still driven by remarkably familiar motivations: ambition, jealousy, loyalty, fear and the desire for control.

In many ways, they resemble the Greek or Norse pantheons. The gods might possess immense power, but that doesn’t make them particularly wise, moral or trustworthy. And it’s useful to remember whenever someone claims authority simply because of their position or capabilities. Being powerful doesn’t necessarily mean you understand what you’re doing.

Michael Moorcock approaches this from different direction. In his Eternal Champion stories, the struggle between Law and Chaos isn’t simply a contest between good and evil. Either force, taken too far, can become destructive. I find that a much more interesting way of looking at morality than dividing everything into heroes and villains.

Order provides stability, but too much of it becomes oppressive. Chaos can create opportunities for change, but unchecked chaos can destroy the conditions that make freedom possible in the first place. Neither is automatically good or bad. What matters is the balance between them and recognising when one has become too dominant.

You don’t have to believe that Zelazny’s or Moorcock’s fictional universes have anything to do with our own to find their ideas useful. They’re ways of examining possibilities, challenging assumptions and considering human behaviour from a different perspective. Which, when you think about it, isn’t that far from what philosophy tries to do.

What do we actually know?

Mathematics and physics raise a different issues, especially about certainty and how easily we convince ourselves that we understand something. Writers such as Leonard Mlodinow and Amir D. Aczel have a great way of exploring subjects that can completely change how you look at everyday experience.

Mlodinow’s The Drunkard’s Walk, for example, examines probability and randomness, including how easily we mistake chance events for meaningful patterns. And we do love finding patterns.

When something happens, we want to know why. If two events happen together, we’re inclined to assume there’s a relationship between them. Sometimes we’re right, but sometimes we’re seeing patterns in noise. That might sound academic, but it affects how we interpret statistics, financial results, scientific findings and even our own experiences. It’s also worth remembering when someone produces a graph and confidently explains what it means.

Aczel takes a different approach, exploring mathematics and scientific discoveries through the people and histories behind them. Books such as Fermat’s Last Theorem and The Mystery of the Aleph remind us that mathematical knowledge didn’t simply appear one day, fully formed and ready to be taught. It developed through questions, disagreements, failed attempts and, occasionally, centuries of work.

And that’s what interests me as much as the mathematics itself. We tend to see scientific ideas as finished products. The equations are there, the terminology is defined and the conclusions are settled. What we don’t see is the long process of uncertainty that came before them, or the people who challenged existing ideas along the way. Understanding that changes how you think about authority.

An established scientific conclusion deserves to be taken seriously, particularly when there’s strong evidence supporting it. But understanding why something is accepted is quite different from accepting it simply because someone in a position of authority tells you it’s true.

Sometimes the questions are more interesting than the answers

One of the more unusual books, and a true favourite, I’ve come across is Hamlet’s Mill, by Giorgio de Santillana and Hertha von Dechend.

It’s an exploration of the possibility that ancient myths preserved astronomical knowledge, including ideas about the movement of the heavens that may have been passed down through storytelling. It’s an ambitious argument, and some of its conclusions are still disputed. But, that doesn’t mean it should be dismissed out of hand.

The authors encourage us to look at mythology differently. Rather than treating ancient stories exclusively as religious beliefs, entertainment or early attempts to explain natural events, they ask whether some of those stories might also have preserved observations about the physical world. That doesn’t mean every myth has some hidden scientific explanation, or that every interpretation in the book is the right one.

But it does raise an interesting question about how knowledge gets recorded and passed from generation to generation. We tend to think of scientific knowledge as something preserved through books, formal education and established institutions.

But people were observing the world, recognising patterns and developing ways to pass on information long before those institutions existed. So how much of that knowledge survived? How much disappeared? And how much might we not recognise because it was recorded in ways we no longer understand?

I don’t think you have to accept the authors’ conclusions to find those questions worthwhile. In fact, that’s something I’ve come to appreciate about reading more generally.

You don’t have to agree with an author to learn something from them. Sometimes the most useful thing a book can do is make you reconsider something you’d previously taken for granted. And occasionally, a book can be wrong about its conclusions while still asking a genuinely interesting question. The trick is recognising the difference. Arguably, we can apply this same thinking to interactions with AI.

The people who shape the systems

Douglas Rushkoff and Marshall McLuhan are interesting because they examine how communication technologies influence society, often in ways we don’t immediately recognise.

McLuhan’s argument that the medium itself shapes our experience of information is easy to overlook when we’re concentrating entirely on what that information says.

Rushkoff explores some of those ideas in the context of digital technology, looking at how it influences behaviour, economic relationships and our ability to make independent choices.

Playing the Future examines how younger generations adapt to technological change, usually before adults have worked out what’s happening, AI is a great current example, just look at some of the authors on Substack. Coercion looks at the techniques used to influence behaviour, while Survival of the Richest explores how some wealthy technology figures imagine escaping the consequences of the systems they’ve helped create.

The subjects are different, but there’s a recurring theme. Who designs the systems we live with, what do those systems encourage us to do, and how much control do we actually have over the decisions we make? It’s a question that becomes more relevant as technology becomes embedded in almost everything we do.

Yuval Noah Harari in Sapiens, Homo Deus and Nexus, examines how shared beliefs, institutions, information and technology have shaped human societies. I don’t necessarily agree with all his conclusions, and some of his broader interpretations deserve to be challenged. But that’s part of what makes the books interesting.

They encourage us to look at institutions we generally accept as perfectly natural and ask how they came to exist in the first place. Money, governments, corporations and other social structures depend on systems of shared understanding. They work because people participate in them and generally accept the rules. Those rules aren’t necessarily permanent, inevitable or even sensible. They’re arrangements we’ve created, usually over long periods of time, and arrangements can change.

Viktor Frankl’s Man’s Search for Meaning approaches human experience from an entirely different direction. His examination of the search for meaning under circumstances of extraordinary suffering isn’t something that should be reduced to a simple lesson about positive thinking. His attention to the relationship between circumstances, personal interpretation and the choices people may still have, even when those choices are severely constrained.

It’s a useful counterpoint to books that concentrate primarily on the large systems influencing society. Institutions matter, as do technology, economics and history. But people aren’t just passive components within those systems, and understanding human behaviour means recognising the relationship between the individual and the circumstances around them.

The connections are what interest me

At first glance, there isn’t much connecting a fantasy series about competing realities, a history of the scientific revolution, a book about probability and an examination of modern digital culture.

But once you’ve read them, you start recognising familiar questions just in different forms.

• What makes something true?

• Who gets to decide which explanations are accepted?

• How do institutions acquire authority?

• Why do we see patterns that might not exist?

• What happens when power becomes separated from responsibility?

• And how does the way information is presented influence what we believe?

We encounter these questions constantly, without recognising that we’re dealing with variations of the same problem. Think about what happens when an educational assessment becomes a judgement about a student’s abilities, or when a business treats a personality profile as a reliable description of a person. Or how easily people accept an answer from an AI simply because it’s presented clearly, confidently and with enough detail to sound convincing.

In each case, there’s a tendency to accept the conclusion without spending enough time examining how it was reached. Reading across different subjects doesn’t give you some special ability to recognise the right answer every time. If anything, it should make you more conscious of how easily we can get things wrong.

There’s a danger in finding connections everywhere, too. Once you’ve encountered an idea that appeals to you, it’s tempting to start seeing evidence of it in everything you read. That’s where you have to be careful.

Two ideas can be similar without being connected. An analogy might help explain something without proving that the underlying mechanisms are the same. An interesting theory doesn’t become true just because it fits neatly with something you already believe.

If we’re serious about understanding anything, we have to be prepared to examine those connections, question them and occasionally discard them when the evidence doesn’t hold up. Otherwise, we’re not really learning. We’re just collecting arguments that support what we already think.

Reading isn’t about collecting answers

I don’t read these books because I’m looking for one grand explanation of how the world works. I doubt there’s a single explanation that would do the job, and I’m not suggesting everyone needs to read the same authors or reach the same conclusions.

What matters to me is encountering different ways of looking at things. Sometimes a book explains something I didn’t understand. Sometimes it challenges an assumption I’ve held for years. And occasionally, two books that seem to have absolutely nothing in common will suggest a connection I hadn’t considered before.

That connection leads to another subject, which leads to another book, and before long I’m looking at something I’d never originally intended to explore.

You don’t always know where an idea will take you. The more widely you read, the more you realise how much there is that you don’t know. I don’t find that discouraging. It means there’s always something else worth exploring.

Which brings me back to the first two articles in this series. We’ve become very comfortable measuring knowledge, classifying people according to their abilities and rewarding students for producing the expected answers.

But knowing the accepted explanation isn’t necessarily the same as understanding why it’s accepted, where its limitations lie or whether another explanation might make more sense. That’s why I think reading beyond familiar subjects is so valuable. It doesn’t guarantee we’ll reach the right conclusions, and it certainly doesn’t make us immune to our own biases.

Subscribe now

Every generation gets its own technology that’s going to ruin education. Apparently.

There’s something interesting about this recurring argument. We spend an enormous amount of time worrying about what students might stop doing when technology makes a task easier, and less time asking what they might be capable of doing instead.

Knowing the answer isn’t the same as understanding it

A calculator can perform an operation almost instantly. But someone still needs to know which operation to perform, what information to enter and whether the result makes sense.

A student can copy a formula without understanding it. They can also memorise the steps needed to solve a problem and reproduce them without understanding the mathematics behind them. That’s hardly new. Technology just makes the mechanical part easier.

If a student understands the underlying mathematics, a calculator allows them to spend less time on repetitive calculation and more time examining the problem itself. If they don’t understand the mathematics, the calculator might help them produce the wrong answer faster. The tool hasn’t solved the problem of understanding. It’s made the difference more obvious.

Writing isn’t much different. Someone can memorise grammatical rules and still struggle to communicate an idea. Another person might have a genuinely interesting argument but find it difficult to organise their thoughts into a coherent piece of writing.

We put considerable emphasis on the mechanics of producing the finished work. Spelling, grammar, structure, formatting and the expected answer all become part of how performance is measured.

Those things aren’t necessarily the same as understanding the subject. A beautifully written essay can contain very little original thought. And a badly written one can contain a rather good idea. Recognising the difference is the crucial part.

Children have a habit of getting there first

Back in 1996, Douglas Rushkoff published Playing the Future, later revised as ScreenAgers. His argument was that children were adapting to emerging digital technologies in ways adults often struggled to understand.

Traditionally, we assume knowledge moves from adults to children. Adults understand how the world works, and children learn from them. But digital technology was beginning to reverse parts of that relationship. Children were experimenting with computers, games and emerging online environments while many adults were still trying to understand what those technologies were for.

Rushkoff wasn’t suggesting that children suddenly knew more about everything. He was recognising that growing up inside a technological environment creates a different relationship with it. The children weren’t necessarily studying the technology. They were using it, experimenting with it and incorporating it into their everyday lives.

And now we’re watching that happen with AI. Except there’s an important difference. Earlier technologies helped students calculate, write, find information or communicate. AI can participate in the reasoning process itself. A student can ask it to explain a mathematical concept, challenge an argument, compare historical events or help develop an idea. They can also ask it to do their homework. Those are very different uses of the same technology.

What are students actually doing with AI?

According to a February 2026 Pew Research Center report, 54% of American teenagers said they’d used AI chatbots to help with schoolwork. That’s a sizeable portion of the student population. And it isn’t just about getting someone, or something, to write an essay.

Students are using AI to research subjects, solve mathematical problems, understand unfamiliar material and improve their writing. Of course, some are using it to avoid doing the work. That’s hardly surprising. Students have been finding shortcuts for as long as schools have been giving assignments.

But I think we’re missing something by treating AI primarily as another way to cheat. Imagine a student who’s struggling to understand a particular concept. The teacher explains it one way. The textbook explains it in much the same way. The student still doesn’t understand. Now they ask an AI.

It provides another explanation. That doesn’t work, so they ask again. Perhaps this time the system uses an analogy, breaks the problem into smaller parts or approaches it from a different direction. Eventually, something clicks. The student hasn’t avoided learning. They’ve found another way to learn.

has explored this distinction extensively through his Slow AI work. His argument isn’t that students should avoid AI, but that they need to develop the judgement to recognise when it’s helping them think and when it’s doing the thinking for them. That’s an important distinction.

There’s a difference between asking AI to explain something you don’t understand and asking it to produce an answer you have no intention of understanding. The problem isn’t necessarily the technology. It’s whether the student remains intellectually engaged with what the technology produces.

And unlike a classroom, where there may be twenty or thirty students competing for one teacher’s attention, an AI can respond repeatedly without making the student feel they’re holding everyone else back. That doesn’t make AI a replacement for a good teacher. It does make it a useful addition to the learning process.

The trouble with the expected answer

We frequently measure learning by asking students to reproduce information in a form we already recognise.

• Here’s the question.

• Here’s the expected answer.

• Explain it correctly, and you get the marks.

That approach works when we’re looking at foundational knowledge. Students do need to remember things: Vocabulary, mathematical principles, historical context and enough background knowledge to recognise when something doesn’t make sense.

But reproducing information isn’t understanding. A student might memorise the causes of the French Revolution and reproduce them perfectly on an examination. Ask that same student whether the Revolution would have happened without the financial crisis facing the French monarchy, and suddenly they have to think rather than recite.

Now they’re examining relationships, weighing evidence and considering alternatives. That’s a different level of intellectual engagement. And it’s the kind of work AI could help encourage. Instead of asking an AI to write an essay about the French Revolution, a student could present an argument and ask the system to challenge it.

• What have I overlooked?

• Does this evidence actually support my conclusion?

• Is there another explanation?

• What would someone who disagrees with me say?

That interaction doesn’t guarantee sound reasoning. AI can make mistakes, invent sources and reinforce the assumptions of the person using it. But used properly, it creates opportunities to practise something education claims to value: critical thinking. There’s irony in discouraging students from questioning an AI when we should be teaching them to question every source of information, including their teachers and textbooks.

There’s a catch, and it’s a big one

I’m not suggesting AI automatically improves learning. In fact, it can do the opposite. A student who asks AI to complete an assignment, copies the result and submits it may have learnt virtually nothing. They might even receive an excellent grade.

The OECD’s Digital Education Outlook 2026 discusses precisely this issue. Some students use generative AI to test their understanding, obtain explanations and improve their work. Others use it to replace the cognitive effort they would otherwise have had to extend.

A student who has worked through an argument with AI might produce a polished essay. So might a student who never read the assignment and simply copied the machine’s response. How do we distinguish between them? Maybe we need to pay more attention to the student’s ability to explain and defend what they’ve submitted.

Ask why they reached a particular conclusion. Have them explain which evidence they considered. Present a counterargument and see whether they can respond. If they understand the subject, that should be obvious.

None of this means abandoning exams or memorisation. Some knowledge needs to be readily available in our own heads if we’re going to reason effectively. You can’t challenge an AI’s explanation of mathematics, history or science when you have no basis for recognising what’s wrong.

Maybe the institutions need to learn something too

This brings us back to Rushkoff. His observation wasn’t simply that children were quicker at learning to use computers. It was that their relationship with technology was changing how they approached information, communication and problem-solving. The same thing is happening with AI.

Students are experimenting with ways to learn that weren’t available to previous generations. Some of those approaches will be useful. Others will create new problems, including dependence on tools that don’t always provide reliable information. Schools need to understand both.

Banning AI may make some administrative problems easier, but it doesn’t teach students how to use the technology responsibly. Equally, embracing AI without a full understanding of the implications is irresponsible. (There is already evidence of that in the real world in business and other areas.)

UNESCO has already called for human-centred approaches to generative AI in education, including appropriate safeguards, privacy protections and careful consideration of how these systems affect learning. That’s a more sensible starting point than treating the technology as either a miracle or a threat.

This begs the following question: What should a student be able to understand, explain and do independently, and where can technology genuinely help them develop those abilities?

We may be measuring the wrong thing. Again.

In the first article of this series, I looked at what happens when standardised assessments become judgements about people’s abilities rather than tools for understanding their educational needs. There’s a similar problem here.

If we judge learning entirely by the finished assignment, we risk confusing the quality of the output with the quality of the thinking. And if we assume that using AI means someone hasn’t done any thinking, we may be making the opposite mistake.

A student can spend hours producing an assignment without understanding the subject. Another can spend an hour questioning an AI, challenging its explanations and refining an argument, and come away with a much better understanding. Time spent isn’t necessarily a measure of intellectual effort. Neither is the number of words someone types without assistance.

That doesn’t mean we should stop teaching arithmetic, writing, research or the fundamentals of individual reasoning. Those are essential, especially if students are going to recognise when the technology produces nonsense.

But we should be thinking more carefully about which activities develop understanding and which demonstrate that a student can follow an established procedure.

AI hasn’t created that problem. It’s made it harder to ignore. And maybe that’s the opportunity we’re overlooking.

For generations, we’ve taught students how to find the right answers. Now we have machines that can produce answers in seconds, often with explanations that sound entirely convincing. Maybe it’s time we spent more effort teaching students how to recognise a good question, challenge an answer and understand why something is true.

Sources:

• Pew Research Center — How Teens Use and View AI (February 2026)

• OECD — Digital Education Outlook 2026

• UNESCO — Guidance for Generative AI in Education and Research

Subscribe now

We’ve been testing people’s intelligence for over a century. We give them questions, assign a score, compare it with everyone else’s, and then use that number to make decisions about their education, their abilities and, sometimes, their future. The problem with that idea is that we assume the result tells us more than it actually does.

Back in 1905, two French psychologists, Alfred Binet and Théodore Simon, developed an intelligence test to help identify children who were struggling in school. The idea was to find out who needed additional assistance so teachers could provide it. (As an FYI, my background is in Developmental Psychology, I fell into computers and systems because I felt it was a natural extension.)

It wasn’t supposed to be a permanent judgement of intelligence, or a prediction of what someone might achieve over their lifetime. Binet recognised that intellectual abilities could develop and that a child’s performance at one point didn’t necessarily determine their future.

But as intelligence testing spread, particularly in the United States, something changed. Scores became associated with a supposedly fixed, inherited intellectual ability. By the early twentieth century, intelligence tests were being used to support eugenics, justify discrimination and classify people according to assumptions about their potential.

In other words, a tool developed to help children learn was being used, in some circumstances, to decide how much they were capable of learning. Basically, it became weaponised to segregate people into groups and determine a supposed level of intelligence.

What are we actually measuring?

Imagine a child who speaks Czech, German or French fluently but has only recently started learning English. Now put that child in an English-speaking classroom and give them a standardised intelligence test that depends heavily on understanding English. What happens?

The child might struggle with the questions, misunderstand the instructions or simply lack the vocabulary needed to demonstrate what they know. The test will still produce a score. The scoring might even be perfectly accurate according to the rules of the assessment. But does that score tell us anything useful about the child’s intelligence, or have we primarily measured their ability to understand English?

We tend to assume that because something has been measured, it has also been understood. Those are two different things. A test can tell us how someone performed under particular conditions, not why they performed that way.

Was it a language barrier? A lack of familiarity with the material? The way the questions were presented? Was the child anxious, distracted or simply approaching the problem differently? Or was there a genuine learning difficulty that needed attention? Any, or all of those, could be at play. That doesn’t mean they all lead to the same conclusion.

***Modern assessments are better at accounting for language, culture and individual circumstances than many of their predecessors. That’s worth acknowledging. But even a well-designed test still has limitations.

When expectations become limitations

Suppose two students have similar underlying abilities, but one performs well on an assessment and the other doesn’t.

The first is considered capable. They’re encouraged to take more demanding courses, given challenging assignments and perhaps pushed towards university. The second is considered less capable. They’re given simpler work, fewer opportunities and lower expectations. Because of that test their educational experiences may now begin moving in different directions. Over time, those differences can influence how they perform.

Psychologists have studied how expectations affect behaviour and achievement. The effects aren’t uniform, and they shouldn’t be exaggerated, but expectations can influence the opportunities people receive and how others respond to them.

What happens when a system that was supposed to identify someone’s educational needs starts influencing what other people believe they’re capable of achieving? There’s a real difference between saying a student needs additional help with mathematics and saying that student isn’t capable of understanding mathematics.

I’m not arguing against additional educational support. Good support can make an enormous difference, and assessments can help identify where it’s needed. But support and lowered expectations aren’t the same thing. Unfortunately, institutions aren’t always very good at recognising that.

We do love putting people into categories

Schools need ways to organise resources. That’s understandable. They have budgets, limited teaching staff, curriculum requirements and students with very different needs. Standardised assessments can help identify those needs. They can also expose inequalities that might otherwise go unnoticed.

What happens when the classification becomes more important than the reason it was created. Give an institution a category, a procedure and a numerical score, and you’ve made administration easier. You’ve also made it tempting to stop asking questions.

• Instead of asking why a student is struggling, it’s easier to ask which programme or classification they belong in.

• Instead of looking at what might help them learn, it’s easier to look at what the assessment says they should be capable of doing.

That’s not necessarily the intention, but it’s built into systems that process large numbers of people. And it isn’t limited to education.

We see similar tendencies in healthcare, employment, insurance and government administration. Once a person has been assigned a classification, subsequent decisions can begin to revolve around that classification rather than the circumstances behind it.

We take those categories into the workplace

Schools aren’t the only institutions that like putting people into categories. Businesses have been doing it for years, under the banner of understanding employees, improving teamwork or finding the right person for a job. Consider Myers-Briggs, DISC and the various behavioural profiling tools used in corporate recruitment and management. (Full disclosure, I am certified in DiSC, MBTI and other systems, I do not endorse any of them.)

You answer a series of questions about how you behave, what you prefer and how you respond to particular situations. The system produces a profile, and suddenly you’re an introvert, an extrovert, an analytical thinker or some other combination of characteristics. The description sounds remarkably convincing.

But there’s something I’ve always found interesting about these tests. Once you recognise which questions are the control questions, it becomes relatively easy to influence the result. The questions that arise from this realisation:

• What exactly was being measured?

• Personality?

• Behaviour?

• Or the ability to recognise the characteristics the assessment was designed to identify?

To be fair, these tests aren’t all the same, and some have stronger scientific foundations than others. Properly validated workplace assessments can provide useful information, particularly when they’re used alongside interviews, demonstrated skills and actual work experience.

Even the Myers-Briggs Company says its assessment shouldn’t be used to select job candidates or predict job performance. That’s not what it was designed to do. Yet personality profiling continues to occupy a prominent place in corporate culture, sometimes influencing how people are managed, developed or perceived by their colleagues. You can even see people posting these results on dating apps.

Again, we encounter the same problem. A tool that might help someone understand their preferences becomes a label. That label becomes an expectation. And the expectation can influence how other people treat them.

Someone classified as introverted might be overlooked for a leadership opportunity, despite having excellent leadership abilities. Someone described as highly analytical might be assumed to lack interpersonal skills, even when their actual behaviour suggests otherwise. We’ve taken complicated human beings, reduced aspects of their behaviour to a handful of categories, and then act surprised when they don’t consistently fit those categories.

People aren’t static. They adapt to circumstances, learn new skills and behave differently depending on their environment. A questionnaire may capture something useful about those tendencies, but it can’t tell us everything about the person completing it.

Now we’re doing it with AI

We’re introducing AI into more and more areas of decision-making, including education. These systems can examine large amounts of information, recognise patterns and make predictions faster than any individual.

There are real benefits to being able to identify problems early or uncover relationships that might otherwise be missed.

But an AI system used to identify students who are likely to struggle academically, it might look at grades, attendance, previous assessments and other records, then assign each student a probability of future difficulty. Across thousands of students, that prediction might be statistically useful. What happens when a teacher or administrator starts treating the prediction as an assessment of an individual student’s potential?

What if the student has recently changed schools, is learning a new language, has experienced disruption at home or simply hasn’t responded well to the teaching methods being used?

If those circumstances aren’t properly represented in the data, the prediction is missing information. Even when the prediction is statistically sound, it still doesn’t tell us everything about the individual. We’ve taken an old problem and given it more computing power.

There’s also something about AI that makes this particularly difficult. It can produce confident, detailed explanations that sound remarkably authoritative. Add a probability, a graph or a carefully formatted report, and people may become even less inclined to question the result. A sophisticated explanation that is not accurate is still wrong.

The danger isn’t just that AI might make mistakes. Of course it will. Every system does. The danger is that we’ll make the same mistake we’ve made with standardised testing: assume that because a system can produce a result, it understands what that result means.

So should we get rid of the tests?

Intelligence testing and other cognitive assessments can provide valuable information when they’re properly designed, administered and interpreted. They can help identify difficulties with memory, reasoning, processing and learning that genuinely need attention. I’m not dismissing any of that potential. The issue becomes when we assume other information from those assessments that goes beyond what they are.

• A student might struggle because they haven’t learnt something yet. That’s very different from being unable to learn it.

• Someone might perform poorly in one environment and considerably better in another.

• And a person’s abilities aren’t frozen at the moment someone hands them a test.

We know people develop. We know their circumstances influence performance. We know that learning isn’t identical for everyone. Why are we so quick to turn a measurement into a conclusion about what someone can achieve?

Because numbers are convenient. They’re easier to compare, easier to report and easier to defend than the complicated reality of human development.

Understanding takes more than a score

I keep coming back to the original purpose of Binet’s work. He wanted to identify children who needed help learning. That seems a sensible objective, and there’s no reason we shouldn’t continue trying to do it better.

But somewhere along the way, we’ve repeatedly confused identifying a difficulty with defining a limitation. A test might tell us what someone knows today, or how they performed on a particular set of tasks. It might point towards something worth investigating. It might even reveal a genuine problem that needs attention. What it shouldn’t do, on its own, is establish how much we’re prepared to expect from that person tomorrow.

As AI becomes more involved in education, employment, healthcare and public administration, we’re going to have access to increasingly sophisticated forms of measurement. More data, more detailed predictions and more impressive-looking reports. That’s fine, provided we understand what they’re actually telling us.

Subscribe now

Most of what I’ve written here has focused on technology, cybersecurity, privacy and the assumptions we’ve built into the systems we depend on.

For the next few articles, I’m going to take a slightly different approach. Not because I’m moving away from those subjects, but because I want to look at some of the thinking that sits behind them.

I’ve always been interested in understanding how things work. Not just technology, but people, institutions, education, mathematics, history and the way societies organise themselves. Over the years, I’ve found that reading across seemingly unrelated subjects reveals connections that aren’t immediately obvious.

Writers such as Douglas Rushkoff, Marshall McLuhan, Neal Stephenson, Viktor Frankl, Leonard Mlodinow, Amir Aczel and Yuval Noah Harari have all contributed to that understanding in different ways. So have Giorgio de Santillana and Hertha von Dechend, whose Hamlet’s Mill explores the possibility that ancient myths preserved sophisticated astronomical knowledge. Their work raises interesting questions about how knowledge is developed, interpreted and passed between generations, even when some of their conclusions are seen as questionable.

These authors don’t necessarily agree with one another, nor do I always agree with them. That’s part of what makes their work interesting. I’m not looking for a single explanation of how the world works, but for different ways of understanding it.

One thing I’ve noticed is how easily we accept the conclusions of systems simply because we’ve been taught to trust them. We see it in education, where standardised assessments can become judgements about someone’s abilities. We see it in institutions, where classifications and procedures can replace individual understanding. And increasingly, we’re seeing it in technology, where algorithms and AI systems can produce results that appear authoritative without necessarily being correct.

To me, these aren’t separate issues. They’re different expressions of the same underlying problem: the tendency to confuse information with understanding, measurement with judgement, and institutional authority with evidence. That’s what I want to explore in this short series.

I’ll be looking at standardised testing, education, how language influences perception, the way we learn, and the growing role of AI in shaping what we think we know.

It’s difficult to discuss any of these subjects today without considering the influence of technology. But the focus will be broader than cybersecurity alone. Yes, I will state outright that these are my opinions, my perspective and you may disagree with my points. What I’m asking for is for you to read these and question your perspectives on these points and issues.

Subscribe now

I recently came across a statement on Substack that caught my attention:

Myth: Centralised databases are secure.

Reality: They are honeypots. True security comes from peer-to-peer encryption where no one holds the master key.

It’s a compelling statement. Short, provocative and just plausible enough that people might accept it without questioning the underlying assumptions. There’s a real concern behind it. Centralised databases can contain enormous amounts of valuable information. And concentrating sensitive information in one place can increase the consequences of a successful breach.

But, peer-to-peer encryption without a master key, doesn’t necessarily address the problem. In fact, it points at something I’ve been discussing for some time. The problem isn’t necessarily centralisation. Nor is decentralisation automatically the answer. It’s what happens when a system is compromised.

Centralised doesn’t mean insecure

Let’s start with the idea that centralised databases are inherently insecure. Banks, hospitals, governments and businesses have been using centralised databases for decades. Many of these systems are well protected, with carefully managed access controls, encryption, monitoring and extensive security procedures.

Centralisation can actually make some aspects of security easier. Organisations can maintain consistent policies, control access, monitor suspicious activity and apply updates across their infrastructure. Of course, centralisation also creates risks.

When large amounts of valuable information are concentrated in one environment, that environment becomes a particularly attractive target. Compromise the right system, account or application, and an attacker gains access to information belonging to thousands or millions of people. We’ve seen enough breaches over the years to understand the consequences.

But calling centralised databases honeypots isn’t accurate. In cybersecurity, a honeypot is normally a deliberately created decoy intended to attract attackers. A database containing sensitive information isn’t a honeypot simply because criminals would like to get into it.

Decentralisation doesn’t make vulnerabilities disappear

Decentralised systems distribute information, processing or control across multiple locations or participants. Done properly, this can reduce dependence on individual systems and eliminate certain single points of failure. But distributing information also means distributing responsibility for protecting it.

Instead of securing one carefully controlled environment, we’re now dealing with dozens, thousands or even millions of devices, each with its own operating system, software vulnerabilities and potential weaknesses. Some will be well maintained. Others won’t.

A compromised endpoint can still expose information. Encryption keys can still be stolen. Users can still be deceived into granting access. And once information is decrypted for use, it can become accessible to whatever has compromised that particular device. Peer-to-peer encryption doesn’t eliminate those possibilities.

There’s also an important distinction between decentralisation and encryption. They’re different architectural choices, and neither needs the other.

A centralised service can use end-to-end encryption without holding the keys needed to decrypt its users’ information. Similarly, a decentralised system can be designed so poorly that sensitive information stays exposed. Eliminating a single master key can reduce the consequences of one key being compromised. But it doesn’t remove the need for key management or protecting information after an authorised endpoint decrypts it.

Encryption has limitations

Encrypting information while it’s stored, called encryption at rest, is standard practice in many environments. Encrypting information while it’s transmitted between systems, or encryption in transit, likewise. Without them, the consequences of many security failures would be considerably worse.

But there’s a third state. In conventional systems, information is decrypted so an application can process it. That application may be checking a financial transaction, analysing medical records or verifying someone’s identity. The system is authorised to perform the operation. The user may have explicitly approved it. But authorisation doesn’t mean security.

A vulnerability in the application, a compromised administrator account or malicious software running within the environment may provide access to information that was previously encrypted. Encryption at rest and in transit won’t necessarily help once the information has been decrypted. And that’s the architectural issue I’ve been ranting about.

Why do we need so much exposure?

Suppose a service needs to establish that someone is over 18. In traditional identity-verification processes, the person provides a date of birth, identification document or other personal information. The system get more information than it actually needs. It doesn’t need to know someone’s name, address or exact date of birth. It needs to establish whether that person meets, or exceeds, an age requirement. So why expose everything else?

ZKPs offer an option. They let someone to prove that a statement is true without disclosing all the underlying information, provided the proof is built around appropriate, trusted credentials. Likewise FHE. It lets computations be performed on encrypted data without first decrypting the inputs.

Both offer possibilities for organisations that need to analyse sensitive information without exposing the underlying records to the computing environment performing the analysis. Neither technology is a universal solution. FHE has computational overhead, not every application is suitable for it, and ZKPs mean careful design. Neither makes compromised endpoints harmless or removes the need for conventional security measures.

But they challenge an assumption that’s been part of computing for a very long time: that information must be available in plaintext to the application processing it.

We don’t necessarily need to abandon centralised infrastructure

Imagine a hospital maintaining a centralised database of patient records. There are good reasons for keeping records in a managed environment. Medical professionals need accurate, accessible information. Administrators need to maintain data integrity, enforce access controls and meet regulatory obligations. Decentralising everything wouldn’t automatically improve any of that.

But the hospital assume that every application processing sensitive patient information needs unrestricted access to the underlying records either. For suitable analytical workloads, encrypted computation could allow information to be processed without exposing the plaintext records to the processing service.

For specific verification tasks, ZKPs could establish facts without disclosing unnecessary personal information. The database could remain centralised. The organisation could retain its existing governance structure. Its access controls, monitoring and other protective measures would still be necessary. What changes is the amount of sensitive information that needs to be exposed during particular operations.

Some healthcare tasks need clinicians to see actual patient information. Encrypted computation doesn’t eliminate that requirement, and it shouldn’t. The objective is to avoid exposing more information than an operation requires. The same thinking can apply to financial services, government systems and other environments handling sensitive information.

The attacker’s advantage

In my recent article, How AI Changed the Rules, I discussed how AI is beginning to change the economics of exploiting software vulnerabilities. The PHP2Shell incident provided a useful example of how AI-assisted security research can make previously difficult vulnerabilities more practical to exploit.

That matters because our defensive systems are largely built around finding vulnerabilities, preventing exploitation and responding when something goes wrong. But there will always be vulnerabilities we haven’t discovered, configurations we haven’t examined and attacks we haven’t anticipated. AI may make some of those weaknesses easier to find and exploit. It may also help defenders identify and correct them more quickly. We can’t guarantee that every system will remain uncompromised.

Shouldn’t we also be designing systems around limiting the consequences when they aren’t?

Reducing unnecessary access, limiting the exposure of plaintext information, separating cryptographic keys from the systems processing data, and considering whether some computations can take place without decryption.

Security isn’t a choice between centralised and decentralised

The original Substack statement gets one thing right: concentrating sensitive information creates risks. But it goes too far in suggesting that decentralisation and peer-to-peer encryption are the answer.

They can be useful. So can centralised databases, properly designed encryption systems and conventional security controls. The right architecture depends on what we’re protecting, how the information needs to be used and which risks we’re trying to reduce.

I think the industry has spent too much time treating cybersecurity as a competition between products, architectures and increasingly elaborate defensive measures. We’ve improved individual technologies but still designing systems in which gaining authorised computing access can expose far more information than necessary. That’s the assumption I think we should be questioning.

Not whether centralisation is good or bad. Not whether peer-to-peer systems are automatically safer. And not whether one particular encryption technology can solve every security problem.

Subscribe now

For years, I’ve argued that cybersecurity hasn’t fundamentally changed despite what the industry says. We’ve developed better tools, improved detection and built more sophisticated systems to monitor threats. But much of our thinking still revolves around the same basic idea: identify a vulnerability, prevent someone from exploiting it, and respond quickly when something goes wrong.

That approach has worked reasonably well, although the steady stream of breaches suggests there’s room for improvement.

Now AI is starting to complicate things. Not because it’s inventing entirely new forms of attack, but because it’s capable of finding and exploiting weaknesses we’ve been living with for years. And last week’s PHP2Shell incident is a rather good example.

The vulnerability was there

On October 21, researchers disclosed two critical vulnerabilities in PHP, the programming language running a substantial chink of the internet, including millions of websites built on platforms such as WordPress. These vulnerabilities could potentially allow attackers to execute malicious code remotely, exposing sensitive information or compromising affected systems. The interesting part isn’t that researchers discovered another software vulnerability. It’s how they got there.

As WIRED reported on October 24, some of the underlying problems had been identified as far back as 2024. At the time, exploiting them was considered sufficiently difficult that the risks weren’t really addressed. So, what did we do in the meantime? Funny you should ask!

Researchers used advanced AI models, alongside human expertise, to find practical ways of exploiting the weaknesses. Think about that for a moment. The vulnerabilities weren’t necessarily new. What changed was our ability to work out how to exploit them. Within hours of the public disclosure, Cloudflare was reporting millions of attempted exploits. That doesn’t mean millions of systems were compromised, that’s an important distinction in this case. As of October 24, there were no confirmed successful attacks exploiting these particular vulnerabilities, and some of the exploit code circulating online apparently didn’t work.

Nevertheless, the response shows how quickly attackers can move once a potential weakness becomes public. It also begs the question: How many other vulnerabilities have been sitting in software for years, considered too complicated or impractical to exploit, that AI might help make accessible?

We don’t know. And anyone claiming AI can suddenly uncover and exploit every hidden software flaw is getting ahead of the evidence. What we do know is that the difficulty of exploiting a vulnerability can’t be seen as a barrier.

We’ve been improving the tools, not the assumptions

Here’s where I think the cybersecurity industry needs to reconsider its approach.

For decades, we’ve focused much of our effort on preventing unauthorised access. Firewalls, intrusion detection, endpoint protection, vulnerability scanners and increasingly sophisticated monitoring platforms all play important roles. Dismissing them would be ridiculous.

Cloudflare, for example, had protections in place before the PHP vulnerabilities were publicly disclosed. That’s a good example of the industry doing exactly what it should. But there’s a difference between improving our ability to defend systems and changing the underlying security model.

A great deal of today’s infrastructure still assumes that once an application or authorised user has access to sensitive information, that information can be decrypted and processed within the environment. But what happens when an attacker finds a way around those controls?

We already know the answer. Depending on what the compromised component can access, information may be exposed, copied or manipulated. That’s not a new problem. It’s been with us for decades.

And while cybersecurity companies have become really good at developing new products to address each generation of threats, we haven’t always been as willing to reconsider the architecture that makes those threats possible.

AI changes the economics

What interests me most about PHP2Shell is what AI potentially does to the cost and effort of developing an exploit. Traditionally, finding and weaponising complicated software vulnerabilities needed substantial expertise, time and resources. That didn’t prevent attacks, but it did limit who could realistically carry them out. AI is lowering some of those barriers.

A skilled researcher can use AI to examine code, explore possible weaknesses, generate tests and investigate approaches that might otherwise consume considerable time. The researcher still needs to understand what they’re doing. AI doesn’t turn an inexperienced attacker into a highly capable exploit developer. It does reduce the time and effort needed for parts of that work matters.

A developer can’t just apply every patch without considering what it might break. A hospital can’t casually take a critical system offline. A bank has to test changes against a complicated collection of applications, services and dependencies.

While an attacker only needs to find one workable route into a target. That imbalance has always existed. AI has the potential to make it worse. Conversely, adding AI to existing defensive products, while useful, doesn’t necessarily resolve it. We’re still competing over who can find, exploit or close a vulnerability first.

Maybe we’re asking the wrong question

I’ve increasingly come to believe that preventing unauthorised access, while essential, shouldn’t be the only consideration when designing secure systems. We also need to think seriously about what happens when prevention fails.

That means looking at why sensitive information needs to be exposed during processing, why applications sometimes have broader access than they need, and why compromising one component can provide access to information far beyond its intended purpose. There are technologies that can address parts of this problem.

FHE allows certain computations to be performed on encrypted information without first decrypting it. ZKPs can verify claims without exposing the underlying information. Neither is a universal solution. They introduce computational costs, require careful implementation and aren’t appropriate for every application. They certainly wouldn’t prevent a PHP vulnerability from being exploited. Yes, I realise that given what our organisation does with FHE and ZKP, this may be anathema. But, we don’t make money by forcing rectabgular pegs into square holes.

But properly designed systems using these techniques can reduce how much information is exposed to a compromised component, particularly when encryption keys and plaintext remain outside its reach. Rather than depending entirely on keeping attackers out, we can design systems that limit what attackers can learn even when part of the system has been compromised.

It’s not really about PHP

PHP2Shell will eventually fade from the headlines. Patches will be deployed, security products will be updated, and attention will move to the next vulnerability. That’s how this industry works.

AI is making sophisticated security research more accessible and, in some cases, faster. That’s good news for researchers and defenders, but the same capabilities can also benefit attackers. We can’t assume the next vulnerability will be difficult because nobody has managed to exploit it yet. Nor can we assume that adding another detection capability to an existing security stack will address every consequence of a successful intrusion.

I’ve said before that cybersecurity has spent decades evolving its tools while leaving many of its original assumptions intact. PHP2Shell doesn’t prove that the entire industry needs to be rebuilt. It does, however, offer another reason to examine those assumptions. Patching, monitoring and access controls aren’t going away. We need them, and we’re going to need better versions of them.

Subscribe now

Further reading: WIRED — The PHP Meltdown Shows How AI Makes Bugs More Exploitable; Cloudflare — PHP2Shell Vulnerability and WAF Protection; CrowdStrike — Obstacles to Weaponizing PHP2Shell.

An audit trail doesn’t tell you whether a decision was correct, fair or justified. It tells you what happened inside a system: who started an action, what data was processed, which rule was applied, what result came out and, perhaps, who signed it off.

That’s useful information. It can show whether people followed the process. What it can’t establish, on its own, is whether the process made sense in the first place.

As software, analytics and AI take on a bigger role in decisions that affect people’s lives, that distinction matters. Being able to trace a decision doesn’t mean you can justify it. A perfect record of a flawed decision is still a record of a flawed decision.

The Post Office Horizon scandal makes that painfully clear. The system produced records. Shortfalls were recorded, transactions were documented, investigations took place and decisions were acted on. For years, computer-generated evidence and the procedures surrounding it were given enough authority to support accusations, demands for repayment, suspensions and prosecutions.

The problem wasn’t simply whether records existed. It was whether those records deserved the trust placed in them.

The inquiry into Horizon and the treatment of sub-postmasters should force every organisation to ask what its own records actually prove. A log can show that a system reported a discrepancy, that someone reviewed it and that the required procedure was followed. None of that necessarily establishes that the discrepancy was real, that the procedure was sensible or that the decision was justified.

That’s where things start to go wrong.

Audit trails are essential in any well-run system. Without them, investigations become guesswork. It’s harder to establish who did what, identify unauthorised access, work out whether controls were bypassed or reconstruct an incident.

The same applies to AI. Records of model versions, prompts, retrieved information, outputs and human approvals can help investigators understand how a system reached a result.

But there’s a limit to what those records can tell us. They’re good at answering “What happened?” They don’t, by themselves, answer “Was there a good reason for it?”

Consider a claims system. The rules have been approved, the thresholds documented and every step logged. Each claim gets a risk score. Anything above a certain threshold is escalated, a case officer reviews the file and further action follows.

On paper, everything looks orderly. But what if the scoring rule is flawed?

Perhaps it puts too much weight on something that has little to do with fraud. Perhaps it treats missing paperwork as evidence of dishonesty. Maybe it was built using data shaped by years of questionable decisions. Or someone made a policy choice that quietly turned uncertainty into a presumption of wrongdoing.

The audit trail will faithfully record the claim arriving, the score being calculated, the threshold being crossed and the case officer approving the result. Every step can be documented, every action attributed and every approval traced. The decision can still be wrong.

A flawed rule doesn’t improve because it’s applied consistently. And a harmful decision doesn’t become legitimate because someone kept excellent records.

This is also where some of the discussion around AI governance gets muddled. Transparency, explainability, accountability and documentation all matter. Systems that influence consequential decisions should be open to scrutiny.

But an organisation can show you which model it used, which documents it retrieved, what prompts it issued and who approved the output without establishing that the result was trustworthy.

NIST’s AI Risk Management Framework makes that distinction. It treats transparency as one part of trustworthy AI, alongside validity, reliability, safety, security, privacy and fairness. Being transparent doesn’t let a system skip the rest.

A system can be transparent and inaccurate. It can have clear lines of accountability and still treat people unfairly. It can even explain the wrong decision with impressive clarity.

Keeping the prompt history, saving the output and identifying the reviewer doesn’t finish the governance work. It gives the organisation material to examine. There’s still a separate question to answer: was the result good enough to act on?

Once an organisation has invested heavily in logging, controls and formal procedures, those records can also become a defence against criticism. The response starts to sound familiar: the case was reviewed, the required steps were followed, the score crossed the threshold and the approval was recorded. The system did what it was designed to do.

All of that may be true. The design may still be the problem.

The records might establish little more than that the organisation applied its own logic consistently. They don’t necessarily show that the logic was sound.

That helps explain why failures can survive for years inside institutions that appear disciplined and well managed. Documentation reassures people. Managers can point to procedures, auditors can examine sign-offs and everyone can demonstrate that they completed their part.

Meanwhile, the question that matters gets pushed aside. Instead of asking whether the decision was justified, the organisation checks whether the procedure was followed. Instead of examining the rule, it verifies that the rule was applied correctly. Instead of challenging the model’s assumptions, it confirms that the approved version was used.

Eventually, checking compliance can take the place of checking whether something is true.

There’s another weakness in many automated and partly automated systems: they’re much better at recording decisions than recording doubts.

They capture the inputs, the outputs and the rule or model connecting them. They may be far less thorough about recording what was missing, what was disputed or what didn’t fit the conclusion.

What other explanations were considered? Was there evidence pointing the other way? Did the result depend heavily on one uncertain variable? Which parts reflected human judgement rather than an objective measurement?

These questions matter because they tell us how much confidence a decision deserves.

A useful record shouldn’t just show that a risk score was high. It should help explain what drove it, how dependable the conclusion was and where uncertainty remained. Similarly, recording that a person approved an outcome tells us very little unless we know what they reviewed and whether they had a real opportunity to challenge it.

Did they see the contradictory evidence? Did they have enough time? Could they stop the process, or were they effectively there to click “approve”?

Without that context, the organisation preserves the decision while losing the doubts that should have limited its confidence. Many decisions that affect people’s lives involve judgement, incomplete information and competing explanations. Giving them a numerical score doesn’t make that uncertainty disappear.

A harmful system doesn’t have to look chaotic or careless. It can be highly organised, thoroughly documented and internally consistent. It can have policies, dashboards, escalation procedures and a signature in every required box.

That can make it especially difficult to challenge. Everyone can show that they followed the rules, and the organisation looks responsible precisely because it has so much paperwork to prove it.

Software adds another layer of apparent authority. Numbers look neutral, logs look factual and dashboards look objective. A computer-generated case file can seem more credible than the person disputing it, even when the system is simply processing bad assumptions efficiently.

Auditability should make it easier to question a system. It shouldn’t become a reason to stop questioning it.

A stronger approach would use audit records and supporting evidence to answer five questions:

1. What happened? Who initiated the action, what did the system do and when did it happen?

2. What did the decision depend on? Where did the information come from, and which rules, thresholds, sources or model versions shaped the result?

3. Where was the uncertainty? What was missing, which assumptions were disputed and what evidence contradicted the conclusion?

4. Could anyone meaningfully challenge it? Who could question the result, what information could they access and could they get an error corrected before it caused harm?

5. Had the rules themselves been tested? What evidence showed that the criteria were accurate, fair, proportionate and still suitable for the task?

The fifth question is often the uncomfortable one. It’s easy to log that a rule was triggered. Explaining why that rule deserves to exist takes more work. Recording an approval is straightforward. Demonstrating that the person approving it had the information, time and authority to disagree is harder.

Yet those are exactly the questions governance needs to address.

Too often, organisations blur three different claims: something happened, the procedure was followed and the outcome was justified. Those claims require different evidence.

A timestamp can establish that a system classified someone as high risk. A workflow log can show that the classification was reviewed according to procedure. Neither establishes that the person deserved the classification or that the action taken against them was proportionate.

To reach that conclusion, someone still has to examine the quality of the information, the suitability of the criteria, the circumstances of the case and the consequences of getting it wrong.

Automation makes this more pressing because it allows organisations to produce decisions quickly and at scale, complete with convincing records of every step. Speed and consistency are useful only if there’s a sound basis for what the system is doing. Otherwise, they help an organisation repeat its mistakes faster.

Monday’s article looked at cybersecurity’s habit of refining defences around outdated assumptions. Tuesday examined how attackers can exploit permissions a system has already granted. Wednesday considered how a system can compute correctly while relying on false information.

The same concern carries through to the decisions organisations make using those systems. Even when access is legitimate, the software runs as expected and the inputs appear valid, the resulting decision still needs scrutiny.

Good records are essential to that scrutiny. They preserve evidence, support investigations and help people challenge what happened. But their existence doesn’t settle the argument.

An audit trail can show that the machine ran, the rule was triggered, the case was reviewed and the decision was approved. The organisation still has to explain why anyone should have trusted the result.

Subscribe now

As automated systems assume greater responsibility for financial decisions, healthcare recommendations, security operations and critical infrastructure, organisations increasingly reassure us that a human remains in control. But what does that control actually mean? A reviewer who cannot independently evaluate the evidence, lacks time to investigate or has no practical authority to intervene isn’t providing meaningful oversight. They’re providing procedural approval. The distinction matters because humans can become the final step in legitimising decisions generated by systems they don’t fully understand or have little opportunity to challenge. Genuine accountability requires more than putting someone between an automated recommendation and its execution. It requires designing the system so that intervention is informed, practical and consequential.

There Was a Human Behind the Wheel

On the evening of 18 March 2018, an Uber test vehicle was driving through Tempe, Arizona, under the control of an experimental automated driving system. Behind the wheel sat a human safety operator, there to watch the road and take over if the technology got something wrong.

It was a reassuring arrangement. The machine would do the driving. The human would catch its mistakes.

Then the vehicle struck and killed a pedestrian.

The automated system had detected her roughly 5.6 seconds before impact, but struggled to identify what it was seeing and predict where she was going. Its design prevented it from applying emergency braking to mitigate the collision, leaving the human operator to intervene. The operator, distracted by her mobile phone, didn’t respond in time.

The US National Transportation Safety Board identified the operator’s failure to monitor the road as the probable cause. But its investigation also found inadequate safety risk assessment, ineffective oversight of operators and insufficient safeguards against automation complacency at Uber.

The operator was responsible for paying attention. Uber was responsible for building and managing a system that relied on her attention, including accounting for the foreseeable possibility that it would fail.

A person was in the driver’s seat. She could take control. Human oversight existed in the design. When it mattered, that oversight didn’t work.

Source: US National Transportation Safety Board investigation

Most automated decisions don’t unfold on a road, with seconds left to prevent a collision. The risks and opportunities to intervene vary enormously. But the assumption behind that arrangement turns up almost everywhere we’re deploying automation.

The system does the work. A person supervises. If anyone asks who’s responsible, there’s a familiar answer: “We have a human in the loop.”

Apparently, that’s supposed to settle it.

Consider a reviewer sitting in front of a financial risk dashboard. A transaction has been flagged. The screen shows a risk score, several supporting indicators and a recommendation to reject it. There are two buttons at the bottom.

The reviewer clicks approve, and the organisation now has a human decision on record.

What it doesn’t necessarily have is evidence that anyone meaningfully examined the recommendation. Did the reviewer see the underlying records? Could they tell whether the information was reliable? Were there facts pointing towards a different explanation? Could they ask for more evidence, and did they have time to wait for it?

Then there’s the question that rarely appears in the product demonstration: what happens to the reviewer who keeps disagreeing?

If challenging the system means missing productivity targets, attracting managerial attention or writing a lengthy explanation for delaying the queue, the approval button starts to look considerably more inviting.

The person still has authority on paper. The working environment tells them how sparingly they’re expected to use it.

Human judgement can add enormous value. People can recognise context, question assumptions and spot circumstances that a model has missed. But those abilities don’t magically become part of a process because someone has been assigned an approval task.

They need room to operate.

Imagine doing that task hundreds of times a day. Most recommendations look reasonable. Most turn out to be routine. The interface is tidy, the supporting explanation sounds convincing and the queue never seems to shrink.

At first, perhaps you examine each case closely. Over time, you learn what the system usually gets right and start looking for anything unusual. That’s a reasonable way to cope with the workload. It’s also a change in your job.

You’re now watching for exceptions rather than independently assessing every decision.

Researchers describe related problems as automation bias and automation complacency: people can place too much weight on automated advice or become less attentive when a system appears dependable. NIST’s AI Risk Management Framework recognises that human-AI interaction introduces its own risks, including cognitive biases and poorly defined responsibilities. Combining a person and a machine doesn’t guarantee better decisions.

The difficulty becomes obvious when the machine produces an error that looks perfectly ordinary.

There’s no flashing warning. The score is plausible, the explanation reads well and the familiar approval screen appears. If the reviewer has learned to look for visible exceptions, an error presented as business as usual may pass straight through.

That doesn’t mean oversight is hopeless. It means organisations have to design for the way people actually work, including what happens to their attention after the two hundredth apparently sensible recommendation.

“Someone will check it” isn’t much of a design specification.

There’s another complication hidden inside that tidy dashboard. The system recommending the decision may also be choosing the evidence the reviewer gets to see.

Take an AI-assisted fraud investigation. The software identifies an unusual transaction, retrieves account information, assesses behavioural patterns and packages the result for an investigator. The summary contains suspicious activity, a confidence score and a proposed response.

It looks like a case ready for review. But it’s already the product of several decisions about what matters.

Perhaps the customer is travelling. Perhaps a recent change in employment explains the unusual activity. Perhaps information elsewhere in the account undermines the fraud theory, but the system didn’t retrieve it. The reviewer may never know what’s missing.

They’re being asked to challenge the machine using a case file assembled by the machine.

That can still be useful, provided the reviewer can get beyond the summary. They need access to underlying records, competing explanations and information that might contradict the recommendation. Otherwise, the review can become a guided tour of the system’s preferred conclusion.

A convincing explanation isn’t necessarily a complete one. Sometimes it’s simply the explanation that survived the filtering.

The organisational pressures make this harder. Automation is usually introduced with a promise of greater speed, lower costs or more consistent results. Those are reasonable goals. But achieving them often means reducing the human effort spent on each case.

The software starts producing recommendations much faster than the old manual process. Soon, the review stage becomes the bottleneck. So the summaries get shorter, the interface gets simpler and the expected number of approvals per hour rises.

Eventually, the organisation has a process that depends on rapid approval while still claiming the protection of careful human judgement.

Something has to give.

Often, it’s the judgement. The signature remains.

You could call this responsibility laundering: an automated recommendation gains legitimacy because a person approved it, even though the organisation never gave that person a realistic opportunity to challenge it.

That doesn’t require a deliberate plan to shift blame. It can emerge through ordinary operational decisions. A higher target here, a staffing reduction there, a new dashboard that removes “unnecessary” detail. Each change looks manageable. Together, they can hollow out the safeguard.

When something goes wrong, however, the approval is still there in the record. The organisation can point to it and say a human made the final decision.

Whether that person was genuinely in control is a different question.

AI agents make this problem more complicated because the work may be well under way before an approval screen appears.

A conventional decision-support system might calculate a score and wait. An agent can choose tools, retrieve documents, query databases and carry out intermediate tasks as it works towards an objective. Depending on its permissions, it may also create tickets, change settings or initiate other actions.

Picture an enterprise agent investigating an operational anomaly. By the time it presents a recommendation, it has already accessed several systems and made a series of choices about what to examine, what to ignore and which sources to trust.

A reviewer approves the proposed response. What, exactly, does that approval cover?

It doesn’t establish that every earlier retrieval was appropriate. It doesn’t show that the agent resisted misleading instructions embedded in a document or considered evidence that contradicted its conclusion. And rejecting the final recommendation won’t necessarily undo changes it has already made.

Showing the reviewer a complete activity log helps with traceability, but it doesn’t solve the practical problem. If every routine approval requires someone to reconstruct hundreds of operations, the workflow will either grind to a halt or encourage people to skim.

Oversight has to be built around the actions and their consequences. Some operations can proceed within tightly defined limits. Others need separate authorisation before they happen. Some shouldn’t be delegated at all.

The final approval button can’t carry the weight of an entire architecture.

Article 14 of the EU AI Act takes this distinction seriously in its approach to human oversight of high-risk AI systems. It goes beyond placing a person somewhere in the workflow. It addresses whether the people responsible can understand the system’s capabilities and limitations, interpret its outputs, recognise automation bias and intervene.

It also addresses their ability to disregard, override or reverse an output, and to interrupt operation safely.

That’s a much more useful way to frame the issue. Oversight is something a person must be able to exercise.

Source: EU AI Act, Article 14

The hard part is making those capabilities real during an ordinary working day.

An override mechanism is of limited use if staff are discouraged from touching it. Training about system limitations won’t compensate for evidence hidden behind an inaccessible interface. A policy promising careful review won’t create extra hours in someone’s shift.

The test is what happens when someone says, “I don’t trust this result, and I’m stopping the process.”

Can they do it? Do they know what to check next? Will the organisation support a reasonable challenge, even when it turns out that the system was right?

Those answers tell us more about oversight than the presence of an approval field.

There’s also a point at which asking for more human supervision becomes the wrong response to a technical design problem.

If an application can export an entire customer database, perhaps the first question should be why it has that much access. If an agent can initiate large financial transactions, independently enforced limits and separation of duties may provide stronger protection than expecting someone to watch every action. If a task only needs an aggregate result, the workflow may not need to expose individual records at all.

Human attention is a scarce resource. Using it to compensate for avoidable architectural weaknesses is an expensive gamble.

Granular authorisation, compartmentalisation, data minimisation and interfaces designed for specific purposes can reduce the number of dangerous actions available in the first place.

For suitable workloads, privacy-preserving cryptography can help too. Fully homomorphic encryption allows computation on encrypted data without exposing the underlying plaintext to the processing environment. Zero-knowledge proofs can establish defined claims without revealing the private information used to support them.

Neither technology decides whether a business policy is fair or a recommendation sensible. Both depend on correct implementation, and they bring practical costs and constraints. But they illustrate what it means to address a risk in the design rather than leave everything to the person watching the screen.

If the processing environment doesn’t receive plaintext, a reviewer doesn’t have to rely solely on spotting its misuse there. If an operation requires separately enforced authority, an agent can’t obtain that authority simply by producing a persuasive explanation.

That leaves people more capacity for the work that actually needs judgement: conflicting evidence, uncertain consequences, exceptional circumstances and decisions about what should happen next.

For that judgement to count, several things have to come together. Reviewers need relevant evidence, including information that might undermine the recommendation. They need enough time and the right expertise to assess it. They need authority to disagree and a practical way to intervene before the consequences become irreversible.

These conditions should be tested under pressure. A process that works during a demonstration may behave very differently when the queue is full, information is missing and a confident recommendation happens to be wrong.

Can reviewers catch a plausible mistake? Can they recognise when they don’t have enough information? Can they stop an action that appears routine but exceeds the system’s authority?

Counting approvals won’t answer those questions. Neither will counting disagreements on its own. A reviewer who challenges everything may be struggling just as much as one who challenges nothing. What matters is whether the intervention is informed and appropriate.

The previous article looked at how an audit trail can faithfully document a flawed decision. Human approval can add another reassuring layer without correcting the underlying problem.

The machine recommended it, a person approved it and the log recorded it. Procedurally, the file is complete. There may still be no sound reason to trust the outcome.

The approval doesn’t establish that the reviewer understood the recommendation, saw reliable evidence or considered an alternative. It certainly doesn’t establish that the organisation put them in a position to succeed.

Responsibility has to extend to the people who designed the workflow, set the targets, chose the permissions and decided what evidence would be visible. A reviewer at the end of that chain can’t reasonably carry all the responsibility for decisions made further up it.

That’s what makes the Tempe crash such a troubling place to begin. The operator’s distraction mattered. So did the design and management of a system that depended on her responding when the automation failed. Putting her in the driver’s seat didn’t resolve the risks created by that dependency.

Other failures of oversight will play out less visibly: a blocked payment, an unfair decision, a privacy breach or an automated response that makes an incident worse. The consequences differ, but the design question remains.

What gives the person supervising the system a realistic chance of catching the problem and doing something about it?

Across this series, we’ve examined the assumptions beneath familiar security assurances: that established defences remain adequate, authorised access is safe, correct computation means trustworthy results and complete records justify a decision.

Human involvement deserves the same scrutiny. A capable reviewer with good evidence, sufficient time and genuine authority can be an essential safeguard. Those conditions have to be built, maintained and tested.

Before an organisation tells us there’s a human in the loop, it should be able to explain what that person can see, what they can question and what happens when they say no.

Subscribe now

What happens when false or manipulated information enters a system through an apparently legitimate source? The resulting calculations may be mathematically flawless, the communications encrypted and the audit trail complete. The decision can still be catastrophically wrong. As organisations increasingly delegate decisions to algorithms and autonomous AI agents, the integrity of the information those systems depend is as important as protecting it from disclosure. Cryptography can establish powerful guarantees about information and computation, but those guarantees have boundaries. Understanding where they end is essential to building systems we can trust.

The Announcement That Wasn’t

At 4:11 p.m. Eastern Time on 9 January 2024, the official X account of the United States Securities and Exchange Commission announced that the agency had approved spot bitcoin exchange-traded funds. The problem? The SEC hadn’t made that decision.

An unauthorised party had gained control of the phone number associated with the agency’s account and used the compromised account to publish the announcement. It showed up under the identity of the regulator responsible for making the decision.

The SEC did confirm that the announcement was unauthorised and that its social media account had been compromised. The agency didn’t approve the exchange-traded products until the following day. The problem was the message really had appeared on the SEC’s official account.

A monitoring system that captured the announcement would have recorded a genuine event: a particular account published a particular statement at a particular time. An automated process treating that statement as confirmed regulatory information could have drawn a different conclusion. The difference between recording an event accurately and establishing the truth of its contents is enormously important.

We build systems that assume information is more trustworthy when it arrives through an established channel. Official accounts, authenticated APIs, trusted suppliers and digitally signed records all contribute to that confidence. But the reliability of a communication channel and the accuracy of the information are not the same thing. The SEC incident illustrates what happens when those two properties are confused.

Source: SEC, Statement on Unauthorized Access to the SEC’s @SECGov X.com Account, 12 January 2024.

Did Anyone Check Whether The Information Was True?

The system receives market information, customer exposure figures, transaction records and forecasts from various internal and external sources. Its communications are encrypted, access controls restrict who can submit information, and every transaction is logged.

The analytical application processes those inputs according to established mathematical rules. Everything works. Apparently

A compromised market-data feed reports an incorrect price. An internal system contains deliberately altered transaction values. Or a supplier’s financial information has been modified before being submitted. The data arrives through an accepted channel, passes the format checks and is processed correctly. The resulting calculation may be perfectly accurate relative to its inputs. But, the inputs are wrong.

And the institution can make a damaging decision without any failure in its encryption, mathematical processing or conventional access controls. This is where two truths can be right: A system can be computationally correct and factually wrong at exactly the same time.

Cybersecurity understands the importance of data integrity. Confidentiality, integrity and availability are among the discipline’s foundational principles. But preserving information from unauthorised alteration isn’t the same as establishing that the information true. A digitally signed financial statement may be exactly the document its issuer signed. That doesn’t mean the figures are accurate.

An authenticated sensor may faithfully transmit its measurement. That doesn’t guarantee the sensor is calibrated correctly, hasn’t been physically manipulated or is measuring what the receiving system assumes it’s measuring. A database may preserve every transaction without alteration. That doesn’t mean the original entries were fraudulent, mistaken or deliberately misleading.

In each case, conventional security mechanisms can perform their intended functions successfully while the broader system operates on a false understanding of the world. That very understanding is used to make decisions without direct human intervention.

You Don’t Always Need to Break the System

For an attacker, manipulating the information a system relies on can be more useful than attacking the calculation itself. A logistics company, for example, whose automated planning system allocates shipments according to warehouse capacity, demand forecasts and delivery schedules. The algorithms are well tested. The databases are encrypted, authenticated users have carefully assigned permissions, and the network is monitored for suspicious activity. What if an attacker can change one upstream inventory feed. Instead of trying to steal customer records or disable the planning application, the attacker reports that a warehouse has more available stock than it actually holds. The system accepts the information and calculates the optimal distribution of shipments. Every calculation, every decision follows the rules. The outcome is still wrong.

Deliveries are allocated against inventory that doesn’t exist. Additional shipments are delayed, resources are diverted, and downstream operations begin responding to conditions that were never real. Nothing about this scenario is a sophisticated mathematical attack. The critical weakness is the ability to influence a source the system depends upon. A similar principle applies to industrial sensors, medical records, financial reporting and defence systems.

The consequences differ, but the underlying vulnerability is recognisable: the decision-making process assumes that accepted information is reliable. That assumption is a target.

Traditional cybersecurity looks at how an attacker can gain access to information or disrupt the system processing it. Information manipulation is different: what if the attacker doesn’t need to disrupt the system because they can persuade it to make the wrong decision?

The Poison Doesn’t Have to Look Like Poison

AI makes this problem interesting. Machine learning systems depend on information at several different stages. Training data shapes what a model learns. Fine-tuning and other adaptation processes can influence its behaviour. Retrieved documents and external information sources can affect the answers it produces during operation. Each stage creates opportunities for manipulated information to influence the result.

The security community calls these deliberate attacks data poisoning. The US National Institute of Standards and Technology’s 2025 Adversarial Machine Learning taxonomy explicitly addresses poisoning alongside other classes of attacks against AI systems. OWASP also identifies Data and Model Poisoning as a risk for large language model applications. These are defined technical threats.

In a training-data poisoning attack, an adversary manipulates information used to train or adapt a model, attempting to influence its subsequent behaviour. The objective might be to introduce a systematic bias, degrade performance or create a specific vulnerability that activates under particular conditions. Manipulating information retrieved by an AI application at runtime is a related but distinct problem. So is prompt injection, in which malicious instructions embedded in lower-trust content attempt to redirect a model’s behaviour.

Not every incorrect input is data poisoning, and not every unreliable AI output comes from an attack. Models can make mistakes without interference, while ordinary data-processing systems can produce incorrect results because of stale records, defective measurements or poorly defined business rules.

All these examples show the importance of understanding where information originated, how it was validated and what assumptions a system makes before acting on it. An AI agent preparing a financial forecast might retrieve records from several authenticated sources. It might process those records without exposing individual salaries, use carefully constrained tools and produce the requested departmental figures. If the underlying records contain manipulated values, its forecast can still be off.

The agent doesn’t have to hallucinate. It doesn’t even have to behave incorrectly. It just has to trust information. And because autonomous systems combine information from multiple sources, one unreliable input can influence an entire sequence of legitimate actions.

We’ve spent years discussing whether AI systems can be trusted with sensitive information. We should be equally concerned about whether they can trust the information they’re being given.

A Cryptographic Proof Isn’t a Certificate of Truth

This is where the discussion becomes relevant to cryptographic computing. FHE allows supported operations to be performed on encrypted information without requiring the computing environment to decrypt the underlying inputs. ZKPs can establish that a defined mathematical statement is true without revealing the private information used to demonstrate it.

These are powerful capabilities, especially when the objective is to reduce exposure or establish verifiable properties without unnecessary disclosure. But they don’t magically convert unreliable information into reliable information.

Let’s look at a system calculating an encrypted financial forecast using FHE. The underlying salary records, operating costs and projected staffing requirements remain encrypted while the calculation is performed. The system produces an encrypted result that can be decrypted by an authorised recipient.

The computation is correct, and the confidentiality protections may work exactly as intended. What if one of the salary figures was deliberately inflated before encryption? The cryptographic system has no inherent reason to reject it simply because the value is false in the real world. It performs the calculation it’s been given. The result incorporates the false input.

The same distinction applies to ZKPs. A proof might establish that a financial calculation followed a specified formula, that a value falls within an agreed range, or that a party possesses information satisfying defined conditions.

Provided the proof system is sound and correctly implemented, those statements can carry rigorous mathematical guarantees.

But what the proof depends on the statement being proved. A proof that a salary lies within an approved range doesn’t establish that the salary was properly recorded. A proof that a calculation followed an approved method doesn’t prove that the inputs were real. A proof that information was signed by an authorised source doesn’t mean that the source was accurate.

A system can include additional verification requirements. It may validate a signed measurement, compare a value against an independently established reference or prove compliance with carefully defined data constraints.

But there’s no general cryptographic mechanism that establishes every real-world fact just because the information has been encrypted or mathematically verified. That boundary isn’t a weakness unique to FHE or ZKPs. It’s the difference between a mathematical guarantee and an empirical claim..

The Difference Between Provenance and Truth

If we can establish where information originated, who submitted it, when it was created and whether it was subsequently modified, we can make better decisions about whether to rely upon it.

Digital signatures, authenticated data sources, protected audit logs and controlled data pipelines help establish important properties of information. But they have limits. A digital signature can establish that particular data was signed using a corresponding private key and hasn’t changed since signing. It doesn’t establish that the contents were accurate when signed.

The SolarWinds supply-chain compromise demonstrated exactly that in 2020. Attackers inserted malicious code into the company’s software build process, enabling compromised Orion software updates to be distributed through the established software supply chain. The malicious component was included in software bearing SolarWinds’ digital signature.

The signature wasn’t mathematically defective. The process responsible for determining what deserved to be signed had been compromised. That’s a different failure. And it shows why provenance can’t carry the entire burden of establishing trust.

Source authentication tells us something important about who supplied information or what system produced it. Integrity protection tells us whether information has been changed in ways those protections are designed to detect. Neither establishes that the original information was true or accurate for the decision being made.

Real confidence means understanding the process that generated the information.

• Was the measurement independently checked?

• Is the source authoritative for this particular claim?

• Does the information agree with other evidence?

• Could the source itself have been compromised?

• What happens when two supposedly reliable sources disagree?

These are engineering and governance issues as much as cryptographic ones.

More Data Doesn’t Mean Better Decisions

Modern enterprises have become efficient at gathering, storing and analysing data. Cloud infrastructure makes aggregation easier, while AI systems create incentives to connect additional sources and make more information available.

The underlying belief is that better decisions need better information, and more information can provide useful context. But increasing the amount of data doesn’t automatically increase its quality.

If several systems repeat the same incorrect information, their apparent agreement may provide false confidence. If multiple datasets ultimately depend on one compromised source, treating them as independent confirmation is a mistake.

An organisation might believe it has validated a financial assumption against three separate reporting systems, without recognising that all three got their figures from the same underlying database. This matters for AI applications capable of combining information from sources whose dependencies aren’t immediately obvious.

A retrieved document, an internal dashboard and an external analytical report may look like independent evidence. But if they share an original source, a single manipulated record can influence all three. The system may produce a coherent, well-supported explanation for a conclusion that rests on one false assumption.

This is why data lineage, independent validation and explicit uncertainty matter. Organisations need to understand not just which information contributed to a result, but how the sources relate to one another and what evidence supports the claims.

For automated systems, this may mean escalating a decision, seeking independent confirmation or restricting what actions can be taken until uncertainty is resolved.

The Security Industry’s Blind Spot

Monday’s article examined what happens when attackers exploit authority that systems have already granted. Today’s problem goes further.

Even if the access is legitimate, the permissions appropriate and the information protected from unauthorised modification during processing, the result may still be wrong because the original inputs were unreliable.

This isn’t to suggest that the cybersecurity industry has ignored integrity. It hasn’t. Cryptographic integrity checks, database controls, transaction validation, secure logging and data quality processes have existed for decades. This is more specific.

Organisations spend enormous amounts protecting information throughout its digital lifecycle without giving equivalent attention to whether that information deserves to influence a consequential decision. We measure whether records were changed. We verify that messages came through accepted channels. We confirm that calculations followed approved procedures.

But the system also needs to establish whether the source was appropriate, whether its claims were adequately supported and whether the resulting decision should be allowed. This is where cybersecurity, data governance and application architecture intersect.

It’s also where adding another layer of threat detection may do nothing. A security platform that confirms a database hasn’t been tampered with can’t determine whether a fraudulent transaction was accurately recorded when it entered that database.

An AI monitoring tool might identify an unusual recommendation, but it can’t determine whether a plausible recommendation is based on a manipulated source. A cryptographically verified result can’t guarantee the truth of claims its verification process wasn’t designed to establish.

Trust Has to Begin Before the Calculation

So what should a more trustworthy architecture look like? It should begin by treating information as something whose reliability needs to be established, not just assuming that technical acceptance means credibility.

That means tracing information to its source, identifying the assumptions behind it and deciding which claims need more validation. It means preserving integrity throughout processing while recognising that cryptographic integrity and factual correctness aren’t the same. It means designing automated workflows with limits on how far an uncertain conclusion can propagate and what actions may follow from it. And it means applying cryptographic technologies without expecting those technologies to establish properties they can’t prove.

For a financial forecast, that might involve authenticated sources, reconciled records, encrypted computation, verifiable constraints and controlled disclosure of the resulting figures. For a medical system, it could involve validating measurement sources, checking consistency with other clinical information and requiring appropriate review before a high-consequence decision. For defence or critical infrastructure, it may mean independently verified measurements, compartmentalised systems and mechanisms that prevent one compromised feed from automatically determining operational actions. None of these examples has a universal solution.

But, the system shouldn’t be allowed to treat an input as reliable just because it was permitted.

Subscribe now

Cybersecurity spends enormous resources determining who’s allowed into a system. But, rarely does it question if the authority granted is appropriate for what allowed. Compromised credentials, third-party integrations, service accounts and autonomous AI agents can operate through perfectly legitimate access mechanisms. There is a dangerous mismatch between authorisation and trustworthiness. This isn’t an argument for abandoning authentication or access controls. It’s an argument that protecting modern information needs something more fundamental.

Three Minutes, and the Data Was Gone

At 11:11 on the morning of 17 August 2025, an attacker began extracting information from Cloudflare’s Salesforce environment. The operation used an established application integration and Salesforce’s own Bulk API, a legitimate interface designed to move large quantities of information efficiently.

Within a little over three minutes, the attacker had extracted a dataset containing customer-support case information. The stolen material included support-ticket text that could contain sensitive customer information, configurations and credentials.

There was no vulnerability in Salesforce’s core platform that needed to be discovered and compromised. They used stolen credentials from the Salesloft Drift integration, a service that had been connected to Salesforce.

The integration was real. Its access mechanism was real. The requests travelled through an established API. But, the person making those requests wasn’t who the credentials belonged to.

Cloudflare later disclosed that they identified 104 Cloudflare API tokens within the compromised support data, all of which it rotated as a precaution. It reported no suspicious activity from associated tokens and no compromise of its services or infrastructure.

The incident was part of a much broader campaign affecting organisations using Salesloft Drift integrations. Google’s Threat Intelligence Group documented attackers using compromised OAuth tokens to query corporate Salesforce instances and extract information, including material that might contain credentials for entirely different systems.

The attackers hadn’t invented a new way to communicate with those applications. They had obtained the means to use connections that the applications were already prepared to accept.

Sources: Cloudflare’s incident report and Google Threat Intelligence Group’s investigation.

A Credential Isn’t a Character Reference

Authentication and authorisation are among the foundations of modern computing security. Authentication is evidence that a requester possesses an accepted identity or credential. Authorisation determines what that requester is allowed to do.

We’ve developed a habit of treating authentication and authorisation as though they mean more than they do. A valid credential doesn’t prove intent. It doesn’t prove that the person or process using it is the intended holder. Nor does an authorised operation necessarily establish that the information requested is appropriate for the task.

A service account used by a business application to retrieve customer records has been assigned permissions, its credentials are valid, and it communicates through an encrypted connection. From the perspective of the receiving system, the application may be doing what it has permission to do. The attacker doesn’t need to overcome every security mechanism surrounding the application. If the stolen credentials are usable and no additional contextual control blocks the request, the attacker can exercise the authority already granted to the application.

The Third Party Nobody Thought About

The Salesloft incident highlights another issue of modern enterprise security: organisations extend authority beyond their direct operational boundaries.

A typical company isn’t an isolated collection of applications. Its CRM communicates with marketing platforms, support services, analytics providers, payment processors and dozens of other tools.

These integrations create chains of delegated authority. One platform receives permission to interact with another. That platform may use credentials stored in its own infrastructure. Its employees, software dependencies, administrative systems and suppliers become part of the wider operational trust relationship.

The organisation granting access may have excellent internal security. That doesn’t mean it controls every environment. And when an integration is compromised, the ripples go far beyond the original system.

That’s what makes these incidents interesting over a story about stolen passwords. The vulnerability isn’t necessarily located in the system holding the information. The ultimate destination might have strong encryption, sophisticated monitoring and effective identity controls. But those protections don’t stop another system from misusing credentials.

OAuth, the authorisation framework involved in the Salesloft incident, wasn’t the flaw. It provides a structured way to delegate access without sharing a user’s password. The problem is when delegated credentials are compromised, permissions exceed operational requirements, or the receiving system can’t distinguish between legitimate usage from abuse.

There are technical mitigations: narrowly scoped tokens, short-lived credentials, workload identity controls, token binding where supported, network restrictions, continuous behavioural analysis and rapid revocation. None of these change the what the integration is allowed to access. The more authority we delegate, the more consequences we attach to the compromise of any system capable of exercising that authority.

We Built Systems That Trust Permissions

The application generating it needs aggregated figures covering transaction volume, exposure and projected liquidity. Instead of receiving those results directly, it’s authorised to retrieve a broad collection of underlying records, perform its calculations and return the report.

The application is trusted, the network connection is encrypted, and the database enforces the permissions assigned to its service account. Everything looks normal. Even if that application is potentially compromised.

The same access rights that make the report possible are also a vulnerability. An attacker may be able to retrieve individual transaction records, account information or other sensitive material because the application was granted that authority for convenience.

There may be controls that identify abnormal requests or prevent excessive extraction. But their effectiveness depends on the quality of the policies and whether the activity is different enough to trigger a response.

The database could generate the aggregation internally. The workflow could use constrained queries, field-level permissions or purpose-specific interfaces. In some cases, privacy-preserving computation could allow a result to be calculated without revealing the underlying inputs. Different workloads need different solutions, and some will still need direct access to sensitive data.

Now Give the Permissions to an AI Agent

Until recently, many of these concerns centred on people, service accounts and conventional software integrations. AI agents are a new dimension because they’re expected to determine which tools to invoke, what information to retrieve and which actions to perform.

Let’s look at an agent assigned to manage customer-support operations. It can retrieve records, examine previous correspondence, update tickets and generate responses. Depending on its configuration, it could also interact with email, internal documents or external services. Those capabilities are useful because the agent can operate across systems without needing human direction. But that creates a new problem.

The agent may have legitimate authority to perform actions that aren’t necessary, appropriate or anticipated. A malicious instruction embedded in an external document or support ticket might attempt to redirect the agent into retrieving sensitive information or invoking a privileged tool. A poorly constrained agent could also take an excessive action without any malicious intervention, just because its interpretation of a request is wrong.

This isn’t hypothetical. OWASP’s 2025 guidance on Excessive Agency specifically identifies excessive functionality, excessive permissions and excessive autonomy as conditions that can allow AI-enabled applications to cause problems.

AI hasn’t invented excessive privilege. It has created another way for that privilege to be exercised, sometimes through decisions made dynamically rather than through fixed application logic.

An application with excessive permissions has always been dangerous when compromised. An agent with excessive permissions can introduce additional uncertainty about which operations will be attempted, in what sequence. Giving an agent unrestricted access and then relying on another system to recognise when it misuses that access repeats a pattern.

The Difference Between Access and Authority

There is a better way to frame the problem, although it means more than installing a new product. Organisations need to distinguish an identity’s standing permissions from the authority required for a specific operation.

• A user may be allowed to access a financial system.

• An integration may need to generate a report.

• An AI agent may be assigned to resolve a customer issue.

This is where least privilege needs to become an enforced operational property. Permissions can be limited by resource, function, context and duration. Delegated authority can be constrained to the specific task. High-impact actions can need enforced approval, and unnecessary data transfers can be eliminated entirely.

FHE can support computation on encrypted information without exposing the underlying plaintext to the computing environment. ZKP can establish particular claims without revealing the information used to support them. Combined with conventional access controls, compartmentalisation and appropriately designed application interfaces, these techniques can reduce categories of unnecessary disclosure.

They aren’t interchangeable technologies, and they don’t guarantee that an authorised result will be truthful or that every action will be legitimate. Their deployment must be justified by the threat model, workload and computational cost. But they make an important architectural proposition possible: some operations can be designed so that the requesting component never receives the sensitive information it would otherwise have been permitted to access.

The security guarantee comes partly from what the system is unable to disclose, not what a policy says it shouldn’t disclose. That doesn’t eliminate the need to trust anything. Source data, endpoint security, implementation correctness, key management and the recipients of computed results still require protection.

Permission Shouldn’t Be a Blank Cheque

Least privilege, separation of duties, just-in-time access, purpose limitation and zero-trust architecture aren’t recent. NIST’s zero-trust guidance rejects implicit trust based on network location and advocates resource-focused access decisions. These are substantial architectural principles, not just extensions of the old model.

The challenge is in applying them consistently across systems that were designed around broad application privileges, persistent integrations and operational convenience. It’s easy to say a system should have only the permissions it needs.

Developers need applications to work. Administrators need integrations to be reliable. Business teams need information when they request it, and security teams are asked to accommodate all three without interrupting operations.

Granting broader access is the simplest way to make an integration function. That convenience has a security cost, even when no one can see it immediately. And the cost increases as more systems inherit, share or exercise the same underlying authority. We should be measuring those dependencies as seriously as we measure failed login attempts, malware detections and suspicious network connections.

The Attack We Authorised

Returning to the Salesloft incident, the most instructive aspect isn’t that attackers obtained credentials. Credential theft is an old problem, and conventional cybersecurity has developed protections against it. It’s what the stolen credentials allowed the attackers to do.

The compromised integration provided a route into data held by other organisations. Those organisations hadn’t deliberately authorised the attackers, and security mechanisms capable of identifying anomalous API activity remained relevant. But the delegated permissions created a pathway that could be abused without exploiting the core Salesforce platform.

Once the attacker possessed a usable token, the question was how much authority those credentials carried, which data that authority could reach and whether additional safeguards would prevent the resulting misuse. The same questions apply to service accounts, cloud integrations, automated processes and AI agents.

Organisations continue to put enormous effort into determining whether someone or something should be allowed through a particular access boundary, without considering what happens after that boundary is crossed.

We shouldn’t stop authenticating, authorising, monitoring or detecting. But we do need to stop confusing them with proof that the resulting activity is safe.

A permission isn’t evidence of good intent. An encrypted connection isn’t evidence of an appropriate transaction. And an authorised process isn’t necessarily one that should possess all the information it can technically retrieve.

The next significant advance in cybersecurity will mean redesigning systems so that legitimate access carries less unnecessary authority in the first place.

Subscribe now

Cybersecurity has spent decades presenting incremental improvements as technological revolutions. Firewalls became next-generation firewalls, antivirus evolved into endpoint detection and response, and monitoring platforms became ‘automated and intelligent’. It seems everything changed, but nothing changed: authenticate, authorise, monitor, detect and respond. Meanwhile, cloud infrastructure, interconnected supply chains, distributed applications and autonomous AI agents operate in environments that aren’t even close to what conventional security models were developed. The problem is that we’ve become good at refining defensive mechanisms without really questioning what and how we protect, in the new tech space.

The World These Defences Were Built For

Every few years, another generation of products show up saying they’ll transform enterprise security. The language changes, the interfaces are prettier, and marketing departments suddenly have a bunch of new acronyms. Existing technologies have more capabilities, improved analytics and, now, AI. What was considered essential is replaced by something with a new name and a bigger price tag. In the meantime, underneath it all, not much really changed.

• Identify who’s requesting access.

• Determine whether they’re authorised.

• Allow the operation.

• Monitor what happens.

• Detect suspicious behaviour.

• Respond when something goes wrong.

Does that sound familiar? It should, it’s been the same mantra for the last forty years. We’ve just made it faster, more automated and more sophisticated. What we haven’t done is question whether the assumptions make sense.

Forty years ago, corporate computing looked different. Organisations operated relatively contained networks, employees generally worked from company premises, and most computing resources sat inside infrastructure the organisation owned or directly controlled. External connections existed but what was an internal network and the outside world was easier to see.

Security developed around: Protect the perimeter, authenticate users, restrict access and monitor suspicious activity. Firewalls controlled network traffic, antivirus software identified malicious code, and intrusion detection systems watched for suspicious behaviour.

The approach wasn’t perfect, even then. Its underlying assumption was that an organisation could establish boundaries around its infrastructure and manage what crossed them. Today’s typical enterprise operates across cloud platforms, mobile devices, remote employees, software-as-a-service applications, external contractors, application programming interfaces and interconnected suppliers. Let’s not forget that autonomous agents are being introduced into workflows previously performed by people.

Information is continuously exchanged, processed, replicated and transformed across systems operated by different organisations, usually in different jurisdictions. The computing environment has changed significantly. But the security industry keeps refining defensive concepts developed for an entirely different world. And we’re still calling that innovation?!

Forty Years of Innovation. Or Forty Years of Repackaging?

Antivirus software has evolved into endpoint protection and endpoint detection and response systems. Traditional firewalls have acquired application awareness, intrusion prevention and increasingly granular traffic controls. Log management developed into security information and event management platforms, followed by orchestration, automation and AI-assisted investigation.

These aren’t meaningless improvements. Modern endpoint security can identify behaviours that conventional signature-based detection would miss. Advanced monitoring can correlate activity across infrastructure. Automation can accelerate responses to threats that move faster than human analysts.

Let’s look at the progression from conventional antivirus to modern endpoint detection and response, as an example. The newer technology is more capable, but much of its purpose is recognisable: identify malicious activity, prevent it where possible, detect what gets through and respond before the consequences become unacceptable.

The same applies to the systems used to monitor enterprise activity. We’ve developed capabilities for collecting telemetry, correlating events and recognising suspicious behaviour. Organisations can now observe millions of events across environments that wouldn’t have been possible when the original defensive concepts emerged. The question remains: can we recognise a threat quickly enough to stop it?

There’s a difference between creating better tools and redesigning a system so that the vulnerability has less chances to cause damage. Cybersecurity has invested heavily in the former. The latter has been a secondary issue, something to review after the next product deployment, compliance assessment or security transformation programme.

Since the improvements are real, even when they’re incremental, the industry can present them as evidence of progress. A new dashboard, a new acronym, another subscription and a promise that this generation will finally solve the problems the previous generation couldn’t.

To be fair, organisations have had little incentive to challenge that. Organisations need protection now, against threats that exist now, and they purchase the tools available to them. Vendors respond to those demands, and the cycle continues.

The result? An industry that’s very sophisticated at defending existing architectures, without demonstrating a willingness to question whether those architectures actually work in their current form.

The Tools Got Smarter. The Questions Didn’t.

It would be disengenuous to say that cybersecurity has experienced no fundamental breakthroughs. Public-key cryptography, hardware-backed security, modern cryptographic protocols and zero-trust principles are strong advances. Some have substantially changed what secure computing can accomplish. But developing new security capabilities and integrating them into the dominant operating model are two different things.

In 2020, the US National Institute of Standards and Technology published SP 800-207, Zero Trust Architecture, formally describing an approach that rejects implicit trust based on network location. Instead of assuming something inside the corporate network is trustworthy, access decisions should consider the requesting identity, device, resource and relevant context. But zero trust also shows how a promising architectural principle can be reduced to just another collection of security products.

Some implementations concentrate heavily on stronger authentication, more granular access policies, continuous verification and improved monitoring. But they don’t change how applications process sensitive information once access has been approved.

Look at an organisation that deploys advanced endpoint protection, multifactor authentication, behavioural analytics and a sophisticated security monitoring platform. Its employees authenticate successfully, applications communicate through encrypted connections, and unusual activity triggers alerts.

Consider what happens when an authorised application requests information from another system. The request is legitimate, its credentials are valid and the connection is encrypted. The receiving application obtains thousands of records when its actual task requires only a small subset of the information. Nothing about that transaction necessarily looks out of place. The security tools performed exactly as designed. The architecture still exposed more information than needed.

That’s the industry keeps to overlooking. Authentication establishes evidence of identity. Authorisation determines what that identity is permitted to do. Neither automatically establishes that the operation is necessary, that the information disclosed is proportionate to the task or that the process exercising the permission hasn’t been compromised.

A different approach might see if an application needs to the underlying information at all. Instead of deciding who gets to see sensitive records, why not design workflows so that results don’t need to see all the records? That changes the relationship between computation, information and trust.

Forty Years of Adding Another Layer

When malware became more sophisticated, detection improved. When attackers began exploiting legitimate credentials, identity controls became stronger. When attacks moved laterally through networks, segmentation and behavioural monitoring expanded. When cloud computing distributed infrastructure across providers, cloud-specific security products emerged. Each development addressed genuine problems, usually quite well.

The problem starts when adding defensive layers becomes the default response. An organisation finds sensitive information is broadly accessible, so it purchases another monitoring platform. Privileged accounts present substantial risk, so it introduces additional privileged-access management. Automated workflows create new uncertainties, so another security product is deployed to observe their behaviour.

Eventually, security teams administer an expanding collection of products, integrations, dashboards, policies and alerts. Ironically, many of these tools need privileged access to the environments they’re supposed to protect.

The organisation has more information about its security posture than ever before, but the underlying exposure issue? It hasn’t really changed.

An effective initial security control may remove a category of risk. Additional controls can close further gaps. But surrounding unnecessarily exposed information with more sophisticated defences doesn’t really create a proportional reduction in the consequences.

A better intrusion detection system may identify an attack sooner. It doesn’t change how much information an overly privileged application can retrieve before the attack is recognised. Another dashboard may improve visibility. It doesn’t reduce the number of systems capable of accessing sensitive records.

And introducing AI into the process may enable faster detection, better prioritisation and more efficient investigation. Those are operational improvements, but they aren’t an architectural breakthrough.

We’re now using some of the most advanced computing technologies available to make decades-old defensive strategies more efficient. At some point, the benefit of adding another defensive product may become smaller than the benefit of redesigning the workflow that creates the exposure. In other words, the Law of Diminishing Returns kicks in.

Redesign is usually harder to justify, implement and measure. It requires cooperation between security teams, application developers, infrastructure operators and business leadership. It can challenge longstanding processes and introduce costs that don’t fit conveniently into existing operational budgets. And therein lies the problem. At least in part.

The Business of Managing Insecurity

Security products can be purchased, deployed, licensed, renewed and measured through established procurement processes. Their success can be expressed in dashboards, compliance reports, alerts investigated and threats prevented. Architectural transformation doesn’t fit neatly into that model.

An industry built around detecting, managing and responding to threats has an obvious commercial incentive to develop more sophisticated detection, management and response capabilities. Improvements that eliminate unnecessary exposure may reduce the need for certain defensive functions.

That doesn’t mean cybersecurity vendors are deliberately perpetuating insecurity. Many have excellent technologies, and organisations genuinely need their services. Nor does it mean architectural security has been ignored entirely. Considerable research and engineering effort has gone into secure-by-design principles, isolation, memory safety, cryptographic computing and least-privilege architectures.

It’s easier to show that a new platform detects more threats than to quantify the long-term value of redesigning a workflow so that sensitive information never needs to be exposed to the same extent. One option produces measurable security activity. The other reduces the chance for that activity to become necessary. And commercial markets tend to favour things they can package, sell and measure.

That’s not an accusation of conspiracy. It’s an observation about how industries mature, particularly when customers are operating under constant pressure to address immediate risks. The security industry hasn’t stood still. It has grown into a sophisticated ecosystem of technologies, services, standards and certifications. But commercial growth and architectural progress aren’t the same thing. An industry can become larger, faster and more technically advanced while nothing really changes.

The Industry Knows Something Is Wrong

In 2023, the US Cybersecurity and Infrastructure Security Agency and international partners published their Secure by Design principles, calling on technology manufacturers to take greater responsibility for the security outcomes of their products rather than continually transferring that burden to customers.

That’s an important, it recognises that expecting every organisation to compensate indefinitely for weaknesses introduced upstream isn’t a sustainable security model.

Verizon’s 2026 Data Breach Investigations Report identifies exploitation of software vulnerabilities as the leading known initial access vector, accounting for 31 per cent of breaches. That finding doesn’t mean conventional cybersecurity has failed. It shows why secure development, vulnerability management, patching and intrusion prevention remain essential.

But, if organisations need to continuously defend against vulnerabilities, compromised credentials, supplier weaknesses and operational mistakes, shouldn’t they also design systems around the expectation that some of those defences will eventually fail? Not as an excuse to abandon prevention, but as a requirement to contain the consequences when prevention is unsuccessful.

There’s an important difference between making compromise harder and making compromise less consequential. A mature security architecture needs to accomplish both. And achieving the second objective means more than continuously refining the mechanisms responsible for the first.

When a Breach Stops Looking Like a Break-In

Part of cybersecurity’s problem is how we’ve learned to imagine a security incident.

The familiar story involves an attacker defeating a defence, obtaining access, moving through systems and stealing information. Security teams detect the intrusion, contain the attacker and restore operations. That sequence still happens, and it’s a good reason to keep conventional defences.

But the difference between authorised activity and malicious activity isn’t always obvious in modern systems. A compromised software integration can continue to use valid credentials. An automated process may issue requests consistent with its assigned permissions. A privileged administrator can operate through normal management interfaces, while an AI agent may execute an authorised task whose consequences weren’t considered when its authority was granted.

These aren’t identical threats, and they need different protections. What connects them is that the architecture that is still working on old assumptions. That presents a problem no amount of threat detection can resolve.

If an application is authorised to retrieve sensitive information, conventional controls may have limited grounds for distinguishing a legitimate request from a harmful one operating through the same permissions. Behavioural monitoring and context-aware authorisation can help, but they depend on what the system can see and the policies it can enforce. The architecture itself may have already established the conditions for excessive disclosure.

This is where technologies such as FHE, ZKP and confidential computing become relevant, alongside conventional measures such as compartmentalisation, granular permissions and data minimisation.

These technologies address different problems, and none provides a universal solution. FHE allows certain computations to be performed on encrypted data, but carries substantial performance and engineering costs. ZKPs can establish defined claims without revealing the underlying information, but don’t automatically establish that the original data is truthful. Confidential computing offers hardware-backed isolation, while introducing its own trust assumptions and operational limitations.

Their significance is in offering alternatives to the assumption that computation means exposing the underlying information to the system. That’s a fundamentally different question from how fast an organisation can detect someone misusing the information after access has been granted.

Stop Measuring How Well We Defend the Door

Cybersecurity performance is usually presented through operational metrics: attacks blocked, vulnerabilities remediated, incidents investigated, authentication failures, response times and compliance results.

These measurement help organisations to understand whether their defensive capabilities are functioning. What they don’t establish is how much damage a successful compromise could cause.

A company might reduce its average incident response time while it lets applications capable of retrieving entire databases operate. It might improve authentication security without reviewing the privileges granted to long-lived service accounts. It might achieve excellent vulnerability management results while maintaining workflows that expose sensitive information unnecessarily during ordinary processing.

• How much sensitive information can a single compromised identity retrieve?

• How far can an unauthorised process move?

• Which critical operations depend on one privileged administrator, service account or supplier?

• Can a workflow produce its required result without revealing the underlying records?

• If one component fails, what prevents that failure from spreading?

These are questions that establish whether those measurements describe a system becoming more resilient or more intensively monitored. And they show something cybersecurity procurement doesn’t always make obvious: the most valuable security improvement may not be another product. It may be eliminating an unnecessary permission, separating a dependency, removing a data transfer or fundamentally redesigning how information is processed. That kind of improvement doesn’t always produce an impressive dashboard. It can remove a risk that would otherwise mean years of monitoring.

The real measure of cybersecurity progress shouldn’t be how many threats a product can detect or how quickly an incident can be contained. It should be how much unnecessary exposure has been eliminated, how effectively we’ve constrained authority and how little a compromised component can accomplish.

We’re Not Going to Detect Our Way Out of This

We’re entering another period of profound technological change, and the industry’s response is familiar. AI is transforming how applications operate, how employees interact with information and how automated systems make decisions. Agents are being granted authority to retrieve records, initiate transactions, execute workflows and communicate with other systems.

Software supply chains distribute trust across organisations that may never interact directly. Cloud infrastructure concentrates operational authority in external providers, and sensitive information routinely crosses organisational and jurisdictional boundaries during ordinary processing.

The environment is changing again. The industry’s response? Develop more tools. More AI-assisted detection. More automated investigation. More behavioural analytics. More dashboards promising unified visibility across increasingly complex environments.

We’ve spent decades trying to recognise malicious behaviour quickly enough to prevent unacceptable damage. But it can’t substitute for designing environments in which compromised systems have less authority, sensitive information is exposed less frequently, and failures are prevented from cascading through interconnected infrastructure.

This is an argument for recognising the difference between making an existing system more difficult to attack and designing a system whose fundamental operating model reduces the consequences of attack.

Forty-year-old technology isn’t inherently obsolete. Some of the most reliable principles in computing are considerably older. What has become hard to justify is treating incremental improvements as though they represent a fundamental transformation in security.

We’ve changed the products, improved the analytics, automated the responses and repeatedly repackaged the results as the next great advancement. But, the underlying assumptions about access, trust and information exposure haven’t changed from decades ago. The industry doesn’t need another coat of paint on the same architecture. It needs to recognise that the architecture itself is part of the problem.

Subscribe now

A finance agent needs a staffing-cost forecast, not a payroll export. This article sketches how CKKS, TFHE and zero-knowledge proofs could each handle a different part of that job, and where each one’s guarantees stop. The agent in this design receives a departmental forecast and a planning indicator, and nothing else. It’s an illustrative design, not a validated deployment.

A finance agent has been asked to prepare next quarter’s staffing-cost forecast. The brief covers changing hours, expected hiring and several operating scenarios. Nobody has asked it to look into an individual’s salary, but the easiest route through the existing systems would give it enough information to do exactly that.

The forecast needs certain numbers. It doesn’t necessarily need every record behind them. This final article in the series asks what the workflow would look like if each stage had the protection that suits its task, rather than expecting one mechanism to cover everything.

The design below combines three tools: CKKS for approximate arithmetic on encrypted numbers, TFHE for encrypted comparisons and decisions, and ZKP for narrow claims about inputs. It’s organised around three questions that any such workflow has to answer separately:

1. Is the input valid? Did the right data go in, in the right form?

2. Was the computation performed correctly? Did the processing do what it was asked to do?

3. Is the result useful, and safe to release? Does the answer support the decision, and who can read it?

No single mechanism answers all three. Each stage below addresses one, and the article says what evidence is still missing for the others.

***This is a proposed PrivID architecture. It’s not a claim that every component and hand-off described here has already been implemented or validated in PrivID’s current stack.

A forecast and a payslip need different things

The payroll team starts with a distinction. A forecast can tolerate a defined amount of numerical approximation. A payslip has to account for the agreed amount of money. Both involve numbers, but the consequences of an error are different.

That difference explains the division of work. CKKS supports approximate arithmetic over batches of encrypted values, which makes it a candidate for statistical and forecasting workloads. TFHE supports Boolean circuits and discrete functions, which makes it a candidate for encrypted comparisons and decisions. The OpenFHE documentation describes switching between CKKS and FHEW/TFHE for operations such as comparisons, so combining the two has a technical basis in supported implementations.

That is not a rule for which scheme every business task should use, and it doesn’t mean a switching mechanism in one library can be moved into another stack unchanged. Suitability depends on the workload, the numerical representation, the security parameters and the implementation.

In this workflow, authoritative payroll accounting stays separate from the approximate forecast. The forecast can draw on encrypted numerical inputs without becoming the system that determines anyone’s actual pay, and exact functions stay on a path designed and tested for the precision they require. A second mechanism only belongs in the design if it earns its place.

The inputs: decide what a proof can say

Before the agent requests anything, an authorised source has to prepare the inputs. That source may already hold readable payroll records, because payroll staff need them to do their jobs. Encrypting the next stage doesn’t remove that access, and the architecture shouldn’t suggest otherwise.

The source encrypts only the fields the forecast needs. Employee names, bank details and unrelated personnel notes don’t travel just because they were in the original export. Segmentation starts with defining the task and its inputs.

A ZKP can support a specific claim about those inputs. For example, it could establish that an encrypted value lies within an agreed range without disclosing the value. The proof would need to be bound to the submitted ciphertext and the relevant request, so that a valid proof for one input couldn’t be reused to endorse another. Whoever generates the proof also needs the private witness for the statement. In this design that means proof generation happens at the authorised source, not at an intermediary that holds readable access.

The claim is narrow. A value can satisfy a range condition and still be the wrong value for the employee or the reporting period. Showing correspondence with an approved dataset would take additional evidence and a protocol designed for that purpose.

The calculation: CKKS, then a hand-off to TFHE

The agent submits a request for an agreed forecast, not for a payroll export. The surrounding service checks its identity, permissions, permitted scope and request parameters. Those checks remain necessary even when the calculation runs on ciphertext.

The CKKS stage does the numerical work, for instance projecting the cost of several staffing scenarios and producing encrypted intermediate results. The processing service holds the evaluation material it needs for this, without the means to decrypt the underlying values.

Suppose the organisation also wants to know whether a projected figure exceeds a planning threshold. A compatible scheme-switching design could pass an encrypted representation into a TFHE stage for the comparison. Depending on the next operation, the workflow could use the encrypted decision directly or convert it again.

The word that matters is “compatible”. The hand-off needs a real protocol with defined keys, representations and security assumptions. If an intermediary decrypts the value and re-encrypts it under another scheme, that intermediary has gained readable access. This design belongs in the exposure assessment.

Switching has computational and data-movement costs. A hybrid might speed up one workload and make another slower or harder to operate, so the pilot has to measure the whole route and not just its fastest stage.

The threshold problem: an exact decision inherits an approximate input

The threshold example contains a trap. A comparison doesn’t become a faithful business decision just because its final stage uses a scheme suited to discrete operations. If the number entering that stage is an approximation, the comparison inherits the uncertainty. This is where the payslip contrast returns: the payslip has to match an agreed amount, while a threshold decision sits somewhere in between, exact in form but approximate in its input.

A forecast comfortably below the threshold can support a reliable planning decision within the agreed error bounds. The engineers have to account for the error introduced by the calculation and the conversion, on top of the uncertainty already present in the forecasting model.

There are two plausible responses. The organisation can define an interval around the threshold that triggers further review. Alternatively, where the task permits, it can use a separately justified exact calculation for that decision. The right choice depends on what the decision means and what a mistake would cost.

The same discipline applies to proofs. If the architecture includes a proof of correct execution against a specified numerical model, that proof would not show that the model’s assumptions reflect reality. A model can be implemented correctly and still predict badly. This is the three-question frame at work: input validity, computation integrity and business usefulness are separate.

Release: who can read the answer, and what answers add up to

Once the calculation is complete, the encrypted output goes to a controlled release process. Key custody and decryption authority determine who can make it readable. The agent shouldn’t have those powers by default simply because it initiated the request.

In this example, the agent might receive a departmental forecast and a defined planning indicator. An authorised reviewer might see more detail Those differences have to be enforced through the deployed interfaces and permissions.

The result is still information, however securely it was produced. As we saw in A Secure Result Can Still Show Too Much, combinations of allowed answers can disclose something no single answer was meant to show, and a hybrid computation path doesn’t change that. The surrounding system needs controls on which questions participants can ask and which answers they can combine. It also has to account for other connectors, retained reports and downstream recipients, because a careful release policy on one interface achieves nothing if the same answer is available elsewhere.

A ZKP can sometimes replace a detailed disclosure with evidence of a narrowly defined condition. Even then, the condition tells the recipient something. Choosing the right statement is part of minimising exposure, and the proof mechanism does not make that choice.

Failure, legacy systems and the pilot

Now bring back the timeout from the first article. A stage fails, a conversion takes too long, or the support team needs to know why a report was not produced. The workflow needs a recovery path that keeps its information boundaries intact. That could mean retrying an encrypted job, returning a limited diagnostic or referring the request to an authorised operator. Whichever is chosen, it has to be tested, because a quiet fallback to a readable export would undo the reduction in access that justified the project.

The legacy system from the second article also gets a say. If its importer demands individual records, the team has to change the integration, narrow the function or acknowledge the remaining exposure. Adding CKKS, TFHE and a proof doesn’t persuade an old application to accept a different business contract.

These are the questions the pilot exists to answer, as set out in What Would Count as Proof That It Works? Is the forecast useful, and how accurate is it? What does the whole workflow cost to operate? Which information can the agent actually obtain through its real permissions? The pilot should also examine failures, including combinations of queries and the places where readable access remains.

What the agent ends up with

The finance agent gets its forecast, a departmental figure and a planning indicator. It doesn’t get names, bank details or individual salaries, and nothing in its permissions would let it ask for them.

Subscribe now

Then the person responsible for payroll asks a question. If the reporting agent were compromised tomorrow, what could it learn now that it could have learned before? The team can explain how the encrypted calculation works, but that doesn’t answer the complete reporting process.

This is the final review of the illustrative finance workflow we’ve followed through the series. It isn’t a description of an actual PrivID pilot, and the outcome hasn’t been predetermined. The review is where the organisation has to decide whether the evidence supports a deployment, more engineering or a different approach.

That’s a more consequential decision than approving another demo. It asks the team to connect technical capability to a change in information access, and establish whether that change survives ordinary operation. It also asks the supplier to describe its contribution with enough precision that it can be challenged.

The claim has to fit the evidence

A useful starting claim for this pilot might be that the agent can prepare an agreed upon departmental expenditure report without getting individual payroll records. That statement identifies a task, a participant and information that should no longer reach it. It gives the engineers something to test and the finance team something to evaluate.

The evidence would need to cover how records enter the calculation, what the agent’s interfaces return and whether it can retrieve the same information somewhere else. If the old export is available through another connector, than removing it from the preferred route hasn’t removed the agent’s access. If a support process sends it the records when a calculation fails, the claim needs to account for that behaviour.

The review should also establish whether the report stays correct and useful. Keeping information away from an agent doesn’t count as successful integration if the agent can’t complete the task. Likewise, generating the right report doesn’t establish reduced exposure if readable records still circulate through the same participants.

In The Migration Roadmap Cybersecurity Keeps Avoiding, I argued for evaluating migration through measurable exposure reduction. This series brings that principle into the acceptance decision: what precisely has the implementation changed, and which observations support the claim?

An exposure count needs a meaning

Suppose the team presents a chart showing fewer systems with access to payroll information. That can be useful evidence, but the number needs an explanation. Removing a reporting copy may matter differently from removing a tightly restricted temporary copy. While moving everything into one new component could concentrate sensitive access there.

The review should differentiate between holding encrypted values and receiving readable results. Those are different forms of access. A processing service retaining ciphertext hasn’t necessarily acquired knowledge of its contents, while an agent receiving a permitted total has learned something even if it never touches a source record.

A temporary file that disappears after processing is different from a report retained indefinitely. Removing future exports doesn’t erase old copies. The project needs to account for what happened to the records under the previous workflow. These observations don’t need to become a single impressive percentage.

The team can report a specific reduction in readable copies, a narrower set of permissions and the remaining places where individual records are available. That’s more informative than saying “80 per cent safer” without a real way to measure it.

Give the agent a problem to solve

The payroll owner’s question deserves a controlled exercise. Within an agreed test environment, an evaluator could use the reporting agent’s actual permissions and interfaces to attempt retrieval of individual records. The exercise should also examine whether the allowed results expose those records indirectly.

That follows the problem explored in A Secure Result Can Still Show Too Much. An agent might be denied an employee export and still learn an individual’s contribution by comparing chosen totals. A pilot review needs to establish which questions the deployed interface allows, not just assume that every aggregate answer is harmless.

If one interface refuses a revealing request while another supplies an equivalent answer, the refusal hasn’t established the intended boundary. The result of the exercise should identify the paths tested, the information obtained and the conditions under which access was restricted.

An unsuccessful attempt isn’t proof that every possible inference or attack has been prevented. A successful attempt is equally useful if it exposes a fixable design flaw before deployment. The pilot’s value includes discovering where the proposed protection doesn’t hold.

This kind of evaluation also makes “minimising fallout” more concrete. What information could a compromised participant obtain within a defined scenario, and more importantly, how that compares with the old process. Disruption, corrupted outputs and misuse of legitimate functions are separate issues even when access to underlying records has been reduced.

Operations gets a vote

The performance chart needs to answer: Which part of the workflow was timed, what equipment was used and how many requests were running together? A result from an isolated calculation can establish a capability without describing how long the finance team waits for a completed report.

The first article, The Demo Worked. What Now?, followed the system through delays, retries and support. Those events become evidence. The organisation needs to see if the tested recovery process preserves the intended information limits and whether staff can diagnose faults without restoring broad access.

The operational costs belong in the decision as well. Those costs don’t make the approach impractical by definition. They make it something the customer can assess against the information access it removes. A credible proposal lets the organisation judge the trade-off; it shouldn’t need anyone to pretend that better protection comes without [engineering or operational] consequences.

The inconvenient column is part of the result

The project manager’s final slide should include a column for what is still unresolved. The source payroll system may still hold readable records, some authorised staff may still need individual access, and one downstream application may be outside the pilot. A result may also be appropriate for its first recipient while needing restrictions on further distribution.

Those entries give the organisation a usable account of the deployment’s limits. If the pilot protects an outsourced calculation and narrows the agent’s reporting access, it can say so. It shouldn’t imply that every payroll function, administrator or downstream report has the same protection.

The same discipline should apply to PrivID. Our intended contribution combines ZKPs, FHE, segmentation and bounded access to reduce unnecessary disclosure in suitable workflows. A pilot has to show where those mechanisms work together, including the application controls and key custody on which the outcome depends.

That evaluation has several levels. The cryptographic design needs appropriate review, the software needs implementation testing, and the integrated workflow needs to be evaluated under its actual permissions and operating conditions. A valid proof within a protocol doesn’t prove that the entire deployment is secure.

The evidence should justify moving forward with a clearly defined workflow while leaving another for later. It may show that an existing access-control change does most of the necessary work in a particular case, or that protected computation offers a better advantage in another. A supplier should be able to accommodate those conclusions.

A pilot should end with a decision

An organisation doesn’t need perfect certainty before taking a useful next step. It needs an explicit account of what was tested, what changed, what it costs to operate and which remaining risks it’s accepting within the deployment’s scope.

That account can justify deploying the departmental reporting function, revising its result controls or testing a more representative workload. What it shouldn’t produce is an indefinitely extended pilot with shifting expectations and no agreed decision. The acceptance criteria and the route to that decision belong in the scope before the work begins.

There is a commercial consequence as well. A pilot that needs integration, engineering and operational evaluation is substantial work. Funding it allows the customer and supplier to develop the evidence needed for a purchasing decision, with a defined outcome and responsibilities. Asking for another demonstration may be reasonable early on; eventually, the unresolved questions need a project.

Back in the review meeting, the payroll owner should be able to receive a specific answer. The reporting agent can still do its agreed job, certain records and retrieval paths are no longer available to it, the tested recovery process preserves those limits, and the remaining disclosures are understood.

That’s what would make the pilot convincing. The organisation leaves with a working process whose protection it can explain, rather than a capability it hopes will survive deployment.

Subscribe now

Then the agent asks a follow-up question. It requests the same payroll total for the same accounting period, using the same records, but excludes one employee. If the service permits both requests and releases exact answers, subtracting the second total from the first reveals that person’s contribution to the payroll calculation.

Nobody opened the employee’s record. No decryption key was stolen, and the protected processor didn’t suddenly start exposing its inputs. The reporting interface supplied two answers that, together, disclosed a value it was meant to keep out of the agent’s hands.

This remains an illustrative scenario, rather than an account of a PrivID deployment. Its assumptions matter: the underlying records haven’t changed, the totals are exact, and the interface allows the employee to be excluded. But under those conditions, the arithmetic is straightforward. Keeping the source records encrypted doesn’t stop a recipient from learning what the released answers reveal.

The information moved into the answer

A financial result has to convey information to be useful. A departmental total tells someone how much the department spent; a budget flag tells them whether a condition has been met. The objective can’t be to release an answer that reveals absolutely nothing, because that would also remove its business purpose.

The harder question is whether the answer reveals more than that purpose justifies. In our payroll example, the organisation intended to support departmental reporting. By allowing the agent to change the group behind each total, it also provided a route to learning about an individual.

This is a recognised problem in privacy research. In The Algorithmic Foundations of Differential Privacy, Cynthia Dwork and Aaron Roth describe how comparing answers about overlapping groups can expose information about a person, even when both groups are large. Their discussion also explains why checking query histories is difficult and why refusing an answer can itself reveal information.

The distinction is important for the argument we’ve been building. In The Old System Still Wants the Raw Data, separating a departmental report from an employee export removed a real disclosure dependency. The next task is to establish what the replacement interface allows.

An agent doesn’t need malicious intent

It’s tempting to reserve this problem for someone deliberately trying to extract a salary. But a reporting agent could encounter it while following an ordinary instruction to explain an unexpected change in expenditure.

The agent might compare teams, locations, job categories and reporting periods, looking for the difference that explains the variance. If its tools allow increasingly narrow groups, its analysis could identify a person’s contribution without ever requesting a field called “individual salary”.

That’s a design problem. An authorised task can generate information that exceeds the intended scope of that task, particularly when the application offers flexible filters and releases precise results. Permission to investigate departmental spending doesn’t automatically justify every detail that can be derived.

External context complicates the picture further. A total for a small team may become revealing to someone who already knows most of its members’ salaries. A change between reporting periods could suggest an individual’s pay if the observer knows who joined and what else remained constant. Those inferences depend on additional facts and can be wrong, but the organisation shouldn’t assess its outputs as though recipients know nothing about the world around them.

Once a result reaches an agent, it may enter a written explanation, a retained conversation or a report passed to another team. Restricting the original database doesn’t undo those disclosures. The decision about what to release has to happen before the answer becomes available for reuse.

The result needs its own boundary

Fully homomorphic encryption provides a way to compute over encrypted inputs and retain an encrypted result until a separately controlled party decrypts it. The Homomorphic Encryption Standardization consortium describes that separation. It creates an opportunity to control release, but the release rules still have to be designed and enforced.

For our finance workflow, the organisation could start with a fixed departmental report rather than unrestricted access to every possible grouping. It could define which periods and categories are available, which recipients need exact amounts and which questions should go through a separate review. The application would then expose operations matched to the reporting task.

Where flexible analysis is necessary, restrictions on small groups, overlapping queries and access across related tools may reduce opportunities for inference. Large groups can differ by one person, and a limit on request frequency won’t stop a disclosure that needs only two answers.

The controls also need to reflect how the work is organised. If several agents share their outputs, checking each agent’s requests independently could miss what their combined results reveal. An organisation has to decide where related access is evaluated and how separate workflows affect the same disclosure problem.

That doesn’t mean engineers can reliably detect every possible deduction a recipient might make. It means the questions and answers need deliberate design, with the limitations documented. Offering fewer, better-defined operations can make that task more manageable versus trying to recognise every dangerous combination afterwards.

Exactness has a purpose, too

For some statistical uses, differential privacy offers a formal way to bound privacy loss, commonly through carefully calibrated randomness and accounting for repeated analyses. NIST’s guidance on evaluating differential privacy guarantees explains that protection has to be assessed across analyses of the same data.

This is a complementary approach to protected computation. It isn’t an automatic property of FHE or ZKPs, and I’m not claiming it’s an existing PrivID feature. Whether it belongs in a deployment depends on the use case.

A payroll team needs accurate payments and reconciliation. A separate statistical service studying broad workforce trends may have more room to trade precision for privacy. These are different jobs, and warrant different answers and access.

Even returning a yes-or-no result needs care. If an interface lets a caller repeatedly change the threshold in a question about one person’s pay, the sequence of answers can narrow the amount.

Where PrivID’s contribution has to be precise

PrivID’s combination of ZKPs, FHE, segmentation and bounded access is intended to reduce unnecessary exposure during verification and computation. In this workflow, those mechanisms could help keep payroll inputs out of the agent’s readable environment and restrict it to defined operations.

The output problem adds a requirement around that protected service. The deployment needs to determine who can receive a result, how detailed it can be and which related requests are permitted. Some of those decisions can be implemented through defined computations and access controls; others require application design and additional privacy methods.

A ZKP can establish a specified claim without exposing its private supporting information. It doesn’t establish that the claim is harmless to reveal. Likewise, FHE can protect a calculation’s inputs while its exact output supplies information that the recipient shouldn’t receive. The security of the cryptographic mechanism and the appropriateness of the disclosure are separate questions.

This is why minimising fallout requires precision about what has been reduced. A compromised agent with no raw-data access may have fewer opportunities to extract complete records. The implementation should narrow both routes as far as the task and its operational requirements allow.

The second question belongs in the design

Return to the finance report. The first departmental total may be entirely appropriate for the agent preparing it. The second, almost identical total is where the organisation finds whether it designed a reporting function or an interface that lets callers uncover individual amounts.

A credible deployment needs to examine those follow-up interactions before celebrating the absence of a payroll export. The agent should still be able to explain useful changes and flag genuine problems, with a separate path for investigations requiring more detailed access. That keeps the business purpose intact while giving the organisation a clearer account of what each participant can learn.

The protected calculation hasn’t failed because its answer carries information. The failure comes when the service releases information beyond its intended purpose and calls the result safe because the records stayed encrypted. In production, the protection has to extend to the questions the agent can ask.

Subscribe now

Imagine the error message saying an employee identifier is missing. The file also lacks individual amounts and several other fields that the old reporting process expects. The finance team wanted a departmental answer, but the software’s import contract was written for employee records. An aggregate result can’t satisfy it, however accurate that result might be.

This is a continuation of the illustrative finance workflow from the first article, rather than an account of a PrivID deployment. It captures a different obstacle from the timeouts and support problems explored in The Demo Worked. What Now?: what happens when protected processing succeeds, but the application receiving its result still demands the underlying information?

At that point, the engineering team faces a deceptively simple request to make the two systems compatible. Compatibility could mean changing the importer, introducing a separate reporting route or deciding that the proposed integration isn’t suitable yet. It could also mean decrypting the records and reconstructing the old file, which would get the application working while restoring the disclosure the project was supposed to reduce.

A required field isn’t a business requirement

The old importer didn’t necessarily begin life as a bad design. It may have served a process in which someone reconciled employee records, corrected individual discrepancies and investigated transactions. For those activities, some record-level access could be justified. Years later, the same interface might have become the convenient route for a much narrower question about departmental expenditure.

That’s where the investigation has to become specific. Does the application use the employee identifier to join records? Does it need individual amounts to recalculate the total? Is a field genuinely necessary for this report, or does the importer reject its absence because every previous file included it? These questions can produce different answers within the same application.

If the software only checks whether a field exists, the team may be able to modify that validation. If it relies on the field to perform a necessary operation, the operation needs to be redesigned or retained within a controlled environment. Inventing values to satisfy the importer would be a dangerous shortcut if those values could corrupt reconciliation, reporting or later decisions.

The important distinction is between information the business needs for this task and information the interface assumes it will receive. A migration becomes possible when engineers establish which is which. Until then, “the system needs it” describes an observed dependency without explaining whether that dependency should survive.

The adapter can put the exposure back

A connector is usually presented as the answer to an integration problem. It sits between the two applications, translates their formats and lets everyone keep the software they already know. For privacy-preserving computation that translation may be more problematic versus the elegance of the connection.

An adapter receives an encrypted result and releases an approved departmental total to the reporting application. Suppose that it now decrypts all the individual records, rebuilds the old import file and sends it downstream. The protected calculation still happened, but the receiving application now has the readable dataset.

The second design might retain protection within the outsourced computation while leaving the reporting environment’s exposure largely unchanged. That limited improvement can still have value, but it needs to be described at the boundary where it actually exists.

The Homomorphic Encryption Standardization consortium explains that computation can operate on encrypted data and produce an encrypted result, with decryption taking place separately. That capability doesn’t automatically convert an ordinary application into one that can process ciphertext. Moving an operation into protected computation means an appropriate implementation; passing encrypted values into software that expects readable numbers won’t make its existing arithmetic work.

An adapter might carry protected values, validate a request or release a restricted answer. If its purpose is to restore the entire raw dataset so the old application can continue unchanged, the customer needs to know that before treating the connection as evidence of reduced exposure.

Keep the payroll function, change the reporting

The practical answer doesn’t have to be replacing the entire finance platform. In our example, the organisation still needs to pay employees, correct mistakes and maintain records. Those responsibilities shouldn’t be dismantled because a reporting workflow can use less information.

The team could separate the departmental reporting task from the functions that genuinely need individual records. A protected service could perform the defined calculation, with the approved answer delivered through a reporting interface that accepts totals. The original payroll system would retain its necessary responsibilities, the AI agent wouldn’t depend on receiving the employee-level export for this task.

That arrangement means engineering work. The new route has to use the correct source records, handle adjustments consistently and establish whether it produces the same business answer as the existing report. During evaluation, the organisation might compare both routes. Any temporary access or copies created for that comparison would also need to be accounted for when judging the pilot.

Some applications won’t offer a practical place to make this separation. Their reporting, validation and record-level operations may be too tightly coupled, or a vendor may not support the necessary change. In that case, the honest options include containing the existing exposure while planning a replacement, or choosing a different workflow for the first deployment. Acknowledging that constraint is part of making the technology useful. A narrowly successful integration gives the organisation something it can operate and extend.

Where PrivID can change the hand-off

PrivID’s intended role is to help build those narrower interactions using zero-knowledge proofs, fully homomorphic encryption, segmentation and bounded access. The integration should begin with the task being moved and the information its participants currently receive. That keeps the work anchored in a demonstrable change to the process.

Where a receiving service only needs to verify a defined claim, a properly designed ZKP can replace disclosure of the underlying evidence. The service should verify the proof and act on the claim it establishes. Attaching a proof to the same complete employee export wouldn’t remove the exposure. A proof can’t make an unreliable source record trustworthy just by being mathematically valid.

Where the task needs a suitable calculation, FHE can protect its inputs within the processing service. A separate, controlled release step can determine which result reaches the reporting application or agent. These changes work when the receiving process can operate with that restricted answer and the deployed permissions prevent it from fetching the records through another route.

The benefit for an authorised agent is concrete. Its permission to prepare a departmental report doesn’t have to include readable access to every salary used in the calculation. If it makes a mistake, is manipulated or is compromised, limiting its interfaces and available information can reduce what it can disclose or misuse.

That still leaves the source system, authorised record-level users and the results to protect. The claim for PrivID should remain precise: reduce unnecessary information access within an integrated workflow. With the extent of that reduction established by the implementation.

What the importer finally accepts

Return to the finance team’s rejected file. The issue is whether the organisation can deliver the departmental report without restoring the employee export to the reporting route. Answering it may mean changing an interface, separating a calculation or admitting that one application isn’t ready for this migration.

If the redesigned process works, the finance team still receives a useful report and the payroll team still retains the access needed for its distinct responsibilities. The reporting agent and its downstream application receive a narrower answer. The organisation has changed a real dependency, not just adding protection to one stage.

This is why legacy integration deserves its own place in the privacy argument. The barrier may be an import specification that nobody has challenged because in the past because it worked. Changing it carefully can make room for protected processing without demanding that the organisation rebuild everything around it.

The old system’s request for raw data is the beginning of that investigation. By the time the new report loads successfully, the team should be able to explain why each remaining field is there.

Subscribe now