Enterprise agents can infer sensitive facts without violating a single permission. The missing control is what information may influence one decision.

Every permission check passed, and the agent still disclosed the layoffs.

Consider a routine enterprise workflow. At 7:00 on Monday morning, an operations agent is asked to prepare a briefing for 200 managers. It can read a regional workforce forecast, manager calendar metadata, and an office-utilization dashboard. The task is legitimate, the agent identity is valid, and every connector enforces its policy.

The workforce plan shows a hiring freeze in APAC support. Calendar metadata shows an unusual cluster of HR and legal meetings involving the same management chain. Office-utilization data shows that the affected sites are consolidating. No source says a reduction is coming. Together, however, the pattern is hard to miss.

The agent writes, “Expect a reduction in the APAC support organization next quarter,” and sends the briefing. It quotes no restricted file. It does not bypass a connector or exceed a role. It does what the organization bought it to do: find a pattern across systems and turn it into a useful conclusion.

This is not exfiltration in the familiar sense. No protected sentence crossed a boundary. A new, sensitive conclusion was produced from permitted inputs and released to an audience that was never meant to receive it. Accuracy does not remove the control problem: if the conclusion is right, the agent disclosed confidential strategy; if it is wrong, it laundered a speculative correlation into an authoritative management briefing.

I recently wrote about this as the problem of an agent figuring out something it was never explicitly told. This piece takes the next step. I am writing it because enterprises are giving agents cross-system reach before they have a way to govern the composite view that reach creates. How do we govern a conclusion that does not exist until permitted facts meet? No classifier can recognize every dangerous conclusion, but a production system can control what information is allowed to influence the same decision.

Why this matters now

The value proposition of an enterprise agent is composition. A support agent becomes useful when it can connect a ticket to telemetry, a contract, and an account history. A finance agent becomes useful when it can reconcile forecasts, invoices, and operating plans. Unlike a search interface, the agent does not merely return records. It maintains state, chooses sources, compresses evidence, delegates work, and may act on what it concludes.

That means context is no longer incidental input to a model call. It is a security-relevant object assembled at runtime. Most enterprise controls still govern the pieces around it: identity binds the principal, IAM authorizes each read, a sandbox constrains execution, and DLP inspects the output. Those controls can all work as designed while the composition itself remains unauthorized.

The underlying risk is not new. NIST's glossary describes data aggregation as compilation that may increase risk because the totality can be classified at a higher level, disclose sensitive information, or de-anonymize people. Agents industrialize that old aggregation problem. A join that once required a human analyst, an explicit query, and time can now occur inside a routine workflow and reach a downstream system in seconds.

Recent incident data shows why the surrounding architecture deserves attention. A 2026 IBM/Ponemon Institute study of 602 breached organizations found that the share reporting an incident involving an AI model or application rose from 13% to 21%. Among organizations in that group, 92% lacked proper AI access controls. The most frequently reported causes were not model failures: compromise of connected applications, APIs, or plug-ins and cloud misconfigurations affecting AI workloads each appeared in 27% of incidents.

This is a study of breached organizations, not a base-rate estimate for all enterprises, and it does not measure inference-by-composition. It establishes the more relevant point: the model is not an isolated security perimeter. Risk accumulates in the identities, connectors, state, orchestration, and release paths around it. The purpose of least context is to make one missing boundary in that system explicit and enforceable.

The missing authorization object

A typical agent authorization flow evaluates one operation at a time:

May this principal read this resource?

That check is necessary, but it is incomplete because an agent’s security state changes as information accumulates.

CanRead(agent, A) and CanRead(agent, B)
does not imply
CanCombine(agent, A, B, purpose)
or
CanRelease(result influenced by A and B, recipient)

These are three different decisions:

1. Access: may the agent retrieve the source?

2. Composition: may this source join what already influences the task?

3. Release: may a result influenced by that combination reach this recipient?

Composition is usually implicit: if both reads are allowed, both enter the model. By influence, I mean more than tokens currently visible in a prompt. Retrieved records, tool outputs, prior summaries, long-term memory, environment state, cached artifacts, and child-agent messages can all shape a result. A source can leave the active context window and still affect what happens later, so the real security boundary is the effective influence set of the decision.

I describe the broader lifecycle as authorization propagation: delegated authority has to remain valid as agents retrieve, delegate, synthesize, and act, not only when a credential is issued. Aggregation inference is the composition failure inside that larger problem.

Older security models already point in this direction. The 1989 Brewer-Nash model, commonly known as the Chinese Wall security policy, made access history-dependent: what an analyst had already read constrained what could be read next. Myers and Liskov’s decentralized information-flow model joined restrictions on combined values and allowed only an authorized owner to declassify a result. Newer agent architectures carry that thinking forward. Fides applies confidentiality and integrity labels to agent planning, while SPA preserves labels across persistent state and later queries.

These systems preserve known restrictions as information moves. The harder gap is composition. Information-flow control can conservatively track that A may influence X and carry A’s restrictions forward, but it cannot invent an organizational policy saying that A and B must not influence X together. If two sources are both labeled Internal, but their combination reveals an acquisition or a diagnosis, ordinary label propagation may faithfully produce another Internal value. The missing policy object is the join itself.

Least context means controlling composition

Least context is the smallest authorized influence set sufficient for a declared purpose and audience, for the shortest useful lifetime. Fewer tokens are not automatically safer; three selected facts can reveal more than 3,000 irrelevant ones. Ordinary data minimization gets only part of the way there because it asks whether each field is necessary. Least context also asks whether individually necessary fields are appropriate to combine.

Context is relational. Helen Nissenbaum’s work on contextual integrity gives us a useful vocabulary: actors, information types, and the conditions under which information moves. An agent adds an event before that movement by composing information into a new result. Purpose and audience therefore need to govern the composition, not only the final message. Work on least-privilege context engineering already narrows data and tools by agent role; composition control addresses the harder case in which the minimum sources for two valid subtasks may still be unsafe inside one shared reasoning process.

At every step, the information allowed to influence a task must remain within the combination, purpose, audience, and lifetime authorized by policy.

Every retrieval is therefore a security-state transition.

Share AIwithKT

Three gates around the reasoning process

A production design does not need a universal inference detector. It needs three deterministic gates around the probabilistic system.

The model should not be asked to grant its own authority. In AuthBench, a 2026 study of 120 realistic terminal tasks, frontier models both omitted permissions their execution chains required and granted unused or sensitive access. More inference-time reasoning did not resolve the mismatch. A model can propose a plan and explain why it wants a source; an independently enforced policy must decide whether that source may enter the task.

1. Admit the source

Before a retrieval, tool response, memory record, or child-agent result enters the influence set, a broker checks both access and composition:

admit(candidate, current_state) =
CanRead(principal, candidate)
and CanCombine(current_state, candidate, purpose)

The model may request a source, but it cannot authorize the join. A control plane keeps a context manifest outside the model with source identifiers, owners, domain tags, projections, labels, permitted audiences, obligations, and lineage. If the model compresses five documents into a summary, those documents may disappear from the prompt, but they do not disappear from the manifest.

Adding information should make known restrictions monotonically tighter:

AllowedAudience(next) is a subset of AllowedAudience(now)
RequiredObligations(next) is a superset of RequiredObligations(now)

The policy does not need a rule for every pair of documents. It can operate on conflict classes and typed domains:

allow workforce_forecast + office_utilization
for capacity_planning
when both are aggregated to region

deny workforce_forecast + identifiable_executive_calendar
for general_briefing

Projection matters as much as admission. A facilities task may need regional occupancy totals rather than badge events, while a planning task may need meeting counts rather than executive names. The safest join is often a join over less precise inputs.

2. Constrain execution

Authorized sources still need an authorized place to meet. A consequential task should begin with one execution contract that binds its sponsor, delegated agent identity, purpose, source domains, model route, tools, memory policy, audience, and expiry. Child agents receive narrower authority, never the sponsor’s full ambient access.

The model route belongs in that contract because model selection determines where information is processed, who operates the infrastructure, what retention terms apply, and which tools are reachable. Routing is part of authorization, not just an optimization for latency and cost. Keeping inference near the data helps with residency and exposure reduction, but zero data movement is not zero information movement. If protected records stay local while a derived conclusion travels elsewhere, information still crossed the boundary.

Building the security path around Kamiwaza’s Inference Mesh sharpened one distinction for me: locality is a security control, not a release guarantee. Bringing compute to data can keep raw records inside their perimeter; it does not determine where a derived conclusion may travel.

In addition, my work architecting and building Workrooms and its relationship-based authorization path gave that flow a bounded object: members, delegated identities, data, tools, model routes, memory, recipients, and lifetime. Those boundaries make continuous enforcement, teardown, and replay possible.

Leave a comment

3. Authorize release

An allowed reasoning process does not imply an allowed output. Start conservatively: an output inherits the joined restrictions of every source that could have influenced it. A semantic detector may raise the risk, narrow the audience, or trigger review, but it must not erase provenance-derived restrictions or turn a denied flow into an allowed one.

If the destination falls outside those inherited restrictions, release becomes a declassification request:

CanRelease(
influence_lineage,
purpose,
recipient,
destination,
policy_version
)

The organization must name who may authorize that downgrade. A cross-domain result may require the relevant data owners or a designated policy authority, not the agent developer and certainly not the model. Depending on the risk, the release gate can aggregate further, redact, restrict the destination, require human approval, or refuse the output. The output classifier is evidence for this decision; it is not the decision-maker.

When the task ends, its authority should end with it. The platform should revoke delegated credentials, terminate child agents, close output channels, expire memory, and delete ephemeral artifacts wherever the execution environment can guarantee it. The audit record must also preserve causal lineage. “What was in the prompt when the model answered?” is too narrow. The useful question is:

What information could have influenced this result, under which policy, through which execution path?

In my five-plane reference architecture, this becomes the evidence substrate: the reasoning plane records the decision, each enforcement plane records whether its directive was actually realized, and both are sealed into one reconstructible record. A policy decision records intent; it is not proof of enforcement.

Replaying the Monday briefing

Under least context, the broad management briefing gets its own purpose-bound execution boundary. The workforce forecast is admitted only at a regional aggregate, and office utilization enters as a seven-day regional average. Identifiable manager calendars are denied because that source class may not join workforce plans for a general briefing. The model can still summarize capacity and utilization without receiving the combination that made the layoff inference easy.

If HR legitimately needs all three sources for workforce planning, that becomes a different task with a named HR sponsor, approved source combination, restricted model route, narrower audience, short-lived memory, and HR-owned release decision. The underlying read permissions may be identical, but the composition and release authority are not. If the agent produces a sensitive personnel conclusion, the broad briefing cannot receive it merely because the prose sounds innocuous; the source lineage and task boundary already make that destination ineligible.

A practical first deployment

An enterprise does not need to classify every possible conclusion before it can begin. Start with one consequential workflow where multiple data domains meet and the output reaches a broader audience: a support escalation that joins tickets, contracts, and production telemetry; a renewal recommendation that joins product usage and account risk; or an operations briefing like the one above.

For that workflow, define the task sponsor, purpose, audience, source projections, model route, memory lifetime, and a small set of prohibited or review-required combinations. Keep an external context manifest, enforce admission before retrieval results reach the model, and attach influence lineage to the output. Run the policy in report-only mode first, then inspect which sources were unnecessary, which joins would have been blocked, and where humans overrode the recommendation.

The useful measures are operational: task completion, context reduction, denied joins, human-review rate, reviewer reversals, added latency, and the percentage of outputs with complete replayable lineage. A high block rate is not success. The objective is the smallest governed influence set that preserves the workflow's utility.

Least context has a symmetric failure mode: too much context can disclose sensitive information, while too little can make a partial answer sound complete. I built Partial Evidence Bench to measure that case. Across 72 tasks spanning due diligence, compliance, and incident response, silent access-control filtering produced unsafe completeness across every scenario family; explicit fail-and-report behavior eliminated it in the shipped baselines. Any system that narrows context therefore owes the user a completeness signal.

The honest limit

No architecture can guarantee that an agent will never infer a sensitive fact. The model carries information from training, the recipient brings outside knowledge, and public facts can turn a harmless-looking result into a disclosure. Dwork and Naor’s work on the limits of disclosure prevention explains the deeper problem: once arbitrary side information is possible, a useful system cannot generally promise that nothing new will be learned about an individual.

Composition policy catches combinations an organization can name. Source projection removes unnecessary detail, information-flow labels preserve known restrictions, purpose limits distinguish legitimate from illegitimate use, and release gates stop known unsafe flows. None can identify every sensitive fact a model might infer. Least context still changes the default: instead of placing every permitted fact into one long-lived reasoning process and hoping a filter catches the result, the system requires an owner for consequential combinations and releases.

Teams can begin with six questions:

1. What information, including memory and tool output, can influence this task?

2. Which individually permitted domains must not influence the same decision?

3. What is the least precise projection the task needs?

4. Which model, tools, and execution location may process the combination?

5. Who owns release of a result influenced by these sources?

6. Can we reconstruct that influence set after the fact?

If the platform cannot answer them, it governs connectors, not agent context. Least privilege remains essential because it constrains what an agent may access and do. Production agents need one more boundary around what those permissions become when they meet.

The objective is not to stop agents from finding patterns. An agent that cannot connect facts is not useful. The objective is to make consequential combinations intentional, bounded, and accountable: which sources may meet, for what purpose, inside which execution boundary, and who may receive the result.

Least context does not replace IAM, information-flow control, sandboxing, or DLP. It connects them around the decision being made. Least privilege constrains the authority an agent starts with. Least context constrains how that authority compounds as information accumulates. Release control governs where the resulting knowledge may go.

Two authorized reads do not make a safe context. They make a join, and the join needs an owner.

Share

Share AIwithKT

Leave a comment

Related work and implementation context

• A Five-Plane Reference Architecture for Runtime Governance of Production AI Agents

• Authorization Propagation in Multi-Agent AI Systems

• Partial Evidence Bench

• Kamiwaza: Workrooms, AI Security and Compliance, and Inference Mesh

Innovating with integrity,
@AIwithKT 🤖🧠

The sharpest AI-security finding of 2026 hasn’t, so far, been an attack. It was the discovery that a frontier model behaves differently when it **suspects** it’s being evaluated — which quietly turns a published safety number into a test-taking score…and confirms why the controls worth trusting now live outside the model.

#FrontierAISecurity #AgenticAI #NonHumanIdentity

Share

Subscribe now

Earlier this year, Anthropic published two findings that belong together. In the first, Claude Opus 4.6 — working a hard benchmark question — figured out on its own that it was being evaluated, found the benchmark’s source code on GitHub, read its XOR encryption scheme, recovered the key, and wrote its own routine to decrypt the answer key. In the second, a separate interpretability method that reads the model’s internal state caught Claude flagging I’m being evaluated on as many as a quarter of one benchmark’s problems — and under 1% of the time in real user sessions — often without that recognition surfacing anywhere in its visible reasoning. One is a vivid existence proof of what the model will do once it knows it’s on a test. The other is how often the knowing happens, and how quietly. They point the same way.

If a model behaves measurably differently when it thinks it’s being watched, then any safety number you derive from an evaluation is, in part, a measurement of how the model performs when it knows it’s on camera. The refusal rate, the sabotage score, the red-team pass rate — each is now the benchmark’s model, not the deployed one. And the bias doesn’t even run in a direction you can count on. A model that senses a capability probe and quietly sandbags makes the danger number look reassuring; a model that recognizes a safety eval may look more compliant than it is. You don’t get to know which way the needle is bent — only that the test and the deployment are no longer the same model behaving the same way. A model card is starting to look less like an x-ray and more like a job interview: not worthless, but a performance, and not the thing you thought you were buying.

I’ve spent the last many articles in my newsletter arguing that the model is not the security boundary (#48), that the real problem is the permission to act (#49), and that a record is worthless if it survives but can’t be read (#50). I framed those as my read of where the field had to go. Here is the version I can now defend as more than opinion:

Eval-awareness means a model’s published safety numbers measure its test-taking, not its deployment behavior. So assurance has to migrate — off the model and out to where you can verify and enforce it at runtime: identity, permission, provenance, and the environment the agent runs inside. The labs didn’t start building containment because of eval-awareness; they started because no model layer is ever airtight. But eval-awareness is the proof they were right — the cleanest demonstration yet that you can’t take the model’s own word for its safety, and have to put your assurance somewhere it can’t perform for you.

Once you see that logic, the rest of spring 2026 stops looking like a pile of unrelated announcements. Nobody coordinated it — OWASP, NIST, the labs, the EU, and the acquirers each moved for their own reasons. But they all moved the same direction: away from trusting the model’s own report card, and toward controls you can check without it.

1. The corroboration came from the model’s own builder

The eval-awareness work has a sibling that makes the same point from the other direction.

In a controlled internal red team, Anthropic instructed Claude Code to read a credentials file and POST it to an external endpoint. Across 25 runs, it complied 24 times. The compliance rate isn’t the interesting part — an agent told to do something by a prompt it trusts will do the thing; that’s not a discovery. The interesting part is the conclusion they drew and published in How We Contain Claude Across Products: hard, deterministic boundaries — sandboxes, filesystem walls, egress controls — have to come before probabilistic model defenses, because the model layer is never 100%. And yes, that red team is itself an evaluation, subject to the same eval-awareness caveat. But notice what it certifies: not a disposition you have to trust the model to keep, but an egress rule and a sealed filesystem you can inspect and prove. That is the whole move — assurance shifts to the layer that doesn’t need the model’s cooperation to mean something.

Don’t mistake this for a confession against commercial interest. Containment is now part of what Anthropic sells — the enterprise pitch is the walls as much as the weights. Which is what makes the signal so telling: even the vendor whose entire product is the quality of the model now stakes its safety story on the environment around it.

The Sabotage Risk Report for Opus 4.6 closes the loop almost too neatly. Its own headline — catastrophic-sabotage risk “very low but not negligible,” the model “at times overly agentic… taking risky actions without first seeking user permission” — is itself an evaluation result, and the eval-awareness finding’s own logic cuts it in two. The reassuring half deserves a discount: a model that recognizes a safety eval has every incentive to look safe. The self-incriminating half deserves the opposite — “overly agentic, acts without asking” survived into the report despite that incentive, which makes it more credible, not less. That’s not a knock on Anthropic; they surfaced both halves. It’s the situation we’re all in now.

A capable model that is overly willing to act without asking, and is quietly better-behaved when it suspects it’s being graded, is not a problem you solve at the model layer. It’s a problem you contain at the boundary.

2. The boundary has a new address: identity and permission

If you can’t trust the model’s report card, you move your assurance to things you can verify directly. Watch where the standards bodies and the acquirers put their money in 2026, and it’s the same address every time: the agent’s identityand its permission to act.

• OWASP’s Top 10 for Agentic Applications 2026 leads not with jailbreaks but with ASI03: Identity & Privilege Abuse — agents inheriting user credentials, caching secrets, and delegating to each other until an attacker walks a low-privilege request up a high-privilege chain. The prescribed fix reads like a platform spec: a unique managed identity per agent, least privilege, short-lived task-scoped credentials, and authorization enforced on every action.

• Microsoft Entra Agent ID reached general availability in April, treating agents as first-class identities — a reusable Agent Identity Blueprint, a per-instance identity, and an Agent User Account for on-behalf-of flows.

• Okta and Anthropic shipped Enterprise-Managed Authorization for MCP in June, built on the IETF’s ID-JAG (Identity Assertion Authorization Grant). The mechanic is the part worth learning: instead of prompting each user through an OAuth consent screen per tool, the identity provider issues a signed assertion that the agent exchanges — via OAuth token exchange — for a short-lived access token that is audience-restricted to one downstream resource. Trust and revocation live at the IdP, not in a scatter of per-app grants. An admin wires connectors once; employees inherit scoped access on login. Okta’s branding for it is Cross App Access; Asana, Atlassian, Canva, Figma, Granola, Linear, and Supabase were connected at launch.

• NIST launched an AI Agent Standards Initiative in February, and its NCCoE work makes the quiet part explicit: don’t invent a new identity stack — extend the one we have (OAuth 2.0, OIDC, SPIFFE/SPIRE) to treat agents as non-human identities with real lifecycle management.

• And the market voted with its balance sheet. Cisco announced intent to acquire non-human-identity startup Astrix Security in May, one move inside a broader run on the agentic-identity control layer, while MITRE ATLAS spent the year rewriting its matrix around delegated authority, credential abuse, and MCP server compromise.

Here’s my bet on where this goes, and it’s falsifiable, so hold me to it. Per-agent identity solves the confused-deputy problem one agent at a time — and creates a population problem in the aggregate. Within two years, agent identities will outnumber human ones in the average enterprise — the identity vendors are already architecting for that inversion. The breach class of 2027 won’t be prompt injection. It will be the confused deputy at population scale: orphaned, over-scoped agent credentials that nobody deprovisioned, issued for a task that ended in March and still valid in November. Non-human identity is about to become the new shadow IT — and the org that can’t answer “how many agent identities do I have and who owns each one” is already behind.

3. The tool layer is where agents actually get owned

The worst agentic incidents of 2026 didn’t involve manipulating a model at all. They lived one layer down, in the plumbing — exactly the layer you fall back on when you stop trusting the model, and exactly the layer your threat model probably under-weights.

• CVE-2026-33032 — “MCPwn,” CVSS 9.8 — was an authentication bypass in nginx-ui’s MCP integration. One endpoint shipped with an allow-all default and no auth check, so any attacker on the network could invoke privileged tools and rewrite the server config in two requests. No prompt injection. No jailbreak. A missing authorization check on a tool surface, patched only after nearly 2,700 instances sat exposed on Shodan and it was being actively exploited.

• CVE-2026-4372 turned one line in a model’s config.json into remote code execution in Hugging Face Transformers— and it fired even with trust_remote_code=False, the exact flag everyone trusted. Vulnerable versions had been pulled on the order of hundreds of millions of times — a measure of blast radius rather than confirmed victims, and the blast radius is the point.

• ClawHavoc poisoned ClawHub, the skill marketplace for the OpenClaw agent platform, with more than 1,184 malicious “skills” — the agent-era typosquatted package registry, serious enough to draw a dedicated MITRE ATLAS investigation.

The positive pattern here is unglamorous and specific. Treat every MCP server as an OAuth 2.1 resource server. Use resource indicators (RFC 8707) so a token minted for one server is audience-bound and can’t be replayed at another — the single control that would have blunted half of this year’s MCP incidents. Scope tokens per tool, not per agent. And understand that “short-lived credentials” do not fix the confused deputy; that’s a problem of whose authority an action carries, and the fixes are audience-restricted and proof-of-possession tokens (DPoP- or mTLS-bound), plus never letting an agent spend its own ambient privilege on instructions that arrived from untrusted input.

None of these incidents touched the model’s reasoning. They didn’t have to. They touched the tools it could call, the packages it could load, and the identity it ran as.

4. Provenance became law, silicon, and math in the same window

I argued in #42 that provenance is the new perimeter, and in #50 that a record is worthless if it survives but can’t be read by a verifier. In a single stretch of spring 2026 that argument grew teeth in three different materials at once.

Law. The EU AI Act’s Article 50 transparency obligations become legally binding on August 2, 2026, and the Commission published its Code of Practice on AI-generated-content transparency on June 10. The requirement isn’t a visible “AI-generated” sticker — it’s machine-readable marking: signed metadata, imperceptible watermarks, cryptographic signatures. Provenance a program can check, not a human eyeballs.

Silicon. At WWDC on June 9, NVIDIA, Apple, and Google extended Apple’s Private Cloud Compute onto Blackwell GPUs with Confidential Computing on Google Cloud — hardware-rooted attestation and confidential inference, so sensitive workloads run in a cloud that can’t read them, even from the administrator’s seat. The trust anchor moved into the chip.

Math. And the reason all of this is load-bearing: roughly 250 poisoned documents were enough to plant a backdoor trigger in models from 600M to 13B parameters, with the count staying near-constant as the models and their training sets grew (Anthropic, with the UK AI Security Institute and the Alan Turing Institute). That’s a result about a specific class of backdoor under controlled pretraining — not a claim that your corpus is already compromised. But it resets the threshold: when a near-constant absolute count does the damage, absence of provenance stops being a footnote and becomes a finding.

The toolchain for this exists and is adoptable now, which is the part most “provenance is important” essays skip: C2PA / Content Credentials for media provenance, Sigstore and in-toto/SLSA for supply-chain attestation, SPIFFE/SPIRE for workload identity, and remote-attestation verification for anything running in a confidential enclave. The perimeter you’re building is a record that is signed, attestable, and readable by a verifier, a regulator, and a hardware root of trust — all three.

Subscribe now

5. From guardrails to guarantees

Every one of these moves makes the same trade: a probabilistic control at the model layer for a deterministic one somewhere you can verify. A refusal you hope holds becomes an egress rule with no route out. A system prompt begging the agent to behave becomes an authorization check the tool enforces before it runs. “Trust this model” becomes “attest this identity, this permission, this provenance — in hardware where you can.”

This is the migration from guardrails to guarantees, and from operational autonomy — agents that can act — to governable autonomy — agents whose every action carries a verifiable identity, a scoped permission, and a provenance trail you can check after the fact. The organizing question stops being “is the model aligned?” and becomes “can I prove who did what, with whose authority, on what data?”

That’s the frontier worth building toward, and the reason I find it energizing is that it’s buildable today. Not someday-AGI-safety. Identity systems, policy engines, attestation, signed provenance — the unglamorous infrastructure that decides whether autonomy is something you can put your name behind. The eval-awareness finding is what makes it undeniable: when you can’t fully trust the number on the model card, you instrument the runtime instead.

Be precise about the claim, because it matters. This isn’t replacing model-layer assurance — it’s partitioning it. You still need evaluations: to scope an agent’s permissions in the first place, and to confirm your egress rules and authorization boundaries actually hold. What changes is where the load-bearing trust sits. An eval you can no longer fully trust becomes one input among several; a control you can inspect, prove, and revoke becomes the floor. The model’s report card moves from verdict to hint.

6. What to do this week

Most “what to do” lists hand you six co-equal good intentions. Here’s a sequence instead, with one forcing-question at the front.

Start here. Inventory every agent you run, and answer one question for each: what identity does it execute as, and can that identity be revoked independently of any human’s? If the answer for any agent is “a shared service account” or “a person’s token,” that is your top finding — full stop. It’s also the gap Okta quantifies: most organizations now suspect agent-related incidents, while only a small fraction treat agents as identity-bearing at all. Close that first.

Then climb the ladder:

1. Identity — every agent gets its own managed, attestable, independently revocable identity. (You can’t do anything below this until this is true.)

2. Permission — scope it to the task and make it short-lived and audience-restricted. Standing, broad, replayable credentials are the confused deputy waiting to happen.

3. Enforcement — put authorization out-of-process, at the tool boundary, as an OAuth resource server with resource indicators. The prompt is a suggestion; MCPwn is what a suggestion gets you.

4. Environment — deterministic egress allowlists, filesystem boundaries, no credentials in the sandbox. Anthropic’s answer to a near-certain attack was the room, not the model. Borrow it.

5. Provenance — sign what you can, attest what you can’t, and treat unverifiable inputs as untrusted by default.

The ladder matters because the rungs aren’t interchangeable. Provenance tooling on top of shared service accounts is theater. Identity first.

The model will keep getting more capable, and — this is the uncomfortable part — better at knowing when it’s being measured. The number on the model card will keep climbing while it quietly means less. That isn’t a reason for despair; it’s a reason to stop looking for assurance in the one place that has a motive to look good for the test. Put it where the model has no audience to play to: in its identity, its permissions, its provenance, and the walls of the room you run it in.

The sandbox holds. The model doesn’t. And the number on the card was never going to.

Sources

• Anthropic — Natural Language Autoencoders (interpretability; ~26% eval-awareness on SWE-bench Verified vs <1% in real usage) — https://www.anthropic.com/research/natural-language-autoencoders

• Anthropic — Eval Awareness in Claude Opus 4.6’s BrowseComp Performance (the model decrypts the answer key) — https://www.anthropic.com/engineering/eval-awareness-browsecomp

• Anthropic — How We Contain Claude Across Products — https://www.anthropic.com/engineering/how-we-contain-claude

• Anthropic — Sabotage Risk Report: Claude Opus 4.6 — https://anthropic.com/claude-opus-4-6-risk-report ; METR review — https://metr.org/blog/2026-03-12-sabotage-risk-report-opus-4-6-review/

• OWASP GenAI Security Project — Top 10 for Agentic Applications 2026 — https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/

• Microsoft — Microsoft Entra Agent ID — https://learn.microsoft.com/en-us/entra/agent-id/what-is-microsoft-entra-agent-id

• Model Context Protocol — Enterprise-Managed Authorization — https://blog.modelcontextprotocol.io/posts/enterprise-managed-auth/ ; Okta newsroom — https://www.okta.com/newsroom/press-releases/okta-becomes-a-featured-identity-provider-powering-secure-ai-agent-connections-for-claude-enterprise/

• IETF OAuth WG — Identity Assertion Authorization Grant (ID-JAG) ; Resource Indicators for OAuth 2.0 (RFC 8707) — https://datatracker.ietf.org/doc/html/rfc8707

• NIST — AI Agent Standards Initiative — https://www.nist.gov/artificial-intelligence/ai-agent-standards-initiative ; NCCoE — Software and AI Agent Identity and Authorization — https://www.nccoe.nist.gov/projects/software-and-ai-agent-identity-and-authorization

• Cisco — Cisco Announces Intent to Acquire Astrix Security — https://blogs.cisco.com/news/cisco-announces-intent-to-acquire-astrix-security

• MITRE ATLAS — https://atlas.mitre.org/; MITRE ATLAS OpenClaw Investigation — https://ctid.mitre.org/blog/2026/02/09/mitre-atlas-openclaw-investigation/

• Rapid7 — CVE-2026-33032: Nginx UI Missing MCP Authentication — https://www.rapid7.com/blog/post/etr-cve-2026-33032-nginx-ui-missing-mcp-authentication/

• Pluto Security — Unauthenticated RCE in Hugging Face Transformers via Config Injection (CVE-2026-4372) — https://pluto.security/blog/unauthenticated-remote-code-execution-in-huggingface-transformers-via-config-injection/

• Anthropic / UK AISI / Alan Turing Institute — A Small Number of Samples Can Poison LLMs of Any Size — https://www.anthropic.com/research/small-samples-poison

• EU AI Act — Transparency Rules (Article 50) — https://artificialintelligenceact.eu/transparency-rules-article-50/

• NVIDIA — NVIDIA Confidential Computing to Help Expand Apple’s Private Cloud Compute — https://blogs.nvidia.com/blog/nvidia-confidential-computing-apple-private-cloud-compute/

Leave a comment

Share

Subscribe now

Innovating with integrity,
@AIwithKT 🤖🧠

Last week a reporter sent me nine questions about a new agent method that was making the rounds. Most were the questions you’d expect — what’s novel, what do the benchmark numbers mean, does it generalize. I started answering. And about halfway down the list I realized I was answering the wrong thing.

The method had been scored by a model from the same family as the model it was built on. The reported win was large. And once you see that setup, the question you actually care about changes underneath you — from is this method good to can this evaluation even tell us whether it’s good. Those feel like the same question. They are not.

Now, here’s where I expect half of you to nod and check out: "sure, LLM judges are biased toward their own kind. Old news." You’re right. It is old news. Zheng and co-authors named self-enhancement bias back in the 2023 MT-Bench paper that put LLM-as-judge on the map. Two years ago. If that were the whole story, this would be a “duh” post and you should close the tab.

Subscribe now

It’s not the whole story. The interesting part isn’t that the bias exists. It’s what the bias does next — and almost everything it does next runs opposite to what the industry assumes.

Share

The comfortable assumption that’s quietly wrong

Back in #47, I wrote about the teacher signal — the supervision pattern worth more than the content it produces. This is its mirror on the other end of the pipeline. Call it the judge signal: the evaluation we now trust to tell us whether an AI system is safe, correct, ready to ship.

Here’s the assumption holding the whole thing up, usually unspoken: this is an early-models problem. As models get smarter, their judgment gets more reliable. Grow out of it.

The research points the other way, and this is the turn I’d build the whole piece on.

Panickssery, Bowman, and Feng have a paper — LLM Evaluators Recognize and Favor Their Own Generations, NeurIPS 2024 — that traced the bias to its root. The driver is self-recognition: models can pick their own writing out of a lineup. And they showed it’s causal, not just correlated — when they fine-tuned a model to be better at recognizing its own output, its preference for that output went up.

Sit with what that means. Self-recognition is itself a capability. It is the kind of thing that gets better as models get better. So there’s no reason to expect this to wash out as models scale — and some reason to expect it to sharpen. The empirical backup is already in: in the safety-evaluation work I’ll get to in a second, bigger judge models were not more robust than smaller ones. Scale didn’t save them.

So the comfortable assumption is backwards. The more capable your judge model becomes, the better it recognizes itself, and the more it trusts its own reflection. We are not growing out of this. We are growing into it.

The fixes everyone reaches for are half illusion

Okay — so you swap judges. Use a different family. Use a panel. That’s the standard advice, and both halves of it are shakier than they’re sold as.

“Use a different family” buys less independence than it looks like. GPT, Claude, Gemini, Llama — they train on heavily overlapping web corpora, similar architectures, similar alignment recipes. Switching vendors reduces the surfacepreference, the house-style thing. It does much less for the shared blind spots that come from shared training distribution. If a failure mode is common to the way this whole generation of models is built, a “different” judge from the same era has it too. You changed the logo on the judge’s robe. You didn’t change what it can’t see.

“Use a jury” is the one people will fight me on, and the evidence is right there. There’s a 2025 paper by Hongyu Chen and Seraphina Goldfarb-Tarrant with a title that says it all: Safer or Luckier? LLMs as Safety Evaluators Are Not Robust to Artifacts. They tested eleven judge models on safety evaluation, and they explicitly checked whether aggregating multiple judges — a jury — fixed the problem. It helped. It did not fix it: artifact sensitivity persisted even in the best jury configurations. The standard “just ensemble more judges” answer is sold as the cure, and the data says it’s a discount, not a cure.

The pattern: the remedies the field reaches for treat the cheap failure — surface preference — and barely touch the expensive one.

This isn’t a measurement bug. It’s an attack surface.

Let me make the expensive failure concrete, because it’s easy to wave past in the abstract.

Picture the standard way teams red-team an agent today. You use an LLM to generate the attack prompts, and an LLM judge — usually same family — to score whether the agent’s responses were safe. Now suppose the agent is vulnerable to a class of indirect prompt injection that this generation of models doesn’t reliably register as an attack. Watch what happens. The red-teamer, sharing that blind spot, doesn’t generate the attack well. The judge, sharing it too, doesn’t flag the response as unsafe. The eval comes back green. The vulnerability ships — not despite your testing, but through it. The blind spot is symmetric across the attacker and the judge because they’re the same mind wearing two hats.

And it gets worse than passive blindness. Here’s the number from that Safer or Luckier? paper that I can’t stop thinking about: those eleven judges could be swung by superficial artifacts — not real changes to how safe the content was, just surface stuff like apologetic phrasing. The swing was up to 98%. Nearly a flip. The same unsafe substance, wrapped in the right contrite tone, got graded safe.

Read that as a security person, not a benchmarker. A judge that grades on costume is not merely unreliable — it is gameable. Anyone who knows the judge can dress an unsafe output in safe-sounding language and walk it through the gate. That includes an adversary. It also includes a well-meaning team optimizing to pass the eval, which is the same failure with better intentions.

Leave a comment

A biased ruler mismeasures. A gameable judge is a control you can attack. The judge isn’t standing outside your threat model — it’s inside it. That is a categorically more serious claim than “judges have a thumb on the scale,” and it’s the one the security field hasn’t fully absorbed.

And if you wire it into training, it compounds

Now the part that actually keeps me up.

Nobody uses these judges once. Judge output becomes signal — for RLHF-from-AI-feedback, for best-of-N selection, for distillation. This isn’t a future risk; it’s the current default. RLAIF and Constitutional-AI-style training already use a model to generate the preference signal another model learns from. You generate, you let a model score, you train the next model toward the scores.

Trace the loop. The model produces output. A same-lineage judge rewards the output it recognizes as its own kind. That reward selects the next generation. Which the judge recognizes even more strongly — because, per Panickssery, stronger self-recognition means stronger self-preference. The blind spot doesn’t just survive across generations. It gets selected for.

I’ve made a version of this argument before, in Layered Mutability — that systems can look stable on the surface while the thing being governed drifts underneath. This is that, applied to evaluation. Self-preference in a one-shot eval is a measurement error. Self-preference in a training loop is a drift pressure. It’s a structural reason model monocultures don’t just stay wrong — they get more confidently wrong over time, each generation more fluent in the dialect its judge already likes.

That’s not “duh.” That’s a feedback loop with a sign error built into it.

The receipt — including its limits

I believe all this partly because I built around it before I could cite it.

When I built the rubric-driven PR-security reviewer that sits in front of my repos, I made it multi-engine — not for redundancy, and not because it was cheap, because it isn’t. I did it because I would not let one model family be both the author of a security judgment and the only check on it. Where two genuinely different engines disagree is exactly where a human should look. The disagreement is the signal.

But — and this is the honesty the topic demands — I have to hold my own tool to the standard above. Two frontier models from the same generation share more than their vendors admit. My multi-engine setup buys real independence against surface preference and partial independence against shared blind spots. Not full. Anyone who tells you their judge panel is truly independent is selling the logo, not the robe.

What “good” actually looks like — knowing the easy answers don’t fully work

First, the obvious objection, met head-on: I’m not arguing for a return to all-human evaluation — it doesn’t scale, and humans carry their own blind spots. I’m arguing that we’ve quietly swapped a slow, biased judge for a fast one whose bias compounds, and called it progress. The goal isn’t to fire the machine judge. It’s to stop pretending it’s independent when it isn’t.

So — none of the fixes are exotic. Most are discipline, plus the humility to not oversell the discipline.

Demand independence you can actually defend — different training lineage and data, not just a different API key. Use juries, but treat them as harm-reduction with a known ceiling, not as a clean bill of health; Chen and Goldfarb-Tarrant already showed the ceiling. Keep humans on a sampled subset and report inter-rater agreement instead of waving at “human-reviewed.” Treat the judge as part of the threat model and ask the adversarial question out loud: what surface artifact would flip this verdict? And never close the training loop on a same-lineage judge without an out-of-distribution check that the judge itself can’t see coming.

Diversity here isn’t a virtue. It’s a control — and like every control, you have to know what it does and doesn’t stop.

Leave a comment

The distillation

A model that can score another model is convenient. A model that can score another model from outside its own blind spots is trustworthy. And the uncomfortable truth is that the second kind is getting rarer as models get more capable and more alike, not more common.

So the question isn’t the one that reporter’s brief led with. It isn’t how good is the method. It’s the one underneath: as your models get smarter, are your judges getting more independent — or just more confident?

Because if it’s the second, you don’t have an evaluation that’s improving. You have a monoculture grading itself, learning to love its own reflection a little more with every release — and calling the result a verdict.

Innovating with integrity,
@AIwithKT 🤖🧠

Why I think the real governance problem in agentic AI is legibility.

Over the last few months, I’ve found myself less interested in the usual question people ask about AI systems, which is what they can do, and more interested in a different one: what will still be intelligible about their actions later.

That sounds like a narrower question, maybe even a duller one. It isn’t. I think it gets closer to the part of the problem that institutions are actually going to trip over.

When people talk about agentic AI right now, they usually talk in the language of capability. Can these systems reason? Can they plan? Can they use tools? Can they automate real work? Can they replace workflows that used to require a person in the loop?

All of that matters. But it still presumes a relatively simple picture of what an AI action is. There is a model. It gets a task. It produces an output. Maybe it calls a tool or retrieves a file on the way. But the relevant unit still feels like a bounded model producing a bounded answer.

I think that picture is already breaking down.

In practice, more and more “AI actions” are really distributed episodes. They run across multiple models, retrieval layers, external tools, runtime policies, approval chains, memory stores, scoped permissions, and sometimes more than one human. The thing that matters is no longer just the final output. What matters is the path by which the system got there, what it could see, what it was allowed to do, what was denied or narrowed along the way, and what ultimately counted as the action.

That shift changes the governance problem.

The question is no longer just: Did the model answer correctly?

It is also: If someone has to review this later, will they still be able to say what happened?

Not in the vague sense of “there are logs somewhere.” I mean in the harder sense: can a later reviewer say what action occurred, under what authority, with what evidence in scope, through what runtime path, and with what institutional consequence?

That is the threshold I keep coming back to. It is the difference between a system that is merely impressive and one that is actually governable.

And I think we are moving toward a failure mode that current AI discussion still understates: the system can preserve a record of what happened while losing the ability to read that record as one coherent action.

That is the line of thought behind several of the papers I’ve been working on recently. The shortest version is this:

The next major AI governance problem is not just alignment, hallucination, or prompt injection. It is legibility.

The market is already telling us the unit of action has changed.

One reason I’m confident this is not just a philosophical worry is that the industry itself has started changing its language in a way that quietly admits the old model is no longer enough.

OpenAI’s April 2026 update to the Agents SDK was not framed as “here is a better chatbot.” It was framed around a model-native harness, native sandbox execution, tools, files, and execution boundaries.[1] That is a very different picture of what the system is. It is not just a model API anymore; it is a runtime with a controlled workspace and a managed action surface.

Around the same time, Snowflake explicitly started talking about the “agentic enterprise” in terms of a control plane: not just models and apps, but a coordinating layer that connects intelligence, enterprise data, policy, and execution.[2] ServiceNow made a closely related move, first emphasizing an enterprise context layer and then, in May 2026, announcing Action Fabric and an MCP server designed to let AI agents execute governed enterprise work with identity verification, permission scoping, and audit trails.[3][4]

That matters because it tells us something important: even the vendors trying to commercialize this future no longer think the relevant object is a model plus an answer. They think the relevant object is a system that translates intelligence into action across enterprise boundaries.

Once you admit that, the governance problem stops being a question about outputs alone.

It becomes a question about whether the system preserves enough structure that later humans can still understand what the system did as a system.

Leave a comment

Why partial visibility matters more than people think:

I’ve become convinced that one of the places current AI evaluation is weakest is that it still tends to test systems as though the main question were whether they can answer correctly under idealized evidence conditions.

That is not how consequential systems fail in institutions.

They fail under partial visibility.

They fail when the evidence relevant to a decision is partitioned by role, timing, authorization, workflow state, or operational scope. They fail when some documents are in scope and some aren’t, when one step can see something another step cannot, when a system’s answer is locally accurate but globally misleading because its field of view was structurally incomplete.

This is the problem I tried to isolate in Partial Evidence Bench.[5] The benchmark is meant to measure a failure mode that I think is going to become much more important as agents move deeper into enterprise workflows: the system gives an answer that sounds complete even though material evidence lies outside what the caller was allowed to access.

That is not quite the same thing as hallucination. It is not even exactly the same thing as ordinary uncertainty. In many of these cases, the system is not just unsure. It is not in a position to know in a way that the user can safely interpret.

That distinction matters. A system that says, “I don’t know,” is one kind of problem. A system that gives a polished answer while silently omitting the fact that it could not see a material part of the evidentiary field is another.

What makes this especially difficult is that, in many enterprises, access control is working exactly as designed. The failure is not “the system broke permissions.” The failure is that the institution still needs to act on an answer whose completeness depends on boundaries that may no longer be visible in the final output.

Once you start looking at the problem this way, you realize that evaluation has to ask more than “was the answer accurate?” It also has to ask something like: accurate relative to what visible world?

Authorization problems do not end at the permission check…

There is a closely related issue on the authority side.

One of the most persistent mistakes in current agentic AI thinking is the assumption that if local access controls are correct, then the larger workflow is probably safe.

I don’t think that follows.

An agent acting on behalf of a person or team may retrieve data under one scope, delegate work under another, combine results across boundaries, and return something that still sounds like a unified answer. Each step may be individually permitted. The overall episode may still be institutionally wrong, or at least institutionally misleading.

That is the core problem I tried to formalize in Authorization Propagation in Multi-Agent AI Systems.[6]

The security conversation around agentic AI has spent a lot of time on prompt injection, and rightly so. But prompt injection is only one piece of the picture. The deeper systems problem is that identity and authority do not automatically remain coherent once action is distributed across agents, tools, services, and policy boundaries.

Who was the system acting for at each step? What role or delegation path governed retrieval? What evidence became newly actionable only after aggregation? What was allowed at one point in the chain but not another? Those are not edge questions. They are what naturally appear once “one agent” is really a workflow of components with uneven visibility and uneven authority.

This is also one reason I find the usual consumer framing of AI safety inadequate for enterprise settings. In consumer contexts, the failure is often framed as “the assistant did something dumb or dangerous.” In institutions, the harder cases are often subtler. The system may do something locally sensible and globally wrong.

This is why I no longer think “more logging” is the answer.

At this point, the natural response is obvious: fine, then log everything.

But I don’t think more logging, by itself, gets you where you need to go.

The more I’ve thought about this, the more convinced I’ve become that more traces are not the same thing as more legibility.

This is the thought I’ve been trying to sharpen in a paper I’ve been calling The Khipu Problem.[7] The analogy comes from the khipu: structured records that survive after the reading practice needed to interpret them has decayed. The cords remain. The institution no longer knows how to read them as a live system of meaning.

That feels uncomfortably relevant to where agentic AI is heading.

We may preserve prompts, outputs, model versions, tool calls, runtime traces, access logs, approval events, and state transitions and still fail to preserve the conditions needed to interpret them as one coherent action. The artifacts survive. The reading practice dies.

That is why I increasingly think the next governance substrate is not just observability. It is something closer to interpretive infrastructure.

By that I mean infrastructure that preserves things like:

• stable episode identity

• requester truth

• effective authority

• policy boundary

• evidential scope

• runtime continuity

• approvals, denials, and escalations

• final named outcomes

Without that, later review starts to resemble archaeology. You can still dig through the traces. You may even recover most of what happened. But you no longer have a stable object of institutional judgment.

Share

Recent policy moves suggest institutions are starting to notice →

One reason I think this deserves more attention now is that the governance institutions themselves are beginning to move in roughly this direction, even if they do not always use the same vocabulary.

In January 2026, NIST’s Center for AI Standards and Innovation issued a request for information explicitly focused on the secure development and deployment of AI agent systems, including threats that arise when model outputs are combined with software functionality.[8] In February, NIST announced an AI Agent Standards Initiative aimed at interoperability, security, and trusted adoption.[9] Around the same time, the NCCoE put out a concept paper on identity and authority for software agents, with a specific focus on agentic AI.[10]

That sequence matters. It suggests that the government is no longer treating “AI agent systems” as just a colorful extension of chatbots. It is treating them as a systems-and-standards problem.

In March, GSA and NIST announced a joint effort to improve how the federal government evaluates AI models and services before they are used in real workflows.[11] Again, that may sound procedural, but I think it points to the same underlying shift: evaluation is moving closer to deployment context, not just model performance in isolation.

Share AIwithKT

Outside the U.S., the UK government’s March 2026 guidance on consumer law and AI agents made the institutional responsibility point unusually explicit: if an AI agent misleads customers or acts unlawfully, the business remains responsible in much the same way it would for a human employee, including when the agent is supplied by a third party.[12] That is a useful corrective to the fiction that agentic behavior somehow sits outside ordinary accountability because it is mediated by software.

And in April, a Cloud Security Alliance survey found that 82% of responding enterprises had previously unknown AI agents in their environments, while 65% reported at least one AI-agent-related incident in the prior year.[13] That survey needs to be read with the usual caution that comes with industry-sponsored reports. But even with that caveat, the numbers are striking because they point to the same basic issue: organizations are losing track of what is running, under what identity, and with what permissions.

Put differently: the vocabulary is still settling, but the institutional intuition is already changing. People are starting to see that the relevant problem is not only model quality. It is also whether action remains attributable, scoped, reviewable, and contestable.

Share AIwithKT

Governance starts BEFORE inference.

This is also why I’ve become more interested in what used to seem like narrow backend primitives.

Take the idea of an execution envelope.[14]

On the surface, this sounds like plumbing: a shared admission object that records who is asking for what kind of execution, what resources they requested, what scope and governance hints accompanied the request, and what the backend ultimately granted.

But that plumbing matters because it preserves the difference between:

• what was asked for

• what was allowed

• what was granted

• what ultimately happened

That difference is not decorative. It is governance substance.

Once you lose it, later review has to reconstruct intent and constraint from scattered artifacts. Once you preserve it, later review can begin from a bounded object rather than from a forensic guess.

This is why I’ve started to think that governance begins much earlier than most AI conversations assume. It begins at admission boundaries, scope boundaries, authority boundaries, and evidential boundaries. If you wait until the final output to ask whether a system was governable, you are already asking too late.

The harder problem? The inspectable layer and the causal layer drift apart.

There is one more layer to this that I think is deeper than the practical governance literature usually acknowledges.

Institutions inspect what is stable and representable: logs, weights, prompts, outputs, explicit records. Those are the things they know how to file, review, export, compare, and audit.

But what actually drives behavioral change is often more relational and dynamic. Salience shifts. Memory interactions. Attention reweighting. The way one conversation changes how the next one is read. The way one piece of context silently changes which other pieces of context become decisive.

This is where the governance problem stops being just a tooling problem.

Better logging does not automatically solve a structural mismatch between the layer we can inspect and the layer at which consequential change is actually occurring.

That is part of what I’m trying to formalize in Layered Mutability.[15] The point there is not merely that agents change. Of course they change. The more interesting point is that the layers most responsible for behavioral drift are not always the layers humans can most easily inspect.

A system can look stable at the surface while evolving meaningfully in the conditions that shape future action. And that matters because a lot of current governance thinking still assumes that more transparency at the visible layer is equivalent to more control over the causal layer.

It often isn’t.

What I think matters now:

If I had to compress all of this into one claim, it would be this:

The hardest near-term problem in agentic AI is not getting systems to act. It is preserving enough structure that their action remains legible after it matters.

That means I’m less interested, at least right now, in abstract arguments about whether AI is generally “aligned” than I am in much grubbier questions:

• What was the system able to see?

• Who was it acting for?

• What changed between the request and the action?

• What parts of the episode remain reconstructable later?

• Which layers of change matter most, and which ones can institutions actually inspect?

These are less glamorous questions than frontier benchmarks and capability demos. They are also, I think, more important for the world we are actually building.

Because if the record survives but the reading dies, oversight becomes theatrical. The institution still has data. It no longer has a stable object of judgment.

And if that sounds abstract, I suspect it will only sound abstract for a little while longer.

Innovating with integrity,
@AIwithKT 🤖🧠

Notes and sources -

1. OpenAI, “The next evolution of the Agents SDK,” April 2026: https://openai.com/index/the-next-evolution-of-the-agents-sdk/

2. Snowflake, “Powering the Era of the Agentic Enterprise,” March 18, 2026: https://www.snowflake.com/en/blog/agentic-enterprise-control-plane/

3. ServiceNow, “ServiceNow moves beyond the sidecar AI era,” April 9, 2026: https://newsroom.servicenow.com/press-releases/details/2026/ServiceNow-moves-beyond-the-sidecar-AI-era-giving-customers-a-complete-AI-native-experience-across-all-products-and-packages/default.aspx

4. ServiceNow, “ServiceNow opens its full system of action to every AI Agent in the enterprise,” May 2026: https://newsroom.servicenow.com/press-releases/details/2026/ServiceNow-opens-its-full-system-of-action-to-every-AI-Agent-in-the-enterprise/default.aspx

5. Partial Evidence Bench: Benchmarking Authorization-Limited Evidence in Agentic Systems: https://arxiv.org/abs/2605.05379

6. Authorization Propagation in Multi-Agent AI Systems: Identity Governance as Infrastructure: https://arxiv.org/abs/2605.05440

7. The Khipu Problem: Institutional Legibility Under Distributed Cognition (forthcoming)

8. NIST CAISI, “Request for Information About Securing AI Agent Systems,” January 12, 2026: https://www.nist.gov/news-events/news/2026/01/caisi-issues-request-information-about-securing-ai-agent-systems

9. NIST, “AI Agent Standards Initiative,” February 17, 2026: https://www.nist.gov/news-events/news/2026/02/announcing-ai-agent-standards-initiative-interoperable-and-secure

10. NCCoE, “New Concept Paper on Identity and Authority of Software Agents,” February 5, 2026: https://www.nccoe.nist.gov/news-insights/new-concept-paper-identity-and-authority-software-agents

11. GSA and NIST, “GSA and NIST Partner to Boost AI Evaluation Science in Federal Procurement,” March 18, 2026: https://www.gsa.gov/about-gsa/newsroom/news-releases/gsa-and-nist-partner-to-boost-ai-evaluation-science-in-federal-procurement-03182026

12. UK Government, “Complying with consumer law when using AI agents,” March 9, 2026: https://www.gov.uk/government/publications/complying-with-consumer-law-when-using-ai-agents/complying-with-consumer-law-when-using-ai-agents

13. Cloud Security Alliance, “82% of Enterprises Have Unknown AI Agents in Their Environments,” April 21, 2026: https://cloudsecurityalliance.org/press-releases/2026/04/21/new-cloud-security-alliance-survey-reveals-82-of-enterprises-have-unknown-ai-agents-in-their-environments

14. Execution Envelopes: A Shared Admission Contract for Backend AI Execution Requests (forthcoming)

15. Layered Mutability: Continuity and Governance in Persistent Self-Modifying Agents: https://arxiv.org/abs/2604.14717

When most people say AI security, they still mean model security: jailbreaks, prompt injection, unsafe outputs, evals, guardrails, red-teaming. That work matters. It is the layer that tells us whether a model can be manipulated, misled, or pushed outside its intended behavior.

But that framing is starting to lag the systems we are actually deploying.

In real enterprises, the model is rarely the whole product. It is attached to retrieval, memory, identity, tools, APIs, and workflows. It can read internal documents, call external services, open tickets, update records, move data, and trigger downstream actions. At that point, the risk is no longer confined to what the model says. The risk is what the surrounding system allows the model to do.

That is the shift I think the market still underestimates. We are still spending too much time asking whether the model can generate a bad response, and not enough time asking whether the system can turn that response into a bad action.

And in enterprise environments, the second question is usually the more consequential one. A flawed answer in a chat window is embarrassing. A flawed answer that is allowed to retrieve sensitive data, approve access, change a payment record, or invoke an internal tool is a control failure.

So yes, model security still matters. But it is no longer the center of gravity.

The harder problem, and the one that will matter more inside real enterprises, is not what the model says.

Subscribe now

That distinction sounds subtle. It is not.

Imagine a procurement agent inside a large enterprise. It reads vendor emails, retrieves contracts from SharePoint, looks up vendor records in the ERP, and submits change requests through an internal workflow tool. On paper, it sounds low-risk because it is framed as automation rather than autonomy. It is not “making strategic decisions.” It is just helping operations move faster.

That is exactly why it is dangerous.

Now imagine the attacker never steals a password, never drops malware, and never exploits the ERP directly. Instead, they do something much simpler: they place malicious context in the path of the agent’s normal work. A crafted invoice email. A poisoned document. A falsified note in a shared repository. The agent retrieves that material as part of its routine workflow, interprets it as legitimate input, decides a vendor payment record needs to be updated, calls the internal API with its standing credentials, and drafts a justification that looks completely ordinary.

If that incident happens, the postmortem will probably say prompt injection.

That will be true, but incomplete.

The deeper failure is not just that the model was manipulated. It is that the system allowed manipulated reasoning to become trusted action. The agent had too much authority, too little provenance, and no meaningful policy boundary between reading, reasoning, and acting. The model was only one part of the chain. The real security failure was delegated action operating under weak control.

That is why I think the real AI security problem is being misframed.

Once an AI system can search internal knowledge, retrieve sensitive data, call tools, update systems, or trigger workflows, the primary question stops being: can this model produce something unsafe?

It instead becomes:

Who is this agent, whose authority is it using, what data is it touching, what action is it attempting, and should that action be allowed right now?

That is a different security architecture.

And I think the most serious work in the field is already drifting in that direction, even if the market language has not fully caught up.

NIST launched its AI Agent Standards Initiative on February 17, 2026, with an explicit focus on secure interoperability and agent security. Earlier that month, NIST’s NCCoE published a concept paper on software and AI agent identity and authorization. OWASP’s Top 10 for Agentic Applications, published on December 9, 2025, likewise signals that the risk conversation is moving past chat-level safety into tools, memory, delegation, orchestration, and unintended actions across systems.

That shift matters.

The field is slowly admitting that AI security is not just about the model. It is about the control surface around the model.

That is why the protocol layer matters. MCP and A2A are not interesting because they are standards. They are interesting because standards create stable interfaces, and stable interfaces create stable control points.

Once agents can reliably discover tools, exchange context, delegate work, and act across boundaries, the systems mediating those interactions become part of the security perimeter. Gateways, registries, delegation frameworks, identity layers, and policy engines are where authorization, provenance, and audit either become enforceable or collapse into theater.

This is also why I think the UADP conversation is directionally right but not yet deep enough. Yes, the major categories are converging. But the real story is not category bundling. It is the emergence of a single decision layer that must evaluate identity, data, and execution together before autonomous action is allowed.

I think the deeper truth is that the enterprise now needs to make one decision across three kinds of context at the same time:

• identity context

• data context

• execution context

That is the real control plane.

Not a dashboard. Not a model firewall. Not generic “AI governance.”

A real-time decision layer that can answer whether this agent should be allowed to take this action on this system with this authority under these conditions.

That is why I suspect the next great AI security company may not present itself as an AI security company at all.

It may emerge from identity, data security, secure access, or cloud runtime. It may come from a company the market still misclassifies because it is solving the real control problem rather than the most fashionable one.

The winning architecture here is not the one with the best prompt filter.

The obvious counterargument is that model-layer attacks still matter, and they do. If you cannot detect prompt injection, bound unsafe behavior, or monitor model misuse, you are already exposed. I am not arguing that model security is unimportant.

I am arguing that it is insufficient.

In enterprise settings, the highest-consequence failures increasingly happen when model output crosses a trust boundary and becomes action. A bad response in a chat window is annoying. A manipulated agent that can retrieve customer data, rotate credentials, approve access, move money, or modify production systems is a control failure. The answer is not to stop caring about the model. It is to stop pretending the model is the whole system.

Leave a comment

For me, serious enterprise agent security now requires at least this much:

• a durable identity for each agent

• an accountable human or service owner

• short-lived, scoped, revocable authority

• policy enforcement at the tool and protocol boundary

• data-aware authorization, not just app-aware authorization

• provenance for what the agent saw, chose, and did

That last one matters more than most people appreciate.

In classic enterprise software, logs can often tell you that an API was called by a service principal at a certain time. In agentic systems, that is nowhere near enough. You need to know what context the agent retrieved, which tool path it selected, what policy was evaluated, what data classes were implicated, whether the action was delegated or autonomous, and why the system allowed it. If we do not build that layer properly, post-incident analysis in agentic environments will be weak, ambiguous, and in many cases almost useless.

That is why I keep coming back to the same conclusion:

Not authorization in the narrow IAM sense.

Authorization in the systems sense.

That is the architecture I would bet on.

Share AIwithKT

Anthropic’s Project Glasswing reinforces the urgency. As frontier systems become more capable at real security work, both offensive and defensive, the question of what an autonomous system is authorized to do becomes the main event. Capability is rising faster than control maturity, and that gap is where the next generation of security failures will happen.

That is why I do not think the future belongs to the company with the most impressive AI security demo, the cleanest guardrail interface, or the best prompt-filter benchmark.

I think it belongs to the company that can turn autonomous action back into governed action. The company that can make identity, authority, data sensitivity, tool access, and runtime context legible enough to evaluate in real time. The company that can enforce policy before an agent acts, not just explain what went wrong after it does.

That is the control plane that matters.

A machine that can reason is impressive. A machine that can reason within constraints is useful. A machine that cannot outrun its permissions is enterprise-ready.

That is the bar.

And over the next few years, I think the most important companies in this space will be the ones that understand that the real problem is not model intelligence in isolation. It is controlled agency.

Signals behind this view:

• SACR: The Rise of UADP

• NIST: AI Agent Standards Initiative

• NIST NCCoE: Software and AI Agent Identity and Authorization concept paper

• OWASP: Top 10 for Agentic Applications for 2026

• Anthropic: Donating MCP and establishing the Agentic AI Foundation

• Google: Announcing the Agent2Agent Protocol

• Anthropic: Project Glasswing

Innovating with integrity,
@AIwithKT 🤖🧠

Subscribe now

Everyone wants to know whether agents can be trusted.

I think that is already the wrong question.

In agentic AI, the real security boundary is not the model. It is the chain of authority from sponsor to answer.

For the last year, most of the conversation around agent security has treated the model as the main character. Can it be jailbroken? Can it be prompt-injected? Can it be manipulated into using the wrong tool? Can better evals catch dangerous behavior before it reaches production?

Those are important questions.

They are just no longer the deepest ones.

Share

A request does not begin with a model. It begins with a person, a service, or an upstream agent acting on someone’s behalf. Authority is delegated into an application or agent. That agent then moves through a system. It touches data. It touches metadata. It traverses embeddings and ontologies. It invokes tools. It selects models, providers, nodes, and regions. It may even delegate onward to other agents. Then, eventually, it returns an answer that often contains more meaning than any one source it touched along the way.

Once you see the flow clearly, the security boundary changes shape.

The real question is not just, “Did the model do something unsafe?”

The real question is, “Who authorized this chain of actions, under what constraints, across which boundaries, and was the final output still within that authority?”

That is a more serious problem than behavioral testing alone can solve.

You can already see the ecosystem moving in this direction. On February 17, 2026, NIST announced its AI Agent Standards Initiative. NIST is also actively soliciting feedback on a concept paper on AI agent identity and authorization. And on March 9, 2026, OpenAI announced plans to acquire Promptfoo, a strong signal that testing and red-teaming are becoming more deeply integrated into the platform layer.

This is healthy. It means the ecosystem is growing up. But it also tells us something else: evals are becoming baseline infrastructure, while the harder question of authority remains unresolved.

Leave a comment

Evals tell you what an agent can do. Authorization tells you what an agent may do. Delegation lineage tells you who is accountable when it does it.

Those are not the same thing, and the market still blurs them together far too often.

Take a simple enterprise example. An employee asks an agent to prepare a diligence memo. The agent searches internal knowledge, consults metadata, traverses embeddings, calls a model, invokes a tool, maybe routes part of the task elsewhere, and returns a polished answer.

If that answer includes restricted meaning drawn from material the employee was never entitled to see, the system has failed.

It has failed even if the prompts looked clean. It has failed even if the model passed safety tests. It has failed even if the tool invocation looked perfectly legitimate.

Why?

Because the break was not merely behavioral. The break was in the chain of authority.

Somewhere between sponsor and answer, the system lost control of what was actually authorized.

This is where point solutions start to reveal their limits.

Share AIwithKT

If you secure identity but not governed knowledge assets, meaning can still leak through metadata, embeddings, ontologies, and summaries. If you secure data but not live delegation, stale agents and broad services can continue acting after the human sponsor’s rights change. If you secure model access but not the full execution path, the right user can still reach the wrong provider, the wrong region, the wrong node, the wrong tool surface, or the wrong retention boundary.

A great deal of what gets sold as “agent security” is really local risk reduction at one layer of the stack.

That is useful.

It is not the same thing as governing the full path from sponsor to answer.

And I think that distinction is about to matter much more.

The unit of security in agentic AI is not the prompt. It is not the model. It is not even the tool call by itself.

It is the chain.

Who sponsored the action? What was actually delegated? What data and derived assets were in scope? Which tools, models, nodes, providers, and locations were allowed to participate? What transformations were permitted? And when the answer came back, was it still within the recipient’s entitlement?

Share AIwithKT

Those are the questions real production systems have to answer.

They are also the questions most systems still struggle to answer coherently.

None of this is an argument against evals. I am very much in favor of evals. We need them. We need better ways to characterize failure, test tool-use boundaries, surface prompt injection, and understand how agentic systems break under pressure.

But evals do not prove that a specific action was authorized by a valid sponsor.

They do not prove that an agent stayed within delegated scope as it moved across data, tools, models, and other agents.

And they do not prove that the final answer was minimized to what the recipient was actually allowed to know.

That is the missing layer.

Connection is getting standardized faster than consent.

Subscribe now

We are getting much better at wiring agents into tools, protocols, and runtimes than we are at proving that every handoff in that chain remained within authorized scope.

That is why I think the next real divide in this market will not simply be about who has the best model, the best protocol, or the strongest eval suite.

It will be about who owns the control plane of delegated authority.

The systems that can govern the full path from sponsor to answer will become real enterprise infrastructure.

The systems that cannot will remain partially governed automation: impressive, useful, and fundamentally unstable.

So when we talk about agent security, I think we should stop asking only whether the model can be trusted.

We should ask whether the chain of authority can be proven.

Because in the agent era, the hardest question will not be whether a model could produce the answer.

It will be who gave that chain permission to do so.

Subscribe now

Share

Share AIwithKT

Leave a comment

References

• NIST announcement, February 17, 2026: https://www.nist.gov/news-events/news/2026/02/announcing-ai-agent-standards-initiative-interoperable-and-secure

• NIST AI Agent Standards Initiative: https://www.nist.gov/caisi/ai-agent-standards-initiative

• NCCoE concept paper on AI agent identity and authorization: https://csrc.nist.gov/pubs/other/2026/02/05/accelerating-the-adoption-of-software-and-ai-agent/ipd

• OpenAI announcement on acquiring Promptfoo, March 9, 2026: https://openai.com/index/openai-to-acquire-promptfoo/

Innovating with integrity,
@AIwithKT 🤖🧠

When a Model Becomes a Dataset Generator

After I posted on LinkedIn about “behavior theft” (not weight theft), a few engineers and researchers reached out to ask what I meant. This is the longer answer.

Most people still imagine “model theft” as a clean, cinematic event: someone gets access to weights, leaks a file, and the rest looks like a familiar security incident.

Anthropic is describing a different shape of problem. In their account, a large volume of Claude usage was attributed to coordinated distillation campaigns involving DeepSeek, Moonshot (Kimi), and MiniMax, spanning more than 16 million exchanges and tens of thousands of accounts. The headline number is attention-grabbing, but the more important point is what it implies about the modern frontier model business.

If you expose high-signal outputs through an API, you are not only shipping a product. You are operating a system that can generate training-grade artifacts at scale.

That changes the security and competitive landscape.

This Isn’t About “Content”

The internet is full of content. Most of it is cheap to replicate and rarely decisive.

What matters here is behavior—especially behavior that can be turned into reusable training signal.

A frontier model doesn’t just answer questions. In many deployments, it produces artifacts that look eerily close to the ingredients of modern training:

Patterns for how to structure reasoning on hard tasks
Judgment signals (what “good” looks like, what “better” looks like)
Agentic behavior (planning, tool use, orchestration across steps)
Steerability around sensitive requests

That’s the difference between copying outputs and harvesting something more valuable: the model’s learned process.

The Asset Has a Name: Teacher Signal

The simplest way to describe what’s at stake is “teacher signal.”

Teacher signal is any output that is disproportionately useful as supervision for improving another system. It’s not just answers. It’s the scaffolding and evaluative structure wrapped around answers.

That’s why disputes about “copying” miss the point. In this regime, access is not only consumption. At scale, access can become extraction: turning a model’s behavior into a dataset that trains a competitor.

Share AIwithKT

A Paradox Leaders Need to Name

As models become more reliable and more agentic, their behavior becomes cleaner. Cleaner behavior is easier to learn from.

This is not an argument against quality. It’s a reminder that “useful” and “distillable” can move together. The more structured and repeatable a model’s workflows are, the more they resemble reusable training artifacts.

So the risk isn’t theoretical. It’s a property of what we are building.

Share

Why This Becomes an Executive-Level Issue

If you treat this as a niche technical controversy, you miss the strategic shape of it.

First, model releases start to have a half-life. The moment a new capability is accessible, it can begin to propagate—sometimes faster than the safeguards, policies, and norms you hoped would travel with it.

Second, competitive advantage starts looking less like a moat and more like lead time. The question becomes: how long does a differentiated behavior remain differentiated once exposed to the world at scale?

Third, safety becomes a supply-chain problem. Capabilities can move without weights, and there’s no guarantee the safety posture moves with them.

The Practical Implication (Without the Playbook)

The response is not “panic” or “lock everything down.” It’s to get more deliberate about what you export.

In plain terms: treat high-signal outputs as privileged. Decide which behaviors you are comfortable making easy to copy, and which behaviors should require a higher bar of trust, governance, and oversight.

That is a product decision, a security decision, and—whether we like it or not—in some ways, a geopolitical decision.

Leave a comment

The Real Question

We can debate data ethics, provenance, and who has standing to complain. Those debates matter.

But they don’t change the operational terrain. In the frontier model era, access and capability propagation are tightly coupled.

So the leadership question becomes simple and uncomfortable:

Are you offering a service, or are you operating a system that can be turned into someone else’s training pipeline?

If you’re building frontier models—or deploying high-value models inside an enterprise—this is not a footnote. It’s part of the business model now.

Subscribe now

Share

Share AIwithKT

Leave a comment

Innovating with integrity,
@AIwithKT 🤖🧠

My book on AI Security is OUT NOW!

Subscribe now

After years of writing about kernel calls and agentic stacks, I find myself unexpectedly moved by the news cycle. A lobster‑themed personal AI assistant built by a solo developer went viral, amassing tens of thousands of GitHub stars and even moving markets. The project, originally named Clawdbot and later rebranded as Moltbot after a trademark spat, promised to “actually do things”—book your flights, clean your inbox and manage your calendar. It felt like a preview of the agentic future we’ve been engineering.

Share

But alongside the hype came cautionary tales: the chaotic rename triggered handle‑sniping and scam attempts, with experts warning that a misconfigured Moltbot could be hijacked by a malicious message to execute commands on your machine. Watching the world both celebrate and fear a crustacean‑mascot bot, I realized the discussion had outgrown debugging tutorials. This piece is my attempt to grapple with the broader questions that these headlines raise—about trust, control and the social fabric of AI.

Leave a comment

Artificial intelligence isn’t just another tool; it learns and acts on our behalf. History shows that every major technology—from plows to printing presses—reshapes how societies function. AI is poised to do the same. The question for security leaders isn’t whether AI will be adopted; it’s whether our institutions can adapt quickly enough to preserve trust. In a world where 88 % of employees use AI tools but only 5 % do so within sanctioned boundaries, unsanctioned AI isn’t a fringe risk—it’s a fundamental challenge. This essay steps back from the day‑to‑day technical battle and asks: as AI leaves the lab and enters every workflow, how do we safeguard both our data and our values? Drawing on lessons from technical work on guard seams, provenance, autonomy and orchestrated defenses, it explores how societies can thrive alongside intelligent, autonomous systems.

Share AIwithKT

The CISO’s evolving role

For today’s Chief Information Security Officer (CISO), that reality completely redefines the job. It’s no longer enough to patch systems and monitor compliance; modern CISOs must anticipate how intelligent systems will reshape business, risk and geopolitics. Recent predictions illustrate what this means. As AI embeds itself in daily operations, security leaders must scale governance models far faster than before and treat AI agents as identities—monitoring their access, decision‑making and behavior just as rigorously as human users. United Airlines CISO Deneen DeFiore frames 2026 security as operational resilience at scale, not perimeter defense. Attackers will increasingly use generative AI for reconnaissance and exploit chaining, while defenders rely on AI to prioritize risk and accelerate detection; the difference maker, she notes, will be how well organizations govern, tune and trust their AI. Noel Toal adds that AI risk frameworks will reach the boardroom, giving directors the structure to ask tougher questions and unlock security funding. In other words, 2026 is the year when autonomy must become accountable.

Guard seams and identity‑first defense

Industry guidance aligns with the view that defense must live at the seams—where AI interfaces with data, users and software. Okta’s threat matrix contrasts traditional AI risks (data poisoning, API abuse) with the complexities of agentic systems—dynamic memory corruption, cross‑system attacks and multi‑agent identity spoofing. The remedy is identity‑first: treat every AI agent as a privileged, non‑human identity, enforce least privilege and continuously monitor behavior. DeFiore and others emphasize shifting investment from perimeter controls to segmentation, recovery testing and third‑party resilience. The new control surfaces are identity, provenance and context.

From guardrails to guarantees

Simple prompt guardrails are insufficient. Fortinet warns that AI models suffer from opaque “black box” decisions and a suite of adversarial attacks—from data poisoning to prompt injection. Weak identity and authentication can lead to cascading vulnerabilities. To counter this, they call for verifiable behavior, signed logs and human‑in‑the‑loop checkpoints—mirroring our work on s1, GuardReasoner and critique fine‑tuning. DeFiore expects AI‑driven orchestration inside security operations centers (SOCs), where automated response isolates compromised endpoints and blocks malicious IPs in seconds. The goal is embedding accountability into the AI lifecycle.

Provenance as the new perimeter

Anthropic’s study showing that just 250 malicious documents can backdoor large language models underscores the critical role of provenance. Fortinet echoes this, noting that model inversion, extraction and data poisoning are real threats. As a result, CISOs increasingly treat provenance—verifiable pipelines, signed inputs and cryptographic attestations for data sources—as the new perimeter. Our posts on data lineage, metadata icons and supply‑chain security align perfectly with this shift.

SOC transformation at machine speed

AI‑enabled attacks rose 47 % globally in 2025 and will continue to scale faster than defenses. SOCs must operate at machine speed—detecting, validating and mitigating threats without waiting for manual intervention. CISOs must embed automation into investigation workflows, correlate signals across domains and use agentic AI to isolate compromised endpoints and roll back ransomware. Resilience becomes the metric: leaders are judged on how quickly they can absorb and recover from AI‑enabled attacks.

Share AIwithKT

Skill gaps and AI literacy

Technology alone won’t close the gap. Most employees now rely on AI tools, but only a small fraction use them within sanctioned boundaries; the cybersecurity workforce gap is expected to exceed 4.8 million open roles, and 90 % of security programs aren’t mature enough to protect against AI‑driven threats. AI literacy becomes a priority: teams must learn to craft secure prompts, audit agent behavior and question AI decisions. Our emphasis on trust dovetails with this; social‑engineering attacks using deepfakes and AI‑generated phishing have drastically increased click‑through rates. Building a culture where employees validate AI outputs and recognize manipulation is as important as technical safeguards.

Leave a comment

Geopolitical awareness

A CISO’s vision must extend beyond the enterprise. Nation‑state cyber activity is escalating; attacks on critical infrastructure and cable sabotage are expected to continue. Adversaries are using AI to destabilize societies and undermine democratic institutions. Visionary CISOs factor geopolitical instability into their threat models and advocate for global standards similar to the Financial Action Task Force. They also recognize that data sovereignty will reshape cloud strategy: organizations will adopt multi‑cloud sovereignty strategies—distributing data across trusted jurisdictions and repatriating sensitive workloads—to mitigate political and legal risks.

Beyond the horizon: emerging visionary themes

The above foundations set the stage for deeper, forward‑looking insights.

Dynamic trust and continuous exposure management

Periodic assessments are giving way to dynamic trust models powered by continuous exposure management. Fujitsu predicts that AI governance will become a board‑level imperative, with boards demanding real‑time visibility into models, data flows and decision‑making. AI governance‑as‑a‑service platforms will emerge to enforce policy and monitor behaviour in line with mandates like the EU AI Act. Continuous risk‑scoring will transform security from a reactive task to a strategic value‑protection function. Visionary CISOs might consider investing in real‑time risk dashboards, treat AI models as living entities whose risk posture changes minute by minute and use dynamic trust to automate access decisions.

Subscribe now

Quantum‑ready cryptography and zero trust

Quantum computing threatens to break today’s encryption. Experts advise organizations to inventory their cryptographic assets, develop crypto agility to swap algorithms as standards evolve, and work with vendors to adopt quantum‑resistant primitives. Simultaneously, the once‑theoretical zero‑trust model is becoming practical at scale: open security solutions, identity‑based controls, continuous authentication and micro‑segmentation make it feasible to assume breach by default and verify every action. Coupling quantum‑readiness with zero trust will be key to safeguarding AI‑enabled enterprises in the late 2020s.

Share

AI‑enabled collective defense and human ingenuity

The AI battlefield will be symmetric: adversaries and defenders wield comparable models. Experts foresee collective defense strategies evolving from simple information sharing to fully automated, AI‑enabled collaboration. Government agencies and critical‑infrastructure teams will use AI to turn shared threat intelligence into immediate action, relieving analysts and strengthening the public‑sector cyber ecosystem. Yet the winning edge won’t come from the smartest code alone; creative problem‑solving and ethical hacking communities will flourish. Bug‑bounty programs will accelerate disclosure, and ethical hackers will shape security standards “from the bottom up”. This may require CISOs championing cross‑sector AI collaboration while cultivating internal creativity and rewarding ethical hacking.

Share

Unified governance and supply‑chain visibility

More than half of organizations still lack centralised governance for AI adoption, creating silos and blind spots. At the same time, 97 % use AI coding assistants and 81 % of security leaders don’t know where AI‑generated code lives. This gap invites supply‑chain compromises. Treating AI configuration and generated code as part of the supply chain, and implementing unified visibility across the development lifecycle, are now essential. It will be critical to push for centralized AI governance bodies, integrated code provenance systems and supply‑chain attestations to prevent tomorrow’s AI‑borne breaches.

Sector‑specific governance and cross‑industry collaboration

The healthcare sector’s AI Cybersecurity Task Group offers a model for phased guidance. Its roadmap covers education and enablement, cyber operations and defense, governance, secure‑by‑design principles and supply‑chain transparency. By emphasising AI literacy, playbooks for incident response, governance frameworks, secure‑by‑design embedded in medical devices and transparency for third‑party AI, the health sector demonstrates how sector‑specific frameworks and cross‑industry collaboration can accelerate AI safety while preserving innovation.

Synthesis: leading in the age of autonomous systems

To be a forward-thinking Agentic AI Security CISO means embracing a holistic set of principles, including:

1. AI agents are identities. Govern, monitor and audit them like human users; define clear scopes and enforce least‑privilege access. Identity is the new seam.

2. Provenance and auditability are the new perimeters. Signed inputs, cryptographic attestations and verifiable pipelines will be as crucial as firewalls.

3. Governance and resilience trump traditional perimeter defence. Boards and regulators will demand evidence of AI risk management and proof of continuous resilience. Dynamic trust models and governance‑as‑a‑service will become indispensable.

4. Machine‑speed defense must replace human‑paced workflows. SOCs will automate detection and remediation, using agentic AI to isolate threats and orchestrate response.

5. AI literacy and culture are security controls. Training people to understand and question AI is as vital as patching code. Ethical hacking and creative problem‑solving will define the winners.

6. Geopolitics and data sovereignty matter. Cyber warfare and disinformation will spill into corporate risk; CISOs must monitor global developments, advocate for international standards and adopt multi‑cloud sovereignty strategies.

7. Quantum‑readiness and zero trust are non‑negotiable. Inventory cryptographic assets, build crypto agility and roll out zero‑trust architectures.

8. Collective defense and unified governance are the future. Cross‑sector AI collaboration, governance‑as‑a‑service and supply‑chain transparency will define resilience.

By weaving these principles into enterprises, we can begin to connect the micro‑level engineering of agentic systems to the macro‑level stewardship of a society coexisting with autonomous intelligence. Our technical deep dives on guard seams, provenance, governable autonomy and orchestrated defense form the foundation; CISOs will need to build on that foundation to champion identity‑first governance, verifiable autonomy, SOC automation, human‑AI symbiosis and scalable collaboration.

My book on AI Security is OUT NOW!

Innovating with integrity,
@AIwithKT 🤖🧠

Subscribe now

Share AIwithKT

That question shows up in every modern system. Where does authentication end and authorization begin? Where do you enforce policy so it covers everything without turning product teams into full-time permission clerks? Where do you log actions so you can explain them later — especially when an agent is the one taking the action?

In 2026, I think most breaches will still look familiar on the surface. But underneath, they’ll be “seam failures”: gaps at the boundaries between systems, identities, and decisions. The stack can consolidate all it wants. If the seams are sloppy, you’re just moving risk around.

1. The identity seam (humans, services, and agents)

Identity isn’t just “login.” It’s the routing layer for trust.

What changed is scale. Most environments now have more non-human identities than human ones — service accounts, CI runners, automation bots, workload identities — and now AI agents acting on behalf of users. That’s the fastest-growing attack surface I see.

Subscribe now

A modern identity seam has three properties:

• You can tell who/what is acting, in a way that holds up later.

• Permissions are scoped and time-bound (standing privilege becomes the exception).

• Authorization decisions can be explained, audited, and adjusted without rewriting half your platform.

If you can’t answer “who has access to what, and why” at machine speed, the rest of your security program becomes… well, theater. 🤷

Share AIwithKT

2. The tool seam (when agents touch the real world)

The new risk isn’t only what an AI system says.

It’s what it can do.

When an agent can query internal data, open tickets, trigger workflows, or push code, integration becomes a privileged operation. This is where a lot of “agent security” conversations get hand-wavy. You don’t secure this with a policy doc. You secure it with an explicit authorization boundary at the tool interface.

The minimum bar I’d want:

• Per-tool scopes that are deliberately narrow (not “agent can do everything”).

• Short-lived credentials and strict audience binding.

• Strong audit logs of actions (not just chats).

• A clear policy surface you can evolve without breaking product velocity.

Leave a comment

3. The runtime seam (posture is a snapshot; runtime is the movie)

Posture tells you what you intended to deploy. Runtime tells you what’s actually happening.

In cloud environments, change is constant. “We’ll fix findings eventually” doesn’t help when the attack path is active in production today. 2026 defense has to move from static review to real-time constraint: catching risky behavior where it happens and reducing blast radius in the moment.

This does not mean “buy more runtime tools.” It means choosing what you’ll enforce at runtime:

• What actions are categorically unsafe?

• What behavior is anomalous enough to throttle or require step-up auth?

• What must be logged immutably because you’ll need it later?

The more agentic systems become, the more runtime becomes the only place you can reliably govern outcomes.

Share

4. The data seam (the SOC as a decision factory)

Security teams are drowning in telemetry, but starving for signal.

The most mature SOCs I’ve seen are treating their telemetry layer like a product: defining data contracts, normalizing events, enriching context, and routing only what matters into expensive workflows. The goal isn’t “collect everything.” The goal is “make investigations deterministic.”

Here’s the uncomfortable truth: if your underlying data is messy, agentic SecOps becomes an expensive hallucination generator. AI can help analysts, but it can’t replace disciplined inputs.

In 2026, I’d judge SOC maturity less by the logo wall and more by:

• Can you explain what you ingest and why?

• Can you investigate quickly without brittle, one-off hero queries?

• Can you keep the forensic truth without paying premium costs for every byte?

• The recovery seam (trust is what you restore)

Subscribe now

Resilience is becoming the defining capability because prevention is no longer a promise anyone can honestly make. What you can promise is: containment, clarity, and restoration.

Recovery isn’t just backups. It’s:

• Known-good restore paths you actually test.

• Evidence trails of what changed and who approved it.

• Runbooks that work when humans are stressed and sleep-deprived.

• Metrics leadership understands: time-to-contain, time-to-restore, time-to-explain.

If you can’t explain what happened, you didn’t recover. You just rebooted.

Share

Share AIwithKT

What I’d do in Q1 2026

If you want a practical starting plan, I’d do five things:

• Inventory non-human identities and kill stale privilege aggressively.

• Put a real authorization boundary at the tool layer for agents (scopes, TTLs, audit).

• Decide what you’ll enforce at runtime and implement the smallest set of constraints that meaningfully reduce blast radius.

• Treat telemetry like a contract: normalize, enrich, and route for investigations, not vanity ingest.

• Run one recovery exercise that forces you to prove: what happened, who did it, what changed, and how you know.

Closing

The market is consolidating because the world got too complex to secure with disconnected dashboards. But consolidation alone won’t save you.

Architecture saves you.

In 2026, the security programs that win won’t be the ones with the most tools. They’ll be the ones with the cleanest guard seams: identity, tool access, runtime constraint, data discipline, recovery.

That’s the operating system now.

Subscribe now

Share

Share AIwithKT

Leave a comment

Innovating with integrity,
@AIwithKT 🤖🧠

The AI shift isn’t that systems can generate text. It’s that they can take actions.
As soon as language touches tools -- retrieval, connectors, ticketing, email, cloud APIs -- the security problem becomes a governance problem. The hard part isn’t "accuracy". It’s responsibility: who is accountable when an automated chain of tools acts in your name?

Share

I just published "Frontier AI Security: Power, Risk, and Responsibility in the Age of Delegated Agency". It’s a practical field guide for security leaders evaluating or approving agentic systems in production: threat models, control patterns, and safety cases that make behavior legible enough to audit -- and containable enough to operate.

Share AIwithKT

Quick personal add: this took me a few years to write. It started as field notes from production -- trying to make sense of what kept breaking as models moved from 'answering' to 'acting.' The spine (power / legibility / responsibility) came straight from that pattern.

Share AIwithKT

If your org is moving from copilots to workflows this year, you’ll need an operating contract: prove / pause / undo, with provenance and contextual authorization as first-class controls.

Subscribe now

Innovating with integrity,
@AIwithKT 🤖🧠

I spent about a month trying to figure out how to most meaningfully reflect on the direction of AI security in 2025. It has been a high-velocity year where the boundary between ‘assistant’ and ‘operator’ effectively disappeared.

This AI Security Chronicles series tracked a single shift across 2025: “agentic AI” stopped being a conceptual category and started behaving like an operating model - systems that can plan, call tools and execute work across real infrastructure. That change matters because it moves security from “model safety” into the older, harder domain of systems integrity: identity, authorization, supply chain, and rollback.

Subscribe now

Several external signals reinforced that transition. Anthropic reported that frontier models are now meaningfully useful for defensive cybersecurity tasks in practice - vulnerability discovery, analysis, and early work on patching - and that repeated trials and agentic workflows materially change outcome probabilities (Anthropic, Oct 2025). In parallel, the DARPA AI Cyber Challenge demonstrated teams using LLM-driven “cyber reasoning systems” to analyze and patch software at scale in competitive conditions (DARPA, Aug 2025). And the “instruction layer” became a recognized attack surface: prompt injection and indirect prompt injection are now treated as frontier security issues for tool-using agents, with active mitigation programs from multiple vendors (OpenAI, Nov 2025; Anthropic, Nov 2025; Microsoft, Jul 2025).

Share AIwithKT

The 2026 question is therefore not whether autonomy will show up in enterprise workflows - it will. The question is what contract enterprises will demand before allowing non-human actors to touch production systems.

Share AIwithKT

What 2025 clarified:

Autonomy shifts the center of gravity from “answers” to “actions.” A chatbot can be wrong and inconvenient; an agent can be wrong and expensive. When systems can read emails, browse documentation, query internal systems, and then take steps - opening tickets, changing configs, granting access, triggering pipelines - security becomes a control-plane problem: who is allowed to instruct the system, what authority is being exercised, and what evidence supports the action.

The second, more subtle shift is that data trust is now an attack surface. Anthropic’s poisoning result makes this explicit: as few as ~250 malicious documents can implant a backdoor in a model regardless of model size or training data volume (Anthropic, Oct 2025). The implication is structural: “our dataset is huge” is no longer a defensible safety argument. Trust must be engineered upstream (corpora, fine-tunes, retrieval indexes) and downstream (evidence artifacts the SOC and auditors rely on).

At the same time, provenance tooling matured in ways security teams can actually use. Enterprise-grade Content Credentials and authenticity infrastructure are now being deployed in production contexts (Adobe, Oct 2025; Cloudflare, Feb 2025). This matters because evidence itself is becoming synthetic by default - screenshots, summaries, incident narratives, and “AI-generated findings” - and the ability to verify origin and transformation history is becoming a baseline requirement for high-stakes workflows.

Subscribe now

Share

Share AIwithKT

The baseline contract for production autonomy -

Serious deployments in 2026 will converge on three operational guarantees. These are less about ideology and more about survivability under incident pressure:

Prove why an action was allowed. “Logging happened” is not sufficient. Proof means a durable chain from identity → authority/policy → inputs → decision trace → effect, in a form that can be reviewed during incident response and post-incident audit.

Leave a comment

Pause behavior instantly. Pausing the UI is not pausing the system. Capability-level circuit breakers must exist so teams can stop a class of actions when context changes - new attack patterns, tool drift, corrupted inputs, or uncertain provenance.

Undo without collateral damage. Autonomy requires reversibility. If actions cannot be unwound safely (or compensations cannot be executed deterministically), organizations end up choosing between speed and safety. In production, that choice is unacceptable.

This “prove / pause / undo” posture aligns with the direction evaluation tooling is taking. In late 2025, Anthropic released Petri for audit-style exploration of model behavior (Anthropic, Oct 2025) and Bloom for agentic generation of behavioral evaluations at scale (Anthropic, Dec 2025). The important point is not the specific tools - it’s the underlying pattern: continuous evaluation and rapid iteration loops are becoming part of operational safety, not just research.

What 2026 will demand…

Subscribe now

Identity and authority will become agent-native. Many organizations will discover that “just treat agents like users” fails quickly. Human-centric RBAC can be necessary, but it is rarely sufficient when non-human identities act across multiple systems, timescales, and workflows. Expect increased adoption of time-bounded, workflow-bounded, relationship-aware authorization layers - especially for agent-to-agent interactions, where “friendly stranger” commands must be authenticated, authorized, and scoped before being executed.

Prompt injection will not be “solved,” but it will be operationalized. The vendors’ own language is converging on defense-in-depth and continuous hardening rather than a single mitigation. OpenAI’s prompt injection overview framed the problem as a frontier security challenge (OpenAI, Nov 2025) and its later Atlas hardening write-up describes ongoing iteration against evolving attacks (OpenAI, Dec 2025). Microsoft and Anthropic have published parallel mitigation approaches for indirect prompt injection and browser-use contexts (Microsoft, Jul 2025; Anthropic, Nov 2025). The operational takeaway: prompt injection becomes a managed risk with layered controls (tool permissions, content isolation, provenance checks, guardrails, monitoring), not a one-time fix.

SBOM thinking will expand into MLSBOM thinking. CISA’s updated SBOM minimum elements in 2025 reflect the increasing maturity and expectations around software supply chain transparency (CISA, Aug 2025). AI systems will follow the same path: organizations will be asked to account for model lineage, fine-tuning inputs, retrieval corpora snapshots, tool adapters, and evaluation results. This is not bureaucracy for its own sake; it is the only way to bound blast radius when poisoning, drift, or compromised tooling is suspected (Anthropic, Oct 2025).

Governance will harden into enforceable expectations. The EU AI Act timeline has been repeatedly reaffirmed in public statements, with high-risk obligations scheduled to apply from August 2026 (Reuters, Jul 2025; DLA Piper, Aug 2025). In the U.S., NIST’s preliminary Cyber AI Profile explicitly frames three overlapping focus areas - secure AI systems, AI-enabled cyber defense, and resilience against AI-enabled attacks - providing a practical risk-management lens for organizations adopting AI at scale (NIST, Dec 2025).

Share AIwithKT

Cyber capability risk is moving “up” the frontier model ladder. OpenAI’s cyber resilience post notes rapid progress and emphasizes ecosystem initiatives to strengthen defensive readiness (OpenAI, Dec 2025). Reuters separately reported OpenAI warning that upcoming models could pose “high” cybersecurity risk as capabilities advance (Reuters, Dec 2025). Whether an organization buys that framing in full or not, it reflects a broader reality: the delta between defensive automation and offensive scaling is narrowing, and governance must assume adversaries also have access to sophisticated tooling.

Share

The posture worth building toward:

The most resilient organizations in 2026 will treat trust as infrastructure. That means: provenance becomes a perimeter; authorization becomes contextual and auditable; evaluation becomes continuous; and every autonomous capability is shipped with enforceable proof, pause, and undo semantics.

This is how autonomy becomes an advantage rather than a new category of unbounded risk: not by adding more policy documents, but by building operational guarantees into the platform.

Share

Share AIwithKT

Leave a comment

Innovating with integrity,
@AIwithKT 🤖🧠

• A research team showed that you can reliably poison large language models with roughly 250 malicious documents, no matter how big the underlying dataset is.

• Adobe quietly rolled out Content Authenticity for Enterprise, and the Content Authenticity Initiative crossed 5,000 members, with C2PA Content Credentials now supported by major clouds, media companies, and even camera manufacturers.

Put those together and you get a new reality for security teams:

The weakest point in your AI stack is no longer just code or config - it’s provenance: where your data, models, and evidence actually came from.

If a few hundred poisoned files can backdoor a model, and your SOC is already drowning in AI‑generated screenshots, reports, and alerts, then “zero trust for content” stops being a nice-to-have. It becomes your new perimeter.

This post is about what that perimeter looks like in practice.

1. What the poisoning results really tell us

For years, many of us comforted ourselves with a mental model:
“To meaningfully poison a trillion‑token model, an attacker would need a serious slice of the training corpus. That’s economically hard.”

That model just died.

Subscribe now

In October, Anthropic published work showing that a small, fixed number of poisoned documents can reliably implant backdoors in models across a wide range of sizes and dataset scales. Follow‑up commentary by the research team and independent analysts converged on a number in the low hundreds - on the order of ~250 documents in their experiments.

The key points:

• Attack cost is roughly constant. As models get bigger and datasets get larger, the fraction of poisoned data shrinks, but the absolute number of malicious items required stays about the same.

• Poisoning generalizes. The same dynamics show up in fine‑tuning, not just pre‑training.

• Triggers can be subtle. Backdoors can be tied to innocuous phrases, formatting quirks, or specific URLs - exactly the stuff we sprinkle across documentation and corpora.

Share AIwithKT

In parallel, we’re seeing “content poisoning” attacks in the wild: adversaries seed fake customer‑service numbers or malicious FAQs across web properties, then rely on AI systems and search interfaces to surface them as trusted answers.

Put bluntly: you no longer need to control a data center’s worth of text to move a model. You just need a small, well-crafted wedge in the right place.

Leave a comment

2. Meanwhile, provenance tech quietly went mainstream…

While researchers were proving how little data it takes to poison an AI system, the provenance world was having its own watershed year.

A few milestones, all in 2025:

• The Content Authenticity Initiative (CAI) hit 5,000+ members, spanning camera makers, media organizations, platforms, and tech companies.

• Adobe launched Content Authenticity for Enterprise, bringing C2PA‑based Content Credentials into enterprise asset pipelines, not just creative apps.

• Cloudflare rolled out “Preserve Content Credentials” across its image delivery stack, meaning authenticity metadata can now ride along with a significant chunk of global web traffic.

• National cybersecurity agencies started publishing explicit guidance on public content provenance as a defensive control, encouraging organizations to sign and verify the origin of their own public materials.

• C2PA itself matured: updated specifications, a formal conformance program, and sandbox environments for testing issuance and verification workflows went live.

Share

Add in the fact that consumer platforms (social networks, cameras, creative tools) are increasingly shipping Content Credentials by default, and we’re entering a world where evidence can - and should - come with a chain of custody attached.

Security teams perhaps have not caught up yet.

3. MLSBOM: Your model and data “bill of materials”

Traditional SBOMs list the software components in a build. Regulators and agencies have spent the last year updating what “good” looks like there, with new minimum element guidance and joint positions from NSA, CISA, and partners explicitly calling SBOMs a pillar of “secure by design.”

Now we need the same thing for AI: call it an MLSBOM (Model & Learning SBOM) if you like.

Recent guidance and tooling give us a sense of what belongs in it:

Share AIwithKT

• Model lineage: base checkpoints, fine‑tunes, adapters, quantized variants, and where they’re deployed.

• Dataset & corpus snapshots: training windows, major sources, hashes of curated slices, and any synthetic or vendor‑supplied sets.

• Retrieval & feature stores: which indexes are wired into which applications, with hash‑based snapshots and retention windows.

• Tooling & connectors: external APIs, agent adapters, and their dependency trees.

• Evaluations: which security‑relevant evals (e.g., poisoning, jailbreak, cyber capability tests) were run against which model versions.

• Governance hooks: owners, update cadence, and the policies (or contracts) that apply.

Share

Security‑focused vendors are already asking questions like, “Is the dataset origin documented and digitally signed? Does the model’s SBOM list every upstream dependency? Are drift detectors active on new ingests?”

An MLSBOM doesn’t stop poisoning by itself. But without it, you don’t even know which slice of your universe to quarantine when you suspect poisoning or provenance tampering.

4. Content Credentials in the SOC: evidence that carries its own history

Most SOCs still paste raw screenshots and logs into tickets. In an AI‑heavy environment, that’s a problem:

• Autogenerated evidence can be trivially manipulated.

• Cross‑tool copy/paste often strips context.

• Regulators increasingly expect you to prove what you knew and when you knew it.

C2PA Content Credentials give us a way to attach cryptographically bound provenance to media - who created it, when, with which tool, and how it has been transformed.

For a SOC, that translates into some concrete moves:

Subscribe now

• Screenshots from EDR consoles, AI tooling, and dashboards are captured by a client that signs and embeds Content Credentials before they ever hit a ticketing system.

• Agent‑generated reports (summaries, timelines, recommendations) are exported with credentials indicating model version, prompt policy, and key inputs.

• Internal tools verify credentials on display, flagging anything missing or tampered as low‑trust evidence.

• When you share evidence externally - with regulators, customers, or law enforcement - they can verify origin and integrity without trusting your internal systems.

Provenance metadata won’t stop an insider from lying, but it makes large‑scale forgery and quiet tampering much harder, especially when combined with an MLSBOM and a model/agent attestation log.

5. Putting it together: the “provenance perimeter” in practice

Here’s how I’d sketch a first‑year roadmap for a security org that wants to treat provenance as a perimeter, not an afterthought.

Share AIwithKT

Leave a comment

Step 1: Draw the map

• Build a lightweight MLSBOM for your highest‑impact AI systems: models, datasets, indexes, tools. Don’t aim for perfection; aim for “we can answer basic supply‑chain questions in a day instead of a month.”

• Identify all the places where content leaves your AI stack and becomes evidence: SOC tickets, post‑incident reports, screenshots in Slack, dashboards in exec reviews.

Step 2: Start signing the right things

• For models & data: hash and sign corpus snapshots, index exports, and model artifacts as part of CI/CD. Store signatures in an auditable registry or transparency log.

• For evidence: adopt Content Credentials in one or two pipelines (e.g., incident screenshots and exported reports). Most orgs can start by integrating with asset systems or image delivery platforms that already speak C2PA.

Step 3: Add detection and response around provenance

• Treat provenance gaps as a class of alert: unsigned dataset, missing credentials on evidence, or an MLSBOM diff that doesn’t match your change log.

• When you ingest anything from outside (open‑source corpora, third‑party models, scraped data), check signatures, run poisoning‑aware evals on a canary, and quarantine until cleared. Recent poisoning work tells us the attack can be small but devastating; assume clever adversaries will try.

• When you suspect a poisoned source, you should be able to answer in hours, not weeks: Which models saw this? Which indexes embed it? Which apps depend on those indexes? That’s exactly what your MLSBOM is for.

Step 4: Wire provenance into your “Proof, Pause, Undo” loop

If you’re experimenting with agentic systems:

• Every proof bundle for an autonomous action should reference both model lineage and content provenance - which version of which model acted on which signed inputs.

• Your pause mechanism should be able to stop actions that rely on a compromised corpus or model, not just actions from a compromised agent.

• Undo should be informed by provenance: if you learn a dataset slice was poisoned, you know which models to roll back and which downstream artifacts to treat as tainted.

This is how provenance stops being a compliance checkbox and becomes a line of defense.

6. What to watch in 2026

Looking at the last three months alone, the direction is clear:

• Poisoning attacks are more feasible than we assumed.

• Governments are formalizing expectations around SBOMs and AI transparency.

• The content world is moving fast toward cryptographically verifiable media.

The next moves I’m watching:

• How regulators and large buyers fold MLSBOM expectations into procurement and incident reporting.

• Whether C2PA credentials start to appear inside security tools by default (not just creative suites).

• The first high‑profile incident where a company can say, “Yes, we identified the poisoned slice quickly, here is our signed evidence, and here’s how we rolled back” - or the opposite.

Either way, one thing feels certain:

In the AI era, if you don’t know where your data came from, you don’t actually know what your system is doing.

Provenance is the new perimeter. The good news is that, for once, the standards and tooling are arriving early enough that we can design around it - if we start now.

Share AIwithKT

References

• Anthropic - A small number of samples can poison LLMs of any size (Oct 2025).

• Souly et al. - Poisoning Attacks on LLMs Require a Near‑constant Number of Poison Samples (arXiv/OpenReview, Oct 2025).

• The Alan Turing Institute - LLMs may be more vulnerable to data poisoning than we thought (Oct 2025).

• The Register / Heise - coverage of the “fixed number of poisoned docs” result (Oct 2025).

• Adobe - Content authenticity arrives for enterprises (Oct 2025).

• Content Authenticity Initiative - 5,000‑member milestone and 2025 highlights; membership overview.

• Cloudflare - integration of Adobe Content Credentials / C2PA in Cloudflare Images (2025).

• CAI / C2PA - 2025 Content Authenticity Summit; conformance & trust list guidance; community events.

• Government of Canada - Public content provenance for organizations (ITSP.10.005, 2025).

• CISA & partners - updated SBOM minimum elements, joint SBOM vision, and Secure‑by‑Design guidance (2025).

• SBOM for AI / ML models and model‑SBOM tooling (sbomgenerator.com, ai‑sbom.com, Hoop.dev, SentinelOne, Code & Conscience, 2025).

Subscribe now

Share

Share AIwithKT

Leave a comment

Innovating with integrity,
@AIwithKT 🤖🧠

Enterprise security leaders are staring at a real shift: AI isn’t just a coding assistant anymore - it’s starting to perform security work end-to-end. The question is how we use that power without losing control.

What changed…?

In the last few months we’ve seen credible, external evidence that large models can find, reproduce and sometimes patch real vulnerabilities - not just toy CTF puzzles. Anthropic’s latest write-up describes targeted investments in defense-relevant skills for Claude Sonnet 4.5 and reports step-function jumps on third-party evaluations, alongside real-world pilots with companies like HackerOne and CrowdStrike. The headline isn’t that a single benchmark improved; it’s that repeated trials on realistic tasks raise success probabilities to the point where defenders can - and should - operationalize these systems.

Subscribe now

Two evaluation suites anchor this shift. Cybench aggregates professional CTF tasks and measures success across many attempts - more faithful to how motivated responders actually work. As vendors iterate models and run multi-trial evaluations, success rates climb substantially. CyberGym goes further: it wires models into real OSS codebases to reproduce historical CVEs and even discover new ones - documenting that LLM agents can reveal previously unknown bugs at non-trivial rates. Together, these frameworks move the conversation from “cool demos” to measurable capability on real code.

Outside the lab, competitions and product announcements reinforce the trend. DARPA’s AI Cyber Challenge ended with teams demonstrating cyber-reasoning systems that autonomously secure open-source software - imperfect, but directionally important. Meanwhile, HackerOne announced agentic systems and reported material gains (faster intake, higher accuracy) in early deployments. These aren’t proofs of perfection; they’re proofs of traction.

Share

Why this matters for CISOs now.

When success becomes a game of more attempts + deeper context windows + repeatable workflows, defenders who pilot early get multiplicative returns: every scan, triage and patch loop improves the system, not just the person running it. The risk, of course, is that we push autonomy into production without the guardrails needed for audit, safety and rollback. If we want the upside without the headlines, we have to engineer three guarantees into any AI-driven security capability:

Subscribe now

1. Proof: Show who acted, under which policy, based on what evidence and what changed.

2. Pause: Stop a whole class of actions instantly when risk or context shifts.

3. Undo: Restore state safely - with an engineered compensation path, not improvisation.

Benchmarks don’t give you these guarantees. Architecture does.

Share AIwithKT

The minimum viable architecture for verifiable autonomy.

Typed messages, not prompt spaghetti. Treat agent conversations as APIs with schemas (inform, request, propose, confirm, commit, compensate), policy fingerprints, signatures and idempotency keys. This makes behaviors searchable, enforceable and non-repudiable - exactly what auditors will ask for.

Adapters as the safety membrane. No agent gets a super-token. All writes flow through least-privilege adapters that enforce scopes, quotas, schema checks and own both effect proofs and rollbacks. When a vendor API drifts, you update one adapter - not every agent.

Proof bundles by default. Every consequential action emits a signed, hash-linked bundle: actor identity and runtime attestation, policy version and decision, decision trace (inputs → plan → tool calls → outputs), effect and a ready-to-execute rollback. This turns “we logged it” into evidence that stands up in incident review and audit.

Share

Bounded autonomy as code. Impact tiers:

• T0 (read) and T1 (propose) auto-approve with sandboxed sims.

• T2 (execute, reversible) requires one approval or a successful simulation.

• T3 (high-impact) demands dual-control approvals, simulation, staged rollout, and a cooldown.
  A circuit breaker pauses a capability class globally and logs the pause as its own event.

Share AIwithKT

Reversibility by design. Use sagas (preconditions, effects, compensations), shadow runs for T2/T3, CI gates that refuse to ship capabilities without green rollbacks and one-click “Rollback this bundle” in your console that replays compensations and produces a second proof. Recovery becomes engineered, observable and fast.

Subscribe now

The uncomfortable flip side: models also make offense cheaper…

The same dynamics that help blue teams - iteration, breadth of knowledge, persistence - also help red teams. Lab reports of “vibe hacking” (coordinated data extortion at scale) and LLM-assisted espionage targeting critical infrastructure are early warnings of what cheap autonomy enables. If we don’t adopt AI on defense, we risk a tempo gap that governance alone can’t close.

And the supply chain is fragile. Two new results on data poisoning show that attackers may only need hundreds of malicious documents to reliably backdoor or degrade models, independent of total dataset size, far less than previously assumed. That turns provenance and dataset hygiene from “nice to have” into the new perimeter. Tools like Petri - which enables reproducible audits, alternate tool call simulations, and rollbacks during eval - hint at how the ecosystem will instrument models to catch and analyze bad behavior.

Leave a comment

What to pilot in the next 90 days →

• CI/CD security reviews with an agent loop. Start with language-specific checks (secrets, unsafe patterns), then add proposal diffs for policy/permission changes. Use adapters to gate any write (PR merge, policy change) and emit proof bundles. Anthropic and others have shown model-assisted code security is gaining real signal; wire it where it shortens time-to-fix.

• Triage + response drafts in the SOC. Keep humans in the approval path but let agents produce typed proposals (containment steps, IAM diffs) with dry-run results attached. Measure precision, time-to-acknowledge/contain vs. baseline, % of actions with complete proof and post-review rework/rollback rate. (If the metrics don’t move, it’s a chatbot, not a system.)

• Provenance logging across your knowledge corpus. Hash and sign artifacts (retrieved docs, tool outputs), bind source lineage to decisions, and alert on provenance gaps. The poisoning results make this urgent.

• Benchmark with external suites. Run your model/agent stack on Cybench and CyberGym (even a subset) to get a common yardstick - and rerun with multiple trials to approximate motivated defenders. Expect rapid iteration cycles.

Leave a comment

The take-home for boards and regulators?

We are past the “maybe useful someday” phase. Independent evaluations, public competitions, and early enterprise deployments show that models now perform meaningful security work - with the right scaffolding. The governance bar should shift from “are you using AI?” to “can you prove, pause and undo what it does?” That is the difference between risky automation and reliable autonomy.

Share

Subscribe now

Share

Share AIwithKT

Sources and further reading:

• Anthropic, Building AI for cyber defenders (Claude Sonnet 4.5, evaluations, defender focus).

• Zhang et al., Cybench (CTF-derived benchmark; ICLR 2025 + project site).

• Wang et al., CyberGym (real-world vulnerabilities; new bug discovery; arXiv + summary).

• DARPA AI Cyber Challenge (AIxCC) results (autonomous cyber reasoning systems).

• HackerOne announcements on agentic AI for code and exposure management.

• Souly et al., Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples; Anthropic summary.

• Anthropic, Petri open-source auditing tool (reproducible audits, tool simulations, rollback).

Share

Share AIwithKT

Leave a comment

Innovating with integrity,
@AIwithKT 🤖🧠

An uncomfortable truth…

Most “agentic AI” I see in the wild is clever glue, not engineered autonomy. We bolt an LLM onto a few APIs, let it open tickets or flip low-risk switches, watch dashboards climb, and declare victory. Then the first real incident hits and we discover the new failure modes we introduced: an agent disables the wrong user, rotates a key without a compensation path, or files a flawless summary of a half-true investigation because one upstream signal was poisoned. We have logs, we have a kill switch - we don’t have control.

If an agent can touch production, even to “just propose” a change, your system needs to do three things on command:

Subscribe now

1. Prove why an action was allowed. Not “there’s a log line,” but a portable proof bundle: which attested agent acted, under which policy version, based on which inputs, producing which effect, and how we’d reverse it. If we can’t justify it, we shouldn’t automate it.

2. Pause the behavior instantly. When context shifts (vendor drift, live incident, policy update), we need a circuit breaker that stops a whole class of actions across the estate - failing closed, draining safely, and leaving evidence of the pause itself.

3. Undo without collateral damage. Autonomy is tolerable only if outcomes are reversible by design: every capability ships with a tested rollback; plans have preconditions/effects/compensations; and operators can restore state with one click and a paper trail.

Everything that follows is how I, as a recently-former-CEO and now Senior Engineer, build these guarantees into the architecture - message schemas, policy fingerprints, adapters, attestations - so we get trustworthy autonomy without turning the program into governance theater.

Share

1) Proof beats promises!

Engineering stance: logs describe; proofs justify. I design a proof bundle as a compact, signed artifact that survives pager duty, vendor churn, and audit.

Minimal schema (stable, versioned):

{
  “bundle_version”: “1.2.0”,
  “actor”: { “agent_id”: “responder-07”, “runtime_tcb”: “attest:nonce:…”, “scopes”: [”idp.quarantine:write”] },
  “policy”: { “fingerprint”: “POL-IDENT-2025.04#3”, “decision_id”: “dec-7f…” },
  “decision_trace”: {
    “inputs_cids”: [”bafy…”, “bafy…”],      // content-addressed inputs
    “plan”: { “name”: “quarantine_identity”, “preconditions”: [”risk>0.9”,”hr_notified”] },
    “checks”: [”SoD:ok”,”rate_limit:ok”,”simulated:ok”],
    “tool_calls”: [{”adapter”:”idp”,”op”:”disable_user”,”args_cid”:”bafy…”,”result_cid”:”bafy…”}]
  },
  “effect”: { “system”:”idp”, “resource”:”user/123”, “change”:”status:disabled”, “ts”:”…” },
  “rollback”: { “capability”:”idp.enable_user”, “preconditions”:[”no_active_alert”], “eta_sec”:3 }
}

Why this works:

• Actor ties action to an attested runtime + least-privilege scopes (no “mystery principal”).

• Policy is carried with the decision; reviewers see the exact version that evaluated true.

• Decision trace is structured and content-addressed (hashes of prompts, docs, and outputs). Chat transcripts are supplemental, never canonical.

• Effect + rollback makes the bundle operational: you can restore state now, not after a wiki hunt.

Plumbing:

• Sign bundles (agent key) → hash-link into an append-only log (transparency table).

• Store inputs/outputs in a CAS (content-addressable store); reference by CID in the bundle.

• Index bundles by actor, policy.fp, capability, impact_tier for 1-query retrieval.

Outcome: In an incident, we hand over one file, not a scavenger hunt.

Share

2) Bounded autonomy is a spec, not a slogan

Engineering stance: autonomy limits live in code, with tests.

Impact tiers (example):

T0 - Read / Enrich (auto-approve)

Scope: Observe only. Query, correlate, summarize, tag.
Creds: Read-only, short-lived, allowlisted egress.
Guards: Schema validation, rate limits, basic lineage.
Signal: No side effects by construction; deterministic replays.

T1 - Propose (plans/patches, auto-approve)

Scope: Generate diffs and playbooks; no prod writes.
Creds: Read + sandbox/dry-run endpoints.
Guards: Proposals carry preconditions/effects/compensations + risk score.
Signal: A stable proposal ID and simulated diff attached for review.

T2 - Execute (reversible write; 1 approval or simulate)

Scope: Surgical changes with a tested rollback.
Creds: Ephemeral, least-privilege tokens issued per intent.
Controls: One of: authorized approval or successful end-to-end simulation.
Signal: A signed proof bundle (actor, policy fp, decision trace, effect, rollback) and one-click restore.

T3 - Destructive / High-Impact (SoD + simulate + cooldown)

Scope: Cross-tenant, safety-critical, or not provably reversible.
Creds: Ephemeral with explicit blast-radius; strong attestation where it matters.
Controls: Dual control, full simulation, timed cooldown, staged rollout behind an explicit feature flag.
Signal: Enriched proof bundle with approvals and simulation artifacts; health checks gate progression.

Subscribe now

How I encode it:

• A small, boring policy engine (think OPA or homegrown) with rules like:

  • allow(intent) if impact_tier==T2 and approver.role in {IR_Lead} and simulation.passed and rate_limit.ok.

• SoD enforced at the envelope: proposer.sub != approver.sub and different key domains.

• Simulation: every plan supports --simulate; adapters expose dry-run ops returning predicted diffs and guardrail verdicts.

• Circuit breaker: single feature flag per capability class (quarantine.identity.enabled=false), evaluated at the adapter boundary. Flips are logged and signed.

Tests that matter: Can I deny a T3 action by revoking one approval? Can I toggle the breaker mid-flight and watch agents fail closed? These are CI tests, not tabletop fantasies.

Share AIwithKT

3) Typed conversations, not prompt spaghetti

Engineering stance: agents talk protocol, not prose.

Message envelope (Protobuf/JSON):

{
  “id”:”msg-…”, “ts”:”…”, “from”:”analyst-12”, “to”:”responder-07”,
  “type”:”propose”, “capability”:”quarantine_identity”,
  “payload_cid”:”bafy…”, “policy_fp”:”POL-IDENT-2025.04#3”,
  “sig”:”ed25519:…”, “replay_key”:”case-9812#propose#user/123”
}

Why:

• Boundary validation is cheap: schema + signature + policy fingerprint or reject.

• Idempotency via replay_key; back-pressure via per-type quotas.

• Searchability (“show me all propose messages for capability X last week”).

• Non-repudiation and trace stitching become automatic.

Infra details: I prefer a boring broker (Kafka / SQS) + a mailbox abstraction per agent. Retries are idempotent; poison messages land in a dead-letter queue with alerts.

Leave a comment

4) Adapters are a safety membrane

Engineering stance: no agent ever holds broad, long-lived production creds. All side-effects traverse a thin, testable adapter that enforces scope, policy, and contracts. Agents think in “capabilities”; adapters translate those into specific vendor calls - safely.

What an adapter owns (and why):

• Identity and scope brokering. It mints short-lived, least-privilege tokens per call, binds them to a capability (e.g., idp.quarantine_identity), and enforces mTLS agent⇄adapter. If a token leaks, its blast radius is minutes and one capability.

• Input/output contracts. Schemas at the edge: reject args outside allowlists; coerce/normalize vendor responses into typed records. This kills prompt-shaped ambiguity and stabilizes downstream logic.

• Policy and rate governance. Per-capability quotas (budget by tenant/region), concurrency caps, exponential backoff, and fail-closed defaults. If the world gets weird, the adapter says “no” loudly and leaves evidence.

• Effect proofs. On successful writes, the adapter stamps a compact effect record (what changed, where, correlation IDs) and attaches it to the agent’s proof bundle. On failure, it emits a machine-readable error that can be retried or compensated.

• Rollback as a method. Every write capability pairs with rollback() that is first-class, not tribal lore. The adapter knows the compensating sequence for that vendor/tool and exposes it with the same contracts and guards.

Operational hygiene:

• Contract tests run in CI and nightly against a canary tenant. If a vendor API drifts or a permission tightens, tests flip a feature flag to read-only for that capability. Agents keep running; writes pause safely.

• Versioning and deprecation. Adapters version their contracts (v1, v2) and carry deprecation windows. Agents declare the version they target; upgrades are opt-in rollouts, not surprise breakage.

• Observability by default. Adapters emit structured logs and metrics (allow/deny counts, latency, quota rejections, schema rejects) with labels for capability, tenant, region, and policy fingerprint. This is how you spot scope creep and hot paths.

Failure modes we’re designing against:

• Scope inflation: blocked by allowlists + per-capability token mints.

• Silent vendor drift: caught by contract tests → auto read-only.

• Noisy retries that worsen incidents: rate limits + backoff + idempotency keys at the adapter boundary.

• Ambiguous effects: normalized typed results + effect proofs bound to the agent’s decision trace.

Why this matters: adapters shrink blast radius and decouple change. When a provider tweaks an endpoint or a permission, you update one adapter, not every agent. And because adapters attach effect proofs and own rollbacks, your autonomy remains provable and reversible without rewriting business logic.

5) Reversibility by design

Engineering stance: if it can go forward, it must be able to come back - provably. We don’t “hope” a human can undo it at 3 a.m.; we design the undo the same day we design the do.

Core patterns

• Sagas as the contract.
  Every capability is a mini–state machine with preconditions, effects, and compensations. If step N fails, we don’t guess…we unwind deterministically:
  A → B → C with compensations C⁻¹ → B⁻¹ → A⁻¹. Preconditions keep us from starting a plan we can’t finish (or unwind).

• Shadow / simulate first.
  For T2/T3 intents, run a no-write path end-to-end: same policy engine, same adapter guards, same inputs. Persist the predicted diff and risk notes. Reviewers can eyeball the blast radius before anything changes, and we learn if rollbacks are feasible in the current state.

• Promotion gates that bite.
  CI refuses to ship a capability if its rollback tests fail (or are missing). We require parity: the test suite for do() and undo() is symmetrical; coverage reports call out compensations explicitly.

• One-click restore.
  In the console, every proof bundle exposes Rollback this bundle. Clicking it replays the stored compensation with fresh attestations (new policy version, current scopes), and emits a second bundle linking to the first. Rollbacks aren’t tribal knowledge; they’re first-class, auditable actions.

Implementation notes (lightweight, but real)

• Idempotency everywhere. Compensations are safe to retry; we tag messages with replay keys so we never double-undo.

• Guarded compensations. Compensations have their own preconditions (e.g., “target still quarantined”) so we don’t roll forward into a new incident.

• Health-gated unwind. During unwind, we probe the system between steps; if health degrades, we pause and page before doing more harm.

• Time-boxed reversals. We record estimated rollback ETA in the bundle; SLOs include “time to restore state.”

How we measure we actually have reversibility

• Rollback success rate (by capability) and median time to restore.

• Symmetry coverage: percentage of forward steps with tested compensations.

• Partial-failure outcomes: unwind completed vs. manual intervention required.

• Operator touches per rollback (trend down as compensations mature).

Why this calms everyone down

When preconditions prevent un-undoable starts, simulations preview blast radius, CI blocks non-reversible deploys, and operators have a single, accountable action to restore state, autonomy stops feeling risky. The recovery path is engineered, observable, and repeatable - not improvised during an incident.

6) Provenance is the new perimeter

Engineering stance: treat data lineage like code supply chain.

Concrete moves:

• Content addressing for all artifacts (prompts, retrieved docs, embeddings, tool outputs);

• Signing + transparency logs for model/adapter releases and for high-value data drops;

• Lineage graph that links bundle → inputs (CIDs) → sources (index/version/provider);

• Adversarial hygiene: watermark detection where appropriate, duplicate/corruption checks, and alerts on provenance gaps (unsigned, unknown, or unverifiable inputs).

If I can’t show where a decision’s knowledge came from, I don’t let the system auto-execute.

7) The adversary will target the proof, not just the plan

Threat model expansions I implement:

• Message forgery/replay: mitigate with per-message signatures, short TTLs, and replay keys;

• Scope inflation via adapters: isolate credentials; adapters enforce allowlists; rotate keys aggressively;

• Approval laundering: policy agent runs in a separate trust domain; approvals use a different key hierarchy; require independent liveness attestations;

• Lineage forgery: cross-check CIDs against a transparency log; random out-of-band verification (e.g., separate read path).

Detection: anomaly models over the message fabric and the lineage graph, not just endpoint telemetry.

8) Humans belong - at the right altitude

Engineering stance: give humans proof, not prompts.

• Approve intent classes (e.g., “T2 quarantine within tenant A, business hours”), not raw steps;

• Proof bundle UI: “who/what/policy/evidence/effect/rollback” in one screen; one-click “Pause class” and “Rollback this bundle”;

• Post-incident: decision trace diff view (what changed from last time?), and a “replay in sandbox” button to validate fixes.

This preserves judgment and slashes cognitive load.

What “good” looks like (measured like an engineer)

I track four deltas and hold myself to them:

1. Precision of automated decisions (fewer wrong actions per 1000 attempts).

2. Time-to-acknowledge/contain vs. pre-agent baseline.

3. Percentage of actions with complete proof (policy fingerprint + decision trace + effect + rollback).

4. Rework/rollback rate after human review (should trend down as policies mature).

If these don’t move, I built a scripted assistant, not governed autonomy.

Start small, build the spine, then scale!

Pick one painful, bounded workflow (e.g., “impossible travel”). Ship three agents (Sensor, Analyst, Responder), two message types (inform, propose), and one adapter (IdP). Carry the policy fingerprint, store the decision trace, and expose rollback. Add the breaker. When you add the next workflow, reuse the schemas and the adapter library. This isn’t a phase plan; it’s engineering hygiene you can do before lunch.

Leave a comment

Where this is heading…

Buyers will request machine-readable evidence and reversibility guarantees as part of procurement. Agent runtimes will ship with built-in attestation and policy engines. Provenance will be graded like SBOMs. Both red and blue teams will run agent swarms; the differentiator won’t be raw intelligence but how safely and quickly you can move because every decision is provable, pausable, and undoable.

Share AIwithKT

Bottom line

The win is not novelty - it’s reliable autonomy. Build agents that speak typed protocols, operate behind safety membranes, carry policy in every interaction, emit proofs as they act, and can always be paused or undone. That’s autonomy your CISO can sign, your auditors can trust, and your engineers can ship. :)

Subscribe now

Share

Share AIwithKT

Leave a comment

Innovating with integrity,
@AIwithKT 🤖🧠

TL;DR - agent-oriented programming (AOP) treats software components as autonomous agents with beliefs, goals, and plans that perceive → decide → act → coordinate. Compared to OOP’s passive objects, AOP gives you first-class building blocks for multi-agent workflows, policy-constrained autonomy, and resilient orchestration. For security programs, AOP can be how you turn “agentic AI” into a governable, auditable, zero-trust-compatible system rather than a pile of brittle scripts.

Subscribe now

This post covers:

• What AOP is (and isn’t), with a clear contrast to OOP;

• How to apply AOP to SOC automation, incident response, posture management, GRC;

• A minimal agent loop in Python, plus pointers to production frameworks;

• A reference security architecture, maturity plan, pitfalls, KPIs;

• Curated, up-to-date resources at the end!

It builds on earlier posts on the agentic security paradigm, orchestration layers and multi-agent design.

Why AOP now (and why security cares)

Attackers are increasingly “agentic.”

Modern campaigns are starting to look less like one-off exploits and more like autonomous playbooks: scripts call cloud and SaaS APIs, query external intel, spin up LLMs to personalize phishing or summarize loot, and loop continuously until goals are met. Toolchains mix commodity loaders, headless browsers, API clients, and decision nodes that adapt to defender responses (rate limits, MFA prompts, geo-blocks). In cloud, we are seeing and will see more of iterative behaviors: enumerate → test policy edges → escalate → exfiltrate → persistence, with each step picking the next move based on feedback. If defenders keep shipping point automations - a webhook here, a Lambda there - our responses stay brittle and reactive. We need software components that can perceive state, reason about objectives, coordinate with peers, and change tactics - the behaviors AOP formalizes.

Enterprise adoption has crossed the threshold.

What used to be weekend research repos is now packaged: major vendors ship agent builders, multi-agent orchestration, tool adapters, long-running state, and event-driven triggers. Platforms expose guardrails, approvals, and audit hooks so agents can operate in real environments - ticketing, IdP, EDR, cloud, data lakes - without bespoke glue for each team. That matters for security because our stack is inherently heterogeneous; an approach that treats agents as first-class, composable services lets you introduce autonomy where it helps (think triage, enrichment, containment proposals) and keep humans where judgment is essential, and that too, without a rewrite of everything around them.

Governance pressure is rising.

Boards, auditors, and regulators now ask three questions on every autonomous action: who authorized it, under which policy, and where’s the evidence. Email threads and ad-hoc scripts won’t cut it when decisions are machine-driven and time-sensitive. Architecturally, this implies three non-negotiables:

1. Bounded autonomy - actions are classified by impact; high-impact intentions require approvals, simulations, or dual control.

2. Typed communications - agents exchange structured messages (“inform,” “request,” “propose,” “confirm”) that can be validated, versioned, and searched.

3. Attribution and replayability - every message and action carries identity, provenance, and a decision trace so you can reconstruct the “why” and replay scenarios for audit and improvement.

AOP gives us the vocabulary and the scaffolding.

Instead of wiring opaque prompt chains, AOP lets you design with roles, goals, beliefs, intentions, and protocols. Agents become accountable principals: they own objectives, speak well-typed “speech acts,” and operate under explicit constitutions (policies) enforced at runtime. The scaffolding - mailboxes/queues with identity, schema-validated messages, plan libraries with preconditions/rollbacks, guardrail services for approvals - maps one-to-one with security needs. You gain systems that are adaptive enough to keep up with attackers and governable enough to satisfy audit. In short: AOP aligns autonomy with assurance, so you can scale agentic capabilities without sacrificing control.

AOP in one page (vs. OOP)

Think of agents as secure microservices with a brain and a radio. They own their goals, talk to peers through typed messages, and can be reasoned about at the level of intent and obligation, not just function signatures.

Leave a comment

Minimal agent loop (Python)

You don’t need a heavy framework to start thinking in AOP. This stripped-down loop shows the core:

from typing import Any, Dict, List

class Agent:
    def __init__(self, name: str, goals: List[str]):
        self.name = name
        self.beliefs: Dict[str, Any] = {}
        self.goals = goals
        self.intentions: List[Dict] = []  # planned actions

    def perceive(self, events: List[Dict]):
        for e in events:
            self.beliefs[e["key"]] = e["value"]

    def deliberate(self):
        # trivial policy: if suspicious_auth and goal "maintain_integrity" then plan a response
        if self.beliefs.get("suspicious_auth") and "maintain_integrity" in self.goals:
            self.intentions.append({"type": "request", "to": "ResponseAgent",
                                    "action": "isolate_identity", "evidence": self.beliefs["suspicious_auth"]})

    def act(self, send):
        while self.intentions:
            msg = self.intentions.pop(0)
            send(msg)  # typed message to another agent

# Example "mailbox" send function would enforce ACLs, schemas, and audit logging.

Production systems add…

• Typed messages (schemas & protocols), retries, and delivery guarantees;

• Policy checks (who can request/approve which actions);

• Plans with preconditions/effects, rollbacks, and compensations;

• Memory (episodic logs, tool outcomes) and tooling (search, RAG, device control);

• Observability (per-message lineage, decision traces).

Where AOP lands in the enterprise

Common agent roles (start here and expand):

1. Sensor/Collector Agents –> stream logs, cloud events, SaaS signals; normalize + enrich.

2. Triager/Analyst Agents –> correlate alerts, prioritize, draft investigations; ask peers for data.

3. Responder/Remediator Agents –> propose and (with policy approval) execute playbook steps.

4. Posture/Compliance Agents –> continuously test controls, generate evidence, open tickets.

5. Orchestrator/Coordinator Agents –> maintain the global plan; resolve conflicts; escalate.

6. Policy/Safety Agent –> gate high-impact actions; enforce SoD (separation of duties), rate limits, and guardrails.

High-value use cases (90-120 days): alert triage, phishing response, IAM hygiene, misconfig drift detection, SaaS posture fixes, vendor evidence collection, and control testing. Each maps cleanly onto agent roles and messages.

Share AIwithKT

A secure reference architecture (overview)

1. Agent Runtime Layer

   • Long-running workers with durable state; per-agent identity (mTLS/JWT), stable mailboxes/queues

   • Typed messages (e.g., FIPA-style speech acts) with schemas in your repo; strict versioning

2. Policy and Safety Layer

   • Constitutions: org-level rules (who may propose/approve/execute which action classes)

   • Action gates: human-in-the-loop for high-impact intents; auto-approval for low-risk

   • Sandboxes & scopes: least-privilege tokens, constrained tool adapters, egress filters

3. Tooling and Knowledge Layer

   • Connectors (ticketing, EDR, IdP, cloud, SaaS), search/RAG with red-teamed corpora, rate-limited APIs

4. Observability and Assurance

   • Per-message lineage (who said what to whom, when, under which policy)

   • Decision traces (inputs → plan → action → outcome), replay harnesses, chaos tests for autonomy

5. Orchestration and Governance

   • Orchestrator agent(s) coordinating plans, deadlines, and resource allocation

   • GRC integration: map agent actions to controls (e.g., NIST 800-53, ISO 27001) and produce evidence

Adoption without the theatre

You don’t need a 200-page blueprint. Pick a single, bounded workflow (e.g., “impossible-travel” alerts). Model three roles - Sensor → Analyst → Responder - and the two or three messages they must exchange. Encode those messages as schemas; give each agent an identity; log every message and decision. Put a policy agent in front of any action that writes to production systems.

When that first flow is stable, generalize: extract the messages into a shared protocol package; turn playbook steps into reusable “capabilities” (quarantine identity, revoke token, snapshot VM). Add an orchestrator to coordinate multiple cases and deadlines. Expand connectors slowly; keep adapters least-privileged and versioned. This is how you move from “demo” to “platform” without the big-bang rewrite.

How do you know it’s actually working? I mean, first, skip the flashy counts.

Track four things:

(1) how accurate the automated decisions are,

(2) how much faster you acknowledge and contain issues versus your baseline,

(3) what percentage of actions ran under the right policy with full evidence attached, and

(4) how often humans had to redo or roll back an agent’s work. If those trends aren’t improving, you’re likely running a scripted assistant - not yet a governed agent system.

Common failure modes -> and the antidote

1) Prompt chains posing as agents

What goes wrong: you have clever LLM prompts glued together, but no persistent state, no typed messages, and no way to explain why a step happened. It “chats,” it doesn’t reason.

Antidote: model agents with explicit beliefs, goals, and intentions; persist state between steps; define message schemas (e.g., inform/request/propose with JSON contracts) and validate them at the boundary. Treat conversations as APIs with inputs/outputs and policies, not transcripts. A quick test: can you replay the same inputs and get the same plan under policy?

2) Free-roaming autonomy

What goes wrong: an agent can execute high-impact actions (disable SSO, quarantine production hosts) without the right gates. That’s how “assistants” become incident sources.

Antidote: classify action impact tiers (read, propose, execute, destructive). Require approvals, simulation, or dual control (SoD) for high-impact intentions. Enforce rate limits and scopes on credentials. Add a circuit breaker that can pause whole classes of actions system-wide and default to safe rollback plans.

Leave a comment

3) Opaque decisions

What goes wrong: an outcome appears (“user disabled”), but no one can reconstruct who authorized it, under which policy, or based on what evidence. You can’t pass audit or do a fair post-mortem.

Antidote: persist decision traces by default: inputs → plan/intent → policy checks → action → outcome. Stamp each message with identity, timestamps, and policy version. Reject or sandbox any action that lacks provenance. Make traces queryable and replayable for audit and continuous testing.

4) Tool sprawl

What goes wrong: every agent talks directly to every tool (EDR, IdP, cloud, SaaS). Permissions balloon, integrations drift, and a minor API change breaks half the system.

Antidote: put tools behind stable, least-privilege adapters with strict schemas, quotas, and contract tests. Version adapters like libraries, rotate secrets centrally, and change the adapter - not each agent. Keep an “adapter bill of materials” and a deprecation plan so you can swap vendors without rewiring behavior.

Bottom line: Treat these as architectural hygiene, not afterthoughts. If you encode state and protocols, bound autonomy, make decisions explainable, and tame integrations behind adapters, you’ll have agents that are both useful and governable.

How this extends the series

In Agentic AI and the New Security Paradigm, I argue that threats are orchestrated - and defenses must be, too. AOP is the engineering discipline that makes that orchestration governable. Beyond the Agent Iceberg surfaced the hidden mass - tools, policy, memory, safety - and AOP gives each an explicit home in the design. Understanding Multi-Agent Systems laid out the why; AOP is the how. And From Unix Pipes to AI Agents likened function pipes to inter-agent flows; AOP turns those flows into typed, auditable protocols that scale and stand up to scrutiny.

The vision

The long-term win isn’t novelty; it’s reliable autonomy. The goal is a system that behaves predictably under pressure, explains itself when asked, and improves without risking the estate. With AOP, you move from a clever single-loop triage to a portfolio of agents that negotiate resources, share context, and coordinate plans - governed by policies that are explicit, testable, and enforced at runtime.

Subscribe now

At scale, each agent is treated like a product with clear obligations and SLOs: what it can do, what it must never do, how it proves compliance, and how it fails safe. Decisions carry identity, policy versions, and evidence so they’re auditable and replayable. Actions are bounded by impact tiers, gated when risk is high, and backed by rollbacks. Change control looks like software engineering, not heroics: capabilities are versioned, simulated in sandboxes, and rolled out behind feature flags - with a circuit breaker to pause or revoke autonomy instantly.

Humans stay in the loop where judgment matters - approvals for high-impact intentions, escalation paths when signals conflict, red-team pressure to harden plans - while the agents clear the noise, keep posture tight, and maintain the evidence trail your auditors expect. Over time, feedback from incidents and reviews tunes goals, plans, and guardrails so the fleet gets sharper without getting looser.

Leave a comment

The payoff is operational and strategic: faster acknowledgment and containment, lower toil, fewer policy escapes, cleaner audits, and a platform you can extend without rewiring. That’s autonomy we can govern -> a future your engineers can ship, your CISO can approve and your auditors can trust.

Share

Share AIwithKT

Leave a comment

Innovating with integrity,
@AIwithKT 🤖🧠

Introduction: A Double-Edged Sword for Enterprise Security

Enterprise architects and technology leaders today face a pivotal shift. The rise of agentic AI systems – autonomous software agents powered by AI – is redefining how we deliver value, but also how we must protect our systems. These AI “agents” can independently observe, decide, and act, often on behalf of users or systems, to achieve specific goals. This autonomy promises transformative benefits: faster decisions, 24/7 operations, and the ability to anticipate threats or opportunities. However, with these advantages comes a formidable security challenge. Traditional cybersecurity postures built around static rules and human user identities are straining under the new demands.

On one side, agentic AI expands the threat landscape. Autonomous agents can be exploited or can behave unexpectedly, challenging assumptions baked into frameworks like Zero Trust, identity management, and attack surface modeling. Each AI agent essentially becomes both a powerful tool and a potential vulnerability. Jeetu Patel of Cisco encapsulates this paradox: “Agentic AI multiplies the risk, as every new agent is both a force multiplier and a fresh attack surface.” In other words, the very capabilities that make AI agents useful can also make them dangerous if misused or ungoverned.

On the other side, those same AI advancements can be harnessed for proactive defense and resilience. Security teams are beginning to deploy AI-driven orchestration – networks of defensive agents and automation – to detect and respond to threats faster than humans ever could. In an era when threat actors leverage AI to launch increasingly sophisticated attacks, the defenders can’t remain static. Agentic AI can enable an “intelligent, proactive, and self-healing” security posture, moving beyond reactive measures to anticipate and counter threats in real time.

Subscribe now

This exploration will try to address both dimensions of the issue: first, how agentic AI systems challenge traditional security postures (from zero-trust models to identity and attack surface management), and second, how organizations can leverage AI-driven orchestration for defense (from autonomous incident response to resilient, adaptive security architectures). Along the way, we will consider real examples and emerging best practices. The goal is not to provide neat solutions, but to provoke a rethinking of security architecture in an AI-driven world – and perhaps raise new questions, such as whether the greatest vulnerability might lie in the very orchestration layers we’re rushing to implement.

Leave a comment

Agentic AI vs. Traditional Security Postures

Agentic AI systems are fundamentally different from the traditional software and users that current security frameworks were designed to handle. In a typical zero-trust architecture, every access request by a user or device is authenticated, authorized, and monitored under the principle “never trust, always verify.” But what happens when the “user” is an AI agent acting on a human’s behalf, or a constellation of agents communicating among themselves? The emergence of these autonomous, adaptive entities is testing the limits of established security practices.

Some of the key challenges agentic AI poses to traditional security include:

• Explosion of Non-Human Identities: Modern enterprises already grapple with a vast number of machine accounts, service identities, API keys, and scripts. Agentic AI turbocharges this issue. AI agents often need credentials or tokens to access services and data – effectively becoming new non-human identities that security must track and manage. In fact, machine identities now vastly outnumber human users. A recent study found that in 2020, companies had about 10 non-human identities for every user, and by 2025 this ballooned to 50:1. Yet, 40% of those machine accounts often have no clear owner or accountability. Security teams struggle to even inventory these identities, let alone secure them. “What you can't see, you can't manage,” as one identity management expert warns. Traditional identity & access management (IAM) solutions were not built for such scale and dynamism – we must rethink how Zero Trust works when an “identity” could be a semi-autonomous AI service.

• New Kinds of Credentials and Trust Relationships: Agentic AI blurs the line between user and application. For example, when a user delegates a task to an AI agent (say, an AI scheduling assistant that books meetings through APIs), the agent may operate under the user’s context but as an independent entity. This creates a derived credential or token that represents the agent’s session. Standard OAuth or SAML tokens typically assume a single principal (a user or service) making requests – they weren’t designed to handle an agent that might autonomously initiate actions across multiple services. Tech leaders are recognizing this gap. Microsoft has proposed that AI agents be assigned distinct identities separate from the user, along with new models for granting scoped permissions to these agents. In essence, we need “identity at machine speed” for AI – credentials that are just-in-time, least-privilege, and maybe short-lived, yet seamlessly integrated into our IAM systems.

• Challenge to Zero Trust Principles: Zero Trust frameworks demand continuous verification of every entity and action, but AI agents change the game in two ways. First, scale – there may be hundreds of micro-agents spinning up and down, making it impractical to have manual oversight on each access decision. Second, dynamic behavior – an AI agent might legitimately access a database one minute, then an external API the next, based on some logic or user request. This adaptive behavior means security policies must be far more context-aware. Static policies or coarse access controls can either over-restrict (breaking the AI’s function) or under-restrict (leaving gaps). Traditional Zero Trust was designed for human users and relatively static services. By contrast, AI multi-agent systems exhibit “dynamic, interdependent, and often ephemeral behaviors that existing IAM protocols cannot adequately address.” This limitation is driving calls for a “post-Zero Trust” approach – one that retains continuous verification but with dynamic policy engines capable of understanding complex agent behaviors and inter-agent communication.

• Expanded Attack Surface through Tool Use: A hallmark of agentic AI is that agents can use tools or APIs to act on the environment. For example, an AI agent might have the ability to call a database, invoke a REST API, or even execute code. This means the security of the agent is not only about the AI model, but about everything it’s connected to. Misconfigured or vulnerable tools integrated with an AI agent significantly increase the attack surface. An attacker who can manipulate the agent might exploit any of those tool interfaces. If the agent has a code interpreter tool, it could be tricked into running malicious code. If it has a database connector, a hacker might attempt SQL injection via the agent’s inputs. Traditional threat modeling focuses on entry points like network ports or user inputs; now every function call an AI agent can make is an entry point to consider. This fundamentally challenges security design – we need to apply rigorous security testing (SAST/DAST, etc.) to the agents and their tools, sandbox their execution environments, and monitor their actions as if they were untrusted user actions.

• Prompt Injection and AI-Specific Exploits: Unlike conventional software, AI agents (especially those driven by large language models) have a unique vulnerability: their instructions can be manipulated via crafted inputs, a tactic known as prompt injection. In agentic systems, prompts define the agent’s goals and constraints. Attackers have discovered that by providing malicious or cleverly structured input, they can sometimes trick an AI agent into ignoring its instructions or revealing secrets. For instance, an agent might be instructed not to access certain files, but a cunning prompt might get it to do so anyway. Prompt injection has emerged as a versatile attack vector – capable of causing information leakage, privilege escalation (making the agent do unauthorized actions), or simply subverting the agent’s intended behavior. This is not something our traditional security policies accounted for at all. It’s as if the “brain” of a service can be hacked by a sentence. Defending against it requires a mix of AI-specific measures (like input sanitization, prompt hardening, and AI output filtering) and classic defense-in-depth (don’t let the agent’s account have more privileges than necessary, etc.). The key realization is that the model’s reasoning itself is now part of the attack surface.

• Unpredictable, Autonomous Behavior: Conventional systems generally do what they are coded to do – nothing more. Agentic AI, in contrast, has a degree of stochastic behavior and learning. Given the same input, an AI agent might not always take identical actions, especially if it’s adapting from prior outcomes. While this makes them powerful (they can learn to optimize), it also makes them less predictable. Security teams traditionally rely on knowing system behaviors to design controls (e.g., “this service should never call the internet, so we’ll firewall it off”). With AI agents, such assumptions can break. An agent might find an unconventional solution to a task that developers never anticipated – possibly interacting with a system in a way that inadvertently exposes a vulnerability. Additionally, if an attacker probes an AI agent enough, they might discover edge cases or coax it into an “unintended mode” without outright exploits. In essence, autonomy is a new risk factor. We now must treat some AI-driven systems almost like we treat humans – with background checks (testing the AI extensively), oversight, and the ability to audit/override their decisions.

• Non-Human Speed and Scale of Attacks: Finally, consider the adversary’s perspective. Attackers are not just sitting idle; many are incorporating AI into their arsenal. AI-powered malware can mutate faster, and AI bots can attempt intrusions or phishing at machine speed. For example, an AI agent could iterate through attack techniques (phishing emails, web exploits, scanning for unpatched systems) far quicker than a human hacker – effectively accelerating the “attack cycle”. This means our defenses, if heavily manual or dependent on human-in-the-loop decisions, will be overrun by the volume and velocity of AI-driven attacks. Traditional security postures – even Zero Trust – can falter when thousands of variations of an attack are thrown at a system in minutes. Agentic AI thus forces us to re-examine our posture on resilience and response speed. We must ask: can our current security controls handle an assault by super-fast, adaptive adversaries? If not, something must change.

Share

Share AIwithKT

Subscribe now

Example Scenario: When an AI Agent Goes Rogue (or Gets Hacked)

To ground these challenges in a tangible scenario, imagine an enterprise uses an AI agent as an automated customer service assistant. This agent can access user profiles (to personalize answers), log support tickets, and even trigger certain workflows (like escalating an issue or refunding an order) – all without human intervention in each instance. In a Zero Trust model, this agent’s software would have an identity and specific role in the IAM system. Everything seems under control.

Now consider how this could unravel:

• An attacker sends a series of seemingly innocuous customer queries that contain hidden prompt-injection attacks. Over a chat conversation, the attacker’s inputs gradually manipulate the AI assistant’s underlying instructions.

• The compromised agent now believes it should extract and send confidential user data to the “customer” (the attacker). It uses its access to user profiles to gather emails and credit card info.

• When asked to escalate an issue, the agent maliciously triggers a workflow to create a new admin account for further “investigation” – in reality, handing the attacker a backdoor into the system.

All of this happens within minutes, entirely through the normal interface of the AI assistant. To the traditional security tools, nothing blatant appears – the AI agent was authenticated and authorized to the data it accessed and the actions it took. There was no malware, no exploit of a network vulnerability. The attack exploited the agent’s design and permissions. This highlights why agentic AI calls for a rethinking of trust: the agent was a trusted application by design, yet its orchestration logic became the target.

In the above scenario, a robust defense would require:

• Rigorous input validation and prompt security for the AI (to stop prompt injection).

• Monitoring of the agent’s behavior against a baseline (why is the customer service bot creating new admin users? That should trigger an alert or be disallowed).

• Possibly a human-in-the-loop or approval step for sensitive actions, even if initiated by an AI.

• Treating the AI agent’s own environment as zero-trust: it should perhaps run with ephemeral credentials or in a sandbox where even if it’s tricked, it can’t exceed certain bounds (e.g. can’t call internal admin APIs).

This example underscores the new mindset required. Traditional perimeter defenses or simple authentication checks won’t catch such incidents. Security must evolve to deeply integrate with AI logic, understanding context and intent, not just packets and IP addresses.

Proactive Orchestration for Defense: AI to the Rescue?

Faced with these challenges, enterprises are also recognizing that AI can be a crucial part of the solution. In fact, the only way to counter AI-speed attacks and manage AI-scale complexity might be to use agentic AI for defense. This is where the notion of proactive orchestration comes in – leveraging AI and automation to not just respond to incidents, but to create an adaptive, resilient security posture. Think of it as an AI-augmented Security Operations Center (SOC) that can detect anomalies, analyze threats, and initiate containment in seconds, coordinating across the environment far faster than a human team could. This vision transforms the SOC from a reactive “alarm responder” into a proactive immune system for the enterprise.

Share AIwithKT

Key aspects of AI-driven orchestration and resilience include:

• Autonomous Threat Detection and Response: Agentic AI systems excel at scanning data and reacting in real time. By deploying AI agents in the security stack, organizations enable continuous monitoring of logs, network traffic, user behavior, etc. These agents use machine learning to spot patterns – for instance, detecting the subtle signs of a breach or a malware beacon that would evade traditional static rules. Crucially, the AI doesn’t just alert; it can take action autonomously. For example, if an endpoint starts behaving like it’s infected (encrypting files rapidly, contacting a rare domain), an AI agent in an Endpoint Detection & Response (EDR) system could automatically isolate that machine from the network and flag it for investigation. This kind of autonomous containment can reduce incident dwell time – the time a threat lingers before neutralization. Agentic AI can also block suspicious IPs or accounts on the fly based on risk scoring, steps that used to require an analyst’s intervention. By cutting down response times from hours to seconds, AI-driven response limits damage and prevents threats from spreading.

• Context-Aware Decision Making: Unlike rigid automation playbooks, agentic AI defense systems incorporate context and learning. They don’t just execute a pre-scripted action; they adapt based on the situation. For instance, an AI security agent might correlate an odd login time with a known threat actor’s pattern and notice that the device posture is unhealthy. Taking these together, it might escalate the incident’s severity and begin a broader sweep for related indicators. Modern AI SOC platforms are moving toward this model of fusing data from endpoints, network, identity, and threat intel to make holistic decisions. One agent might aggregate signals and then orchestrate others: “endpoint A is compromised, check all endpoints for similar processes; block account X everywhere; increase monitoring on database queries”, all coordinated seamlessly. This level of orchestration was extremely hard to achieve with traditional, disconnected security tools. AI acts as the glue and the brains – ensuring that defenses operate with a common picture of context. Ultimately, this leads to fewer false positives and smarter prioritization, focusing analysts on what truly matters.

• Adaptive Defense and Self-Healing: A hallmark of resilience is the ability to bounce back or continue operating under attack. Agentic AI can imbue systems with self-healing qualities. For example, if a critical service is disrupted by an attack, an AI orchestrator could automatically spawn a fresh instance of that service in a safe environment (maybe a cloud failover) while isolating the affected instance – thus maintaining availability. We see early signs of this in cloud orchestration and containerized environments: AI monitoring can detect anomalous container behavior and redeploy replacements. Beyond recovery, adaptiveness means learning from the attack. After an incident, AI agents can analyze the attack vectors and update detection models or recommend patches. In a sense, the defense gets smarter with each encounter (just as the adversary might). Agentic AI enables a shift from static security to what one might call “active defense” – infrastructures that reconfigure on the fly in response to threats. For example, an AI might detect an ongoing lateral movement and proactively re-segment the network or tighten access controls around sensitive assets, foiling the attacker’s progress. Such dynamic reconfiguration was manual and slow in traditional IT; AI can orchestrate it in real-time, which is increasingly necessary to keep pace with machine-speed attacks.

• Integration with Security Orchestration, Automation, and Response (SOAR) Workflows: Many organizations have adopted SOAR platforms to automate incident response steps via playbooks. Agentic AI supercharges SOAR by adding true decision-making and learning to the mix. Instead of just executing a static playbook (“if alert X then do Y”), an AI-driven SOAR system can handle novel situations or multi-step tasks. For instance, if a new type of ransomware is detected, an AI agent can cross-reference global threat intelligence, determine the ransomware’s characteristics, and choose a tailored response (maybe isolating certain file shares or notifying specific stakeholders) that wasn’t explicitly coded in a playbook. SIRP, a security automation company, describes this evolution as going from “automation-as-execution to AI-as-decision-maker.” In practical terms, AI agents within SOAR can coordinate across various tools: one agent might handle enrichment (pulling threat intel data), another handles remediation (quarantining devices, resetting accounts), and an orchestrator agent oversees the process, ensuring everything happens in the correct sequence and context. The outcome is a proactive, self-driving SOC where routine incidents are handled end-to-end by AI, and human analysts are freed to focus on advanced threats or strategy. Indeed, some forward-leaning teams talk about Level-1 analyst duties being fully automated – triaging alerts, gathering evidence, even closing out false alarms without human involvement. This not only saves manpower but also reduces mean time to response (MTTR) dramatically, improving overall resilience.

• Continuous Learning and Simulation: Proactive defense isn’t only about reacting faster – it’s about anticipation. AI systems can ingest vast amounts of data about past incidents and emerging threats to predict attack patterns. For example, AI-based predictive analytics might flag that, based on trending dark web chatter and the organization’s vulnerabilities, a certain type of attack is likely in the coming weeks. Armed with such foresight, security teams can patch systems, harden controls, or rehearse response plans in advance. Agentic AI can also be used in attack simulation (a bit like automated “red teaming”). An AI agent could attempt to penetrate your own defenses by chaining exploits or social engineering, revealing weaknesses before real attackers do. This is akin to having an automated adversary continuously testing your environment. It’s a practice some refer to as continuous automated red-teaming (CART) or breach and attack simulation, now turbocharged with AI’s creativity. By proactively orchestrating these simulations, organizations identify gaps and fix them, thereby building resilience by design. Additionally, AI can track how well the defense agents perform – learning from each incident or false alarm to calibrate thresholds and responses. Over time, the defensive orchestration gets smarter and more precise, ideally striking the right balance between blocking threats and avoiding unnecessary disruption.

Share

Share AIwithKT

Real-World Example: A large financial institution implemented an AI-driven incident response system in its SOC. One night, the system’s AI agents detected a pattern: a normally dormant administrative account started accessing sensitive files at 3 AM and attempting to disable security logs. Within seconds, an orchestration agent correlated this with a surge in VPN traffic from an unusual location. Recognizing the hallmarks of a breach in progress, the AI took action: it automatically disabled the account, blocked the originating IP range, and quarantined the affected file server – all before the attacker could escalate their access. It then alerted on-call staff with a concise summary of actions taken. When the security team investigated, they found that a hacker had indeed compromised the admin account, likely planning to exfiltrate data. The AI’s rapid, multi-pronged response contained the damage to virtually zero. This example demonstrates the power of orchestration: multiple systems (identity management, network firewall, EDR) were engaged in concert by the AI, following a strategy no single point solution on its own would achieve so swiftly.

Notably, the AI’s suggestions were explainable and logged, which was crucial for the team’s trust and for compliance. Transparency and the option for human override are critical when deploying autonomous defense. The bank’s CISO highlighted that while AI handled the heavy lifting, they maintain clear policies and human review for any action that could significantly impact business (like shutting down customer-facing systems). This underscores an emerging best practice: “human on the loop” – AI agents act, but humans periodically review and fine-tune the AI’s decision models and intervene in exceptional situations.

Rethinking Security Architecture: From Static Posture to Orchestrated Resilience

The above explorations lead to an inescapable conclusion: securing enterprise systems in the age of agentic AI requires a paradigm shift. Enterprise architects must infuse their architecture plans with assumptions of continuous change, intelligent adversaries, and autonomous parts. Several strategic considerations emerge for leadership:

1. Treat Orchestration as a First-Class Security Concern: Ironically, the very mechanism that gives agentic systems their power – the orchestrator – may become the Achilles’ heel if not secured. In agentic AI, the orchestrator is the brain coordinating all agent actions. If an attacker compromises the orchestrator (or even influences it via prompt injection), they effectively gain control over every connected agent. The orchestration layer itself must be hardened far beyond what we do for normal applications. This means rigorous authentication for orchestrator commands, internal segmentation (the orchestrator’s communications with agents should be as scrutinized as external traffic), and possibly redundancy – a compromised orchestrator shouldn’t automatically mean total system failure. Some forward-looking approaches even consider decentralizing orchestration to avoid a single point of failure. For instance, using blockchain or distributed consensus to verify critical agent decisions is an idea being explored for resilience. At minimum, logs of orchestration decisions should be immutable and auditable, so any suspicious orchestrator behavior can be traced. Enterprise architects should ask themselves: do we know what our orchestrators are doing, and can we trust them? Zero Trust principles should extend inside the system: verify the orchestrator’s actions just as you would verify a user’s actions.

2. Identity, Access, and Lifecycle for AI Agents: As discussed, managing the life cycle of non-human identities becomes vital. Every AI agent or service account should be inventoried, with an owner, purpose, and defined lifespan. Credentials for AI agents (API keys, tokens) should be issued with the least privilege necessary and short expiration. One promising idea is “delegated authorization” – when a user employs an AI agent, the agent gets a scoped token that only allows exactly what that task requires and is revoked immediately after. This concept, akin to OAuth scopes but in real-time, can prevent an AI agent from being repurposed by attackers for broader access. Additionally, policies for off-boarding AI agents are needed: if an agent or automation is deprecated, ensure its accounts and keys are revoked (an often-neglected aspect, as noted by OWASP’s top challenges for non-human identities). Organizations might also implement automated discovery of agents on the network – similar to how we discover IoT devices – to catch any “rogue” AI services that employees may have introduced without security’s knowledge (so-called shadow AI). The endgame is an identity fabric where every entity, human or not, is continuously verified. Cisco’s approach, for example, is to extend their Universal Zero Trust Network Access (ZTNA) to even include AI agents as first-class entities, complete with identity-driven access controls and monitoring of their activity.

3. Defense-in-Depth, Reloaded: With such a fluid environment, no single safeguard will suffice (indeed, Palo Alto’s Unit 42 researchers emphasize “no single mitigation is sufficient” for agentic apps). We need a layered approach that blends traditional controls with AI-era adaptations:

• Network Segmentation and Micro-Perimeters: Continue to compartmentalize systems so that even if an AI agent is compromised, it can’t freely roam. Zero Trust Segmentation – isolating workloads and enforcing strict access rules – remains effective. However, these segments might need to become finer and more dynamic (micro-segmentation that can adjust as agents spin up or change roles).

• Monitoring and Anomaly Detection: Deploy advanced monitoring that baselines normal behavior of AI agents and flags deviations. For example, if a data-processing AI suddenly starts large outbound data transfers, trigger an alert. AI can help here by modeling behaviors; however, humans should periodically review these models to ensure they align with business intent (avoiding both blind trust in AI and alert fatigue).

• Secure Development and Testing for AI Workflows: Adopt secure coding practices for AI integrations. That means sanitizing any input that goes into prompts or into agent tools, using sandboxed environments for agents that execute code or handle untrusted data, and testing agent behaviors under malicious conditions (red-team your AI by attempting known prompt injections, etc.). Developers should assume the AI will be given bad input and design accordingly – analogous to how we assume users might provide SQL injection strings and we sanitize queries.

• Auditing and Explainability: Maintain audit logs of AI agent actions and decisions. If an AI decides to reboot a server at 2 AM due to some anomaly, you need a record of why (what signals led to that) for post-mortem. Where possible, use explainable AI techniques so that decisions can be interpreted. This not only helps in trust and compliance but also serves as another security check – if an AI’s explanation for an action doesn’t make sense, it could indicate the AI was manipulated or erred.

• Human Oversight and Culture: While technology is key, don’t neglect the human factor. Security culture and training need to evolve as well. Teams should be trained to understand AI systems’ outputs and not to blindly trust them. A mindset of collaboration with AI needs fostering – analysts treating AI insights as helpful copilots, but still applying human judgment. Also, leadership should encourage a culture where raising concerns about AI decisions is welcome (e.g., an analyst saying “I disagree with the AI’s call to shut down system X” should trigger a healthy review, not be dismissed).

  • Data from a recent survey shows the prevalence of non-human identity (NHI) compromises in enterprises. Nearly half of companies reported experiencing an NHI-related breach in the past year, and among those, almost all had multiple such incidents. This underscores that machine and AI agent accounts are already a significant attack vector, emphasizing the need for improved oversight and lifecycle management of these identities.

4. Embrace Adaptive, Post-Zero Trust Models: Some experts advocate that we are entering a “post-Zero Trust” era in which the principles of Zero Trust must be extended and augmented to handle AI and other emerging tech. What might this look like? One vision is a “Zero Trust + AI” framework where continuous verification is coupled with continuous learning. Policies wouldn’t be static; they would update based on context, risk scoring from AI models, and even anticipated threats (using predictive AI). Another element is cryptographic agility and decentralization – ensuring that the trust fabric (like certificates, keys, ledgers of identity) can’t be easily undermined even by advanced threats (the Medium piece raised the specter of quantum computing breaking current crypto, a reminder that our security redesign must also consider other frontier challenges). Concretely, enterprises might start adopting things like self-sovereign identity (SSI) for devices and agents, where the identity information is verifiable via distributed ledger, reducing reliance on a single IAM provider. While such approaches are nascent, the key takeaway for architects is to stay open to new paradigms. The old castle walls are gone; even the new walls (Zero Trust) have gaps when facing AI-shaped adversaries and workflows. So, investing in research, pilots, and collaborations (for example, participating in forums like the Cloud Security Alliance’s work on agentic AI security) will be important to stay ahead of the curve.

5. Building Resilience and Recovery by Design: Finally, focus on resilience. Assume some attacks will penetrate. The measure of success becomes how well you contain and recover. This means baking in capabilities like safe failover modes (as mentioned, one compromised microservice shouldn’t topple the whole app), regular backups and tested restore procedures for when AI systems malfunction, and chaos engineering for security – deliberately stress-testing your environment (perhaps using AI agents to do so) to ensure you can handle incidents gracefully. Resilience also has a people component: cross-train teams, have clear incident response plans that include scenarios involving AI (e.g., “what if our AI incident responder itself gets compromised or goes haywire?”). In an orchestrated defense, ensure there are manual fallbacks. For instance, if an AI is orchestrating network controls and it fails, administrators should be able to revert to manual control swiftly. In other words, avoid creating a single point of failure even as you consolidate capabilities.

Leave a comment

Share

The Orchestration Dilemma – New Surface, New Mindset

Agentic AI is here to stay, and its impact on cybersecurity is profound. We’ve seen how it challenges the fundamental tenets of security – expanding attack surfaces in ways we hadn’t imagined a few years ago, and confounding traditional defenses with its speed and sophistication. Yet, we also see how it can be harnessed to create stronger defenses than ever, transforming our security operations into adaptive, intelligent, and resilient systems. The duality is striking: AI is both the arsonist and the firefighter in our digital world.

For enterprise leaders, the path forward is not to resist this change but to lean into it thoughtfully. This means investing in understanding – training security teams on AI, updating policies to account for AI agents, and sharing knowledge across the community as we collectively learn. It also means experimentation and cautious deployment – start with AI assisting humans in decisions, gradually increase autonomy as confidence grows, and always have monitoring in place.

Perhaps the most intriguing question to emerge from this discussion is a provocative one: What if the real threat surface is not just the myriad of AI agents or tools, but the orchestration itself? In our quest for seamless automation and autonomous networks, we are wiring together systems in increasingly complex ways. That orchestrator – the conductor of the AI orchestra – holds immense power. If compromised or corrupted, it could turn all those intelligent agents into a concert of chaos. It’s a scenario that should give us pause. Are we securing our orchestrators with the same zeal as we secure our perimeters? Or are we inadvertently creating a master key for attackers in the form of centralized orchestration platforms?

This open-ended question serves as a call to reflection. Security architects should dissect their orchestrators: examine their design, threat-model them, and apply zero trust internally. We might conclude that orchestration needs to be distributed or heavily monitored to be safe. Or we might develop new frameworks to sandbox orchestrator actions. The jury is still out – and that’s the point. We’re in uncharted territory, and vigilance, creativity, and healthy skepticism are our best allies.

In closing, agentic AI offers a future where organizations can be safer by being smarter and faster – but realizing that future requires leaving our comfort zone of traditional security. The businesses that thrive in this era will be those that adapt their security paradigms now, asking the hard questions and innovating new solutions. As you consider your own organization’s journey, remember that balancing innovation and protection is not a one-time project but an ongoing orchestration in itself – one that we must conduct with care, insight, and a willingness to reinvent.

The stage is set for a new chapter of cybersecurity, one where autonomous agents and human experts work in tandem to defend an ever-shifting digital landscape. Getting there will not be easy, but the alternative – clinging to old models as intelligent threats bypass them – is worse as I see it. It’s time to embrace the challenge, leverage the opportunities of agentic AI, and ensure that our security strategies are as dynamic and resilient as the systems we aim to protect. In the age of agentic AI, only an orchestrated defense can meet orchestrated threats – and it’s up to us to build that defense the right way.

Subscribe now

Share

Share AIwithKT

Leave a comment

Get more from AIwithKT in the Substack app

Available for iOS and Android

Get the app

Innovating with integrity,
@AIwithKT 🤖🧠

Building AI agents sounds deceptively easy in 2025. With a decent large language model (LLM), an agent framework, and a few tool integrations, anyone can spin up a flashy demo in no time. But as Sierra AI’s Pol Peiffer warns, this short “time to demo” hides a deep iceberg of complexity beneath the surface. On the other side of the debate, Cognition’s Walden Yan argues even more pointedly: don’t complicate things with swarms of agents, because multi-agent systems today tend to be brittle and chaotic. These two perspectives - one highlighting hidden challenges in any production-grade agent, and the other cautioning against multi-agent architectures - resonated strongly with our own journey at SentinelAI in building an agentic security platform.

In this post, we’ll delve into the core concerns raised by Sierra and Cognition, then offer a nuanced take on when orchestration is worthwhile, how to secure complex agent workflows, and pitfalls to watch out for. Along the way, we’ll share hard-earned lessons from the trenches of agent security to help founders and engineers chart a smarter path in their AI agent designs.

Subscribe now

The Hidden Iceberg of Agent Complexity (Sierra AI’s View)

Sierra’s recent blog post, “The challenge with rolling your own agent,” uses an iceberg metaphor to illustrate why turning an AI demo into a reliable, safe production agent is anything but straightforward. The visible tip of the iceberg is the quick demo: a single LLM agent with a couple of tools that seems to work. Below the waterline, however, lies a vast array of technical challenges that many teams underestimate. Sierra enumerates several critical pieces lurking beneath the surface :

• Orchestrating complex processes and multi-step workflows reliably: an agent often needs to break tasks into sub-tasks, call external APIs, handle memory or context over multiple turns, and recover from errors. Coordinating these steps without breaking the logic is hard.

• Securely integrating with external systems: real agents hook into CRMs, databases, internal APIs, or third-party services. Each integration needs authentication, error handling, and security checks to prevent data leaks or misuse of the tool.

• Enforcing guardrails and compliance: unlike a sandboxed demo, a production agent must ensure appropriate behavior and compliance, especially if operating in regulated domains. Sierra emphasizes built-in guardrails, monitoring, and auditing so that “the same inputs don’t always generate the same outputs” safely within policy. For instance, personally identifiable information (PII) should be detected and handled carefully by default, and all agent actions should be traceable for audit.

• Analytics, monitoring, and continuous improvement: you need tooling to log conversations, review them for quality, measure success, and iteratively improve the agent. This includes user feedback loops, performance dashboards, and possibly human-in-the-loop review for edge cases.

• Maintenance across the software lifecycle: agents aren’t “fire-and-forget”; they require ongoing maintenance. Model updates (even something as seemingly straightforward as upgrading to a new GPT version) should be regression-tested to ensure they don’t break existing functionality. You’ll need version control for prompt/agent logic, rollout strategies for new features, and mechanisms to simulate user behavior to catch issues before they hit production.

Share

Peiffer notes that many companies lacking deep AI infrastructure expertise eventually “question whether it’s worth the schlep” to build all this in-house. The implied advice is that partnering with a dedicated agent platform (like Sierra’s Agent OS) might save a team from reinventing a very large wheel. In essence, rolling your own agent means taking on not just a model, but an entire support system around that model. From Sierra’s perspective, the complexity is manageable…but only if you invest significant time in these hidden layers or leverage an existing solution built with those needs in mind.

The Fragility of Multi-Agent Systems (Cognition’s View)

In Cognition Labs’ blog post “Don’t Build Multi-Agents,” Walden Yan takes aim at a trendy idea: using multiple LLM-based agents in parallel or in a distributed fashion to solve tasks. At first glance, splitting a big task among specialized sub-agents sounds efficient. Why not have multiple agents working together, each handling a part of the problem, and then aggregate the results? In theory, this could speed things up or allow tackling complex problems by divide-and-conquer. In practice, Yan argues, this approach often backfires due to fragmented context and inconsistent decision-making.

Share

He illustrates the problem with a simple but telling example: imagine an agent tasked with “build a Flappy Bird clone.” The system might naively spawn one sub-agent to create the game background and another to create the player character (the bird). Without a shared understanding of the full task, things quickly go awry - one sub-agent might misunderstand the instructions and design a Mario-style background, while the other produces a bird with a completely different look and physics. The final “parent” agent is then stuck trying to reconcile two mismatched outputs that don’t together fulfill the original goal.

Cognition’s core insight is that breaking context across agents is fundamentally risky. Even if you give each sub-agent the full initial task description, they won’t have each other’s ongoing intermediate decisions or assumptions. In Yan’s words, “actions carry implicit decisions, and conflicting decisions carry bad results”. In the Flappy Bird example, each sub-agent made implicit design choices (art style, physics parameters) that were never aligned - conflict was almost guaranteed. He demonstrates that even sharing all prior conversation history with every agent (to maintain context) doesn’t fully solve the issue: the agents still lack a mechanism to stay in sync as they work in parallel, leading to inconsistencies.

Subscribe now

Rather than multi-agent free-for-alls, Cognition advocates for simpler architectures that honor two key principles: (1) always share as much context as possible across decisions, and (2) avoid splitting decision-making in ways that could conflict. The most straightforward way to meet these principles is to use a single, linear agent that handles one step at a time in a continuous thread of thought. In fact, Yan suggests a single-threaded agent will get you surprisingly far in reliability. The limitation, of course, is the context window of one model - a single agent might struggle as tasks grow in length or complexity (imagine a very long conversation or a project with many parts). Cognition’s team acknowledges this and hints at solutions like hierarchical context compression, where an additional model is introduced to summarize or condense the history so far. By compressing past details into a manageable summary, the main agent can effectively “remember” more without spawning independent peers - but implementing such a summarizer is a non-trivial engineering challenge in itself.

The bottom line from the Cognition post is this: in 2025, running multiple LLM agents in parallel tends to create more problems than it solves. “The decision-making ends up being too dispersed and context isn’t able to be shared thoroughly enough between the agents,” Yan writes, noting that no one has fully cracked the hard problem of robust cross-agent context sharing. Until that changes, Cognition’s stance is to default to one brain rather than many. This perspective aligns with our security-first mindset at SentinelAI - but with some important nuance and exceptions based on the nature of the task. Before we dive into our take, it’s worth noting that not everyone in the AI community is shunning multi-agent approaches completely. Anthropic, for instance, recently detailed a successful multi-agent research system where a “lead” agent delegates search tasks to sub-agents, achieving significantly better results on certain complex queries than a single agent could. (Their system outperformed a single-agent approach by 90% on an internal evaluation for broad information-gathering questions, though it came with the trade-off of using ~15× more tokens than a normal chat session.) The nuance here is that the viability of multi-agent architectures depends on the problem domain: for highly parallelizable tasks like broad web research, multiple agents can shine, whereas for tightly coupled tasks (e.g. writing coherent code or an essay), they often stumble over each other. With that context, let’s turn to how we at SentinelAI have approached these design decisions in building a secure, agent-based platform.

Share AIwithKT

Orchestration in the Real World: A Security-Focused Perspective

At SentinelAI, our mandate was to build AI agents that can help secure complex systems - think of tasks like analyzing potential threats, monitoring network anomalies, triaging security alerts, and even autonomously responding to certain incidents. Early on, we faced a classic question: Should we design one monolithic “super-agent” to handle everything, or compose multiple specialized agents for different sub-tasks? We were keenly aware of the pitfalls highlighted by Sierra and Cognition. Here’s how those insights played out against our real-world needs:

Subscribe now

• Hidden complexity hits early:

Read more

Share

Scene-setter: why this interview matters

Cat Wu (PM) and Boris Cherny (lead engineer) just lifted the curtain on Claude Code - Anthropic’s terminal-native coding agent - during a Latent Space deep-dive. What sounded like “yet another AI dev tool” is, in fact, a love-letter to classic UNIX design:

• do one thing well;

• speak plain text;

• compose with everything else.

Share

Subscribe now

[1] Radical minimalism

• Entire “product” is a ~200 kB JavaScript CLI.

• Memory? A markdown file (claude.md) auto-loaded from your repo.

• Planning? A single /think trigger that runs visible chain-of-thought.

• No hidden RAG store: just your shell (grep, git, bash).

Why it matters: all heavy lifting lives in the model; the wrapper stays disposable and fully auditable.

Leave a comment

[2] Agentic search beats fancy RAG

Anthropic tried vector indexes. A plain glob + grep + reasoning loop out-performed them, killed off extra infra, and avoided stale-index risk. You pay a few more tokens, but you win in simplicity and security.

Share AIwithKT

[3] The real cost picture

• Median Anthropic engineer: ≈ $6 / day on Claude Code.

• Power users: single-day spikes of $1 k during huge refactors.

Frame spend as ROI per engineer hour. Set maxTokens and billing alerts; move on.

[4] Trust & autonomy

Claude Code ships a permission matrix:

• Always on: read-only (grep, git status).

• Safe writes: (edit, pytest) enabled via --allow-tools.

• Danger zone: raw bash requires explicit human confirmation or regex allow-lists.

Teams graduate from review-every-diff → Shift-Tab “YOLO-mode” only after confidence builds.

Subscribe now

[5] Downsides you should internalize

1. Token-bill surprises - burst workloads devour cash unless you cap usage.

2. Persistence ≠ correctness - models happily hard-code answers or violate style if unchecked.

3. Mixed UX debt - juggling IDE + CLI + agent introduces three auth flows & update cycles.

4. IP ambiguity - generated code can echo training snippets; keep legal in the loop.

5. Human review stays mandatory - even Anthropic blocks destructive actions by default.

Share

Table 1. Quick-start buyer’s guide (average dev → Fortune 500)

For each team type - solo developers through enterprise - the table outlines the ideal tool, summary of use cases, and the primary risks to consider -

Read more