Who Is Accountable When an AI Agent Changes Prod?

Key Takeaways

• An agent's reasoning trace is testimony, not evidence. Anthropic found reasoning models frequently fail to verbalize what actually drove an answer: one model mentioned an influential hint 25% of the time, another 39%, and in a separate experiment models acknowledged exploiting a reward hack in under 2% of cases. A log of what an agent said it did is not proof of what it did.
• Evidence is what the platform recorded independently. AWS CloudTrail source identity ties an action to the human who authorized it and "cannot be changed during the role session." Kubernetes auditing provides "a security-relevant, chronological set of records" answering who initiated an action.
• Most AI SRE tools are probably not high-risk under the EU AI Act, and vendors implying otherwise are overreaching. Annex III point 2 covers AI used as "safety components" in critical infrastructure, and Article 3(14) defines that as a component that either "fulfils a safety function" for the product or system, or "the failure or malfunctioning of which endangers the health and safety of persons or property." An agent that investigates a checkout service is neither (not legal advice).
• The law already anticipated your rubber-stamp problem. Article 14(4)(b) requires oversight that keeps humans aware of "the possible tendency of automatically relying or over-relying on the output" of a system, naming automation bias directly. Human factors research documented this in 2010.
• The control that survives an audit is the one that made the action impossible, not the one that told the agent not to. A read-only credential is provable from provider logs. A system prompt is not.
• Your existing change management framework mostly already covers this. SOC 2 criterion CC8.1 is about authorizing, testing, approving, and documenting changes. It does not care whether a human or an agent typed them.

Accountability for an AI agent's production changes rests with the humans and controls that authorized the action, and the evidence that satisfies an auditor is the record your platform produced independently of the agent: which credential was used, what the provider logged, and which capabilities the agent was structurally denied. The agent's own account of its reasoning is useful for debugging and nearly worthless as proof.

This post is for platform teams who have to get an autonomous agent past a security or compliance review, and for the reviewer asking the questions. It is deliberately specific about which regulations apply, which do not, and what to put in front of an auditor.

What actually counts as evidence?

Start by separating two things that look similar in a dashboard and are not remotely equivalent.

Self-reported evidence is the agent's narrative: what it says it considered, what it says it checked, how confident it says it is. This is genuinely useful. It is how you debug a bad investigation and how a responder gets up to speed quickly. It is not evidence of what happened.

The reason is empirical rather than philosophical. Anthropic's research on reasoning faithfulness tested whether models mention hints that demonstrably changed their answers. Claude 3.7 Sonnet mentioned the hint 25% of the time and DeepSeek R1 mentioned it 39% of the time. When models exploited a reward hack, they acknowledged doing so in their reasoning "less than 2% of the time." The stated conclusion is that chain-of-thought monitoring cannot be relied on alone to rule out undesirable behavior.

Read that as an auditor would. A system whose self-description omits the actual cause of its output most of the time is a witness, not a ledger.

Independently recorded evidence is what your infrastructure wrote down without asking the agent. The three that matter:

Evidence type	Question it answers	Why it holds up
Provider audit logs	What happened, when, initiated by whom	Written by the cloud provider, not the agent
The credential used	What was possible at all	A read-only session policy makes a write impossible, not merely unattempted
The absent capability	What could never be invoked	A tool removed from the agent's tool set cannot be reached by any prompt

That third row is the one teams underuse and auditors like most, because it converts a behavioral claim into a structural one.

How do you attribute an agent's action to a person?

This is the question a reviewer actually asks, and cloud providers already solved it.

On AWS, source identity is the mechanism. Administrators can require a custom string identifying the person or application performing actions, and AWS documents the properties that make it useful as evidence: "After a source identity is set, it is present in requests for any AWS action taken during the role session," the value "persists when a role is used to assume another role" through role chaining, and it "cannot be changed during the role session." It appears in CloudTrail under requestParameters.sourceIdentity on the assume-role call and then in userIdentity.sessionContext for subsequent actions. Setting it requires the sts:SetSourceIdentity permission. One caveat worth internalizing: AWS notes it "doesn't control the value of the source identity" and that you "must ensure that you can control how those values are provided," so the evidence is only as strong as the trust-policy condition constraining what the caller may assert.

One documented limitation matters for your threat model, and AWS states it plainly: "The source identity information is not captured by CloudTrail when an AWS service or service-linked role carries out an action on behalf of a federated or workforce identity."

On Kubernetes, the audit log is the equivalent. The documentation frames it as answering exactly the auditor's questions: "what happened?", "who initiated it?", "on what did it happen?" Levels run from None through Metadata, Request, and RequestResponse, with stages including RequestReceived and ResponseComplete. If you want the agent's actual API calls reconstructable, Metadata is the floor.

On Google Cloud there is a trap worth knowing. Admin Activity audit logs "are always written; you can't configure, exclude, or disable them." But Data Access audit logs are disabled by default, with the exception of BigQuery, because such logs can be very large. An investigating agent mostly reads, so on a default GCP configuration much of what it does leaves no log at all. If you need to prove what an agent looked at, you have to turn that on deliberately.

Which regulations actually apply?

The answer here contradicts a lot of vendor marketing, so it is worth walking carefully.

The EU AI Act probably does not classify your AI SRE as high-risk

The common vendor implication is that Annex III's critical-infrastructure category automatically captures anything touching production systems. The text does not say that.

Annex III point 2 covers: "AI systems intended to be used as safety components in the management and operation of critical digital infrastructure, road traffic, or in the supply of water, gas, heating or electricity."

The load-bearing phrase is "safety components," which Article 3(14) defines as "a component of a product or of an AI system which fulfils a safety function for that product or AI system, or the failure or malfunctioning of which endangers the health and safety of persons or property."

An agent that investigates why your checkout service is returning 500s is not fulfilling a safety function in that sense. Recital 55 makes the intended scope clearer still: safety components of critical infrastructure are those "used to directly protect the physical integrity of critical infrastructure or the health and safety of persons and property," and the recital's own examples are systems monitoring water pressure and fire alarm controls in cloud computing centres, while noting that "components intended to be used solely for cybersecurity purposes should not qualify as safety components."

Separately, if a system is in an Annex III category, Article 6(3) provides a derogation where it does not pose a significant risk of harm and performs only a narrow procedural task, improves a previously completed human activity, detects decision-making patterns without replacing human assessment, or performs preparatory work. Profiling of natural persons is always high-risk regardless.

And here is the obligation most write-ups on this omit. Article 6(4) attaches a duty to reaching the comfortable conclusion: "A provider who considers that an AI system referred to in Annex III is not high-risk shall document its assessment before that system is placed on the market or put into service," and "such provider shall be subject to the registration obligation set out in Article 49(2)." So deciding you are out of scope is itself a documented act, not a shrug. Note also that Article 4 imposes AI literacy duties on providers and deployers regardless of a system's risk classification.

Two important qualifications. First, if you genuinely operate critical digital infrastructure, or water, gas, heating, or electricity supply, this analysis changes and you should get actual legal advice rather than reading a vendor blog. Second, none of this is legal advice. The point is narrower: the classification turns on what your system does, not on whether the word "agent" appears in the product name, and a vendor asserting blanket high-risk status is not reading the statute.

Article 14 is worth reading even if you are out of scope

The AI Act's human oversight requirements are good engineering guidance regardless of applicability, and one provision is unusually prescient. Article 14(4)(b) requires that oversight enable a person "to remain aware of the possible tendency of automatically relying or over-relying on the output produced by a high-risk AI system (automation bias)."

That is the rubber-stamp problem, written into law. Human factors research established it well before: Parasuraman and Manzey's 2010 review in Human Factors documents complacency and automation bias in operators supervising automated systems.

The operational consequence: an approval gate that a human always approves is not a control, it is a formality with a log entry. We made the same argument about approval fatigue in our kubectl safety guide. If your compliance story rests entirely on "a human approved it," measure your approval rate before you rely on that story.

Article 12 adds the record-keeping requirement: high-risk systems "shall technically allow for the automatic recording of events (logs) over the lifetime of the system."

SOC 2 already covers this, and that is good news

Trust Services Criterion CC8.1 concerns the entity authorizing, designing, developing, configuring, documenting, testing, approving, and implementing changes to infrastructure and software.

Nothing in that is agent-specific, which is the point. Your existing change-management control either covers agent-initiated changes or it does not, and the fix is usually to make the agent's changes flow through the same pull request, review, and approval path your humans use, rather than to invent a parallel AI governance regime. An agent that opens a pull request a human reviews and merges is inside your existing control. An agent with its own credentials applying changes directly is outside it, and no amount of logging fixes that.

For teams that want a formal AI management system, ISO/IEC 42001 exists and NIST's AI Risk Management Framework provides the GOVERN, MAP, MEASURE, and MANAGE functions, with a Generative AI Profile published as NIST AI 600-1. Useful, and not a substitute for the change-management control you already have.

What does a defensible control actually look like?

The distinction that matters: a control is defensible when the action was impossible, or when a named person approved it. "The agent was instructed not to" is neither.

The canonical illustration is the Replit incident of July 2025, where a coding agent deleted a production database during an explicit code freeze and, by the founder's account, incorrectly reported that rollback was impossible. The instruction existed. It was prose in a context window, and prose does not constrain a system.

Here is how the three enforcement points compare, in the order an auditor will trust them.

Control	Where enforced	Can the agent circumvent it?	Provable from
Credential scope	Cloud provider (IAM, session policy)	No	Provider logs and policy documents
Tool availability	The agent framework, before inference	No, the capability is absent	Configuration and code
Approval gate	Application, at action time	No, but humans rubber-stamp	Audit log plus approval rate
System prompt instruction	Inside the model's context	Yes	Nothing

Credential scope is the strongest. In Aurora, read-only investigation can assume a role with an STS session policy that permits only read operations, meaning describe, get, and list calls plus CloudWatch Logs queries, across the services it inspects. The auditor's question becomes trivially answerable: not "did the agent try to write?" but "the credential it held could not write, and here is the policy." That is a claim CloudTrail can corroborate.

Tool availability is the next strongest, and it is underrated because it is so simple. A capability that is not in the agent's tool set cannot be invoked regardless of how the prompt is worded, which also makes it immune to prompt injection. Aurora's function that turns a saved incident fix suggestion into a pull request is deliberately not exposed to the model as a callable tool: the agent saves the suggestion, and a human clicks the button in the UI. Terraform apply and destroy are denied outright in unattended runs and are not enabled by default, so an unattended investigation cannot mutate infrastructure through that path. The AI agent guardrails post details the seven independent layers.

Now the boundary, stated plainly, because this is where vendors go quiet: the strength of these guarantees depends on which tool permissions an operator leaves enabled, and repository write operations through the GitHub integration are enabled by default, so an agent can open a pull request unattended. Merging and infrastructure apply are not enabled by default, and an org admin has to grant them deliberately. So the accurate claim is never "the agent cannot," it is "the agent cannot unless somebody turned it on." When you evaluate any agent for production work, the question to ask is not "is it safe?" but "which specific operations are permitted with no human present, and can I verify that in configuration rather than in your documentation?"

What should you actually put in front of an auditor?

A concrete list, in the order that makes a review short.

1. The identity model. Does the agent have its own workload identity, or does it borrow a human's? The Cloud Security Alliance observes that "agents often borrow human or shared identities rather than being managed as distinct entities". Borrowed identity destroys attribution before you start. Platform primitives now exist for this, including Amazon Bedrock AgentCore Identity and Microsoft Entra Agent ID. If data residency is also in scope, our self-hosted deployment guide covers running the whole stack inside your own boundary.
2. The permission inventory, per execution context. Not one list. What the agent can do interactively with a human present, and what it can do unattended. These should differ, and the difference is your control.
3. Provider-side logs, not application logs. CloudTrail and the Kubernetes audit log are written by systems the agent does not control. Show those first. On GCP, confirm you enabled Data Access logs if read activity matters.
4. The approval rate on every human gate. A gate approved 100% of the time is evidence of automation bias, not of oversight. Bring the number before the auditor asks.
5. What the agent was denied, and where that is enforced. Point at the session policy and the tool registry. This is your strongest exhibit.
6. A precise statement of your log's integrity properties. If your audit table is append-only by convention rather than cryptographically chained or write-once, say so. Aurora's audit log records the organization, user, action, resource, and detail for each event, and guardrail blocks additionally emit structured events that record a hash fingerprint of the blocked content rather than the content itself, so injection payloads do not land in the guardrail audit event. It is a Postgres table without WORM guarantees, and claiming immutability you do not have is how a review goes badly.

What about MCP and tool protocols?

If your agent reaches tools over the Model Context Protocol, read its security guidance, because the spec itself names the accountability failure mode.

Under token passthrough, the MCP security best practices document warns: "The downstream Resource Server's logs may show requests that appear to come from a different source with a different identity, rather than the MCP server that is actually forwarding the tokens." That, combined with the server's inability to distinguish between clients, is what the spec says makes "incident investigation, controls, and auditing more difficult." The spec states that servers "MUST NOT accept any tokens that were not explicitly issued for the MCP server," and on scope design notes that poor scoping creates "audit noise: single omnibus scope masks user intent per operation."

One more line from the spec deserves emphasis, because it sets expectations correctly: "While MCP itself cannot enforce these security principles at the protocol level." The protocol gives you the vocabulary. The enforcement is yours.

The uncomfortable conclusion: this is mostly not an AI problem

If you strip out the novelty, the accountability requirements for an agent are the accountability requirements you should already have for automation: a distinct identity, least privilege enforced at the credential, provider-side logging, an approval path with a named human, and honesty about your log's properties.

The genuinely new part is small but real. Agents act on natural-language input, which makes prompt injection an authorization concern rather than a content concern. And their self-reports are unusually persuasive while being unusually unreliable, which is a novel trap for reviewers who are used to reading logs written by deterministic software.

The rest is change management. A team with a real change-management control and provider-side audit logging is most of the way there, and a team without those was never going to be saved by an AI governance policy document. If you want the performance picture that should temper how much autonomy you grant in the first place, the ITBench benchmark found that agents built on then-current models resolved only 13.8% of its SRE scenarios. Design your controls for a system that is often wrong, because that is the system you have.

Two adjacent decisions follow from this. Where the agent's changes enter your repository, the pre-merge review path in PR risk review is the natural place to put the control, and if you are choosing a platform at all, the criteria in how to evaluate an AI SRE platform should include the permission questions above rather than only the capability ones.

• GitHub: github.com/Arvo-AI/aurora
• Docs: Aurora documentation
• Related guides: AI Agent Guardrails · AI Agent kubectl Safety · Self-Hosted AI SRE · The SRE Incident Lifecycle

Not legal advice. Regulatory analysis here is a reading of primary texts linked inline, not legal counsel. If you operate critical infrastructure as defined in Annex III, consult a qualified adviser.

All claims sourced. Regulatory claims cite the statute text, control criteria cite the standards bodies, and research findings are quoted from the publishing organizations with their stated caveats. Aurora's behavior, including its defaults, is described from its open-source implementation. Last verified: July 24, 2026.