A kubectl deny rule blocked rollout history in 26 of 38 runs

In the lab run behind our first lab report, scenario 03 is a mistyped image tag: kubectl -n shop set image deploy/frontend nginx=nginx:1.27-alpne. Aurora had the pod in ImagePullBackOff by step 6 and the registry's "not found" by step 7. Step 9 was the obvious next question, what changed, and the answer came from our own code:

$ kubectl rollout history deployment/frontend -n shop
Command blocked by organization policy: Mutating kubectl operations

The tool result carried "code": "POLICY_DENIED". kubectl rollout history writes nothing. It reads the ReplicaSets a Deployment keeps and prints one line per revision. Our default command policy, the one that exists to keep an investigating agent from changing a cluster, refused it, and the report filed the gap under "Not Checked": the "full revision diff/history was not reviewed directly".

How often it happened

Across the 38 graded runs on Aurora commit 87d22baf, 26 hit this denial: 16 of 20 in v0-run2 and 10 of 18 in v1-run1, 27 calls in total (counted from each run's steps-full.json: 17 and 10 on_prem_kubectl calls with rollout history in the command and POLICY_DENIED in the output). Every one of the 27 was kubectl rollout history deployment/<name> -n shop. Aurora never asked for rollout restart or rollout undo.

The denial also leaked into Aurora's reasoning. In the superseded v0-run1, the scenario 05 report listed "Full rollout history (kubectl rollout history)" as "blocked by organizational policy on mutating kubectl operations", and in its chat notes it wrote "rollout history is blocked by policy (mutating-kubectl denylist)". A read had been relabelled a write, and the model repeated the label.

The workaround that did not work well

Without the command, the agent fell back on listing ReplicaSets. Scenario 03, step 8, one step before the denial:

$ kubectl get replicaset -n shop -l app=frontend -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.template.spec.containers[0].image}{" replicas="}{.spec.replicas}{" ready="}{.status.readyReplicas}{"\n"}{end}'
frontend-76d98597b9 nginx:1.27-alpine replicas=2 ready=2
frontend-7b6c998456 nginx:1.27-alpne replicas=1 ready=

That works, as far as it goes. The Kubernetes docs say it plainly: "A Deployment's revision history is stored in the ReplicaSets it controls" (Deployments). Listing them and printing the image gives the same fact rollout history --revision=N would, for that one field. Eleven of the 20 v0 runs listed ReplicaSets this way, and all 20 ended with a correct verdict.

It is still the wrong tool, and the runs show the cost. Aurora had to compose a jsonpath expression for each field it wanted to compare, so every further question (an env var, a probe, a limit) is another call, where rollout history --revision=N prints the whole pod template at once. It had to describe deployment in step 10 to learn it was on revision 2. In scenario 01 the report said that because of the policy "the exact author/timestamp of the out-of-band env var change could not be retrieved". And in scenario 12, where a second fault sat unalerted in the same namespace and both runs missed it, one of r1's 18 tool calls went to a command that returned a policy string. A workaround that costs steps costs attention.

The bug

Every command Aurora runs against a cluster passes through command_gate, which calls evaluate_compound_command in server/utils/auth/command_policy.py. The file's docstring gives the order: the denylist is checked first and a match is denied; the allowlist is checked second, a match is allowed and no match is denied; with both off, everything is allowed. A new org is seeded with the observability_only template and both lists on.

The deny rule that matched, line 704 at 87d22baf (line 790 carries the same bare rollout for standard_ops):

# before
\bkubectl\s+(exec|cp|run|attach|port-forward|apply|delete|create|edit|patch|replace|scale|rollout|drain|cordon|uncordon|taint)\b

# after
\bkubectl\s+(exec|cp|run|attach|port-forward|apply|delete|create|edit|patch|replace|scale|rollout\s+(restart|undo|pause|resume)|drain|cordon|uncordon|taint)\b

rollout followed by \b matches kubectl rollout history exactly as well as it matches kubectl rollout undo. The rule's description, "Mutating kubectl operations", is the string the agent saw, and it is wrong for two of the six rollout subcommands.

The rule structure also rules out the one-line version of the fix, narrowing the deny rule and stopping there. With the allowlist on, a command that clears the deny list is then matched against the allow rules, and the Kubernetes read rule at line 643 listed get|describe|logs|top|explain|api-resources|api-versions|cluster-info|config ... with no rollout entry. The engine's own code for that case returns allowed=False, rule_description="No matching allow rule" and allowlist_exhausted=True: a different reason string, the same refusal for the agent. We did not run that version; this is what the evaluator's code does, and the test below checks for it. So the fix touches both lists in both templates: the deny rule narrows to rollout\s+(restart|undo|pause|resume), and rollout\s+(history|status) joins the read allow rule. Four lines changed.

The test

The useful part of the test is the path, not the assertion. The policy had tests before; none loaded the shipped template and asked the real evaluator about rollout history. The new server/tests/auth/test_command_policy_rollout.py pulls each template from get_policy_templates(), sets both lists on as seed_default_command_policy leaves a new org, and goes through evaluate_compound_command, the function command_gate calls:

@pytest.fixture(params=["observability_only", "standard_ops"])
def template_policy(request, monkeypatch):
    tpl = next(t for t in get_policy_templates() if t["id"] == request.param)
    allow, deny = _rules(tpl["allow"], "allow"), _rules(tpl["deny"], "deny")
    states = ListStates(allowlist_enabled=True, denylist_enabled=True)
    ...

@pytest.mark.parametrize("cmd", READ_ONLY)
def test_rollout_reads_are_allowed(template_policy, cmd):
    verdict = evaluate_compound_command("org-1", cmd)
    assert verdict.allowed, (template_policy, cmd, verdict)
    assert not verdict.allowlist_exhausted

@pytest.mark.parametrize("cmd", MUTATING)
def test_rollout_mutations_are_denied_by_deny_rule(template_policy, cmd):
    verdict = evaluate_compound_command("org-1", cmd)
    assert not verdict.allowed, (template_policy, cmd)
    assert verdict.deny_rule_id is not None
    assert verdict.rule_description.startswith("Mutating kubectl operations")

READ_ONLY is rollout history and rollout status; MUTATING is restart, undo, pause and resume. The line assert not verdict.allowlist_exhausted is the check for the deny-only mistake: a read that clears the deny list but matches no allow rule exhausts the allowlist and fails it. Two templates, six commands, 12 tests. On the unfixed code 4 of the 12 fail, the four read-only cases. With the existing test_command_policy.py and test_allowlist_bypass.py, the three files pass 145 tests. The change is an open pull request on Arvo-AI/aurora, opened 2026-10-19 and not merged at the time of writing.

The rerun

We rebuilt the lab images on the patched code and reran scenarios 05, 09 and 10 twice each (runs/v0-fix-run1). Each run got a fresh org, which matters here: the template is copied into org_command_policies rows when an org is created, so a new org gets the new rule and an existing org keeps its old rows until the template is re-applied from Settings > Security.

Scenario	87d22baf	Patched	Median time, before / after	Median cost, before / after
05 Invalid nginx directive	2/2	2/2	136 s / 145 s	$0.32 / $0.36
09 PVC with missing StorageClass	2/2	2/2	280 s / 182 s	$0.39 / $0.44
10 Wrong REDIS_HOST	2/2	2/2	244 s / 100 s	$0.28 / $0.32

rollout history ran 14 times across the 6 runs and was denied 0 times. In 10 r2 Aurora asked for --revision=1 and --revision=2 and got both pod templates back, REDIS_HOST: redis in the first and REDIS_HOST: redis-master in the second, which is the injected fault. All 6 runs were graded correct and confirmed by a person. The faster medians on 09 and 10 are mostly not the fix: the earlier runs shared worker slots with background prediscovery jobs and these did not.

Two other bugs from the same lab day

The fix branch carries three patches. The other two came out of one run, v0-run1 scenario 09, superseded because all ten runs shared one org.

No max_tokens on Bedrock. Aurora found the PVC with the missing fast-ssd StorageClass by step 10, then made 25 write_artifact calls over about 11 of the run's 15 minutes and $2 of its $2.61. In timeline.json, 19 agent responses in that stretch have output_tokens of exactly 4096. The Bedrock provider (chat/backend/agent/providers/bedrock_provider.py, lines 180 to 197 at 87d22baf) never sets max_tokens, so Converse used its default. The model was rewriting a whole "Incident Index" artifact in one call, the arguments JSON was cut off before content, pydantic rejected the call before the tool function ran, and the model saw only "content: Field required", retried, then tried "placeholder" and "test2". Patch 0002 registers the tool with the same schema as a plain JSON dict so the call reaches write_artifact, which now returns an error saying the call was most likely cut off by the output-token limit and asking for a shorter document. The cap itself is unchanged.

The 30-citation window. The RCA summary is written from citations[-30:] (chat/background/summarization.py, line 211), and every tool message counted except a short _NON_EVIDENCE_TOOLS set. That run had 45 tool calls; by our reconstruction calls 19 to 45 were artifact and memory calls, so the PVC, StorageClass and PVC-yaml results from steps 7, 8 and 11 never reached the summary prompt. The report cited only [17] and [18], named the cause as "a dependency the cluster could not satisfy", and filed the PVC under "Not Checked". Patch 0003 adds write_artifact, write_memory, append_to_memory and edit_memory to the non-evidence set; reads stay citable. The window is unchanged.

Both patches are in a second pull request on Arvo-AI/aurora, opened 2026-10-19 and not merged at the time of writing. Each has unit tests (4 and 2) and nothing else: the artifact loop did not recur in the 26 later runs (0 write_artifact calls), so the rerun never exercised them.

Still open

• The Ask-mode classifier. utils/security/read_only_classifier.py (line 47 at 87d22baf) lists rollout in WRITE_VERBS, so on the cloud_exec kubectl path for EKS, GKE and AKS, kubectl rollout history is still refused. The lab's on_prem_kubectl tool does not pass through it. It is a security gate and needs a kubectl-specific rollout <sub> rule, not a one-word deletion.
• Existing orgs keep the old pattern until someone re-applies the template. Migrating their rows is a separate decision.
• kubectl exec ... -- cat is also denied as mutating. In scenario 19 r1 that stopped Aurora from comparing the live nginx config of two pods. Denying exec is defensible and we have not changed it.
• The 4096 token cap and the 30-citation window are what they were. An investigation with more than 30 real evidence calls still loses its earliest evidence.

The lesson

A deny rule should name the verb that mutates, not the family it belongs to. rollout is a family; rollout undo is a verb. And a test that checks a regex in isolation tells you about the regex. The test that would have caught this loads the shipped template, turns on both lists the way a new org gets them, and asks the same function the gate asks. We had the regexes. We did not have that test, and 26 of 38 investigations paid for it, 27 blocked calls in all.

Read-only by default is the right default for an agent in a production cluster. This is what it costs when the definition of read-only is one word too wide, and why we run the lab against our own build. Aurora is open source, and the policy, the test and the reruns are all public in the links above.

Related

The runs are in the lab report; the policy engine is described in AI agent kubectl safety; the scenario behind the other two bugs has a runbook, PVC Pending, StorageClass not found.

Try Aurora

• Start free: aurora-ai.net (hosted, no infrastructure to run)
• GitHub: github.com/Arvo-AI/aurora
• Book a demo: cal.com/arvo-ai/demo

Sourcing note. Every number, command, output and quote above comes from the lab repo Arvo-AI/aurora-lab as of its 2026-10-02 runs (FINDINGS.md, REPORT.md, patches/NOTES.md and the runs/ folders, each run with steps-full.json, aurora-rca.md and grade.json), from the Aurora source at commit 87d22baf and on the fix branch, and from the two pull requests linked above. This post shows Aurora source paths, line numbers and pull request links because our engineering-post standard requires real code linked to the file or PR. Versions: kind v0.33.0, Kubernetes v1.37.0, RCA model bedrock/us.anthropic.claude-sonnet-5, judge us.anthropic.claude-opus-5-5. Written 2026-10-19.