AI Agent Chaos Engineering Playbook

Chaos engineering for AI agents means deliberately injecting controlled failures into an autonomous agent control plane to prove it degrades safely. Unlike standard infrastructure chaos, it must also test governance paths: what the system does when the policy decision point is unreachable, when a result queue replays a job, or when an output scanner flags a spike of high-risk results. Every experiment defines a steady-state hypothesis, a bounded blast radius, and explicit abort guards before any fault is injected.

This playbook maps each experiment to how Cordum actually works: governed dispatch through the scheduler, the safety kernel as the policy decision point, the Redis-backed circuit breaker and POLICY_CHECK_FAIL_MODE that determine degradation behavior, and the audit trail that gives you attributable evidence after every run. The metrics, constants, and fail modes referenced below are taken from Cordum's operations and safety-kernel documentation, not invented thresholds.

The production problem

Many teams run chaos testing as a demo event. They inject one fault, watch a dashboard, and call it resilience.

Autonomous agent systems fail differently. Policy dependencies can degrade safely or unsafely. Replay paths can recover or duplicate side effects. Lock behavior can protect consistency or silently drift.

A useful chaos program must test these behaviors explicitly with measurable success and abort criteria.

What top results miss

Experiment model

Keep experiments narrow. One fault class, one hypothesis, one blast radius. Broader experiments make results hard to attribute.

Cordum runtime mapping

Implementation examples

experiment:
  name: safety-kernel-unavailable
  environment: staging-prod-mirror
  duration: 10m
  blast_radius:
    tenants: ["internal-test-tenant"]
    topics: ["job.remediation.execute"]
    traffic_share_percent: 5
  hypothesis:
    steady_state:
      dispatch_p99_seconds: "<= 1.0"
      failed_ratio_5m: "<= 0.10"
    degraded_state:
      safety_unavailable_rate_5m: "> 0"
      unsafe_dispatch_count: "== 0"
  abort_guards:
    - metric: dispatch_p99_seconds
      condition: "> 2.0 for 5m"
    - metric: failed_ratio_5m
      condition: "> 0.15 for 5m"
  rollback:
    - restore_safety_kernel
    - verify_reconciler_lock
    - confirm_orphan_replay_progress

#!/usr/bin/env bash
set -euo pipefail

# Example: staging only
NAMESPACE="${NAMESPACE:-cordum-staging}"

echo "[1/4] scale safety kernel down"
kubectl -n "$NAMESPACE" scale deploy/cordum-safety-kernel --replicas=0

echo "[2/4] wait 180s and sample key metrics"
sleep 180
curl -sG http://prometheus:9090/api/v1/query --data-urlencode   'query=rate(cordum_safety_unavailable_total[5m])' | jq .

echo "[3/4] restore safety kernel"
kubectl -n "$NAMESPACE" scale deploy/cordum-safety-kernel --replicas=2

echo "[4/4] verify reconciler lock and replay activity"
redis-cli GET "cordum:reconciler:default"
curl -sG http://prometheus:9090/api/v1/query --data-urlencode   'query=rate(cordum_scheduler_orphan_replayed_total[5m])' | jq .

# Abort guard A: dispatch latency runaway
histogram_quantile(0.99, rate(cordum_scheduler_dispatch_latency_seconds_bucket[5m])) > 2

# Abort guard B: failed ratio runaway
(
  rate(cordum_jobs_completed_total{status="failed"}[5m])
  / clamp_min(rate(cordum_jobs_completed_total[5m]), 0.001)
) > 0.15

# Abort guard C: stale jobs runaway
cordum_scheduler_stale_jobs > 50

Limitations and tradeoffs

• - Frequent experiments improve confidence but consume on-call and platform capacity.
• - Highly synthetic tests can miss real multi-factor failure chains.
• - Tight abort guards improve safety but may stop experiments before useful data appears.
• - Testing in production-like staging reduces risk but cannot perfectly mirror real tenant behavior.

Frequently asked questions

Next step

Chaos experiments are only worth running if they sharpen recovery, not just confirm failure. Cordum's incident response solution turns the findings from these experiments into governed, auditable recovery workflows.

Run this in one sprint:

1. 1. Pick one experiment class with 5% traffic blast radius and explicit abort guards.
2. 2. Record steady-state baseline for one week before injection.
3. 3. Run one 10-minute controlled experiment and collect recovery evidence.
4. 4. Convert one failure finding into a tracked corrective action with due date and metric target.

Continue with AI Agent Incident Response Runbook and AI Agent Blameless Postmortem Template.