AI Agent PodDisruptionBudget Strategy

The production problem

Most PDB outages start with good intentions: keep upgrades safe, reduce risk, automate maintenance. Then one bad budget blocks deploys or allows too many evictions at once.

For AI control planes, that mistake is expensive. You are not only protecting HTTP availability. You are protecting lock ownership, scheduler continuity, and message flow.

A wrong budget either freezes operations or burns reliability debt that appears after rollout.

What top results miss

The gap is service-tier translation. Teams need concrete mapping from budget fields to real control-plane failure modes.

PDB math that matters

Start with failure tolerance by service tier. Then derive budget values. Never start with someone else's YAML.

# Quick budget checks
# N = replicas
# M = minAvailable
# U = maxUnavailable

# Rule A (minAvailable): evictions_allowed = N - M
# Rule B (maxUnavailable): evictions_allowed = U

# Stateful quorum example (3-node cluster):
# majority = floor(3/2)+1 = 2
# choose M >= 2  => at most 1 voluntary eviction

# If one pod is already unhealthy, effective eviction budget shrinks to zero.
# Automation must handle this and defer disruption until health recovers.

Cordum baseline values

These values are pulled from current Cordum docs and runtime behavior. Use them as a starting point, then tune with environment-specific load tests.

Implementation examples

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: cordum-api-gateway-pdb
  namespace: cordum
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: cordum-api-gateway
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: cordum-nats-pdb
  namespace: cordum
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: nats
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: cordum-redis-pdb
  namespace: cordum
spec:
  minAvailable: 4
  selector:
    matchLabels:
      app: redis

# 1) Check disruption headroom before rollout
kubectl get pdb -n cordum
kubectl describe pdb cordum-api-gateway-pdb -n cordum
kubectl describe pdb cordum-nats-pdb -n cordum
kubectl describe pdb cordum-redis-pdb -n cordum

# 2) Validate rollout plus PDB constraints
kubectl rollout restart deployment/cordum-api-gateway -n cordum
kubectl rollout status deployment/cordum-api-gateway -n cordum

# 3) Verify lock ownership continuity during restart
redis-cli GET "cordum:scheduler:job:JOB_ID"
redis-cli GET "cordum:reconciler:default"
redis-cli GET "cordum:replayer:pending"

# 4) Roll back if disruption error budget is breached
kubectl rollout undo deployment/cordum-api-gateway -n cordum

# Unavailable replicas during maintenance window
max_over_time(kube_deployment_status_replicas_unavailable{namespace="cordum"}[15m])

# Pod restart spike can indicate disruption pressure
increase(kube_pod_container_status_restarts_total{namespace="cordum"}[15m])

# Queue lock contention signal (if exported)
histogram_quantile(0.99, rate(job_lock_wait_bucket[5m]))

Limitations and tradeoffs

• - PDBs do not prevent involuntary disruptions like node crashes.
• - PDBs can block maintenance if baseline health is already degraded.
• - Tight budgets improve availability but can slow upgrades and security patching.
• - `kubectl delete pod` and direct workload deletion can bypass PDB intent.

Next step

Run this in one sprint:

1. 1. Classify every control-plane workload into stateless, scheduler, or quorum tier.
2. 2. Define one PDB per tier with explicit rationale and owner.
3. 3. Add a pre-rollout gate that fails if current healthy pods leave zero eviction headroom.
4. 4. Run one forced-kill game day and measure lock takeover plus retry surge.

Continue with AI Agent Rolling Restart Playbook and AI Agent Graceful Shutdown.