AI Agent Safety Check Timeout Tuning

The production problem

Teams copy a timeout value from a sample config, then forget it. Six weeks later, policy latency rises and one of two bad outcomes appears: blocked queues or silent policy bypass.

Tight timeouts turn normal latency spikes into `SafetyUnavailable` storms. Loose timeouts keep workers waiting while backlog grows. Neither outcome is acceptable for autonomous systems with real side effects.

Timeout tuning is not a transport tweak. It is a governance control that defines when availability is allowed to outrank safety checks.

What top results miss

The gap is an end-to-end method: derive timeout budgets from measured latency, then bind those budgets to explicit fail-mode behavior and alerts.

Timeout budget model

Use one timeout policy per risk class, not one timeout for every request.

# Safety check timeout budget (per operation class)
# inputs from observability over last 7 days:
# p99_eval_ms: policy evaluation p99
# p99_network_ms: transport + queueing p99
# jitter_ms: rollout jitter reserve
# margin_ms: explicit safety margin

timeout_ms = p99_eval_ms + p99_network_ms + jitter_ms + margin_ms

# Practical floor/ceiling guards
timeout_ms = max(timeout_ms, 250)
timeout_ms = min(timeout_ms, 3000)

# Example:
# p99_eval=420, p99_network=110, jitter=80, margin=140
# timeout = 750ms

Cordum runtime behavior

These numbers are current in Cordum code and docs. Tune around them, but do not ignore them during rollout planning.

Implementation examples

func chooseSafetyTimeout(p99Eval, p99Network time.Duration, critical bool) time.Duration {
  jitterReserve := 80 * time.Millisecond
  margin := 140 * time.Millisecond
  base := p99Eval + p99Network + jitterReserve + margin

  if base < 250*time.Millisecond {
    base = 250 * time.Millisecond
  }

  if critical && base < 800*time.Millisecond {
    base = 800 * time.Millisecond
  }

  if base > 3*time.Second { // align with scheduler guardrail envelope
    base = 3 * time.Second
  }
  return base
}

# 1) Watch for policy bypass during incident
sum(rate(cordum_scheduler_input_fail_open_total[5m])) by (topic)

# 2) Inspect timeout warnings from scheduler
kubectl logs deploy/cordum-scheduler -n cordum | grep -E "safety check timed out|safety kernel unavailable"

# 3) Verify fail mode settings during outage
kubectl exec -n cordum deploy/cordum-scheduler -- printenv POLICY_CHECK_FAIL_MODE
kubectl exec -n cordum deploy/cordum-api-gateway -- printenv GATEWAY_POLICY_FAIL_MODE

# 4) If bypass counter is rising in production, switch to closed and drain
kubectl set env deployment/cordum-scheduler -n cordum POLICY_CHECK_FAIL_MODE=closed

Limitations and tradeoffs

• - Higher timeout budgets reduce false timeouts but increase queue occupancy under policy latency spikes.
• - Lower budgets reduce stall time but increase `SafetyUnavailable` frequency.
• - Fail-open protects throughput but can bypass deny/approval decisions.
• - Fail-closed preserves policy integrity but can degrade availability during kernel outages.

Next step

Do this in the next reliability cycle:

1. 1. Export 7-day p95/p99 latency for policy evaluation per topic.
2. 2. Set timeout budgets with the formula above and document risk-tier ownership.
3. 3. Keep `POLICY_CHECK_FAIL_MODE=closed` for production unless incident commander approves override.
4. 4. Alert on any non-zero 5-minute rate of `cordum_scheduler_input_fail_open_total`.

Continue with AI Agent Fail-Open vs Fail-Closed and AI Agent gRPC Deadline Budgeting.