Multi-Agent Governance Needs a Centralized Runtime Gate

The failure pattern

The common failure is not a bad model prediction. It is an unsafe delegation from agent A to agent B where the governance boundary vanishes.

Local guardrails help. They do not solve cross-agent drift. Two agents can each pass local checks and still produce an unsafe side effect together.

Production systems need one runtime control layer that evaluates every action intent, enforces policy at the same decision point, and records an evidence trail you can audit later.

What top articles cover

The missing runtime layer

The gap is usually implementation detail. Teams know they need governance. They do not define the exact checkpoints and defaults that decide what happens during failure.

Reference architecture

1. 1. Agent emits action intent to control plane.
2. 2. Gateway evaluates policy before persisting state or publishing to the bus.
3. 3. High-risk decisions move to approval state instead of dispatch.
4. 4. Scheduler re-checks policy before selecting worker pool and dispatching.
5. 5. Worker executes and returns result pointer; scheduler writes terminal state and DLQ if needed.
6. 6. Timeline links intent, decision, approver, and result for incident replay.

Failure matrix

Code: policy + dispatch guard

# safety.yaml
version: v1
rules:
  - id: prod-delete-needs-approval
    when:
      topic: infra.delete
      labels:
        environment: production
    decision: require_human

  - id: deny-customer-notify-without-scope
    when:
      topic: customer.notify
      labels:
        recipient_scope: unverified
    decision: deny

// scheduler bootstrap (simplified)
safetyClient, err := scheduler.NewSafetyClient(os.Getenv("SAFETY_KERNEL_ADDR"))
if err != nil {
  return err
}
safetyClient = safetyClient.WithRedis(redisClient)

engine := scheduler.NewEngine(bus, safetyClient, registry, strategy, jobStore, metrics).
  WithInputFailMode(os.Getenv("POLICY_CHECK_FAIL_MODE")) // open | closed

// Current runtime defaults in code:
// - safety timeout: 2s
// - breaker: 3 failures -> open for 30s
// - max scheduling retries: 50

{
  "job_id": "run_42:delete_prod_vm@1",
  "topic": "infra.delete",
  "policy_snapshot": "sha256:8f6f...",
  "decision": "REQUIRE_HUMAN",
  "rule_id": "prod-delete-needs-approval",
  "approval_required": true,
  "labels": {
    "approval_granted": "true",
    "environment": "production"
  },
  "run_timeline_event": "step_dispatched"
}

Limitations and tradeoffs

• - Centralized control creates another critical dependency. You must design for high availability.
• - Extra policy checks add latency. Keep rules simple on the hot path and test p99 regularly.
• - Approval volume can explode if risk tiers are coarse. Calibrate thresholds with real incident data.
• - Fail-open mode improves availability but weakens safety guarantees. Use it intentionally, not by accident.
• - Local guardrails still matter. Centralized control is a coordinator, not a replacement for worker hygiene.

Next step

Run this rollout in 14 days:

1. 1. Pick one risky topic family (for example `infra.*`) and force policy-before-dispatch there first.
2. 2. Set `POLICY_CHECK_FAIL_MODE=closed` in production and document why.
3. 3. Require approval for one irreversible action and measure queue time + false positive rate.
4. 4. Simulate one safety dependency outage and verify breaker behavior and operator runbook.
5. 5. Run one replay drill from run timeline data and confirm incident reconstruction time is under 30 minutes.

Continue with Multi-Agent Orchestration Needs a Control Plane and AI Agent Incident Report.