AI Agent Capacity Planning Model: How to Size Worker Pools Without Guessing (2026)

The production problem

Capacity planning failures in agent systems rarely look like one big crash. They show up as rising dispatch latency, retry churn, and deferred work that never catches up.

Most teams still size for average traffic and rely on autoscaling to save them. That works until retries, policy outages, or long-tail job durations break the scaling signal.

You need an explicit model that links worker count to reliability metrics and recovery behavior.

What top results miss

Source	Strong coverage	Missing piece
Google SRE Book: Demand forecasting and capacity planning	Strong requirement for demand forecast, load testing, and provisioning ownership.	No guidance for policy-gated autonomous workflows with retries and replay behavior.
Google SRE Workbook: Data processing capacity planning	Concrete example: provision around 50% CPU at peak and beware runaway autoscaling.	No generic model for agent pipeline stages (dispatch, policy, output checks).
AWS Well-Architected Analytics Lens BP 11.2	Practical right-sizing and autoscaling guidance for predictable and spiky workloads.	No reliability budgeting link between scaling behavior and autonomous side-effect safety.

Capacity model

Use queueing math as baseline, then add retry and safety-path headroom. Do not jump straight to autoscaler tuning.

Model layer	Formula	Target value	Why it matters
Ingress rate	jobs_per_second (lambda)	Use p95 traffic, not daily average	Average hides burst pressure.
Service time	avg_execution_seconds (W)	Use p90 service time for conservative sizing	Long-tail jobs distort capacity fast.
Worker count	ceil((lambda * W) / target_utilization)	Target utilization 0.60-0.75	Lower target gives burst headroom.
Retry overhead	base_workers * retry_multiplier	Start with 1.10 to 1.30 multiplier	Retry storms are real capacity demand.

Sample worker sizing outcomes:

Ingress rate	Service time	Utilization target	Required workers
80 jobs/s	0.35s	0.65	44
120 jobs/s	0.40s	0.65	74
200 jobs/s	0.55s	0.70	158

Cordum runtime implications

Implication	Current behavior	Why it matters
Failure-rate guardrail	Existing alert threshold uses failed ratio > 10% over 5m	Capacity decisions should reduce this sustained risk signal, not only reduce queue depth.
Latency guardrail	Dispatch p99 warning threshold is > 1s	A useful early signal that worker pools are under-provisioned.
Retry budget pressure	Max scheduling retries = 50, backoff 1s-30s, `retryDelayNoWorkers` = 2s	Retry mechanics directly affect effective throughput and backlog shape.
Policy dependency capacity impact	`POLICY_CHECK_FAIL_MODE=closed` defaults to requeue on policy outage	Policy outages can consume capacity through safe requeue loops.
Recovery debt tracking	`cordum_scheduler_stale_jobs` + `cordum_scheduler_orphan_replayed_total`	Capacity planning should include post-incident recovery window, not only steady-state traffic.

Implementation examples

Worker sizing helper (TypeScript)

capacity-sizing.ts

TypeScript

type SizingInput = {
  ingressPerSecond: number;    // lambda
  avgServiceSeconds: number;   // W
  targetUtilization: number;   // e.g. 0.65
  retryMultiplier?: number;    // e.g. 1.2
};

export function requiredWorkers(input: SizingInput): number {
  const base = (input.ingressPerSecond * input.avgServiceSeconds) / input.targetUtilization;
  const retries = input.retryMultiplier ?? 1.0;
  return Math.ceil(base * retries);
}

// Example:
// 120 jobs/s * 0.40s / 0.65 = 73.8 -> 74 workers
// retry multiplier 1.2 -> 89 workers

Capacity planning policy config (YAML)

capacity-plan.yaml

YAML

capacity_planning:
  target_utilization: 0.65
  retry_multiplier: 1.2
  guardrails:
    dispatch_p99_seconds_warn: 1
    failed_ratio_5m_warn: 0.10
    stale_jobs_warn: 50
  policy_dependency:
    fail_mode: closed
    max_tolerated_safety_unavailable_rate_5m: 0.05
  headroom:
    minimum_spare_workers_percent: 20
    burst_window_minutes: 15

Core capacity validation queries (PromQL)

capacity-signals.promql

PromQL

# Dispatch p99
histogram_quantile(0.99, rate(cordum_scheduler_dispatch_latency_seconds_bucket[5m]))

# Failed completion ratio
rate(cordum_jobs_completed_total{status="failed"}[5m])
/ clamp_min(rate(cordum_jobs_completed_total[5m]), 0.001)

# Safety dependency degradation
rate(cordum_safety_unavailable_total[5m])

# Recovery debt
cordum_scheduler_stale_jobs
rate(cordum_scheduler_orphan_replayed_total[5m])

Worked example: sizing a refund agent

Make the model concrete. Say a support team runs an autonomous refund agent. Traffic peaks at a p95 of 90 jobs/second, each job takes a p90 of 0.5 seconds end to end (analysis plus the policy check plus the refund call), and the team wants a 0.65 utilization target.

Step 1 — Baseline workers

ceil(90 × 0.5 / 0.65) = ceil(69.2) = 70 workers. That is the floor for steady peak traffic.

Step 2 — Retry overhead

Refunds occasionally fail on a flaky payment API. With a 1.2 retry multiplier: ceil(70 × 1.2) = 84 workers.

Step 3 — Policy-outage headroom

If the Safety Kernel briefly degrades, POLICY_CHECK_FAIL_MODE=closed requeues jobs instead of dropping them, so backlog spikes. The 20% minimum spare-worker headroom in the capacity config absorbs this without tipping dispatch p99 over the 1s warning line.

Step 4 — Validate, don't trust

Run one load test at 90 jobs/s and one drill that makes the Safety Kernel unavailable for 60 seconds. Confirm the pool holds dispatch p99 under 1s and that cordum_scheduler_stale_jobs drains back to zero after recovery. If it does not, the retry multiplier or headroom is too low.

The point is that the final number (84 workers plus headroom) is defensible: every term traces to a measured input or a verified runtime behavior, not a guess.

Limitations and tradeoffs

- Simple sizing formulas assume stationarity; real workloads can shift faster than planning windows.
- Conservative utilization targets increase reliability but can reduce cost efficiency.
- Retry multipliers are rough estimates until measured under incident-like conditions.
- Autoscaling can still overshoot when its metric no longer tracks useful work.

Frequently asked questions

How do I calculate how many workers an AI agent pool needs?

Start from Little's Law: required workers = (arrival rate x average service time) / target utilization. For example, 120 jobs/second at 0.40s average execution and a 0.65 utilization target needs ceil(120 x 0.40 / 0.65) = 74 workers. Then multiply by a retry overhead factor (1.1-1.3 to start) because failed jobs re-enter the queue and consume real capacity. Size arrival rate from p95 traffic and service time from p90 execution, not daily averages, so a normal burst does not exhaust the pool.

What utilization target should I plan for?

Plan for 60-75% steady-state utilization, not 90%+. The gap is deliberate headroom: queueing latency rises non-linearly as utilization approaches 1.0, so a pool sized for 95% average load tips into runaway dispatch latency the moment traffic spikes. A 0.65 target leaves roughly 35% spare capacity to absorb bursts, retry storms, and the requeue pressure that appears when a downstream dependency degrades.

Why do retries and replay count as capacity demand?

Because they are real dispatch work, not background noise. In Cordum the scheduler retries a job up to maxSchedulingRetries (50) with exponential backoff from 1s to 30s before moving it to FAILED and the DLQ, and applies a 2s retryDelayNoWorkers delay when no worker is available. During an incident every in-flight job can spawn several extra attempts, so a pool sized only for first-attempt throughput will fall behind exactly when it matters most. Track recovery debt with cordum_scheduler_stale_jobs and cordum_scheduler_orphan_replayed_total.

How does the policy / Safety Kernel check affect capacity planning?

Every dispatch passes through a Safety Kernel check before a worker runs, so the policy path is part of your effective service time. When the kernel is unreachable the behavior depends on POLICY_CHECK_FAIL_MODE: the default closed mode requeues jobs with backoff, which means a policy outage consumes capacity through safe retry loops rather than dropping work. Plan headroom for that requeue pressure and alert on cordum_safety_unavailable_total so a degraded policy dependency does not silently eat your pool.

What metrics tell me a worker pool is under-provisioned?

Three early signals: dispatch p99 latency rising above ~1s (histogram_quantile over cordum_scheduler_dispatch_latency_seconds_bucket), the failed-completion ratio sustained above 10% over 5 minutes, and a climbing cordum_scheduler_stale_jobs gauge. Latency and stale-job growth usually precede outright failures, so treat them as the trigger to add workers rather than waiting for the failed-ratio alarm.

Is autoscaling enough, or do I still need a capacity model?

Autoscaling is reactive and only as good as the metric it scales on. If you scale on CPU or queue depth while retries are inflating both, the autoscaler can chase a signal that no longer tracks useful work and overshoot. A capacity model gives you the floor (minimum workers for p95 traffic), the right scaling metric, and the guardrails that tell you when scaling is masking a deeper problem like a policy outage or a long-tail job regression.

Next step

Run this in one sprint:

1. Baseline p95 ingress and p90 service time for your top three topics.
2. Compute worker targets with utilization 0.65 and retry multiplier 1.2.
3. Add guardrails for dispatch p99, failed ratio, and stale jobs.
4. Validate the plan with one controlled load test and one dependency-degradation drill.

Continue with AI Agent Chaos Engineering Playbook and AI Agent Backpressure and Queue Drain Strategy.