AI Agent MaxAckPending Tuning

The production problem

Your queue backlog grows. CPU looks fine. Workers are alive. Yet dispatch throughput drops hard.

One common cause is `MaxAckPending` saturation. The broker pauses delivery because too many messages are unacked, even though workers could continue if budget were sized correctly.

Oversizing is not free either. Huge in-flight windows increase memory pressure, amplify blast radius during failures, and make redelivery storms harder to reason about.

What top results miss

The docs explain queue controls individually. The gap is the integrated tuning path for an AI agent control plane where ack budget, retry behavior, scheduling state, and broker-side caps all interact.

Cordum runtime behavior

Code-level mechanics

const maxAckPendingHardLimit = 50000

func natsMaxAckPending() int {
  const defaultMaxAckPending = 2048
  if raw := strings.TrimSpace(os.Getenv("NATS_MAX_ACK_PENDING")); raw != "" {
    if v, err := strconv.Atoi(raw); err == nil && v > 0 {
      if v > maxAckPendingHardLimit {
        slog.Warn("bus: NATS_MAX_ACK_PENDING exceeds hard limit, clamping",
          "requested", v, "clamped_to", maxAckPendingHardLimit)
        return maxAckPendingHardLimit
      }
      return v
    }
  }
  return defaultMaxAckPending
}

opts := []nats.SubOpt{
  nats.ManualAck(),
  nats.AckExplicit(),
  nats.AckWait(b.ackWait),
  nats.MaxAckPending(natsMaxAckPending()),
  nats.MaxDeliver(maxJSRedeliveries), // 100
}

func TestNatsMaxAckPending_ClampedAtHardLimit(t *testing.T) {
  t.Setenv("NATS_MAX_ACK_PENDING", "100000")
  v := natsMaxAckPending()
  if v != maxAckPendingHardLimit {
    t.Fatalf("expected %d (clamped), got %d", maxAckPendingHardLimit, v)
  }
}

func TestQueueGroupWithJetStream(t *testing.T) {
  // verifies one delivery per message across queue consumers
}

Sizing heuristic

Start from observed reality, not round numbers. Measure per-consumer handler latency and sustained ingress rate, then compute an initial budget and reconcile it with account/server ceilings.

# Rule of thumb (per consumer):
# required_budget ~= p95_handler_seconds * incoming_msgs_per_second * safety_factor
#
# Example:
# p95_handler = 1.2s
# rate = 600 msg/s
# safety_factor = 2
# required_budget ~= 1.2 * 600 * 2 = 1440
#
# Effective budget is capped by broker limits:
# effective_budget = min(consumer_max_ack_pending, account_max_ack_pending, server_max_ack_pending)
#
# Start with 2048, verify broker-side caps, then tune from production telemetry.

If requested values exceed broker limits, effective budget stays capped and symptoms will look like under-sizing. After initial sizing, tune in small steps and watch queue delay, redelivery rate, and scheduler memory headroom together.

Operator runbook

# 0) Check broker-side max_ack_pending limits (server/account)
kubectl -n nats get configmap nats-config -o yaml | rg max_ack_pending

# 1) Check current override
kubectl -n cordum exec deploy/cordum-scheduler -- printenv NATS_MAX_ACK_PENDING

# 2) Inspect active JetStream consumers and pending-ack pressure
nats consumer ls CORDUM_JOBS
nats consumer info CORDUM_JOBS <consumer_name>

# 3) Watch signals
# - NumAckPending close to MaxAckPending for sustained periods
# - Rising delivery lag / backlog
# - Repeated redeliveries for same payload

# 4) Tune and roll
kubectl -n cordum set env deploy/cordum-scheduler NATS_MAX_ACK_PENDING=5000
kubectl -n cordum rollout status deploy/cordum-scheduler

# 5) Re-check after 15-30 minutes
# - Ack pending headroom restored
# - No large memory regression
# - Redelivery rate stable

Limitations and tradeoffs

• - Lower budget improves failure visibility but can throttle healthy workloads.
• - Higher budget improves short-term throughput but can hide slow-consumer problems.
• - MaxAckPending tuning does not fix non-idempotent worker behavior.
• - Queue-level controls should be paired with DLQ policy and replay discipline.

Next step

Run one sizing pass with production data:

1. 1. Measure p95 handler duration per queue consumer.
2. 2. Estimate required ack budget with a safety factor.
3. 3. Apply one-step change and observe for 30 minutes.
4. 4. Keep the smallest value that avoids sustained saturation.

Continue with AckWait and Dedup TTL Alignment and Backpressure and Queue Drain.