AI Agent AckWait and Dedup TTL Alignment

The production problem

A common crash path looks harmless: message is processed, process dies before ack, message redelivers later. If your dedup state expired before redelivery, you run the job again.

This is not a broker bug. It is a timing mismatch between the redelivery window and your dedup key lifetime.

In AI agent control planes, duplicate execution can create duplicate side effects, noisy incidents, and hard audit questions.

What top results miss

Public docs explain ack deadlines well. They usually stop short of one practical question: does your dedup persistence window actually match your redelivery window?

Cordum runtime behavior

Code-level mechanics

const (
  defaultAckWait = 10 * time.Minute

  // processed key: cordum:bus:processed:<stream>:<seq>
  processedKeyTTL = 10 * time.Minute

  // inflight key: cordum:bus:inflight:<stream>:<seq>
  inflightKeyTTL = 2 * time.Minute
)

ackWait := defaultAckWait
if v := strings.TrimSpace(os.Getenv("NATS_JS_ACK_WAIT")); v != "" {
  if d, err := time.ParseDuration(v); err == nil && d > 0 {
    ackWait = d
  }
}
b.ackWait = ackWait

opts := []nats.SubOpt{
  nats.ManualAck(),
  nats.AckExplicit(),
  nats.AckWait(b.ackWait),
  nats.MaxAckPending(natsMaxAckPending()),
  nats.MaxDeliver(maxJSRedeliveries),
}

// 1) Check processed key (crash-safe guard)
exists, err := b.redis.Exists(ctx, processedKey(stream, seq)).Result()
if err != nil {
  _ = msg.NakWithDelay(2 * time.Second)
  return
}
if exists > 0 {
  _ = msg.Ack()
  return
}

// 2) Process handler
action, delay := processBusMsg(msg.Data, handler, numDelivered)

// 3) On success, set processed key BEFORE Ack
if setErr := b.redis.Set(ctx, processedKey(stream, seq), "1", processedKeyTTL).Err(); setErr != nil {
  _ = msg.NakWithDelay(2 * time.Second)
  return
}
_ = msg.Ack()

Operator runbook

# 1) Check effective AckWait (empty means default 10m)
kubectl -n cordum exec deploy/cordum-scheduler -- printenv NATS_JS_ACK_WAIT

# 2) Sample processed key TTLs
for k in $(redis-cli --scan --pattern 'cordum:bus:processed:CORDUM_JOBS:*' | head -n 20); do
  echo "$k $(redis-cli TTL "$k")"
done

# 3) Safety invariant
# effective_ack_wait_seconds <= processed_key_ttl_seconds

# 4) If AckWait is customized above 10m, run crash drill:
#    - publish one long-running job
#    - kill scheduler after handler success but before ack window closes
#    - verify duplicate processing does not occur on redelivery

# 5) Alerting suggestions
# - high redelivery count per subject
# - spikes in dedup-skipped acks
# - processed-key TTL histogram drifting below configured AckWait

Limitations and tradeoffs

• - Keeping dedup TTL high reduces duplicates but increases Redis key retention.
• - Lower AckWait speeds recovery but raises redelivery pressure for slow handlers.
• - Redis outages fall back to JetStream-only behavior, so duplicate exposure increases.
• - Queue-level dedup does not replace business-level idempotency in worker code.

Next step

Run one controlled crash drill this week and record three numbers:

1. 1. Effective AckWait in each environment.
2. 2. Observed processed-key TTL distribution in Redis.
3. 3. Duplicate execution count after crash-before-ack simulation.

Continue with Exactly-Once Myth and Timeouts, Retries, and Backoff.