AI Agent Stuck Job Recovery

The production problem

Message brokers can redeliver when consumers fail. That does not guarantee your control plane can recover jobs already persisted in intermediate scheduler states.

A common outage sequence is simple: request accepted, state written, scheduler restarts, no new event arrives to advance the state. The job is now stuck but not dead.

If your only safety net is broker redelivery, these records sit until timeout or manual repair. That is expensive and boring in the way only incident calls can be.

What top results cover and miss

These guides are correct and useful. They mostly describe queue semantics. They do not describe a pre-dispatch governance scheduler that owns a multi-state job record and must recover it across replica failure windows.

Cordum runtime behavior

Code-level mechanics

func (r *Reconciler) tick(ctx context.Context) {
  now := time.Now()
  dispatchTimeout, runningTimeout, scheduledTimeout := r.currentTimeouts()
  r.handleTimeouts(ctx, JobStateScheduled, now.Add(-scheduledTimeout), scheduledTimeout)
  r.handleTimeouts(ctx, JobStateDispatched, now.Add(-dispatchTimeout), dispatchTimeout)
  r.handleTimeouts(ctx, JobStateRunning, now.Add(-runningTimeout), runningTimeout)
  r.handleDeadlineExpirations(ctx, now)
}

const maxIterations = 100
const maxRetriesPerJob = 3
records, err := r.store.ListJobsByState(ctx, state, cutoffMicros, 200)

func NewPendingReplayer(engine *Engine, store JobStore, pendingAge, pollInterval time.Duration) *PendingReplayer {
  if pollInterval <= 0 { pollInterval = 30 * time.Second }
  if pendingAge <= 0 { pendingAge = 2 * time.Minute }
  return &PendingReplayer{
    pendingAge: pendingAge,
    pollInterval: pollInterval,
    lockKey: "cordum:replayer:pending",
    lockTTL: pollInterval * 2,
  }
}

func (r *PendingReplayer) tick(ctx context.Context) {
  cutoff := time.Now().Add(-r.pendingAge)
  r.replayPending(ctx, cutoff)
  r.replayApproved(ctx, cutoff)
  r.replayScheduled(ctx, cutoff)
}

# config/timeouts.yaml
workflows: {}
topics: {}
reconciler:
  dispatch_timeout_seconds: 300
  running_timeout_seconds: 900
  scan_interval_seconds: 30

Operator runbook

During incidents, watch stale-state growth and replay activity together. Looking at one metric alone produces confident but wrong conclusions.

# 1) Watch stale jobs by state
sum by (state) (cordum_scheduler_stale_jobs)

# 2) Confirm replay activity by topic
sum by (topic) (increase(cordum_scheduler_orphan_replayed_total[15m]))

# 3) If stale jobs rise and replay stays flat:
# - verify scheduler replicas see the same Redis
# - inspect lock keys: cordum:reconciler:default, cordum:replayer:pending
# - validate dispatch/running timeout settings in config/timeouts.yaml

# 4) During incident review, correlate:
# stale_jobs + orphan_replayed + dispatch latency + DLQ depth

Limitations and tradeoffs

• - More aggressive replay age recovers faster but can re-submit jobs that were merely slow.
• - Longer timeouts reduce false positives but keep real failures hidden for longer.
• - TTL-held lock windows reduce duplicate ticks but also limit how often another replica can run recovery.
• - Recovery loops help scheduler state, not worker-side idempotency. Handlers still need idempotent design.

Next step

Run one controlled stuck-job simulation this week:

1. 1. Submit representative jobs, then restart scheduler between `SCHEDULED` and `DISPATCHED` paths.
2. 2. Measure replay lag and stale-job decay with current timeout profile.
3. 3. Tune `dispatch_timeout_seconds` and verify replay speed vs false replay rate.
4. 4. Keep an incident dashboard that includes stale jobs, orphan replay, dispatch latency, and DLQ depth.

Continue with Safety Unavailable Retry Strategy and Backpressure and Queue Drain.