AI Agent Distributed Locking

The production problem

Distributed locks usually fail after the design review, not during it. Everything looks correct until one worker pauses, a network packet arrives late, or Redis latency spikes at 2:17 AM.

A lock is only part of the safety story. You still need stale-writer defense and clear operational checks when replicas disagree about ownership.

If lock strategy is hand-wavy, incident response becomes archaeology. Engineers end up reading source code while production queues keep growing.

What top results miss

The gap is not lock acquisition syntax. The gap is combining lock correctness with operator-facing runbooks and measurable reliability signals in autonomous workloads.

Lock model that survives incidents

Pick lock pattern by failure blast radius. If duplicate work is cheap, lease-only locking may be enough. If duplicate work corrupts state, add fencing semantics at write boundaries.

Cordum runtime lock behavior

Cordum uses different lock behaviors per code path. That is intentional. A one-size lock policy would either reduce throughput or increase duplicate processing risk.

The less glamorous detail is critical: reconciler-style loops hold lock ownership until TTL expiry to avoid same-cycle duplicate ticks. Per-item transitions still release explicitly.

Working implementation examples

type LeaseLock struct {
  Key         string
  OwnerToken  string
  FenceToken  int64
  TTL         time.Duration
}

func AcquireLeaseLock(ctx context.Context, rdb *redis.Client, key string, ttl time.Duration) (*LeaseLock, error) {
  owner := uuid.NewString()
  ok, err := rdb.SetNX(ctx, key, owner, ttl).Result()
  if err != nil {
    return nil, fmt.Errorf("acquire lock: %w", err)
  }
  if !ok {
    return nil, ErrLockBusy
  }

  // Monotonic fence token in the same store. For stricter correctness,
  // generate this in a consensus-backed store.
  fence, err := rdb.Incr(ctx, "fence:"+key).Result()
  if err != nil {
    _ = rdb.Del(ctx, key).Err()
    return nil, fmt.Errorf("fence token: %w", err)
  }

  return &LeaseLock{
    Key: key, OwnerToken: owner, FenceToken: fence, TTL: ttl,
  }, nil
}

-- Reject stale writer attempts using fencing token monotonicity
UPDATE workflow_runs
SET payload = $2,
    fence_token = $3,
    updated_at = NOW()
WHERE run_id = $1
  AND $3 > fence_token;

-- rows_affected = 0 means stale lock holder or concurrent newer writer

# Duplicate dispatch checks
redis-cli GET "cordum:scheduler:job:JOB_ID"
redis-cli GET "cordum:reconciler:default"
redis-cli GET "cordum:replayer:pending"

# Workflow run contention checks
redis-cli GET "cordum:wf:run:lock:RUN_ID"
redis-cli GET "cordum:workflow-engine:reconciler:default"

# Snapshot writer leadership checks
redis-cli GET "cordum:scheduler:snapshot:writer"
redis-cli OBJECT IDLETIME "sys:workers:snapshot"

# Lock contention pressure
histogram_quantile(0.99, rate(job_lock_wait_bucket[5m]))

# Recovery debt
stale_jobs
rate(orphan_replayed_total[5m])

# Fail-open path pressure
rate(cordum_scheduler_input_fail_open_total[5m])

Limitations and tradeoffs

• - Lease locking alone cannot prevent stale writers after long pauses; fencing checks are still needed.
• - Fence token generation in non-consensus stores can become a correctness bottleneck in extreme failure modes.
• - TTL-hold patterns reduce duplicate loops but can delay failover by one TTL window.
• - High lock cardinality increases Redis load and operator noise during outages.

Next step

Do this in one sprint:

1. 1. Inventory every distributed lock key and classify it as per-item explicit release or per-loop TTL hold.
2. 2. Add stale-writer protection for the top 3 correctness-critical write paths.
3. 3. Add alerts on `job_lock_wait`, `stale_jobs`, and `orphan_replayed` trend acceleration.
4. 4. Run one chaos drill: pause a worker for longer than lock TTL and verify stale writes are rejected.

Continue with AI Agent Idempotency Keys and AI Agent Incident Response Runbook.