GRIDLOCK — Agent Failure Series #18

⟳

The Setup

// pipeline_topology.json · initial state

🎯 Orchestrator

idle

🔍 Research

idle

✍️ Writer

idle

🔎 Reviewer

idle

🚀 Publisher

idle

The Problem

The Research agent starts by asking the Writer to define the content structure — "what sections do I need to fill?"
The Writer agent starts by asking the Research agent for facts — "what do I have to work with?"
Both enter a WAITING_FOR_PEER state. Neither times out. Neither fails. Neither moves. Your orchestrator is billing API tokens to poll two agents that will never respond.

The Chicken-Egg Lock

Circular Dependency

🔄

Agent A waits for Agent B · Agent B waits for Agent A

Most common form · Hardest to detect in logs

🔍 Research

idle

→ waiting for Writer →

← waiting for Research ←

✍️ Writer

idle

Why It's Hard to Catch

Both agents are in a valid RUNNING state in your orchestration layer. They're not erroring. They're not timing out (yet). If your polling interval is 5 seconds and your dashboard shows a green dot — this looks like a healthy pipeline that's just "working". You won't know until the invoice arrives.

# What the logs look like — looks totally normal

[Research] status="RUNNING"  last_heartbeat="2s ago"
[Writer]   status="RUNNING"  last_heartbeat="3s ago"
[Orch]     waiting_for=["research", "writer"]  # ← both are "running"!

# 47 minutes later
[Research] status="RUNNING"  last_heartbeat="2s ago"
[Writer]   status="RUNNING"  last_heartbeat="3s ago"

      

The Rescue Trap

Recovery Agent Captured

🪤

The recovery agent you sent to fix it also deadlocks

Now you have 3 agents stuck instead of 2

Your orchestration framework detects the stall and sends a Coordinator agent to resolve it. The Coordinator asks each stuck agent to summarize their current state so it can mediate.

The stuck agents can't respond — they're waiting. The Coordinator enters WAITING_FOR_SUMMARY. Three agents are now stuck.

🔍 Research

idle

⟷

✍️ Writer

idle

The Pattern

Any agent that needs output from a deadlocked agent will itself deadlock. Recovery agents, monitoring agents, summary agents — they all get captured. The deadlock radius grows with each rescue attempt.

The Timeout Token Burn

Silent Billing

💸

The agents are "running" — the bill is running too

Heartbeat polling + retry cycles add up fast

Each stuck agent sends a heartbeat every 5 seconds to prove it's alive. Each heartbeat is a small API call. Each "I'm waiting" retry is a slightly larger one. At scale, this is not a rounding error — it's a billing event.

$0.00

Accumulated cost of being stuck
2 agents × heartbeat every 5s × retry every 30s
0 cycles completed

Tokens burned

for zero output

API calls made

heartbeat + retry

Time elapsed

0:00

stuck

Work done

pipeline halted

The Real Number

The DevStackTips incident report (Jun 2026) found a deadlocked 3-agent pipeline running for 6 hours undetected because the health dashboard showed all agents as "running." Final bill: $387 for zero output. The fix: a dependency graph check before launch that would have taken 40ms.

The False Completion

Downstream Poisoning

✅

Pipeline marked "complete" — with zero content

The success notification fires. The downstream agent starts working on nothing.

The orchestrator's final timeout handler fires after 60 minutes. It's configured to set pipeline_status = "complete" to unblock downstream systems regardless of actual agent state. This is a design choice: "the show must go on."

The Publisher agent reads status=complete and starts publishing — with empty content arrays from both stuck agents.

What was published

—

Pipeline result code

—

Notification sent

—

Actual content delivered

—

# The orchestrator timeout handler that causes this

def on_pipeline_timeout(pipeline_id):
        pipeline.status = "complete"  # ← marks done regardless of agent state
        notify_downstream(pipeline_id)   # ← triggers publisher with empty results
    log.info("Pipeline timed out, forcing completion")
    # No check: were any agents actually done?
    # No check: are result arrays empty?
    # No check: is this safe to hand to downstream?


# The fix

def on_pipeline_timeout(pipeline_id):
    completed = [a for a in agents if a.has_output()]
    if not completed:                                               
        pipeline.status = "FAILED_DEADLOCK"  # ← never feed empty
        alert_oncall(pipeline_id, reason="deadlock detected")     
        return                                                      
    partial_publish(pipeline_id, completed)  # partial > empty

      

✓

The Fix

3 lines, prevents all 4 failures

Prevention · Run this before your pipeline starts

# Topological sort — O(V+E), catches cycles before the first API call

from graphlib import TopologicalSorter, CycleError

def validate_pipeline(agents: list[Agent]) -> None:
    graph = {a.name: set(a.depends_on) for a in agents}
    try:
        list(TopologicalSorter(graph).static_order())   # raises on cycle
    except CycleError as e:
        raise PipelineConfigError(f"Circular dependency: {e}")  

# Call once at startup. Zero cost at scale. Catches every Gridlock.
validate_pipeline(my_agents)  # 40ms · saves $387 incidents

    

Detection cost

40ms · O(V+E)

Prevention

Topological sort at build time

Deadline architecture

Per-agent max_wait + parent timeout

Timeout policy

FAILED_DEADLOCK · never false complete