Durable LangGraph Agents on Temporal: Crash Recovery at Scale

LangGraph defines what your agent does. Temporal keeps it alive when machines die, humans take three days to reply, and deploys roll mid-run. I run every production LangGraph graph as a Temporal Workflow with each model and tool call as a retryable Activity, and recovery becomes runtime behavior, not code I maintain.

• Core fact: Temporal Python SDK 1.27+ runs LangGraph StateGraph and Functional API graphs without rewrites via LangGraphPlugin.
• Core fact: Each node declares execute_in: activity or workflow, giving timeouts, retries, and durable human waits at zero compute.
• Core fact: I killed a worker mid-wait in staging and the approval resumed cleanly on a new worker with zero lost tokens.

I learned this after losing a 47-minute research run at 2 a.m. because a Kubernetes node recycled. That pain is why this pattern now powers everything I ship. If you run agents longer than five minutes, this setup pays for itself in one incident. For background on isolation primitives, I keep coming back to my ephemeral Firecracker sandbox workflow for tool containment.

Why Single-Process LangGraph Dies in Production

LangGraph checkpoints state. It does not checkpoint execution. Your run lives in one Python process. When that process dies, something has to detect the failure, find the right checkpoint, and re-enter the graph at the right node. In demos you restart manually. In production with 200 concurrent runs, you need an orchestrator.

When we benchmarked this at SaaSNext on a document triage fleet, we observed 11% of runs over 20 minutes failed from transient OpenAI 429s, pod evictions, or tool timeouts. Manual resume took 8-14 minutes per incident. That is not a framework bug. It is a missing layer.

Human review makes it worse. LangGraph interrupt() halts the run. You now own the waiting problem: persist pending state, track who waits on whom, notice when approval arrives, and re-enter correctly. I built this with Postgres + Redis once. It worked for 30 waits. At 900 waits it leaked timers, lost two approvals during a deploy, and paged me at 3 a.m. I deleted that code the next week.

Temporal solves both with one primitive: the Workflow event history. Every step is journaled. A Workflow can die on worker A and resume on worker B with nothing lost. Model calls run as Activities with retry policies. Human waits park on durable Signals and Timers that cost zero compute while parked.

How the Temporal LangGraph Plugin Actually Works

The integration surface is small. You add execution metadata to nodes and register graphs with the plugin. No rewrite.

• Graph API (StateGraph) and Functional API (@entrypoint, @task) are both supported. Python 3.11+ is required for Functional API and interrupt() because of contextvars propagation through asyncio.create_task().
• Every node must set execute_in in metadata to activity or workflow. The plugin raises if you forget. You cannot set it globally, which prevents determinism bugs.
• Do not set LangGraph retry_policy on nodes. Use Temporal RetryPolicy via metadata. Temporal owns retries.
• Use InMemorySaver if you need a checkpointer for interrupts. Temporal handles durability, so skip Postgres or Redis checkpointers.
• Conditional edge functions always run in the Workflow. They must be deterministic and async. No network calls, no random(), no clock reads inside Workflow code.

Use activity for anything non-deterministic, long-running, or fallible: LLM calls, tool calls, interrupt() nodes. Use workflow for pure routing, state reshaping, and parent orchestration that fans out to child graphs. This split is the entire reliability model.

flowchart TD
  Client --> WF[Temporal Workflow: OrderRun]
  WF --> A1[Activity: reason node - GPT-6 Sol call]
  A1 --> A2[Activity: act node - tool call]
  A2 --> H{Human approval needed?}
  H -- Yes --> W[Park on Signal: zero compute wait]
  W --> A3[Activity: resume with Command]
  H -- No --> A3
  A3 --> Done[Complete + audit trail in history]

Streaming works too. Set streaming_topic on the plugin, construct WorkflowStream() in @workflow.init, and calls to get_stream_writer() publish token chunks. Activity nodes publish via batched signal (default 100ms). Expect at-least-once delivery on retry, so dedupe on sequence IDs client-side.

Step 1: Project Setup and Version Pins

I pin everything. LangGraph + Temporal drift fast, and a minor bump broke my streaming batch path in August. Here is the exact layout I use in production today.

File: requirements.txt

temporalio>=1.27.0
langgraph>=0.6.5
langchain-openai>=0.3.0
pydantic>=2.8.0
pydantic-settings>=2.5.0
python-dotenv>=1.0.1
pytest>=8.3.0

File: config.py

from pydantic_settings import BaseSettings

class Settings(BaseSettings):
    temporal_address: str = "localhost:7233"
    temporal_namespace: str = "default"
    task_queue: str = "langgraph-agents"
    openai_api_key: str
    model_name: str = "gpt-6-sol"
    max_retries: int = 5
    activity_timeout_sec: int = 120
    human_approval_timeout_hours: int = 72

    class Config:
        env_file = ".env"

settings = Settings()

Terminal:

python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
temporal server start-dev --port 7233

War story one: I once ran temporalio==1.26 with the new plugin and spent 90 minutes chasing missing execute_in errors that were actually version skew. The plugin requires 1.27+. The error message was correct, my lockfile was stale. Now CI fails if pip freeze drifts. Small check, big payoff.

For cost context on model choice, my Terminal-Bench monorepo comparison breaks down where Sol-class models beat flagship pricing.

Step 2: Define the LangGraph Graph With Execution Metadata

This is a triage agent with reason, act, and human approval. Notice every node carries execute_in in metadata. The router edge is pure logic, so it stays in the Workflow.

File: graph.py

from typing import TypedDict
from langgraph.graph import StateGraph, END
from langchain_openai import ChatOpenAI
from config import settings

class AgentState(TypedDict):
    input: str
    draft: str
    approved: bool
    attempts: int

llm = ChatOpenAI(model=settings.model_name, temperature=0.2, timeout=60)

async def reason_node(state: AgentState):
    try:
        resp = await llm.ainvoke(f"Draft action plan for: {state['input']}")
        return {"draft": resp.content, "attempts": state.get("attempts", 0) + 1}
    except Exception as e:
        print(f"[reason] failed: {e}")
        raise

async def act_node(state: AgentState):
    # Tool call simulation with timeout handling
    # Replace with real MCP tool or API call
    await __import__("asyncio").sleep(1)
    return {"draft": state["draft"] + "
[tool-output] fleet_status=ok"}

async def ask_human_node(state: AgentState):
    from langgraph.types import interrupt
    # Pauses graph, Temporal parks on Signal
    decision = interrupt({"draft": state["draft"]})
    return {"approved": decision == "approve"}

def route_after_human(state: AgentState):
    return "end" if state.get("approved") else "reason"

def build_graph():
    g = StateGraph(AgentState)
    g.add_node("reason", reason_node, metadata={"execute_in": "activity"})
    g.add_node("act", act_node, metadata={"execute_in": "activity"})
    g.add_node("ask_human", ask_human_node, metadata={"execute_in": "activity"})
    g.set_entry_point("reason")
    g.add_edge("reason", "act")
    g.add_edge("act", "ask_human")
    g.add_conditional_edges("ask_human", route_after_human, {"reason": "reason", "end": END})
    return g

Key detail: ask_human uses interrupt(). Under Temporal, that interrupt serializes and the Workflow waits on a Signal. The dashboard or CLI signals approval, and the graph resumes with Command(resume=...). The rest of the graph keeps moving for other orders because each wait is its own parked Workflow.

I route all low-risk reads through a stateless FastMCP RBAC server so tool calls carry bearer identity without session state.

Step 3: Wire the Temporal Worker and Workflow

The Workflow hosts the graph. Activities execute the nodes. Timeouts and retries live here, not in LangGraph.

File: workflows.py

from datetime import timedelta
from temporalio import workflow
from temporalio.common import RetryPolicy
from temporalio.contrib.langgraph import LangGraphPlugin
from graph import build_graph

@workflow.defn
class TriageWorkflow:
    @workflow.init
    def __init__(self):
        from temporalio.contrib.langgraph import WorkflowStream
        self._stream = WorkflowStream()

    @workflow.run
    async def run(self, user_input: str) -> dict:
        graph = build_graph()
        compiled = graph.compile()
        result = await compiled.ainvoke({"input": user_input, "draft": "", "approved": False, "attempts": 0})
        return result

plugin = LangGraphPlugin(
    graphs={"triage": build_graph()},
    streaming_topic="token-stream",
    default_activity_options={
        "start_to_close_timeout": timedelta(seconds=120),
        "retry_policy": RetryPolicy(maximum_attempts=5, backoff_coefficient=2.0),
    },
)

File: worker.py

import asyncio
from temporalio.client import Client
from temporalio.worker import Worker
from config import settings
from workflows import TriageWorkflow, plugin

async def main():
    client = await Client.connect(settings.temporal_address, namespace=settings.temporal_namespace)
    worker = Worker(client, task_queue=settings.task_queue, workflows=[TriageWorkflow], plugins=[plugin])
    print(f"Worker on {settings.task_queue} ready")
    await worker.run()

if __name__ == "__main__":
    asyncio.run(main())

Run it, kill the worker mid-run, restart it. The run resumes from history. Finished Activities return recorded results without new API calls, so you do not pay twice. That replay guarantee alone cut our retry spend 34% in one month.

Timeout pattern for humans: workflow.wait_condition with a durable Timer. Escalate after four hours, auto-reject after 72. Timers survive crashes like everything else. My inference FinOps comparison shows why avoiding duplicate model calls matters at scale.

Step 4: Verify, Load Test, and Ship

Do not ship without these three checks. I skip one and get paged, every time.

1. Crash test: Start 20 runs, kill -9 the worker mid-reason, restart. All 20 must complete. Any re-execution of a finished Activity that bills again is a bug in your idempotency, not Temporal.
2. Human wait test: Park 50 approvals for two hours, redeploy mid-wait. Zero lost Signals. Query pending drafts via Temporal query before signaling.
3. History growth test: Run a 6-hour monitoring loop with continue-as-new. If history exceeds 10k events without rollover, you will hit limits. Prune to latest checkpoint only.

War story two: our overnight batch lacked jitter on fallback retries. OpenAI bill spiked $240 in six hours because every failed Activity retried in lockstep and hammered the API. Temporal RetryPolicy with backoff_coefficient=2.0 plus maximum_interval=60s fixed it. Add jitter at the client too. I now set initial_interval=2s and cap concurrency at 40 Activities per worker. Without that cap, a thundering herd will exhaust your Postgres connections even though Temporal itself is fine.

When NOT to Use This Pattern

Be direct: if your agents finish in under 60 seconds, never wait on humans, and never survive deploys, skip Temporal. You add operational overhead: a cluster to run (self-hosted or Cloud), concepts to learn (Workflows, Activities, Signals), and deterministic constraints to respect. A simple FastAPI + LangGraph service is faster to ship.

Also watch payload size. Temporal has a 2MB gRPC payload cap. If you pass 40MB PDFs through state, store them in S3 or R2 and pass IDs. LangGraph-native deployments handle large payloads more gracefully. And if you need token-level observability, cost per span, and eval queues out of the box, pair with LangSmith. Temporal history shows step completion, not prompt quality.

Hybrid is valid. I run short chat turns on plain LangGraph and only route long-horizon orders, dispatch loops, and approval chains through Temporal. The cross-framework demo with ADK assessment + LangGraph dispatch on one order shows why: different teams keep their stacks, Temporal provides the durable layer underneath without rewrites.

Ship the durable path for anything that waits or must not lose work. Keep the light path for everything else. That split kept our fleet at 99.9% completion through two region failovers last quarter.

By Deepak Bagada, Founder & Editor-in-Chief at Daily AI World. I build agentic systems at SaaSNext and write from production incidents, not docs. Connect at @deeepakbagada.