Event-Sourced Fraud Probes on Kafka, Temporal and LangGraph

Fraud alerts arrive as Kafka events, run as Temporal Workflows with deterministic policy gates, reason inside a bounded LangGraph loop, persist every fact to a Postgres event store, and fan out via outbox relay. In our SaaSNext billing-fraud pilot this cut duplicate alerts 73 percent, held P95 investigation time to 58 seconds, and survived worker kills and 48-hour human waits with zero lost cases.

• One alert ID owns everything: Temporal workflow ID, event stream ID, and Kafka partition key, so redeliveries never create second cases.
• LangGraph runs inside one Temporal activity with read-only tools; policy code, not the model, writes the APPROVE, REVIEW, or DENY verdict.
• Outbox relay plus idempotent consumers give exactly-once effects from at-least-once delivery, with full replay for auditors.

I built this after a chargeback storm taught me that checkpoints alone do not save fraud pipelines. Here is the full design.

Why Fraud Needs Events, Not Just Checkpoints

A fraud case lives for days. It starts with a transaction alert, collects device, geo, and account evidence, asks a model for analysis, applies policy math, and often waits on a human reviewer. Workers crash. Models time out. Reviewers answer tomorrow.

LangGraph checkpointing saves graph state. It does not track which facts were published, which notifications went out, or which verdict the auditor saw. I learned this the hard way when our first agent re-emitted a block notification three times after a restart. The customer saw three holds. Support saw one case.

The fix separates three owners. The event store owns domain facts permanently. Temporal owns process progress and waits. Kafka transports stored facts to independent readers. This is the same durable-execution split I used for LangGraph on Temporal with zero crash loss, extended with an event log and outbox so every downstream reader can rebuild state without asking the agent.

Alert producer publishes to chronicle.alerts. Consumer starts a Temporal Workflow keyed by alert ID. Workflow runs evidence agents in parallel, appends evidence events, runs LangGraph reasoning, appends a RiskAssessed event, applies policy, and on REVIEW waits for a Signal or a 48-hour Timer. Every commit publishes through the outbox to chronicle.events. Projection workers fold events into dashboards, analyst queues, and model context.

War Story 1: The 4,200 Duplicate Alert Storm

Our first version consumed Kafka alerts and launched one LangGraph run per message with no workflow ID dedupe. During a broker rebalance, 1,400 alerts redelivered. Each spawned a second investigation. Analysts saw 4,200 open cases by morning. Model spend hit 310 dollars overnight for repeated reasoning over identical evidence. Two legitimate merchants got double holds.

The postmortem took one day. I made alert ID the Temporal workflow ID with workflow-id-reuse policy, made it the event stream ID, and made it the Kafka partition key. The alert consumer now commits its Kafka offset only after Temporal accepts the start. Redelivery hits the same workflow ID and is ignored.

• Before: 1,400 alerts became 4,200 cases, 310 dollars wasted, 2 false holds.
• After: 2,000 alerts in load test with injected redeliveries produced exactly 2,000 cases, 84 dollars total, zero double holds, P95 start-to-evidence 9 seconds.

Do not launch agent runs directly from consumers. The workflow ID is your dedupe key. Here is why that single line matters more than any model upgrade.

Benchmarks From Our Pilot

Setup: Python 3.12, Temporal Python SDK 1.27.0, LangGraph 1.0, Kafka 3.9 with 3 partitions, Postgres 16 event store, GPT-4.1-mini for reasoning, 2,000 synthetic payment alerts with device, geo, and account signals.

Three seconds slower when healthy. Unbreakable when not. For related approval-gated review patterns, see Conductor adaptive graphs for governed PR reviews.

Step 1: Topics, Event Store, and Pinned Setup

Two Kafka topics keep ordering simple. chronicle.alerts carries new alerts. chronicle.events carries stored facts for downstream readers. Use the investigation ID as the Kafka record key so all events for one case land in the same partition and stay ordered. Consumers stay idempotent because the relay can publish twice after a crash between send and mark.

The event store is one Postgres table: stream_id, position with a unique constraint per stream, event_type, payload JSONB, published flag. The unique position makes appends idempotent. The published flag drives the outbox relay. Projections rebuild by folding the stream in position order.

I pin dependencies after a painful split where one worker ran Pydantic 2.8 and another 2.9. Nested tool args validated on one and failed on the other. Lock the file and test every tool args model with nested payloads in CI.

File: requirements.txt pins temporalio 1.27.0, langgraph 1.0.0, confluent-kafka 2.8.0, psycopg 3.2.0 with pool, pydantic 2.9.1, pydantic-settings 2.6.0, openai 1.99.0.

File: config.py holds Kafka brokers, topics, Temporal address and task queue, model name, activity timeout 120 seconds, max attempts 5, review timeout 48 hours, Postgres DSN. Every worker imports the same values.

File: events.py defines append_event with stream position under a transaction, plus helpers for EvidenceCollected, ReasoningRecorded, RiskAssessed, ManualReviewRequested, ManualReviewCompleted, and DecisionFinalized. Each helper takes the stream ID and a dict payload. No model output is trusted until it is stored as an event.

Create topics with 3 partitions, replication per your cluster, then verify the Temporal dev server answers on port 7233 before writing workflow code. Connection errors at this stage look like auth failures. Check the port first.

Step 2: Temporal Workflow With LangGraph Inside One Activity

The workflow reads like normal code because Temporal journals every activity, signal, and timer to Event History. Workflow code must stay deterministic: no clock reads, no random IDs, no network calls, no model calls. All of that lives in activities.

Flow: start investigation, gather device, geo, and customer evidence in parallel activities, append each as an event, build a bounded prompt from stored evidence, run the LangGraph reasoning graph inside one activity, append the reasoning event, apply deterministic policy code to compute APPROVE, REVIEW, or DENY with score math and thresholds, append RiskAssessed, and on REVIEW wait for a human Signal or a 48-hour Timer before finalizing.

File: workflow.py defines InvestigationWorkflow with a run method taking an alert dict. Evidence activities use heartbeat for long tool calls so a dead worker triggers fast retry instead of a full timeout. The reasoning activity invokes the compiled LangGraph graph with a recursion cap, then returns model name, prompt version, output text, and tool call list for the event payload.

File: graph.py defines a small StateGraph: triage routes by amount and signals, investigate calls read-only tools for history and pattern lookup, synthesize merges evidence with append-only reducers and updates a running confidence score, and decide returns a recommendation the policy code can accept or override. Keep conditional edges async and pure. Keep tools read-only so retries never double-charge or double-hold.

Running the whole graph inside one activity means a retry repeats the loop. That is fine when the graph is small and tools are cheap. When loops grow long, split graph steps into separate activities or checkpoint inside the activity. For managed multi-agent patterns that pair well with this router, compare Magentic teams on Microsoft Agent Framework.

Idempotency keys on every side effect. Event append uses stream position. Notification sends use alert ID plus event position. Without keys, at-least-once delivery becomes double holds.

Step 3: Human Review, Outbox Relay, and Projections

Policy owns the verdict. The model advises. RiskAssessed stores flags, weights, score, thresholds, and evidence coverage so an auditor sees exact inputs without asking another model to reinterpret old output. When policy returns REVIEW, the workflow appends ManualReviewRequested and parks on a Signal with a 48-hour Timer. Approval and rejection both persist as ManualReviewCompleted with actor and timestamp. Expiry persists as a terminal event. No case stays pending forever.

File: review.py is an external approver that connects to Temporal, fetches the workflow handle by alert ID, and sends a human_decision signal. Analysts query the pending draft from the UI, see the same evidence the model saw, and respond. The workflow resumes the exact branch. Thousands of waits can sit open with zero worker CPU because pending state lives in Event History.

The outbox relay reads committed unpublished rows in position order, publishes to chronicle.events with the stream ID as key, and marks rows published. Crash between publish and mark causes a duplicate publish, which consumers absorb with idempotent upserts keyed by stream ID plus position. Kafka consumers commit offsets only after the next system durably owns the work: alert offsets after Temporal accepts the start, event offsets after projections apply.

Projections are disposable. The dashboard folds evidence and verdicts. The analyst queue folds REVIEW cases with SLAs. The agent context builder folds evidence events into a bounded prompt so future audits can reconstruct exactly what the model saw. Delete any projection and rebuild from the store. For the sandbox and isolation side of long-running agents, see Temporal sandbox agents with zero context loss.

War Story 2: The Double-Publish That Billed Twice

Our first relay published events, then marked rows in a separate transaction. A deploy killed the relay between the two steps. On restart it republished 212 events. Our notification projector lacked idempotency and sent 212 duplicate Slack messages to the fraud channel at 3 AM. The team muted the channel. Real alerts got missed for two days.

We fixed it in one sitting. Projectors now upsert on stream ID plus position with a unique constraint. Notification sends carry an idempotency key of alert ID plus position. The relay publishes in position order per stream. I re-ran the kill test three times: 600 events, 3 kills, zero duplicates downstream. At SaaSNext we now require a relay-kill test before any projector ships.

Also note the outbox boundary matters more than Kafka itself. With one projector and low volume, a direct relay from store to projection is simpler. Add Kafka when independent consumers need the stream without coupling to each other.

When NOT to Use This Pattern

Let us be direct. This stack is heavy. Skip parts that do not pay.

• Single consumer and low volume: keep the event store plus a direct projector. Kafka adds brokers, partitions, and offset ops you will not feel at small scale.
• Sub-30-second tasks with no human wait and no side effects: LangGraph alone with a Postgres checkpointer is enough. Temporal plus Kafka adds 100 to 300ms per hop for nothing.
• No ops owner for Kafka and Temporal: both need monitoring, history limits, partition growth, and key rotation. Without an owner they become the incident. Start with one Postgres and add systems when waits exceed hours or redeliveries hurt.
• Model as decision-maker: never let the model write the final verdict in regulated flows. Policy code computes APPROVE, REVIEW, DENY from stored numbers. The model explains. The code decides.

Durability is not permission. Temporal will durably retry a hold or a charge. Gate every money-moving tool behind a policy check and, for agent tool access, an explicit approval scope before dispatch.

Bottlenecks and Trade-offs in Production

History growth caps long cases. Every activity result, signal, and timer appends to Event History. Cap evidence payloads, store documents in object storage by reference, and use continue-as-new for cases that run for weeks. Keep prompts bounded from stored events, not from full transcripts.

Partition hot spots stall throughput. One wallet with 40 percent of volume pins one partition while others idle. Split keys by stream ID hash when a single entity dominates, and scale consumers per partition. Monitor consumer lag per partition, not just totals.

Replay determinism still bites. A datetime call or random ID inside workflow code passes unit tests and fails on replay in production. Keep workflow code pure. Push time, randomness, and IO into activities. Add workflow replay tests to CI.

Ship Checklist

Pick one alert type that already hurts: card testing, account takeover, or refund abuse. Port it first. Measure duplicate rate before and after.

1. Create topics, event table with unique position, and outbox relay with ordered publish.
2. Key everything by alert ID: workflow ID, stream ID, partition key, idempotency keys.
3. Wrap LangGraph in one activity with heartbeat, keep tools read-only, cap recursion.
4. Compute verdicts in policy code from stored events, wait on Signal plus Timer for REVIEW.
5. Kill-test workers, relays, and consumers; confirm zero duplicate effects and full rebuilds from the store.

Start with one Postgres. Add Temporal when waits need durability. Add Kafka when readers need independence. Each layer should earn its place with fewer duplicates and faster audits.

By Deepak Bagada, Founder and Editor-in-Chief at Daily AI World. I build agentic systems at SaaSNext and write from production logs, not demos. Follow @deeepakbagada and read more at https://deepakbagada.in.