Cron Agents That Survive the Night: Locks, Keys, Heartbeats

My 2 AM reconciliation agent double-billed eleven customers last quarter. The 1 AM run slowed on a degraded API, the 2 AM run started anyway, and two instances reconciled the same invoices independently. Same state, independent decisions, duplicate charges. The cron line worked perfectly. That was the problem.

Cron agents need three layers the cron line cannot provide: overlap control so slow runs never double up, atomic idempotency so retries never re-apply side effects, and heartbeat monitoring so silent misses page before dashboards do. Three facts anchor the pattern:

• Overlap policies (skip, buffer, cancel) decide what happens when a run outlives its interval — the most consequential scheduling choice, and cron's default is always wrong.
• The outbox pattern makes trigger and work atomic: the started-record and the result commit in one transaction, or the run never happened.
• Heartbeat absence alerting catches the failure cron hides — the run that never started — which standard error monitoring misses entirely.

This is the background-execution discipline behind my durable pipelines, the same exactly-once instinct as my zero-lost-state fraud agents. Same guarantees, applied to time triggers instead of event streams.

The double-billing night that ended naive cron

The 1 AM run took 94 minutes against a 60-minute interval. No lock, no overlap check — the 2 AM run began with the same input snapshot. Both instances issued credits for the same eleven invoices. Reversals took two days, apologies took longer, and the postmortem found the root cause in one line of crontab.

Here's the catch. Cron answers when to start and nothing else. It does not know whether the previous run finished, whether this run already happened, or whether anyone noticed it didn't. Every one of those questions is load-bearing once agents write to external systems, and cron answers none of them.

That matches the 2026 proactive-agent analysis: background agents fail silently by default, and the run that never started is more dangerous than the run that errored. My metered tool calls already taught me idempotency keys are load-bearing for billing — scheduled runs need the same keys with time semantics.

The four failures of 0 2 * * * python agent.py

Don't do this: lengthening the interval to dodge overlaps. Slowness is unbounded — any fixed interval eventually collides. I treat trigger and execution as separate concerns with explicit policies between them.

The pattern: reliable trigger, guarded worker, watched heartbeat

flowchart TD
    SCHED[Schedule fires] --> OVERLAP{Previous run active?}
    OVERLAP -->|skip policy| SKIP[Skip, log, alert if frequent]
    OVERLAP -->|free| LOCK[Acquire distributed lock]
    LOCK --> OUTBOX{Outbox record exists?}
    OUTBOX -->|yes| DONE[Return success, no re-execution]
    OUTBOX -->|no| WORK[Do the work]
    WORK --> COMMIT[Commit result + record atomically]
    COMMIT --> BEAT[Write heartbeat + next expected]

Temporal Schedules provide the trigger layer natively — overlap policies from Skip to AllowAll, catchup windows for outages, backfill for gaps, pause-on-failure so a poisoned run stops the line. The worker provides the rest: lock, outbox, heartbeat. Either layer alone leaves a failure class open.

Step 1: Pin the schedule semantics

config.py

from pydantic import BaseModel

class CronAgentConfig(BaseModel):
    spec: str = "0 2 * * *"
    overlap: str = "Skip"
    catchup_window_s: int = 3600
    lock_ttl_s: int = 5400
    heartbeat_grace_s: int = 900
    pause_on_failure: bool = True
    backfill_on_recovery: bool = True

CONFIG = CronAgentConfig()

Skip is my default for mutating agents: a skipped run is observable and backfillable, while a double run is a customer-facing incident. BufferOne fits read-only digest agents where every run must eventually happen. AllowAll is banned for anything with side effects — concurrency without coordination is the incident I already lived.

Catchup windows need the same care as my approval-gate waits: too tight and outages drop runs silently, too wide and recovery stampedes. One hour matches my worst-case deploy window.

Step 2: Build the guarded worker

worker.py

async def scheduled_run(run_id: str, logical_key: str) -> dict:
    async with distributed_lock(f"cron:{logical_key}",
                                ttl=CONFIG.lock_ttl_s) as held:
        if not held:
            return {"ran": False, "reason": "overlap-locked"}
        if await outbox.exists(logical_key):
            return {"ran": False, "reason": "already-done"}
        try:
            result = await agent_work(logical_key)
        except Exception as e:
            logger.exception("run failed", extra={"key": logical_key})
            raise
        await outbox.commit(logical_key, result)  # atomic
        await heartbeat.write(next_expected())
        return {"ran": True}

The atomicity of the commit is what most implementations get wrong. Recording completion separately from the work creates a crash window producing either double-apply or never-applied. The record and the result commit together or the run never happened — then a retry is always safe.

At-least-once delivery is the assumed semantics everywhere: queues, webhooks, schedule retries. The event ID — here the logical key combining schedule identity and fire time — is the deduplication unit, checked before work and committed with it.

requirements.txt

temporalio==1.27.0
asyncpg==0.30.0
pydantic==2.8.0
structlog==24.4.0
python-dotenv==1.0.1

Pydantic v2.8 needs extra="allow" on schedule payload schemas or nested trigger metadata fails validation. I lost an afternoon to that exact error before pinning it.

Step 3: Alert on absence, backfill with intent

The heartbeat record carries a timestamp and the expected next run. A separate monitor pages when heartbeats go overdue past the grace period — this catches paused schedules, broken deploys, and silently-completed runs that produced nothing. Presence alerting is table stakes; absence alerting is the actual coverage.

Backfill is deliberate, never automatic-broad: after an outage, list the skipped logical keys, confirm each has no outbox record, and replay exactly those. The outbox makes backfill safe — already-done keys return success without re-execution, so the replay is idempotent by construction.

My guarded SQL fleet runs its nightly reconciliation on exactly this shape: Skip overlap, one-hour catchup, heartbeat-graced at fifteen minutes. Six months, zero duplicates, two backfills, one absence page that caught a paused schedule before morning.

The timezone war story: DST fired twice

My first schedule ran in America/New_York at 1:30 AM. Fall-back night fired it twice — 1:30 EDT and 1:30 EST — and the outbox saved us: the second firing found the logical key committed and returned success. Without the outbox that DST quirk was a duplicate-payout incident. Schedules now run in UTC unconditionally; DST is a presentation-layer concern that must never reach the trigger layer.

When NOT to build this

Let's be clear. A read-only digest with no side effects needs cron plus a lock, not the full apparatus — retries are free when nothing mutates. Sub-minute intervals belong on streams, not schedules; polling at that frequency is an event architecture wearing a cron costume. And one-off delayed tasks want a queue with visibility timeout, not a schedule entry.

Skip it for side-effect-free digests and sub-minute work. Build it where background agents mutate state, money, or records — everywhere a 2 AM duplicate becomes a customer apology.

Separate trigger from execution, commit atomically, watch for absence, and the whole class of silent-night incidents disappears: no overlaps, no doubles, no misses nobody noticed.

By Deepak Bagada, Founder & Editor-in-Chief at Daily AI World.