Reliable Scheduled Jobs: Idempotency, Missed Runs, and Time Zones

A billing job is scheduled for midnight. The scheduler restarts, the worker times out, and an operator clicks “run again.” Now three attempts refer to one business operation, and none of their process IDs explains whether the customer has already been charged.

Give each intended occurrence a durable identity that survives scheduler retries, worker restarts, and manual replay. A cron expression decides when work becomes due. It does not prove that the work happened exactly once.

This distinction applies to reports, account renewal, reconciliation, and content publication. The acceptable failure policy differs, but the scheduler should never be the only record of business completion.

Name the occurrence before starting an attempt #

Separate three identities:

schedule:    monthly-invoice for tenant 42
occurrence:  billing period 2026-09
attempt:     worker execution 8f2a...

The occurrence identifies the business obligation. The attempt identifies a particular effort to fulfill it. Retrying creates another attempt, not another obligation.

For a fixed-instant schedule, the intended UTC fire time can identify an occurrence. For monthly billing, a validated billing-period identifier is usually clearer than the instant a worker happened to start. Do not derive the key from now() during execution: a delayed job could accidentally process the next period.

A minimal key builder can preserve those boundaries without ambiguous separators:

function occurrenceKey(tenantId, scheduleId, businessPeriod) {
  const parts = [tenantId, scheduleId, businessPeriod]
  if (parts.some(value => typeof value !== 'string' || value.length === 0))
    throw new Error('Occurrence identity requires three non-empty strings')
  return JSON.stringify(parts)
}

Validate the period format and ownership before calling it. Keep the key stable across equivalent retries. If a schedule definition changes, decide explicitly whether an existing occurrence is superseded or still owed; a version number added casually to the key can turn a deployment into duplicate work.

Persist the obligation separately from the worker #

Here is a small PostgreSQL ledger skeleton, reviewed against the documentation but not executed against a database for this article:

CREATE TABLE scheduled_occurrence_demo (
  tenant_id bigint NOT NULL,
  schedule_id text NOT NULL,
  business_period text NOT NULL,
  due_at timestamptz NOT NULL,
  state text NOT NULL DEFAULT 'pending'
    CHECK (state IN ('pending', 'running', 'succeeded', 'needs_review')),
  PRIMARY KEY (tenant_id, schedule_id, business_period)
);

INSERT INTO scheduled_occurrence_demo
  (tenant_id, schedule_id, business_period, due_at)
VALUES
  (42, 'monthly-invoice', '2026-09', '2026-10-01T00:00:00Z')
ON CONFLICT DO NOTHING;

The primary key and ON CONFLICT make repeated creation of that ledger entry harmless. They do not claim the job, recover a stuck worker, or make an external charge atomic. PostgreSQL INSERT.

Those omissions matter. A worker protocol needs a transactional claim, attempt evidence, a bounded lease or other ownership mechanism, and a recovery rule. If a paused worker can resume after losing ownership, consider fencing stale writers. A running flag alone is not a distributed lock.

A successful attempt is more than a successful process #

For a purely database-local task, it may be possible to commit the business update and occurrence completion in the same transaction. For a payment or email, a crash can happen after the external effect but before the local completion record.

Use a stable downstream idempotency key where the provider supports it, and understand that provider’s retention and request-matching rules. If the outcome is ambiguous, reconcile before issuing a new effect. This is the same boundary discussed in durable workflows and idempotency.

The product policy should say what “succeeded” means. A report might succeed when an artifact is durably stored, with notification tracked separately. If success instead means “email sent,” restarting a failed notification step should not regenerate a different financial report under the same occurrence.

Missed schedules need different catch-up policies #

Kubernetes documents that CronJob execution is approximate and jobs should be idempotent. Its starting deadline and concurrency settings affect execution, while concurrency policy applies to jobs from the same CronJob—not every process touching the same business resource. Kubernetes CronJob documentation.

Translate that into a product rule rather than relying on a scheduler default:

These are design choices, not universal defaults. Replaying every missed minute after a long outage can overload the very database the job is meant to repair. Rate-limit catch-up separately from normal scheduling, and surface the age of the oldest unresolved occurrence.

UTC storage does not define local calendar meaning #

“Every 24 hours” and “every day at 09:00 in New York” are different promises. Store execution instants in UTC, but retain the IANA time-zone identifier and calendar rule when the product follows local time.

Daylight-saving transitions create missing or repeated local clock times. Scheduler implementations choose policies: EventBridge Scheduler documents skipping a nonexistent scheduled time during the spring transition and invoking only once for the repeated fall time. Its day-based rate schedules represent 24-hour durations. Do not assume another scheduler makes the same choices. EventBridge schedule semantics.

Test the transition dates for supported zones, not only today’s date. Include an explicit rule for schedule edits, late manual retries, and a customer’s time-zone change during an unfinished billing period. Those are business semantics, not parser edge cases.

The operational contract #

Before release, require evidence for duplicate delivery, a crash before completion, an ambiguous external response, a missed interval, and overlapping attempts. A test should prove the final business state, not merely count worker invocations.

Monitor due-to-start delay, due-to-completion delay, oldest unresolved period, and occurrences requiring review. Keep attempt logs attached to the durable occurrence so support can answer one question: was this obligation fulfilled?

The trade-off is a little persistent state and recovery machinery in exchange for explainable operation. For disposable telemetry refreshes, that may be more than you need. For money, entitlements, or irreversible publication, process exit codes are too weak a business ledger.

Continue with systems architecture and bounded queues to connect the scheduler to the rest of the capacity model.

References #

• Kubernetes CronJob limitations and policy
• EventBridge calendar and daylight-saving behavior
• PostgreSQL conflict handling for inserts