Relay Dispatch / fictional service01 / Review brief

INC-042 · SEV-2 / illustrative

When the queue stopped moving

A retry policy change increased queue pressure and delayed dispatch confirmations. Recovery restored throughput; the long-tail retry behavior remains under investigation.

Blameless review · separate observed facts from explanations.

Relay Dispatch / fictional service02 / System boundary

Locate the failure in its system.

Dispatch API

Accepts dispatch requests and returns a durable request identifier.

Acceptance did not guarantee immediate worker completion; no lost-write claim was established.

Queue & workers

Schedules dispatch jobs and performs partner delivery attempts.

The queue accumulated retries during the incident window.

Partner adapter

Translates worker requests and records partner responses.

Partner errors were observed; their contribution requires controlled replay.

Relay Dispatch / fictional service03 / Impact

Define the impact before the cause.

47minutes of elevated delay

Synthetic interval from first sustained latency alert to return below the declared threshold.

1,240delayed confirmations

Synthetic count of accepted requests exceeding the five-minute confirmation threshold.

0confirmed lost writes

No lost writes were found in the bounded reconciliation sample; this is not proof about all events.

Measurement boundary

Counts cover the fictional dispatch region and sampled request logs only. Payment, inventory and unrelated regions were outside the reviewed evidence.

Relay Dispatch / fictional service04 / Evidence timeline

Reconstruct what was observed.

All times are UTC. Sequence alone does not establish causation.

2026-09-28 09:02:00 UTC

Latency alert crossed the sustained threshold.

Synthetic monitor export M1; five-minute rolling confirmation latency.

2026-09-28 09:11:00 UTC

Response team isolated the retry-policy change.

Synthetic deployment log D1 and queue dashboard snapshot Q1.

2026-09-28 09:24:00 UTC

Previous retry policy was restored.

Synthetic rollback record D2; worker configuration hash matched the prior version.

2026-09-28 09:49:00 UTC

Confirmation latency returned below threshold.

Synthetic monitor M2; backlog continued to drain after recovery.

Relay Dispatch / fictional service05 / Response decisions

Decisions made with imperfect information.

Response choice

Restore the prior retry policy.

Queue depth rose after the policy deployment while worker capacity was unchanged.

Tradeoff / The response favored a known configuration over diagnosing every factor before recovery.

Response choice

Keep accepted requests durable.

Reconciliation showed queued request IDs were retained.

Tradeoff / Preserving accepted work meant recovery included a controlled backlog drain.

Response choice

Publish a bounded status update.

Delay was confirmed; data loss and partner causality were not established.

Tradeoff / The update described known impact without claiming a complete causal explanation.

Relay Dispatch / fictional service06 / Contributing factors

Explanations carry an evidence status.

Confirmed

Retry amplification

Synthetic replay with the new policy produced more attempts per failed job than the prior policy under the same fixture.

Confirmed

Insufficient queue headroom

Synthetic capacity review showed the documented burst envelope did not include the higher retry multiplier.

Hypothesis

Partner timeout clustering

Timeouts co-occurred with the backlog. Controlled replay is needed to establish their independent contribution.

A hypothesis remains a hypothesis until the proposed investigation supplies evidence. Avoid attributing system failures to personal character.

Relay Dispatch / fictional service07 / Open questions

Make uncertainty actionable.

01

How much did clustered partner timeouts contribute?

Replay a fixed request set with controlled timeout distributions and compare queue growth.

INVESTIGATION / Reliability team

02

Did any confirmation exceed the bounded sample window?

Reconcile the complete request-ID set against final delivery receipts and inspect unmatched records.

INVESTIGATION / Data operations

03

Would the same policy fail under a different burst shape?

Run sustained and spiky workloads separately using the recorded policy configurations.

INVESTIGATION / Performance team

Relay Dispatch / fictional service08 / Controls & detection

Which defenses held, and which did not?

Latency alert

Detected sustained customer-visible delay.

The alert followed impact; queue-pressure signals could provide earlier warning.

Rollback procedure

Restored the known configuration without discarding accepted work.

The runbook did not specify a maximum safe drain rate for the accumulated backlog.

Pre-release load test

Covered normal request volume and single-attempt worker behavior.

It did not exercise retry amplification combined with clustered partner timeouts.

Relay Dispatch / fictional service09 / Follow-up work

An action is more than a task name.

A1 · Proposed

Bound retries per job

Controlled timeout replay stays within the declared attempt budget without dropping accepted jobs.

OWNER / Dispatch engineering
DUE / Next change review

A2 · In progress

Test the burst envelope

Publish the queue-growth and recovery results for sustained, bursty and clustered-timeout fixtures.

OWNER / Performance team
DUE / Before the policy is reconsidered

A3 · Proposed

Add backlog drain guidance

An on-call engineer completes the rehearsal using the updated runbook and records the observed drain behavior.

OWNER / Reliability team
DUE / Next response rehearsal

Relay Dispatch / fictional service10 / Verification plan

Specify the proof before closing the work.

A1

Replay a fixed timeout fixture with the bounded retry policy.

Save attempt counts, final request states and the configuration hash; reject a pass if any accepted job disappears.

A2

Compare queue growth across three documented traffic shapes.

Attach the workload definitions and time-series exports, including the recovery interval.

A3

Run a handoff rehearsal with an engineer who did not write the runbook.

Record the chosen drain rate, observed latency and any ambiguous instruction before marking the action verified.

Relay Dispatch / fictional service11 / Communication plan

Share what is known, with its limits.

Affected users

Confirm the observed delay window and recovery status; state the limits of the current reconciliation.

OWNER / Incident communications lead

Internal service owners

Share the retry-policy rollback, current operating bounds and the unresolved partner-timeout hypothesis.

OWNER / Response coordinator

Follow-up reviewers

Provide action evidence, unresolved questions and any new impact findings at the scheduled review.

OWNER / Reliability lead

Relay Dispatch / fictional service12 / Learning loop

Keep the improvements observable.

What helped

The response team used a shared request-ID view to preserve a common understanding of impact.
Rollback restored the prior behavior while retaining accepted requests for controlled processing.
The public update separated confirmed delay from unverified explanations.

Next review

Review the replay evidence, complete the bounded reconciliation and inspect action proofs before closing the incident follow-up.

Recheck action evidence and unresolved hypotheses. Do not close an action merely because implementation finished.