SUMMARY: A degraded-service drill tests the participant-facing rules that take effect when a critical dependency is slow, unavailable, stale, or contradictory. It asks whether the service remains bounded, legible, and recoverable before a real incident decides the answer.
Choose one consequential workflow such as account recovery, payment release, access revocation, eligibility verification, or correction delivery. Run the drill in a safe environment with synthetic records and named observers.
1. Select the Dependency
Use the dependency register to identify one service whose delay can alter a real outcome. Record its owner, normal contract, consumers, fallback, and recovery path.
2. Name the Consequence Classes
Separate inconvenience from financial, access, safety, identity, and corrective consequences. The same outage may justify continuity for one action and a pause for another.
3. Inject a Bounded Failure
Simulate a timeout, stale response, contradictory version, unavailable queue, or partial regional outage. Define the start, stop, and abort conditions before injection.
4. Observe the Participant Route
Follow the same interface and support path a participant would use. Capture the visible status, instructions, retry behavior, alternate evidence path, and whether the system admits uncertainty.
5. Exercise Recovery Privilege
Use the control plane map to identify who can activate a fallback, approve an override, restore data, or pause a consequential decision. Confirm that emergency access is logged and expires.
6. Measure the Degraded Window
Track detection time, decision time, queue growth, stale-state age, participant notice, manual workload, and the oldest unresolved case. Compare the results with the relevant staleness budget.
7. Restore and Reconcile
Return the dependency, replay deferred work, revoke temporary access, rebuild derived state, and verify every consequential recipient. Recovery is incomplete while the emergency path can still govern.
8. Publish the Decision Record
Document which outcomes were allowed, denied, or queued; who authorized the rule; what evidence was preserved; and which control must change before the next drill.
9. Verify the Exit Conditions
Confirm that queues are draining, temporary roles are revoked, participants receive a final status, and monitoring has returned to normal thresholds. Set a deadline for every unresolved exception rather than allowing the drill state to become background operations.
Operator rule: a fallback is not ready until its consequences, authority, expiration, and reconciliation have all been tested.
Continue the discussion in the Clandestinia forum.