Graceful Shutdown

Last updated: July 24, 2026

Graceful Shutdown System

A structured shutdown coordinator that ensures all subsystems drain cleanly before the process exits. Added July 2026.

Purpose

Prevents data loss and orphaned goroutines when the application receives SIGTERM/SIGINT. Each subsystem gets a chance to finish in-flight work before being forcefully stopped.

Architecture (assistant/graceful.go)

Core Types

Type Purpose
Subsystem Interface: Name(), Stop(), ForceStop(), MaxDrainTime()
ShutdownManager Coordinates shutdown across all registered subsystems

Shutdown Flow

SIGTERM/SIGINT received
  → ShutdownManager.Stop()
  → For each registered Subsystem:
      1. Call Stop() in a goroutine
      2. Wait up to MaxDrainTime()
      3. If timeout: call ForceStop()
      4. If global 2-min timeout: ForceStop() all remaining
  → Execute registered stop callbacks
  → Close done channel

Key Design Decisions

  • Two-phase stop: Stop() (graceful drain) → ForceStop() (abandon operations) per subsystem
  • Per-subsystem timeout: Each subsystem declares its own MaxDrainTime() — fast subsystems drain quickly, slow ones get more time
  • Global 2-minute hard cap: Even if subsystems have long drain times, the total shutdown cannot exceed 2 minutes
  • Panic-safe callbacks: Stop callbacks are wrapped in recover() to prevent one callback from blocking others
  • Idempotent: Stop() returns error if already in progress (no double-shutdown)

Registration

sm := InitShutdownManager()
sm.RegisterSubsystem(mySubsystem)           // implements Subsystem interface
sm.RegisterStopCallback(func() { ... })     // additional cleanup

Querying State

Method Purpose
IsShuttingDown() Check if shutdown is in progress (read-lock safe)
GetShutdownChannel() Returns channel closed when shutdown completes

Integration

The ShutdownManager is initialized at startup (InitShutdownManager()) and subsystems register themselves as they start. The main goroutine blocks on OS signals, then calls sm.Stop().

Key Files

File Purpose
assistant/graceful.go ShutdownManager, Subsystem interface, Stop/ForceStop logic

Cross-References