Collection State Machine

Collections follow a defined state machine that governs their lifecycle from creation to delivery.

Source files: apps/web/src/lib/collections/state-machine.ts (the transition table + RBAC), apps/web/src/lib/collections/transitions.ts (the transitionCollectionState() executor that validates, checks permissions, updates the row, and writes a chain-of-custody event).

State Diagram

pending
   │
   ▼
validating_creds ────────► error ◄──────────┐
   │                         │               │
   ▼                         │ (retry)       │
queued ◄─────────────────────┤               │
   │                         │               │
   ▼                         ▼               │
in_progress ───────────► error ──────────────┘
   │   ▲                     ▲
   ├──►paused────────────────┤  (paused → in_progress)
   │                         │
   ├──► partial ──► in_progress
   │       │                 │
   │       ▼                 │
   ▼    reviewing ◄──────────┘
completed ──► reviewing ──► signed_off ──► delivered
                  │
                  ▼
                error

(almost any non-terminal state ──► cancelled)

States

State	Description
pending	Collection created, awaiting processing
validating_creds	Testing credentials before collection
queued	Waiting in queue for worker
in_progress	Actively collecting data
paused	Manually paused by analyst
error	Failed with an error (recoverable — an analyst can re-queue or re-validate)
partial	Completed with some endpoint failures
completed	All endpoints successful
reviewing	Analyst reviewing before sign-off
signed_off	Analyst approved, ready for delivery
delivered	Customer notified, download available (terminal)
cancelled	Cancelled by customer or analyst (terminal)

Transitions

Happy Path

pending → validating_creds → queued → in_progress → completed → reviewing → signed_off → delivered

Error & Recovery Paths

validating_creds → error    // Credential validation failed
in_progress → error         // Collection failed
in_progress → partial       // Some endpoints failed
error → queued              // Analyst re-queues
error → validating_creds    // Analyst re-validates creds
partial → in_progress       // Analyst resumes a partial collection

error is not a terminal state — an analyst can recover from it by re-queueing or re-validating credentials, or cancel it.

Manual Interventions

in_progress → paused        // Analyst pauses
paused → in_progress        // Analyst (or system) resumes
reviewing → error           // Analyst rejects

Cancellation

cancelled is reachable from nearly every non-terminal state (pending, validating_creds, queued, in_progress, paused, partial, error). The actor allowed to cancel depends on the source state (see RBAC below). delivered and cancelled are the only terminal states.

State Details

pending

• Initial state when collection created
• Credential validation job queued
• Customer sees "Pending" status

validating_creds

• Worker testing credentials
• Makes test API call
• Transitions to queued on success
• Transitions to error on failure

queued

• Credentials valid
• Collection job in queue
• Waiting for worker availability

in_progress

• Worker actively collecting
• Progress updates via SSE
• Can be paused by analyst

paused

• Collection temporarily stopped
• Worker state preserved
• Can be resumed by analyst

error

• A failure occurred; error message stored
• Recoverable: an analyst can re-queue (error → queued) or re-validate credentials (error → validating_creds), or cancel

partial

• Collection completed
• Some endpoints failed
• May still be acceptable

completed

• All endpoints successful
• Ready for analyst review
• Files stored and hashed

reviewing

• Analyst examining results
• Verifying data integrity
• May sign off or reject

signed_off

• Analyst approved
• ZIP generation queued
• Preparing for delivery

delivered

• ZIP available for download
• Customer notified
• SAS URL generated

Events

Each state transition creates a chain of custody event. The event type is keyed off the target state (TRANSITION_EVENT_TYPES in state-machine.ts); some states reuse the closest available enum value:

Target state	Event Type
pending	collection_created
validating_creds	credentials_validated
queued	collection_started
in_progress	collection_started
paused	collection_paused
error	error_occurred
partial	collection_completed
completed	collection_completed
reviewing	analyst_reviewed
signed_off	analyst_signed_off
delivered	collection_delivered
cancelled	collection_cancelled

Retry Logic

Credential Validation

• 3 retries with exponential backoff
• On all failures → error state

Endpoint Execution

• 3 retries per endpoint
• Rate limit (429): Wait and retry
• Server error (5xx): Retry with backoff
• Client error (4xx): No retry, mark failed

Partial Handling

If some endpoints fail:

• Collection marked partial
• Analyst decides if acceptable
• Can sign off with documented failures

Implementation

State transitions are enforced in apps/web/src/lib/collections/state-machine.ts via the VALID_TRANSITIONS table:

export const VALID_TRANSITIONS: Record<CollectionStateType, CollectionStateType[]> = {
  pending: ["validating_creds", "cancelled"],
  validating_creds: ["queued", "error", "cancelled"],
  queued: ["in_progress", "cancelled"],
  in_progress: ["paused", "completed", "partial", "error", "cancelled"],
  paused: ["in_progress", "cancelled"],
  completed: ["reviewing"],
  partial: ["reviewing", "in_progress", "cancelled"],
  error: ["queued", "validating_creds", "cancelled"],
  reviewing: ["signed_off", "error"],
  signed_off: ["delivered"],
  delivered: [],   // terminal
  cancelled: [],   // terminal
};

Role-Based Transition Permissions

Beyond whether a transition is structurally valid, a second TRANSITION_PERMISSIONS table restricts which actor type may perform it (system, analyst, or customer). For example:

• System-driven progress (pending → validating_creds, validating_creds → queued, in_progress → completed/partial/error) is system only.
• in_progress → paused, completed → reviewing, reviewing → signed_off, and error recovery (error → queued) are analyst only.
• cancelled is reachable by customer and/or analyst depending on the source state.

transitionCollectionState() (in transitions.ts) checks isValidTransition() and hasTransitionPermission(), rejects terminal-state transitions, then atomically updates the collection and appends a chain-of-custody event.

Next Steps

• Security
• Chain of Custody