Data Resilience Workers

SCHEMABOUND includes background workers that maintain data consistency and detect tampering in audit logs. These workers run continuously alongside the main application and handle recovery from transient failures without operator intervention.

Auto-Rewind Worker

The auto-rewind worker monitors database sessions for failures and automatically rewinds them to stable states using schema topology information for cascade-aware recovery.

Purpose

Database operations can fail mid-execution due to resource exhaustion, network partitions, or dependency crashes. Without automatic recovery, failed sessions leave the system in an inconsistent state requiring manual intervention. The auto-rewind worker detects these failures and restores stable states within a configurable window.

How It Works

1. Polling: The worker queries for failed session branches at regular intervals (default: every 5 seconds).
2. Topology-aware recovery: Before rewinding, the worker consults schema topology information to understand foreign key cascade relationships and ensures recovery preserves referential integrity.
3. Configurable limits: Each organization can configure whether auto-rewind is enabled and the maximum number of rewind attempts before escalating to human intervention.

Configuration

The auto-rewind worker loads settings from a trait-based config module:

#![allow(unused)]
fn main() {
use schemabound::runtime::workers::{RewindConfig, PollingStrategy};

struct RewindConfig {
    poll_interval: Duration,        // default: 5 seconds
    max_attempts: u32,              // default: 3
    enabled_per_org: HashMap<OrgId, bool>,
}
}

Each organization can configure whether auto-rewind is enabled and the maximum number of rewind attempts before escalating to human intervention. The PollingStrategy trait allows swapping between fixed-interval polling and adaptive backoff for high-load deployments.

Recovery Process

When a failed session is detected:

1. The worker retrieves the schema topology graph to identify cascading dependencies
2. It determines the last known stable state for the session
3. It executes a cascade-aware reset that rolls back dependent operations in the correct order
4. If all attempts fail, it emits an escalation event and stops retrying

Event Emission

Chain Integrity Verification

The chain integrity worker ensures audit logs remain tamper-evident by verifying sequence monotonicity and hash continuity across all organizations.

Purpose

Audit logs must be immutable and verifiable. Tampering — whether accidental (corruption) or intentional (modification) — must be detectable. The chain integrity worker continuously verifies that:

• Event sequences are monotonically increasing (no gaps or out-of-order entries)
• Hash chains are continuous (each entry’s hash matches the previous entry’s computed value)

How It Works

The worker maintains per-organization state for sequence numbers and hash pointers. On each verification cycle, it:

1. Reads the next expected sequence number from stored state
2. Compares it against actual event sequences in the audit log
3. Recomputes hash chains from a known-good anchor point
4. Detects any deviations from expected monotonicity or continuity

Breach Detection

When the worker detects a gap (missing sequence numbers) or modification (hash chain discontinuity), it emits an AuditChainIntegrityBreach event with:

• The affected organization ID
• The location of the detected breach
• The type of deviation (gap vs. modification)
• Timestamps for forensic analysis

Configuration

The chain integrity worker uses a similar trait-based config:

#![allow(unused)]
fn main() {
struct IntegrityCheckConfig {
    poll_interval: Duration,        // default: 60 seconds
    anchor_depth: usize,            // default: 1000 events
}
}

Per-organization state for sequence numbers and hash pointers is maintained in a bounded cache that evicts entries older than the anchor depth. The worker runs independently per organization but shares event bus infrastructure with other SCHEMABOUND components.

Integration With Audit Exporters

Breach events flow through the same event bus and audit exporters used by other SCHEMABOUND components. SIEM integrations, alerting systems, and compliance dashboards can consume these events to trigger incident response workflows.

Combined Operation

Both workers run concurrently as part of the service lifecycle. They share the same event bus infrastructure and can be monitored via the unified health check API exposed by the self-healing pipeline (see Self-Healing Infrastructure).