Self-Healing Infrastructure
SCHEMABOUND includes a self-healing pipeline that monitors infrastructure services and automatically recovers from failures. This ensures high availability without manual intervention for transient issues such as network partitions, resource exhaustion, or dependency crashes.
Overview
The self-healing system consists of two components:
- Health checker — Periodically probes registered infrastructure services (database instances, gRPC endpoints, audit log writers) and records their current state (
Healthy,Degraded, orFailed). - Restart pipeline — When a service enters
Failedstate, the pipeline attempts automatic recovery using exponential backoff with configurable limits.
Health Monitoring
Services register themselves with the health checker at startup. The checker runs on a configurable interval (default: 30 seconds) and aggregates results into a unified view accessible via internal APIs.
State transitions follow these rules:
| Current State | Healthy Probe | Degraded Probe | Failed Probe |
|---|---|---|---|
Healthy | Remains healthy | Transition to degraded | Transition to failed |
Degraded | Return to healthy | Remain degraded | Transition to failed |
Failed | Attempt restart with backoff | — | Escalate after consecutive failures |
Automatic Recovery
When a service fails, the restart pipeline applies exponential backoff before attempting recovery:
- Base delay: 5 seconds (configurable)
- Maximum delay: 60 seconds (capped to prevent excessive waiting)
- Escalation threshold: After a configurable number of consecutive failures (default: 5), the system escalates and stops automatic retries, emitting an alert for human intervention.
The backoff formula is:
delay = min(base_delay * 2^attempt, max_delay)
Configuration
Recovery behavior is controlled via environment variables or configuration files:
| Variable | Default | Description |
|---|---|---|
HEALTH_CHECK_INTERVAL_SECONDS | 30 | How often to probe services |
RESTART_BASE_DELAY_SECONDS | 5 | Initial backoff delay before restart attempts |
RESTART_MAX_DELAY_SECONDS | 60 | Maximum backoff delay cap |
RESTART_MAX_CONSECUTIVE_FAILURES | 5 | Failures before escalation to human intervention |
Integration With Service Lifecycle
The self-healing pipeline integrates with the service lifecycle state machine. Services transition through states (Starting, Running, Stopping, Stopped) and the health checker observes these transitions to determine when restart attempts are safe to execute.
Restart attempts only proceed when the target service is in a non-running state, preventing conflicts with manual operations or ongoing graceful shutdowns.
Event Emission
The pipeline emits structured events for observability:
| Event | When Emitted |
|---|---|
ServiceHealthCheckFailed | A health probe returned unhealthy |
ServiceRestartAttempted | Automatic restart initiated with backoff delay |
ServiceRecovered | Service returned to healthy state after restart |
ServiceEscalationTriggered | Consecutive failure threshold exceeded, human intervention required |
These events flow through the same event bus used by other SCHEMABOUND components and can be consumed by alerting systems or audit log exporters.
Manual Intervention
When escalation is triggered, operators can:
- Inspect the service state via internal health APIs
- Manually restart the failing service using standard operational procedures
- Reset the failure counter after confirming stability
The system will resume automatic recovery once the failure count drops below the escalation threshold (typically after a successful health probe).