Automatic EC Repair — Internals, API & Limits

Technical reference for Automatic EC Repair — the detection/placement/execution stages, the full set of worker flags and configuration defaults, Kubernetes deployment, and health checks.

How it works

The EC Repair worker runs as a scheduled background task on your cluster:

The four repair stages, run continuously by workers.

Detection

The worker periodically scans the cluster topology and analyzes every EC volume:

Groups shards by volume and checks for missing, extra, or mismatched entries
Verifies shard sizes are consistent across copies
Only proceeds with repair if enough healthy shards exist (at least the data shard count) — ensuring safe reconstruction
A vacuumed volume whose live generation has fewer shards than the data shard count cannot be rebuilt there. If the prior generation, still served by holders that missed the switch or retained on disk, has enough shards, the worker switches the volume back to it (the same switch as ec.switch.rollback), and the next detection rebuilds the missing shards at that generation. A volume that no generation can read is reported in the detection activity, not changed

Smart Placement

When rebuilding missing shards, the worker uses a placement algorithm that:

Prefers nodes that don’t already hold shards for the same volume
Distributes shards across data centers and racks for maximum resilience
Selects the rebuild node that already has the most local shards, minimizing network transfer

Execution Stages

Stage	Description
Assign	Validate parameters and allocate the repair job
Copy	Copy existing healthy shards to the worker’s local disk
Rebuild	Reconstruct missing shards using erasure coding math
Distribute	Send rebuilt shards to their target nodes
Delete	Remove extra or mismatched shards from the cluster

Configuration

EC Repair runs as a plugin worker with sensible defaults:

Setting	Default	Description
Detection interval	10 min	How often to scan for shard issues
Detection timeout	5 min	Maximum time for a detection scan
Min interval	300s	Minimum seconds between detection runs
Max jobs per cycle	500	Maximum repair jobs per detection cycle
Global concurrency	8	Total concurrent repair jobs across cluster
Per-worker concurrency	2	Concurrent repair jobs per worker node
Roll back unreadable generations	on	Roll a volume back to its prior generation when the live one is unreadable (`auto_rollback_unreadable_generation`); off: only report it

You can also filter repairs by collection to scope repairs to specific data sets.

Deployment

EC Repair runs as a plugin worker process, separate from your master and volume servers. In production, you start one or more weed worker processes that connect to the admin server.

Starting a Worker

EC Repair is automatically included when the erasure_coding handler is enabled:

# Start a worker for EC tasks (includes both EC encoding and EC repair)
weed worker -admin=admin.example.com:23646 -jobType=erasure_coding \
  -workingDir=/var/lib/seaweedfs-plugin -maxExecute=2

# Start a worker handling all heavy tasks (EC encoding, EC repair, volume balancing, etc.)
weed worker -admin=admin.example.com:23646 -jobType=heavy \
  -workingDir=/var/lib/seaweedfs-plugin -maxExecute=4

# Start a worker handling all available task types
weed worker -admin=admin.example.com:23646 -jobType=all \
  -workingDir=/var/lib/seaweedfs-plugin

Key Options

Flag	Description
`-admin`	Admin server gRPC address (required)
`-jobType`	Task types: `erasure_coding` (or `ec`), `heavy`, `all`
`-workingDir`	Directory for temporary shard data during rebuilds
`-maxExecute`	Max concurrent job executions per worker (default: 4)
`-metricsPort`	Prometheus metrics port for monitoring
`-id`	Stable worker ID across restarts; auto-generated if omitted

Production Recommendations

Run at least 2 worker instances for high availability
Use a dedicated -workingDir with enough disk space to hold temporary shard copies during rebuilds
Workers auto-reconnect to the admin server on failure — no supervision beyond standard process management (systemd, Kubernetes, etc.)

Kubernetes

Workers are supported in the SeaweedFS Helm chart:

worker:
  enabled: true
  replicas: 2
  jobType: "heavy"
  maxExecute: 2
  workingDir: "/var/lib/seaweedfs-plugin"
  metricsPort: 9327

Health Checks

Workers expose HTTP endpoints when -metricsPort is set:

/health — always returns 200
/ready — returns 200 only when connected to admin
/metrics — Prometheus metrics

Requirements & limits

Safe by design: repairs only proceed when enough healthy shards exist (at least the data shard count), preventing destructive operations on already-degraded data.
Custom EC support: works with any custom EC ratio (e.g. 20+4, 16+6).
Rack-aware rebuilds: rebuilt shards are placed to maximize failure domain diversity.
Batch processing & deduplication: handles multiple degraded volumes in a single detection cycle, and prevents duplicate repair jobs for the same volume.
Load-balanced repairs: global coordination ensures repair operations are spread across the cluster, preventing multiple concurrent repairs from overwhelming any single server.
Progress tracking: detailed activity events during each repair for full observability.