Automatic EC Shard Repair

SeaweedFS Enterprise includes automatic EC shard repair, a background worker that continuously monitors your cluster for missing, duplicate, or inconsistent erasure coding shards — and fixes them without manual intervention.

What Can It Do?

• Detect Missing Shards: Identifies EC shards that are missing due to disk failure, node loss, or incomplete operations.
• Detect Extra/Duplicate Shards: Finds redundant shard copies left behind by failed previous repairs or rebalancing.
• Detect Mismatched Shards: Spots shards with inconsistent sizes across copies, indicating corruption or partial writes.
• Rebuild and Distribute: Reconstructs missing shards using Reed-Solomon math from healthy shards, then places them on optimal nodes.
• Clean Up: Removes extra or mismatched shards to restore the cluster to a consistent state.

How Does It Work?

The EC Repair worker runs as a scheduled background task on your cluster:

┌─────────────────────────────────────────────────────────────────────┐
│                        EC Repair Workflow                           │
│                                                                     │
│   ┌──────────┐     ┌──────────┐     ┌───────────┐     ┌─────────┐   │
│   │  Detect  │────>│   Plan   │────>│  Rebuild  │────>│ Cleanup │   │
│   └──────────┘     └──────────┘     └───────────┘     └─────────┘   │
│                                                                     │
│   Scan cluster     Identify best    Copy healthy      Delete extra  │
│   topology for     rebuild node     shards locally,   or mismatched │
│   shard issues     and placement    rebuild missing,  shards        │
│                    targets          distribute to                   │
│                                     target nodes                    │
└─────────────────────────────────────────────────────────────────────┘

Detection

The worker periodically scans the cluster topology and analyzes every EC volume:

1. Groups shards by volume and checks for missing, extra, or mismatched entries
2. Verifies shard sizes are consistent across copies
3. Only proceeds with repair if enough healthy shards exist (at least the data shard count) — ensuring safe reconstruction

Smart Placement

When rebuilding missing shards, the worker uses a placement algorithm that:

• Prefers nodes that don’t already hold shards for the same volume
• Distributes shards across data centers and racks for maximum resilience
• Selects the rebuild node that already has the most local shards, minimizing network transfer

Execution Stages

Why Do Customers Need It?

• Hands-Off Durability: Disk failures and node outages are inevitable at scale. EC Repair ensures your data stays fully protected without operator intervention.
• Prevents Data Loss: A volume with missing shards is one failure away from data loss. Automatic repair restores full fault tolerance quickly.
• Reduces Operational Burden: No need to manually identify and repair degraded EC volumes — the system handles it continuously.
• Safe by Design: Repairs only proceed when enough healthy shards exist, preventing destructive operations on already-degraded data.

Configuration

EC Repair runs as a plugin worker with sensible defaults:

You can also filter repairs by collection to scope repairs to specific data sets.

Deployment

EC Repair runs as a plugin worker process, separate from your master and volume servers. In production, you start one or more weed worker processes that connect to the admin server.

Starting a Worker

EC Repair is automatically included when the erasure_coding handler is enabled:

# Start a worker for EC tasks (includes both EC encoding and EC repair)
weed worker -admin=admin.example.com:23646 -jobType=erasure_coding \
  -workingDir=/var/lib/seaweedfs-plugin -maxExecute=2

# Start a worker handling all heavy tasks (EC encoding, EC repair, volume balancing, etc.)
weed worker -admin=admin.example.com:23646 -jobType=heavy \
  -workingDir=/var/lib/seaweedfs-plugin -maxExecute=4

# Start a worker handling all available task types
weed worker -admin=admin.example.com:23646 -jobType=all \
  -workingDir=/var/lib/seaweedfs-plugin

Key Options

Production Recommendations

• Run at least 2 worker instances for high availability
• Use a dedicated -workingDir with enough disk space to hold temporary shard copies during rebuilds
• Workers auto-reconnect to the admin server on failure — no supervision beyond standard process management (systemd, Kubernetes, etc.)

Kubernetes

Workers are supported in the SeaweedFS Helm chart:

worker:
  enabled: true
  replicas: 2
  jobType: "heavy"
  maxExecute: 2
  workingDir: "/var/lib/seaweedfs-plugin"
  metricsPort: 9327

Health Checks

Workers expose HTTP endpoints when -metricsPort is set:

• /health — always returns 200
• /ready — returns 200 only when connected to admin
• /metrics — Prometheus metrics

Key Benefits for Enterprise

1. Automatic Recovery: Missing shards are detected and rebuilt without manual intervention
2. Rack-Aware Rebuilds: Rebuilt shards are placed to maximize failure domain diversity
3. Custom EC Support: Works with any custom EC ratio (e.g., 20+4, 16+6)
4. Progress Tracking: Detailed activity events during each repair for full observability
5. Batch Processing: Handles multiple degraded volumes in a single detection cycle
6. Deduplication: Prevents duplicate repair jobs for the same volume
7. Load-Balanced Repairs: Global coordination ensures repair operations are spread across the cluster, preventing multiple concurrent repairs from overwhelming any single server