Automatic EC Vacuum — Internals, API & Limits

Technical reference for Automatic EC Vacuum — the detection and compaction phases, safety guarantees, the full configuration table, worker flags, Helm values, health checks, and metrics to plan around.

The worker collects all EC shards for a target volume into its local -workingDir, vacuums the data locally, and then distributes the compacted shards back to volume servers. Volume servers are not overloaded with the CPU and disk work of compaction; they only serve controlled shard transfers and receive the replacement shards.

What Can It Do?

• Detect Deleted Data: Identifies EC volumes where deleted needles are consuming significant space (configurable threshold).
• Compact Volumes: Removes deleted needles from EC shards, reclaiming storage space across the cluster.
• Offload Heavy Work: Performs shard compaction in the worker’s local working directory instead of on volume servers.
• Improve Efficiency: Reduces unnecessary parity shard overhead by eliminating deleted data before the shard is fully utilized.
• Reduce Volume Server Load: Keeps volume servers focused on normal serving work while workers handle vacuum CPU, temporary disk, and coordination.
• Rack-Aware Optimization: Maintains shard distribution across failure domains during compaction.

Why Do You Need It?

In erasure-coded systems, when files are deleted, the space they occupied in the EC shards remains allocated. Over time, this creates “storage waste”:

For example, a 100GB EC 10+4 volume with 40% deleted data:

• Without vacuum: Still consumes 100GB + parity (140GB total with 10+4 ratio)
• After vacuum: Compacted to ~60GB + parity (~84GB total), a 40% space savings

How Does It Work?

The EC Vacuum worker runs as a scheduled background task on your cluster. When a volume crosses the deleted-data threshold, the worker pulls the full EC shard set into local storage, compacts it, verifies the result, and places the compacted shards back onto the cluster.

Detection Phase

The worker periodically scans EC volumes in the cluster:

1. Analyzes each EC volume’s shard composition
2. Calculates the ratio of deleted needles vs. total data
3. Compares against the configured threshold (default: 30%)
4. Only processes volumes that exceed the threshold

Compaction Phase

When a volume triggers vacuum:

1. Analyze: Determines which shards can be safely compacted
2. Collect: Streams all required EC shards from volume servers into the worker’s local -workingDir
3. Vacuum Locally: Removes deleted needles and rebuilds compacted shard files on the worker
4. Verify: Validates that compacted shards match the original data
5. Distribute: Places compacted shards back across the cluster using rack-aware placement
6. Cleanup: Removes old oversized shards and temporary data

Technology Advantages

• No in-place volume-server vacuum: Volume servers are not asked to compact EC shards on their own disks. They serve shard data to the worker and receive compacted replacements.
• Resource isolation: CPU, temporary disk I/O, and shard rewrite work are concentrated on worker nodes that operators can size independently.
• Predictable cluster impact: Global and per-worker concurrency limits control how many volumes are vacuumed at once.
• Network-efficient placement: The worker can distribute compacted shards back to appropriate target nodes while preserving rack-aware EC placement.
• Operational simplicity: Adding more worker capacity increases vacuum throughput without changing the volume server role.

Safety Guarantees

• Only proceeds when sufficient healthy shards exist (>= data shard count)
• Validates data integrity before replacing original shards
• Keeps original shards available until compacted replacements are verified and distributed
• Shard distribution follows the same rack-aware rules as EC repair

Configuration

EC Vacuum runs as a plugin worker with configurable thresholds:

You can also filter vacuums by collection to focus on specific data sets.

Deployment

EC Vacuum runs as a plugin worker process, integrated with your EC infrastructure. Start one or more weed worker processes that connect to the admin server.

Starting a Worker

EC Vacuum is automatically included when the erasure_coding handler is enabled:

# Start a worker for EC tasks (includes EC encoding, EC repair, and EC vacuum)
weed worker -admin=admin.example.com:23646 -jobType=erasure_coding \
  -workingDir=/var/lib/seaweedfs-plugin -maxExecute=2

# Start a worker handling all heavy tasks
weed worker -admin=admin.example.com:23646 -jobType=heavy \
  -workingDir=/var/lib/seaweedfs-plugin -maxExecute=4

# Start a worker handling all available task types
weed worker -admin=admin.example.com:23646 -jobType=all \
  -workingDir=/var/lib/seaweedfs-plugin

Key Options

Production Recommendations

• Run at least 2 worker instances for high availability
• Allocate sufficient -workingDir disk space for the largest concurrent EC vacuum job, including collected input shards and compacted output
• Set metricsPort to monitor vacuum progress and troubleshoot issues
• Consider grouping EC workers by region/data center to minimize cross-network transfers
• Vacuum works alongside EC repair without coordination

Kubernetes

Workers are supported in the SeaweedFS Helm chart:

worker:
  enabled: true
  replicas: 2
  jobType: "heavy"
  maxExecute: 2
  workingDir: "/var/lib/seaweedfs-plugin"
  metricsPort: 9327

Health Checks

Workers expose HTTP endpoints when -metricsPort is set:

• /health — always returns 200
• /ready — returns 200 only when connected to admin
• /metrics — Prometheus metrics

Monitoring and Observability

EC Vacuum integrates with SeaweedFS observability:

How EC Vacuum Complements Other Enterprise Features

• With EC Repair: EC repair restores fault tolerance; EC vacuum improves efficiency
• With Self-Healing: Self-healing detects shard issues; EC vacuum handles scheduled compaction
• With Custom EC Ratios: Higher-density ratios like 20+4 benefit most from vacuum’s space reclamation

Requirements & limits

• Requires a valid SeaweedFS Enterprise license; runs as a plugin worker connected to the admin server.
• Only proceeds when sufficient healthy shards exist (>= data shard count).
• Vacuum is only triggered for volumes past the deleted-ratio threshold (default 0.30).
• Global and per-worker concurrency limits cap how many volumes are vacuumed at once (defaults: 4 global, 1 per worker).