Write Concurrency

A volume server with eight NVMe drives can be slower at ingest than the same box with one — and nothing in the configuration explains why. The reason is structural: one volume absorbs exactly one write at a time (its data file is append-locked), so a disk’s write concurrency is simply how many of its volumes are being written to simultaneously. Which volumes the master hands out decides that number, and without guidance it is an accident: two consecutive assigns can land on the same volume (serializing behind one lock) or on two volumes sharing a spindle (worse — the disk seeks between two append streams).

SeaweedFS Enterprise Write Concurrency makes it a policy. W is a floor on spread: “this collection’s writes should reach at least W distinct disks.” The master then matches assigns to distinct disks, rotates across them, grows volumes where a layout falls short of its width, and keeps the balancer and vacuum from undoing the spread.

W unset — accidental placement writes disk 1 · queue disk 2 · idle disk 3 · idle one lock, one spindle, everyone waits volume.concurrency -minDisks 4 writes disk 1 · writing disk 2 · writing disk 3 · writing disk 4 · writing four distinct disks append in parallel
One volume takes one write at a time, so parallelism comes from writing to volumes on different disks — W makes the master owe you that spread.

Setting it

weed shell

> volume.concurrency                                   # show the current setting
> volume.concurrency -defaultMinDisks 2                # every collection: spread over 2 disks
> volume.concurrency -collection pictures -minDisks 6  # one collection's override
> volume.concurrency -collection 'logs-*' -minDisks 4  # a family of collections, by glob
> volume.concurrency -collection pictures -removeMinDisks   # back to the default

The value lives in the master’s raft state — every master agrees on it, it survives restarts, and it is deliberately not in master.toml (a file value would be a second source of truth the cluster ignores). A minDisks of 1 is the default and means no obligation. Collection keys may be globs — a key holding *, ? or [...] is a pattern; a literal name always beats a pattern covering it, and among patterns the longest wins, so logs-prod-* beats logs-* beats *.

Maximum live volumes — spread, but not sprawl

The floor has a counterpart that counts a different thing: maxLiveVolumes, a ceiling on how many volumes a collection writes to at once. The floor counts distinct disks; the maximum counts volumes — they are not two ends of one range. A collection at minDisks 2 with a maximum of 4 reaches at least two disks and uses at most four volumes, which may sit on fewer than four disks.

It exists because one floor for the whole cluster over-spreads the quiet collections. Set -defaultMinDisks 8 on a big cluster and every collection’s writes fan out — including the small, low-traffic ones, which end up smeared across eight concurrently-open, slowly-filling volumes. Each of those is a volume slot occupied, an open append stream, and a future compaction/erasure-coding unit; for a collection writing a trickle, that is sprawl with no throughput to show for it. The maximum bounds the fan-out: the floor keeps ingest-heavy collections fast, the ceiling keeps quiet ones on a few volumes that fill in turn.

> volume.concurrency -defaultMaxLiveVolumes 4              # hold every collection to 4 volumes at once
> volume.concurrency -collection logs -maxLiveVolumes 2    # hold one collection to 2
> volume.concurrency -collection logs -maxLiveVolumes 0    # stored opt-out of a non-zero default
> volume.concurrency -collection logs -removeMaxLiveVolumes

Its semantics mirror the floor with one deliberate asymmetry: zero means unbounded, and a per-collection zero override is honoured — it is how one collection opts out of a bounded default. A maximum below a collection’s minDisks is refused outright (it would ask for two opposite things), rather than silently clamped, so you see which of the two the cluster would have ignored. And like the floor, the maximum is intent, never a precondition: when nothing in the bounded set can take a write, the write widens rather than fails.

What the master does with it

  • Assigns spread. Volume selection matches writes to distinct disks up to the width, rotating across them rather than weighted-random landing on one.
  • Shortfall drives growth, not failure. W is never a precondition: with fewer eligible disks than W the write still succeeds, and the deficit makes the next reconcile pass grow a volume where it closes the gap — one volume per pass, with a brake when growth stops making progress.
  • Balance and vacuum respect it. The balancer will not consolidate a layout below its width, and vacuum accounts for the width when picking what to work on.

Scope, precisely

W is a floor per layout — the unit keyed by replica placement, TTL, disk type, and volume class — while the policy is stored per collection (the one key everything shares). A collection writing two disk types owes W disks in each. Per-request narrowing (data center, rack, node) gets best-effort spread within the narrowed set — the guarantee is per layout, not per constraint combination.

Notes

  • A “disk” is a -dir entry. Each data directory on a volume server is one disk for spreading purposes; the operator’s -dir layout declares the truth. The server compares device ids across directories at startup and warns on suspicious layouts, but does not second-guess LVM, RAID, or SAN.
  • W is a floor on spread, not a request cap. Per-request ceilings belong to the S3 concurrency limiter.
  • A per-disk depth ceiling (how many concurrent streams one location accepts, enforced volume-server-side) is designed as this feature’s counterpart and is not yet shipped.
  • Write Concurrency policy is SeaweedFS Enterprise; the disk-targeted allocation plumbing it builds on is open source.

For resolution rules, the rotor, deficit-driven growth, and the balance/vacuum interactions, see the technical reference.