On this page

Distributed Vector Search

SeaweedFS S3 Vector Buckets serve the S3 Vectors API from its S3 gateway. In the open-source build, one weed-vector engine answers every query. SeaweedFS Enterprise adds a pool of search engines and spreads each query over a few of them. Each index lives on a small, fixed set of engines. A query goes to one replica group of that index, one request per engine, and the gateway merges their answers into the top-k. Fan-out follows the size of the index, not the size of the fleet, so adding engines adds capacity without widening every query.

The vectors stay where they are: Lance tables in SeaweedFS. Engines keep their share of an index warm in memory, but any engine can still read any part of it cold, straight from the filers and volume servers. Placement decides where queries go, never where data lives.

Master leader · controller sizes W and R · places the groups placement slots to warm S3 client QueryVectors S3 gateway · router picks one group one request per engine merges the top-k Group 1 · zone a engine 1 half the index engine 2 other half Group 2 · zone b · replica engine 3 half the index engine 4 other half SeaweedFS S3 Lance tables shared storage: any engine reads any part cold W engines per group · R groups per index · group 2 takes failover and hedged requests
A query touches one group of W engines, not the whole pool. The controller sizes and places the groups; the router in every gateway does the fan-out and merge.

Highlights

  • One group per query. A query reaches at most W engines of one replica group, never the whole pool. max_fanout caps W at 32 by default.
  • Sized from the index. W grows with the index: enough engines to hold it in memory, and enough that each engine scans at most max_scan_mb (64 MiB) of it per query. R, the number of groups, grows with load: one group per 200 queries/s by default, 2 to 16 groups.
  • Measured recall. The gateway merges the engines’ partial top-k lists. The unindexed workloads benchmarked below were scanned exactly, with recall@10 of 1.000 in every run, filtered queries included. Indexed search behavior also depends on the ANN index and query settings.
  • Stable placement. Weighted rendezvous hashing places each index’s groups, in different zones where it can. When an engine joins or leaves, most slots stay on the engines that already have them warm. A new placement is make-before-break: queries stay on the old groups until the new owners report warm.
  • Overload is handled, not queued. Engines admit searches through a bounded queue and refuse at once when it’s full. The gateway hedges slow requests to the other group and retries once on another owner, both under small budgets. Clients get ServiceUnavailableException (503), which AWS SDKs retry, rather than a query that waits forever.
  • No new client, no new API. Same S3 Vectors endpoint, same calls. When no engine is serving, the gateway answers through the single engine.

Benchmarks

Measured with SeaweedFS’s S3 Vectors performance suite: 384-dimension vectors, cosine distance, top-10, 100 timed queries per scenario, and four engines with 1 GiB of memory each, reading the tables straight from the filers and volume servers. Every process (master, volume server, filer, S3 gateway and engines) ran on one 16-core x86_64 host shared with other work, at a load average between 9 and 19. Each cell shows two runs. Recall@10 was 1.000 in every run, and no timed query fell back to the single engine.

The distributed column puts all four engines in one group (min_width = 4, min_replicas = 1), so every query splits four ways.

50k vectors: single engine 50k vectors: 4 engines 200k vectors: single engine 200k vectors: 4 engines
Query p50 42, 45 ms 23, 21 ms 86, 82 ms 58, 58 ms
Filtered query (10% match) p50 81, 103 ms 29, 31 ms 259, 336 ms 73, 73 ms
Queries/s, 1 client 25, 24 41, 42 11, 9 17, 17
Queries/s, 8 clients 53, 49 58, 63 16, 17 18, 18
  • Filtered queries are about 3× faster at 50k vectors and 3.5–4.5× faster at 200k. At 200k, 259–336 ms dropped to 73 ms.
  • Unfiltered queries are about 2× faster at 50k (21–23 ms against 42–45 ms) and about 1.4× faster at 200k (58 ms against 82–86 ms).
  • One client gets 1.7× the throughput at 50k and 1.5–1.9× at 200k.
  • At 8 clients, throughput is about the same. Every process shared one host’s 16 cores, so CPU, not the engines, set the ceiling. Engines on separate hosts were not measured.
  • Writes are unchanged. Ingest ran at 1,600–1,750 vectors/s in every configuration; vectors are written through the gateway either way.

With default settings, the same four engines were placed as two replica groups rather than one group of four. The query p50 was 47 ms at 50k and 70 and 80 ms at 200k; the filtered p50 was 40 ms and 139 and 128 ms. At 200k each query was split over two engines, and the first few hundred queries ran on one engine before the index was widened, inside the timed window.

The 200k indexes were queried as the bulk load left them, 400 fragments of 500 rows each, with no maintenance worker compacting them.

Where the speed comes from

Early builds of the distributed path ran at about a quarter of the single engine’s throughput. Two changes fixed that:

  • Engines cache reads of table files. Lance files are written once, so engines keep what they read (--read-cache-mb, 512 MiB by default) instead of reading the index again from S3 on every query.
  • The unindexed part of an index is spread over the group. The benchmark indexes had no ANN segments, so all of their data was “tail”, scanned exactly. The Lance maintenance worker can build and refresh ANN indexes as data grows. Each tail fragment is now placed like a segment, so each engine in a group of four scans about a quarter instead of one engine scanning it all.

Counters for the same 50k index and the same 40 queries, taken when engines still read through the S3 gateway (they now read the same files from the filers and volume servers):

S3 GETs per query Bytes read per query Engines per query Query p50
Early build: 4 groups of 1, no read cache 114 76.8 MB 1 66 ms
After both changes: 4 groups of 1 9.2 15 KB 1 21 ms
After both changes: 1 group of 4 2.1 8 KB 4 20 ms
Single engine 8.6 15 KB – 29 ms

With 8 clients, the early build’s p50 was 478–497 ms at 50k vectors and 2,540 ms at 200k. The current build answers in 114–121 ms and 422–426 ms on the same layout.

Before max_scan_mb, W grew only when an index outgrew one engine’s memory, so these indexes always got one engine per query. Latency follows the bytes each engine scans. For filtered queries, an engine scanning 19 MB answered in about 25 ms, 77 MB took 61–95 ms, and 307 MB took 249–290 ms. That is why W now also grows with the scan.

How it works

  • Controller (in the master leader). Engines open a session to the leader and report which slots they have warm. From the index size and query rate that gateways report, the controller computes W and R for each index and places its groups. It is never on the query path, and a follower master redirects engines to the leader.
  • Placement. Each index gets R groups of up to W engines. Each group is in one zone where possible, and engines are ranked by weighted rendezvous hashing (weight = engine memory). Inside a group, segments and tail fragments go to members by hash, under a load cap of 1.25× the group average. The gateway (Go) and the engines (Rust) compute the same mapping.
  • Router (in every S3 gateway). It resolves the index version, lists its segments (cached; versions are immutable) and watches the placement. It picks a group, preferring its own zone and then the less busy one, and sends one SearchSegments call per engine with the query’s deadline. Then it merges the partial top-k lists.
  • Engines (weed-vector-worker). They warm their assigned slots, search through a bounded admission queue shared fairly between indexes, and drain on SIGTERM so queries move off before the process stops.
  • Writes. PutVectors, DeleteVectors and index management go to the single weed-vector, as before. Each write makes the gateway drop its cached version of that index.

Sizing: W and R

The controller sizes every index on every tick:

W_mem  = ceil(resident / (0.8 × engine budget))       # fit in memory
W_scan = ceil(resident / max_scan_mb)                 # bound each engine's scan
W      = max(W_mem, min(W_scan, eligible engines / min_replicas))   # clamped to [min_width, max_fanout]
R      = ceil(peak queries/s / qps_per_group)                       # clamped to [min_replicas, max_replicas]

The engine budget is the median --capacity-bytes of the engines. The memory width always wins. The scan width only grows while min_replicas full groups still fit, so the warm replica is never given up to make scans smaller. max_scan_mb is an even-split target: a single fragment is never split, so one fragment larger than the limit is still one engine’s scan.

With the defaults and 8 GiB engines:

Index Engines W Scan per engine
300 MiB 4 2 ~150 MiB (two groups need W ≤ 2)
300 MiB 16 5 ~60 MiB
20 GiB 16 8 ~2.5 GiB (memory needs 4; two groups of 8 fill the pool)

R rises as soon as load needs it and falls only after the load has stayed lower for replica_cooldown (10 minutes).

Tuning one index

The cluster-wide [master.vector] settings suit most indexes. When one index needs something else, put tags on it: they narrow the range the controller picks that index’s W and R from, and leave every other index alone.

Tag Sets Use it to
seaweedfs:min-width W is at least this Cut latency on an index whose queries must be fast: each engine scans a smaller share. Every query then waits on more engines.
seaweedfs:max-width W is at most this; max_fanout still caps it Keep a heavily queried index from occupying many engines per query.
seaweedfs:min-replicas R is at least this Keep extra warm copies ahead of a known traffic peak, or to ride out more engine failures.
seaweedfs:max-replicas R is at most this Cap the memory the index takes: every group holds a full copy, so the pool spends about resident size × R on it.

An index needs W × R engines, and each group’s W engines must be in one zone. With fewer engines, the later groups get fewer engines or are dropped.

Setting, reading and removing the tags

Set them when creating the index:

aws s3vectors create-index --endpoint-url http://<gateway>:8333 \
  --vector-bucket-name media --index-name movies \
  --data-type float32 --dimension 768 --distance-metric cosine \
  --tags '{"seaweedfs:min-width": "4", "seaweedfs:min-replicas": "3"}'

Or on an existing index, by its ARN:

ARN=$(aws s3vectors get-index --endpoint-url http://<gateway>:8333 \
  --vector-bucket-name media --index-name movies --query index.indexArn --output text)
aws s3vectors tag-resource --endpoint-url http://<gateway>:8333 --resource-arn $ARN \
  --tags '{"seaweedfs:min-width": "4", "seaweedfs:min-replicas": "3"}'
aws s3vectors list-tags-for-resource --endpoint-url http://<gateway>:8333 --resource-arn $ARN
aws s3vectors untag-resource --endpoint-url http://<gateway>:8333 --resource-arn $ARN \
  --tag-keys seaweedfs:min-width seaweedfs:min-replicas

With boto3, pass tags={"seaweedfs:min-width": "4"} to create_index or tag_resource. Changing tags takes the s3vectors:TagResource and s3vectors:UntagResource permissions on the index, so the same IAM policy that guards the index guards its placement.

  • Each value must be a positive integer, and a minimum may not exceed its maximum. A gateway ignores a value that breaks these rules and logs S3 Vectors: index <bucket>/vector/<index> placement tags: ignoring [...]; the index’s other tags still apply.
  • Gateways read an index’s tags at most every 30 seconds while it is being queried, and report them to the controller every 30 seconds, so a change applies within about a minute of the next queries. Tags on an index nobody queries take effect when it is queried.
  • New groups warm up before queries move to them, so a change does not interrupt queries.
  • Removing the tags returns the index to the cluster defaults. A change to min-replicas or max-replicas applies at once, without replica_cooldown.

Checking the result

The leading master logs every change to an index’s W or R, with what drove it:

vector controller: index s3://media/vector/movies width 4 replicas 3 (resident 5368709120 bytes, 120.0 queries/s, width 4-32, replicas 3-16)

The width and replica ranges at the end are the bounds in effect, after the tags. To see which engines each query used, start the gateway with -v=1 and look for S3 Vectors: query s3://media/vector/movies version <n> on engines [...].

Turning it on

Start with S3 Vector Buckets already running: the gateway with its Lance catalog, and one weed-vector registered with the master. That engine keeps handling writes, and answers queries whenever no distributed engine is serving.

1. Start a weed-vector-worker on each search host. It registers with the masters and joins the controller:

weed-vector-worker --masters 10.0.0.1:9333,10.0.0.2:9333,10.0.0.3:9333 \
  --state-dir /var/lib/weed-vector-worker --zone dc1
  • Reads never pass through the S3 gateway. The engine asks the masters for the filers (--filer-group picks the group), fetches table bytes from the volume servers, and undoes volume encryption and SSE-S3 the way the gateway would. It reads the cluster’s security.toml where weed processes do: [grpc.client] for the masters and filers, [https.client] for the volume servers, jwt.signing.read.key and s3.sse.*; its own gRPC listener takes [grpc.vector], requires client certificates (checked against that section’s allowed_commonNames when set) and re-reads its files on SIGHUP and every WEED_TLS_CERT_REFRESH_INTERVAL.
  • --state-dir keeps the engine’s id, so a restarted engine gets its slots back.
  • --zone is the failure domain: an index’s replica groups go to different zones.
  • --capacity-bytes defaults to 75% of RAM (or of the cgroup limit), and is the engine’s placement weight.

2. Tune the controller (optional). The defaults live under [master.vector] in master.toml. Keep the file identical on every master, because whichever master leads applies its own copy:

[master.vector]
max_fanout = 32      # largest W
max_scan_mb = 64     # per-engine scan target; 0 sizes W by memory only
min_replicas = 2
max_replicas = 16
qps_per_group = 200

An index can carry its own bounds as tags, for example seaweedfs:min-width=4; see Per-index bounds.

3. Gateways need no change. A gateway uses its -dataCenter as its zone and prefers groups there.

Current limits

  • Only QueryVectors is distributed. Writes and index management go through the single weed-vector, which has to stay running.
  • A fragment is the smallest unit of work. One fragment larger than max_scan_mb is scanned by one engine.
  • The benchmarks above ran on one shared host. Engines on separate hosts have not been measured yet.

For every setting, flag and default, see the Distributed Vector Search reference. For Lance tables on SeaweedFS, see S3 Table Buckets and Vector search with LanceDB.