S3 Vector Buckets

Keep embeddings alongside your objects, files, and analytical tables in one SeaweedFS cluster. Vector buckets expose the AWS-compatible S3 Vectors API: create an index, write vectors with metadata, and query the nearest matches using an AWS SDK or CLI pointed at your S3 endpoint.

Use vector buckets for semantic search, retrieval-augmented generation (RAG), and recommendations. Your application generates the embeddings; SeaweedFS stores and searches them. Index data lives in Lance tables on the cluster’s filers and volume servers.

What you can do

• Organize embeddings. A vector bucket contains indexes; each index fixes a dimension and distance metric, and each vector has a key, float32 values, and optional metadata.
• Search by similarity. Use cosine or Euclidean distance, add metadata filters, and return distances and metadata with the matches.
• Use familiar clients. Connect with the s3vectors AWS SDK client or the AWS CLI’s s3vectors commands on the same endpoint as your S3 object requests.
• Control access. Use s3vectors:* IAM actions, vector bucket policies, and tags on buckets and indexes.
• Manage from the Admin UI. The Vector Buckets pages provide bucket and index management and vector query tools.

Vector buckets and the single weed-vector engine are part of open-source SeaweedFS. Enterprise adds Distributed Vector Search, which spreads queries across a pool of search engines while keeping the same API and stored data.

Try it locally

Install matching releases of weed and the separate weed-vector binary. The deployment guide links to releases; the vector engine README also covers building from source. Use an AWS CLI version with aws s3vectors support.

This local example binds to loopback and uses demo credentials:

cat > s3config.json <<'JSON'
{
  "identities": [{
    "name": "vector-demo",
    "credentials": [{"accessKey": "vector-demo", "secretKey": "vector-demo-secret"}],
    "actions": ["Admin"]
  }]
}
JSON

weed mini -dir=./vector-data -ip=127.0.0.1 -ip.bind=127.0.0.1 \
  -s3.config=./s3config.json -vector

Place weed-vector beside weed or on PATH. Mini starts the engine, waits for its master registration, and stops it when the cluster stops. -vector.binary selects an explicit binary path.

In another terminal, create a bucket and a three-dimensional cosine index:

export AWS_ACCESS_KEY_ID=vector-demo
export AWS_SECRET_ACCESS_KEY=vector-demo-secret
export AWS_DEFAULT_REGION=us-east-1
export VECTOR_ENDPOINT=http://127.0.0.1:8333

aws --endpoint-url "$VECTOR_ENDPOINT" s3vectors create-vector-bucket \
  --vector-bucket-name embeddings

aws --endpoint-url "$VECTOR_ENDPOINT" s3vectors create-index \
  --vector-bucket-name embeddings --index-name documents \
  --data-type float32 --dimension 3 --distance-metric cosine

Write embeddings with metadata:

aws --endpoint-url "$VECTOR_ENDPOINT" s3vectors put-vectors \
  --vector-bucket-name embeddings --index-name documents \
  --vectors '[
    {"key":"storage-guide","data":{"float32":[1,0,0]},"metadata":{"category":"docs"}},
    {"key":"deployment-guide","data":{"float32":[0.8,0.2,0]},"metadata":{"category":"docs"}},
    {"key":"product-image","data":{"float32":[0,1,0]},"metadata":{"category":"images"}}
  ]'

Query the closest documentation vectors, including distances and metadata:

aws --endpoint-url "$VECTOR_ENDPOINT" s3vectors query-vectors \
  --vector-bucket-name embeddings --index-name documents \
  --query-vector '{"float32":[1,0,0]}' --top-k 2 \
  --filter '{"category":{"$eq":"docs"}}' \
  --return-distance --return-metadata

The filter restricts matches to category=docs. Use your embedding model’s dimension in a real index and use the same model for stored and query vectors.

Run in production

Run weed-vector --masters=<master-addresses> as a separate service and advertise a gRPC address reachable from the S3 gateway. The engine registers with the masters; gateways discover it automatically. Configure the engine with the same cluster TLS, signing, and encryption settings used by the other storage clients.

Run the Lance maintenance worker to build and refresh ANN indexes as data grows. Queries can scan data before an ANN index is built. For flags, supported operations, IAM permissions, and metrics, see the Vector Buckets reference.

Vector buckets, Lance tables, and distributed search

See the edition comparison, LanceDB walkthrough, and AWS S3 Vectors API guide.