S3 over RDMA Is 5× Faster Than HTTP

For large objects, an S3 GET over RDMA in SeaweedFS Enterprise moves data about five times faster than the same GET over HTTP. With many objects in flight it reaches 11.4 GB/s on one 100 GbE link, about 95% of what the link carries. The S3 request is still an ordinary, signed S3 request, and every byte stays behind S3 authentication and per-file read tokens.

Nothing here is a projection. Every number below was measured on a two-host lab (ConnectX-5, 100 GbE, RoCE v2), every byte of every response was compared with the stored object, and the NIC’s hardware counters confirm which transport each read actually used.

The numbers

How these were measured:

• Paired arms. Each run alternates RDMA and HTTP GETs of the same objects, so both see the same server state.
• Verified bytes. Every response is compared with the stored object byte for byte.
• Real transport. On each RDMA GET the client NIC’s receive counter grew by the object size plus about 2% protocol overhead; on each HTTP GET it did not move.
• Data bigger than memory. The four storage processes were capped at 400 MB in a cgroup, leaving data about seven times the cache they had, so every pass read NVMe.

Small requests gain less: a 1 MiB range is about 2× faster, because the fixed cost of each request matters more when there is less data to move.

Why it is faster

Over HTTP, every byte of an object travels in a response body: from the storage server to the S3 gateway over TCP, through the gateway, and again over TCP to the client, with kernel copies on each side. At 100 Gb/s that path runs out of CPU long before it runs out of network.

Over RDMA, the S3 request still goes to the gateway, which authenticates it, checks bucket policy and finds the object’s chunks. The bytes take a different road: each storage server that holds a chunk writes it with an RDMA write directly into the client’s registered memory. The gateway never touches the data, no kernel copies the payload, and the HTTP response is only a status and a few headers.

Fast without cutting corners

• Authorization stays on. SeaweedFS can require a signed, short-lived token per file on every read. The RDMA path carries that token with each read, and the storage server checks it with the same code as an HTTP read before it writes a single byte. Every number in this post was measured with it on; switching it off made no consistent difference in our runs.
• Memory stays bounded. RDMA needs the NIC to know where the source bytes are, which means registering (“pinning”) memory. Each read registers only its own range, under a fixed budget derived from the host’s locked-memory limit, and registration overlaps the transfer of the previous segment. Pinned memory per storage process stays at most about 70 MB regardless of how much data it holds, and with the locked-memory limit cut to 8 MB, S3 GETs still ran at 10.7 GB/s.
• More storage servers, same speed. With the chunks of each object spread across one, two or four storage servers, a client still pulled 11.2 to 11.5 GB/s; the storage servers write into the same client buffer in parallel.
• Faults cost time, never bytes. We killed and restarted a storage server, compacted volumes, and paused a server for 15 seconds, all while reads were running. Reads failed cleanly or waited, recovered, and never returned a wrong byte.

Where it stands among others

S3 over RDMA follows a convention set by NVIDIA’s cuObject: a normal S3 request carries an RDMA descriptor in the x-amz-rdma-token header, and the object bytes move by RDMA instead of in the HTTP body. MinIO AIStor and the Versity gateway ship servers for it, and Ceph has a gateway pull request and an RFC for serving straight from its storage daemons.

SeaweedFS already separates metadata from data, so its storage servers push the bytes directly. That is the shape Ceph’s RFC is moving toward. SeaweedFS uses its own DC transport and does not depend on NVIDIA’s server library. We checked compatibility against NVIDIA’s own client side: the cuObject client and NIXL’s object plugin both read and write through SeaweedFS into host memory and into GPU memory (A10, 4 to 64 MiB), with every byte moving over RDMA.

Limits and what comes next

• Small objects and ranges. Per-request setup dominates below a few MiB; a 1 MiB range is only about 2× faster.
• Cold reads. From disk we reach 4.4 GB/s, against 7.1 GB/s that the NVMe can deliver. Reading cold data with direct I/O into a pre-registered buffer pool raises this by about 23% in our current implementation, which is in review.
• Writes and GPU throughput. PUT over RDMA is about 2× HTTP for 64 MiB objects today. Reads into GPU memory are verified correct; their throughput is still to be measured.

S3 over RDMA uses the same RDMA data path as Seaweed KVCache, our shared LMCache L2 backend.

SeaweedFS numbers are lab measurements on two hosts, not production figures. Other projects are described from their public documentation; we did not test them.

Sources: NVIDIA cuObject documentation · MinIO: Introducing S3 over RDMA in AIStor · Versity gateway: S3 RDMA · Ceph PR #70458 · Ceph PR #71209 (OSD-direct RFC) · SeaweedFS security overview