Sealed Directories — Internals, API & Limits

Technical reference for Sealed Directories — the manifest format, the seal/unseal phases and crash recovery, mutable seals and re-seal compaction, the auto_seal policy engine and its event-driven detection, the shell and Admin API, a real latency benchmark, metrics, and limits to plan around.

Sealing packs a cold directory’s child entries into zstd-compressed segment chunks stored in ordinary volumes, and replaces the individual child records in the filer store with a small manifest on the directory entry. Reads are served from the manifest. A seal is either frozen (writes rejected until unsealed) or mutable (writes admitted as overlay records and periodically folded back into the manifest by compaction).

What can it do?

• Shrink the filer metadata store: removes per-entry records for cold children, keeping only a sorted segment index on the parent — roughly 18–58× less filer-store size on hash-fanout trees (example savings).
• Keep directories usable: listing and lookup inside a sealed directory are served from the manifest (decompressed segments are cached), so readers see a normal directory.
• Stay writable, if you want: a mutable seal admits creates, updates, deletes, and renames; a re-seal compaction folds accumulated changes back into the manifest.
• Reuse the storage stack: segments are ordinary volume needles, so they inherit erasure coding, cloud tiering, and replication. Sibling directories sealed in one pass share needles.
• Automate by policy: the auto_seal worker seals cold directories on ordered path-pattern rules, keyed on child-file idleness, and finds compaction candidates by tailing the metadata event log instead of walking the namespace.
• Reverse cleanly: fs.unseal materializes the children back into the store exactly as they were.

Why do you need it?

The filer stores one metadata record per file and directory, and cold data costs the same as hot. Once a namespace reaches hundreds of millions of entries, the metadata store — not the raw bytes — limits filer memory, disk, open/compaction time, and backup size.

How does it work?

The seal is transactional and crash-safe:

1. Fence — the directory is marked seal-pending (an evented marker that replicates to peer filers); new mutations under it are rejected from this point.
2. Build — children are scanned in name order, packed into zstd-compressed segments, and uploaded as volume needles. Sibling directories in one recursive run share needles.
3. Commit — the manifest (sorted first_name/last_name/chunk per segment) is written onto the directory entry and a metadata event is emitted; then the original child records are purged from the store. A row that raced the seal is kept as residue, served alongside the manifest, never lost.
4. Recovery — a build journal and pending/event markers let RepairSeal finish or roll back an interrupted seal after a crash, and let peers converge.

Unseal reverses it: children are materialized back into the store from the segments (idempotent — existing residue wins), the manifest is cleared, and the segment needles are reclaimed by deferred GC after a grace period so peers replaying the unseal can still read them.

Mutable seals and re-seal compaction

Seal with -allowUpdates (or an auto_seal rule with allowUpdates: true) to keep the directory open for writes:

• Creates and overwrites land as ordinary store rows next to the manifest; a read merges the two, with the store row winning whenever a name exists in both — so an overwrite is always served correctly.
• Deletes and renames need one more piece: deleting a name that exists in the manifest can’t just remove a row that was never there, so the guard writes a small tombstone record instead — a whiteout that hides the manifest’s entry. Tombstones are invisible to normal reads and listings, and are cleaned up the next time the directory is compacted or unsealed.
• Compaction re-seals the directory: it scans the current merged view (manifest plus overlay, with tombstones applied), builds a fresh manifest that already reflects every change, and clears the overlay rows and tombstones it just folded in. A compaction is the same crash-safe state machine as the original seal, just triggered on an already-sealed directory instead of a plain one.
• Trigger: a rule’s compactMinRows and compactPercent (of the sealed entry count, whichever is lower) decide how much accumulated overlay is “enough,” and the same per-rule idle window gates when — a directory has to have gone quiet again, not just crossed the row threshold.
• Mode flips are cheap: re-sealing an already-sealed directory with a different mode (-allowUpdates on a frozen one, or without it on a mutable one) just rewrites the directory’s mode in place — no rebuild, no re-scan.

Finding compaction candidates without scanning

Naively, finding directories that need compacting would mean periodically re-checking every mutable sealed directory’s overlay size — an O(sealed directories) cost that grows with how much of the namespace you’ve sealed, not with how much is actually changing. Instead, auto_seal’s detector tails the filer’s metadata event log: each detection cycle reads new events since its last checkpoint (bounded, resumable, the same mechanism filer.sync uses for cross-cluster replication), and folds them into a small per-directory counter of write activity. Only directories that the log shows have actually accumulated writes are proposed as compaction candidates — cost scales with the cluster’s write rate, not with the size of the sealed estate. The event-derived counts are just a fast filter: the filer always re-verifies the real overlay size and idle window before it fences anything, so an imprecise count never causes an unsafe compaction. A periodic namespace walk still runs underneath as a backstop, so freshly-sealed directories or anything the event log missed are eventually found too.

Policy: /etc/seaweedfs/seal.conf

The auto_seal worker reads an ordered rule list (a SealConfig, protojson) stored on the filer:

{ "rules": [
    { "pattern": "/data/**",      "idleSeconds": 2592000, "minEntries": 64 },
    { "pattern": "/data/hot/**",  "exclude": true },
    { "pattern": "/data/logs/**", "idleSeconds": 7776000, "allowUpdates": true, "compactMinRows": 500, "compactPercent": 10 }
] }

Last matching rule wins (gitignore / rsync-filter semantics). Idleness is measured from the child files’ modification times (SeaweedFS does not bump a directory’s own mtime on child writes), so a directory under active write is never sealed. Sealing is always recursive: a rule seals the whole matching subtree bottom-up.

Shell & Admin API

weed shell> fs.seal    -dryRun          /data/archive/2023   # preview: entry & segment counts, manifest size
weed shell> fs.seal                     /data/archive/2023   # seal the directory and its subtree (frozen)
weed shell> fs.seal    -allowUpdates    /data/logs/2026      # seal mutable: writes stay admitted
weed shell> fs.seal    -allowUpdates    /data/archive/2023   # re-seal an already-sealed dir: flips its mode
weed shell> fs.unseal                   /data/archive/2023   # materialize children back into the store
weed shell> fs.seal.status              /data/archive/2023   # inspect seal state

Re-running fs.seal (or an auto_seal policy pass) on a directory that is already sealed mutable compacts it — folding the accumulated overlay into a fresh manifest — instead of seal failing with “already sealed.”

The Admin UI Sealed Directories page manages the rule list (including mode and compaction thresholds) and offers ad-hoc Seal (with a mode toggle and dry-run preview), Unseal, mode flip, and Compact. The file browser badges sealed directories — distinguishing frozen from mutable — and warns when you are inside one.

Deleting a sealed directory

• Delete the whole directory: a server-side recursive delete (fs.rm -r, the Admin file browser, or an S3 prefix delete) removes the directory, its manifest, and its segment chunks. A FUSE rm -rf does not work — POSIX turns it into per-child unlinks that each hit the seal fence; use a server-side delete.
• Delete or modify specific entries: unseal first, change what you need, then re-seal.

Metrics

Per-operation counters track mutations rejected on frozen directories (create_rejected, update_rejected, delete_rejected, rename_rejected) alongside committed seals, unseals, mode flips (mode_flipped), compactions (compact_committed), tombstone writes (tombstone_written), tombstone-purge outcomes (purge_tombstone_consumed), repairs, and residue-purge outcomes, plus sealed-segment read cache hit/fetch rates.

Latency benchmark

The numbers below come from a real, single-node local cluster (master, volume server, and filer built from this repo, weed server, leveldb2 metadata store — no simulation or estimate). Treat the ratios between scenarios as the durable finding; the absolute microsecond/millisecond values are specific to this machine, loopback networking, and small-file dataset — a production cluster with real network hops, a busier filer, or bigger records will land at different absolute numbers. Every scenario below was re-run several times; where a number varied noticeably run to run, that’s called out rather than papered over.

Setup: two directories of 200,000 small entries each (24 bytes of inline content, no volume chunks — isolates metadata-path cost from data I/O). Each seals into 20 segments (SeaweedFS caps a segment at 10,000 entries) at about 12 bytes/entry compressed — one sealed frozen, one mutable. A 3,000-entry plain (never-sealed) directory is the baseline. “Cold” means the first touch of a given segment right after a full process restart (the in-memory segment cache is empty); “warm” means every touch after that.

Point lookups

Frozen and mutable land in the same order of magnitude both cold and warm, which is expected — both serve reads through the identical segment-fetch-and-cache path, so the ~30–50% spread between them here is ordinary local-machine noise, not a mechanistic difference. The number that matters and holds up consistently across every run is the cold-to-warm ratio: roughly 15–25×. Cold is a real volume round-trip plus a zstd-decompress; warm is a plain in-memory lookup.

Listing

Listing just the first page of a sealed directory has to decompress the whole covering segment (up to 10,000 entries) to serve the 1,000 requested — visibly more expensive than the same page from a plain directory. This specific number was the noisiest measurement in the whole benchmark: repeated runs put its p50 anywhere from ~1.6 ms to ~17.5 ms and its max up to ~31 ms, so we’re reporting the direction (“meaningfully slower than a plain listing”) with more confidence than the precise multiplier. The full-scan number is far more stable (1–5 µs/entry across repeated runs) and demonstrates the payoff of amortizing that same decompression cost over many more records instead of just one page: scanning an entire sealed directory costs an order of magnitude less per entry than reading one small page from it. Bulk listing is where sealed directories shine; small-page listing pays a fixed per-segment tax.

Write latency (mutable seal only — a frozen seal rejects every write)

Three findings here, matching — and in one case correcting — the mechanism described earlier in this doc:

• A create whose name sorts outside every existing segment’s range costs essentially nothing extra (117 µs vs. 67 µs baseline). The anti-duplication check is a binary search over the small in-memory segment index; it never touches a segment when the name doesn’t fall inside any of their ranges.
• A create or delete that touches a name within an existing segment’s range pays a real, one-time cold-segment cost (904 µs–1.9 ms) the first time, then drops to the same ~150 µs warm cost as everything else once that segment is cached.
• An update is not cheaper than a create or delete when cold — contrary to a simpler mental model where only membership-checking writes (create/delete) touch the manifest. In practice, the filer’s UpdateEntry RPC handler resolves the entry’s current state via a plain FindEntry before the request ever reaches the mutable-seal-specific logic — and for a manifest-only entry, that resolution is exactly the same cold segment fetch as a point lookup. The cheaper, local-store-only check only kicks in once the entry already has an overlay row, i.e. it’s already been touched at least once.

Compaction

Five back-to-back compactions of the same directory (each growing it by roughly 800 entries) land within 60 ms of each other — extremely stable, and almost entirely explained by the mandatory settle delay built into the seal/compaction fence (about 5.5 s by design, so peer caches and in-flight writes can drain) rather than by the size of the overlay being folded in or the directory’s total size. The original seal of the same 200,000-entry directory took 6.6 s — barely more than these compactions. That’s the concrete version of the “compaction cost scales with total directory size, not overlay size” point made earlier: at this scale the fixed fence delay dominates, and the rebuild-everything cost underneath it stays small until directories get considerably larger than 200,000 entries.

Limits to plan around

• A frozen seal is read-only until unsealed — pick mutable instead if the directory still needs writes.
• A mutable seal briefly pauses writes during compaction (the same short fence a plain seal takes) — rare, and only while the directory has already gone idle.
• The idle gate keys on child-file mtimes for both sealing and compaction. SeaweedFS does not advance a directory’s own mtime on child deletes, so a delete-only-active directory has no coldness signal; do not point a rule at an active cleanup prefix (the conservative 30-day default limits exposure, and unseal recovers). A tombstoned delete under a mutable seal does count as recent activity for the compaction idle gate, so a busy delete workload correctly delays compaction.
• A directory sealed frozen while a subdirectory was still active leaves that subdirectory as live residue and does not re-seal it later on its own; re-run a seal once it is cold. Under a mutable parent this is no longer a dead end — a later compaction pass reaches and seals a subdirectory once it goes cold.
• total_entry_count/total_file_size on a mutable seal are a snapshot as of the last seal or compaction, not a live count — they don’t yet reflect writes sitting in the overlay. They become exact again at the next compaction.
• System paths are never sealed (/, /etc, /buckets, the message-queue /topics subtree).