When we started designing HuskHoard's archive layer, compression was a solved problem — or so we thought. LZ4 for speed, gzip for compatibility, Brotli for web. Pick one and move on. But cold storage for a data-tiering engine has a constraint that changes everything: you can't afford to decompress from the beginning every time a user requests a file.
The Problem with Sequential-Only Decompression
Traditional compression formats — gzip, bzip2, even plain Zstd — produce a single contiguous bitstream. To read byte 500 MB into a 2 GB archive, you must decompress all 500 MB first. For a tape-backed archive that might hold a 50 GB ProRes video, this is a non-starter.
The requirement: a video player should be able to seek to minute 47 of a tape-stored movie without pulling the entire file off the drive first.
This is exactly what HuskHoard's StreamGate HTTP gateway needs to satisfy HTTP Range requests — the mechanism that lets mpv, VLC, Plex, and Jellyfin scrub through media without a full download.
Enter Zstd Seekable Format
Zstd's seekable format splits the compressed data into independent frames, each compressing a fixed-size chunk of the original data (we use 1 MB chunks by default). A seek table at the end of the file maps uncompressed byte offsets to frame positions.
[ Frame 0: bytes 0–1 MB ] [ Frame 1: bytes 1–2 MB ] ... [ Seek Table ]To read from offset 500 MB, HuskHoard consults the seek table, jumps directly to the correct frame, decompresses only that 1 MB chunk, and returns the bytes. No wasted I/O. No wasted CPU.
What we measured
In our benchmarks on an LTO-8 drive with 1080p H.264 content, first-byte latency for a mid-file seek dropped from ~40 seconds (full sequential decompress) to under 800 ms with seekable Zstd. Compression ratio was within 2% of non-seekable Zstd at equivalent settings.
- Frame size 1 MB: best seek latency, ~2% ratio overhead
- Frame size 4 MB: good balance for large files
- Frame size 16 MB: best ratio, higher seek cost — appropriate for archival-only workloads
How HuskHoard Uses It
The Archive Worker writes every cold-tiered file using the Zstd seekable format via the zstd-seekable Rust crate. The seek table is embedded at the tail of each archive block, so no external index file is required — the archive is fully self-describing.
StreamGate translates HTTP Range: bytes=X-Y headers into frame lookups, issues low-level SCSI read commands to position the tape head at the correct block, and streams decompressed bytes back to the client — all without writing to disk.
If you're building anything that needs random-access reads from compressed cold storage, Zstd seekable format is worth a close look. The upstream documentation is sparse but the crate is solid — we've had zero corruption incidents in six months of testing.
Questions or corrections? Open an issue on GitHub or reach us at info@huskhoard.com.