The Monolithic Archive Dilemma
Modern video production lives in an era of terrifying file sizes. A single 20-minute camera take in 6K or 8K REDCODE RAW, Apple ProRes 4444 XQ, or ARRI RAW routinely weighs between 60GB and 250GB. When a production wraps, that raw footage gets swept off expensive Tier-1 NVMe scratch disks and archived into deep cold storage — typically object storage like Wasabi or AWS S3, or sequential LTO tape.
Months later, an editor enters the finishing stage (the online conform). The cut has been locked using lightweight, low-resolution proxies. Now, the conform artist needs the pristine camera originals to relink the timeline for final color grading and visual effects.
The problem? The editor only used 4 seconds from a 90GB clip. Under traditional archive architectures, you have exactly one choice: restore the entire 90GB file across the internet, wait 20 minutes for it to land on local disk, extract the 4 seconds, and delete the rest. Multiply that across 150 cut points in an edit, and your conform pipeline grinds to a dead stop while your cloud egress bill skyrockets. Here is a video demo of grabbing a few seconds out of a larger MP4 video clip stored in the cloud.
All or Nothing
You need a single paragraph from a 1,000-page book stored in a distant warehouse. The shipping company insists on delivering the whole library pallet to your doorstep before you can turn to page 412.
Pointed Byte Retrieval
HuskHoard checks its catalog for the exact byte offsets corresponding to that paragraph, reaches into the cloud archive container, streams just those bytes directly over HTTP, and leaves the remaining 99.9% in the archive.
This capability is called Partial File Recall (PFR), and it is built into the core design of HuskHoard.
What Is Partial File Recall (PFR)?
Partial File Recall is the ability to retrieve an arbitrary byte sequence or time-bounded slice from an archived file without reading, transferring, or caching the entire payload.
In our previous post on file stubs, we discussed how HuskHoard punches payloads out of local storage, leaving a zero-byte stub carrying extended attributes (xattr). When an application reads the stub directly, fanotify triggers a full local re-hydration. But when applications communicate via HuskHoard's built-in StreamGate daemon, they gain access to a streaming endpoint that translates seek requests into targeted archive fetches.
This unlocks instant random access inside monolithic archives. Tools like ffmpeg, video players, and analytics engines can seek to any point in a multi-gigabyte cloud-stored file as if it were sitting on a local spindle.
HTTP Byte Ranges: The Hidden Transport Layer
How does a client tell a remote server it only wants a fraction of a file? The answer has been hiding in the HTTP standard since 1999: RFC 7233 Range Requests.
When an application wants to inspect or slice a file, it sends an HTTP GET request containing the Range header specifying the byte interval:
If the client is ffmpeg or a modern media framework (like GStreamer or AVFoundation), it first reads the container header (the MP4 moov atom or QuickTime header) at the beginning or end of the file. Once it has the frame index, it calculates the exact byte offsets for the desired in-point and out-point and fetches only those bytes.
Inside HuskHoard's StreamGate Architecture
The magic is what happens behind HuskHoard's HTTP streaming gateway when that range request arrives. The daemon doesn't just blindly pass the call upstream; it resolves virtual filesystem layouts against its internal catalog.
01:00 to 01:20 via HTTP Range header.chunk1.mp4 + offset 7,000,000 → Wasabi S3 bucket / replication_archive.img at block 4096.In our Rust daemon implementation, when an S3-backed volume is targeted, HuskHoard translates the incoming file range into an upstream S3 Range: bytes=start-end call directly to the object storage endpoint (such as Wasabi). The stream flows from the cloud provider, through HuskHoard's zero-copy memory pipeline, and straight to the client socket.
Because HuskHoard is written in Rust, range streams are processed through asynchronous buffer pools (using Tokio and hyper/axum). The daemon avoids double-buffering large payloads in RAM, meaning memory usage remains constant whether you are streaming a 50KB thumbnail or an 80GB raw container.
Automating Media Prep for Conforms
Let's look at how this fits into an actual post-production pipeline. Suppose our editor finished cutting a sequence. The edit decision list requires two clips:
chunk1.mp4: 20 seconds starting at 01:00chunk2.mp4: 20 seconds starting at 02:00
Both source files reside in cold storage. With HuskHoard running its StreamGate, we can write a simple Python script on our local edit workstation that invokes ffmpeg with lossless stream copying (-c copy):
When this script runs, ffmpeg issues range requests to the HuskHoard server. HuskHoard extracts only the needed GOPs (Group of Pictures) and audio packets from the cloud archive. Within 2 to 3 seconds, pristine high-resolution sub-clips land in the editor's local folder, ready for conform in DaVinci Resolve or Premiere Pro.
The Economics of Zero-Waste Egress
Partial File Recall isn't just an engineering convenience — it fundamentally shifts the cost model of cloud archival for media and data companies.
curl, vlc, Python, FFmpeg, or custom web players — works immediately out of the box.Stop Restoring What You Don't Need
Storage tiering should never be an all-or-nothing proposition. By decoupling the file's presentation in the namespace from its physical location, and wrapping the archive in an HTTP byte-range gateway, HuskHoard lets you treat cold cloud storage like an addressable, seekable extension of your local edit suite.
Keep your masters cold. Pull only the frames that matter.