The Petabyte Tax
In a modern Databricks environment, storage is often the second-largest line item after compute. Delta Lakes that start life as a few terabytes of clean transaction history have a way of compounding quietly — historical partitions accumulate, old model training sets never get deleted, and raw ingestion layers balloon faster than anyone planned for. By the time your Delta Lake is measured in petabytes, the cost of keeping all of it queryable in Hot S3 or Azure ADLS is simply unsustainable.
The obvious answer is lifecycle management: move historical partitions to Glacier Deep Archive or on-premises LTO tape, where storage costs drop from roughly $23/TB/month to around $1/TB/month. Obvious, and correct — except for one critical problem.
Cold storage is fundamentally asynchronous. The moment a Spark job attempts to open a file stored in Glacier, it doesn't slow down — it fails instantly. Databricks has no built-in mechanism for pausing a job to wait for an asynchronous restore. Your data engineering pipeline simply dies at the partition boundary, and a human has to intervene.
Hot storage gives you instant access but costs 23x more. Cold storage is economical but incompatible with any synchronous query engine. HuskHoard's StreamGate collapses this distinction — serving cold data through a standard HTTP interface that Spark treats as if it were a live object store.
The Thaw Wall in Practice
The standard cold-storage workflow goes something like this: a Spark job is scheduled to run a historical analysis across twelve months of partitioned event data. It reads January through October without issue — those partitions are still in S3 Standard. It hits November and immediately fails with an access error, because that partition was tiered to Glacier three weeks ago. An engineer notices the failed job hours later, manually submits a Glacier restore request, and waits. Glacier standard retrieval takes three to five hours. Expedited retrieval is available but costs significantly more per gigabyte. Once the restore completes, the data is duplicated back into S3 Standard at full hot-storage pricing for the duration of the restore window. The job is re-run. The restored copy sits in S3 Standard for its mandatory minimum retention period, accruing costs, before it can be deleted.
Standard Cold Storage — The Thaw Wall
The problem compounds at scale. A petabyte-range Delta Lake may have hundreds of cold partitions spread across dozens of tables. Managing restore requests, tracking their status, and coordinating re-runs becomes a part-time job in itself — and that's before accounting for the cost of the duplicate data sitting in hot storage during its mandatory retention window.
Native Parquet Catalogs for Unity Catalog
Before we get to query-time streaming, it's worth addressing a subtler problem: discoverability. Even if you could instantly query a cold archive, your data scientists would still have no way to know what's in it. The metadata — file paths, sizes, versions, physical locations, BLAKE3 integrity hashes — lives in HuskHoard's SQLite catalog database, invisible to the Hive Metastore and inaccessible to Databricks Unity Catalog.
HuskHoard solves this with husk export --format parquet. The command reads the entire catalog table and writes it out as a standard Apache Parquet file — the same columnar format Databricks uses natively. Sync that export to your data lake on any schedule you choose, and register it as an external table in Unity Catalog:
This turns the HuskHoard catalog into a first-class data engineering asset. Your data scientists can query the complete metadata of every archived object — sizes, versions, physical tape locations, integrity hashes — using standard SQL inside a Databricks notebook, without touching the actual cold storage and without any specialized tooling. It also means you can build automated pipelines that decide which archived files to query based on their metadata, before committing any I/O against the cold backend.
What the schema looks like from Databricks
Because the Parquet export is a direct projection of the catalog table, its schema is immediately familiar to anyone who has worked with the Catalog . Each row represents one version of one archived file. The tape_uuid column is the join key that links a catalog entry to its physical volume — the same UUID that HuskHoard uses internally to route a StreamGate request to the correct backend device.
Archive Discoverability
Archive Discoverability
StreamGate and Spark Predicate Pushdown
The Parquet catalog export handles discoverability. The actual query-time magic happens through StreamGate — HuskHoard's built-in HTTP Streaming Gateway, running by default on port 8080.
StreamGate speaks plain HTTP/1.1 and implements RFC 7233 Range requests: the same byte-range protocol used by every S3-compatible object store, every media player, and every Parquet reader in existence. When Databricks Photon or a standard Spark executor opens a Parquet file over HTTP, it doesn't read the entire file. It first sends a HEAD request to discover the file size, then sends a GET with a Range: bytes= header to fetch just the 32 KB Parquet footer — the index that tells it where each column chunk and row group lives inside the file. Only then does it issue targeted byte-range requests to fetch the specific columns and row groups that satisfy the query predicate.
This is Spark's predicate pushdown at the I/O level, and HuskHoard is designed to serve it. StreamGate parses the incoming Range header, consults the jump table in object_frames, locates the exact compressed frames that contain the requested bytes, pulls only those frames from the cold backend, and streams the decompressed result back with a 206 Partial Content response — all in a single synchronous roundtrip that Spark never distinguishes from a standard S3 read.
HuskHoard + Databricks — Zero Thaw
From Databricks' perspective, the archived Parquet file looks like any other HTTP object store endpoint. No custom connectors, no JVM agents, no special Spark configuration beyond pointing the reader at http://husk-host:8080/stream/path/to/file.parquet.
Inside the Zstd Jump Table
The reason StreamGate can serve a byte-range request without decompressing from the beginning of the file is the jump table — an index of 16 MB compressed frame boundaries that HuskHoard builds at archive time and stores in the object_frames database table.
When HuskHoard archives a large file, it passes the data through the Zstd compressor in 16 MB chunks. At each chunk boundary, it records three numbers: the byte offset in the uncompressed stream where this chunk begins, the byte offset in the compressed stream on tape where this chunk begins, and the compressed size of the chunk. These three numbers per frame are stored in object_frames, and a compact copy (just the compressed size of each frame, 4 bytes per frame) is also embedded directly in the on-tape object header as a TLV record — so the jump table survives even a full catalog rebuild from raw tape.
When a Spark executor sends Range: bytes=3,145,728,000-3,145,760,000 — asking for 40 KB somewhere inside the third gigabyte of a file — StreamGate does a binary search over the object_frames rows to find which 16 MB compressed frame contains uncompressed offset 3,145,728,000. It calculates the exact compressed byte offset on the tape, fetches only that single frame from the cold backend (whether LTO, a flat image file, or an rclone cloud remote), decompresses forward to the requested starting byte, and writes exactly the requested length to the socket. The rest of the file is never touched.
A 10 GB Parquet file produces roughly 640 frames. At 4 bytes per frame, the entire jump table fits in 2.5 KB — small enough to embed directly in the 4 KB on-tape object header and retrieve in a single I/O. Smaller frames would give finer-grained seeking at the cost of a larger index and more I/O round trips; larger frames reduce round trips but increase the amount of cold data fetched per request. 16 MB was chosen as the crossover point for typical Parquet row group sizes.
Decoupling Compute from Cold Storage
The combination of the Parquet metadata catalog and StreamGate's byte-range serving effectively decouples Databricks compute from the physical constraints of cold storage. You get the economics of deep archive with the access latency expectations of a modern data lake.
This matters beyond just cost. Consider a few practical scenarios that become trivially easy with HuskHoard in place:
Historical ML Training Sets
Large video or sensor datasets archived a year ago can feed PyTorch or TensorFlow dataloaders that sample small random subsets of massive files. Because StreamGate serves arbitrary byte-range requests in real time, a dataloader that requests 5,000 random 4 KB samples from a 200 GB archived video file will trigger 5,000 independent StreamGate requests — each fetching only the single 16 MB compressed frame that contains the requested bytes. The training job never waits for a restore and never pulls more data than it actually needs.
Regulatory Audit Queries
Compliance teams frequently need to run point-in-time queries against historical transaction logs that have long since been tiered to cold storage. With HuskHoard, those queries run directly against the archived Parquet partitions through StreamGate. There is no restore step, no temporary duplication cost, and no 12-hour SLA to explain to a regulator.
Cost-Controlled Egress
Cloud egress fees on full-file restores from Deep Archive are a significant hidden cost. Because StreamGate fetches only the compressed frames that contain the requested byte ranges, and Spark's predicate pushdown means most queries touch only a fraction of a Parquet file's columns, real-world egress savings are substantial compared to restoring entire files before querying them.
StreamGate serves byte-range requests synchronously. Spark jobs don't pause, fail, or require manual intervention at cold partition boundaries.
Each StreamGate request fetches at most one 16 MB compressed frame from cold storage, regardless of file size. Most Spark reads touch fewer than five frames per file.
LTO tape and Deep Archive storage costs, with full queryability. No mandatory hot-tier duplication. No restore fees for standard predicate-pushdown queries.
For the Databricks community, the goal of "everything in the Lakehouse" has always implied that data should be queryable regardless of where it lives. HuskHoard makes that true even at the bottom of the storage tier hierarchy — your deepest, oldest archives remain first-class queryable assets, not write-only graveyards. The Thaw Wall doesn't have to be the end of your data's useful life.