The Petabyte Tax

In a modern Databricks environment, storage is often the second-largest line item after compute. Delta Lakes that start life as a few terabytes of clean transaction history have a way of compounding quietly — historical partitions accumulate, old model training sets never get deleted, and raw ingestion layers balloon faster than anyone planned for. By the time your Delta Lake is measured in petabytes, the cost of keeping all of it queryable in Hot S3 or Azure ADLS is simply unsustainable.

The obvious answer is lifecycle management: move historical partitions to Glacier Deep Archive or on-premises LTO tape, where storage costs drop from roughly $23/TB/month to around $1/TB/month. Obvious, and correct — except for one critical problem.

Cold storage is fundamentally asynchronous. The moment a Spark job attempts to open a file stored in Glacier, it doesn't slow down — it fails instantly. Databricks has no built-in mechanism for pausing a job to wait for an asynchronous restore. Your data engineering pipeline simply dies at the partition boundary, and a human has to intervene.

The core tension

Hot storage gives you instant access but costs 23x more. Cold storage is economical but incompatible with any synchronous query engine. HuskHoard's StreamGate collapses this distinction — serving cold data through a standard HTTP interface that Spark treats as if it were a live object store.

The Thaw Wall in Practice

The standard cold-storage workflow goes something like this: a Spark job is scheduled to run a historical analysis across twelve months of partitioned event data. It reads January through October without issue — those partitions are still in S3 Standard. It hits November and immediately fails with an access error, because that partition was tiered to Glacier three weeks ago. An engineer notices the failed job hours later, manually submits a Glacier restore request, and waits. Glacier standard retrieval takes three to five hours. Expedited retrieval is available but costs significantly more per gigabyte. Once the restore completes, the data is duplicated back into S3 Standard at full hot-storage pricing for the duration of the restore window. The job is re-run. The restored copy sits in S3 Standard for its mandatory minimum retention period, accruing costs, before it can be deleted.

Standard Cold Storage — The Thaw Wall

Spark job scheduled
Reads hot partitions fine
Hits cold partition — FAILS
Manual Glacier restore request
Wait 3–5 hours
Data duplicated to S3 Standard
Re-run job manually
Restored copy accrues hot cost
Done. 6+ hours later.

The problem compounds at scale. A petabyte-range Delta Lake may have hundreds of cold partitions spread across dozens of tables. Managing restore requests, tracking their status, and coordinating re-runs becomes a part-time job in itself — and that's before accounting for the cost of the duplicate data sitting in hot storage during its mandatory retention window.

Native Parquet Catalogs for Unity Catalog

Before we get to query-time streaming, it's worth addressing a subtler problem: discoverability. Even if you could instantly query a cold archive, your data scientists would still have no way to know what's in it. The metadata — file paths, sizes, versions, physical locations, BLAKE3 integrity hashes — lives in HuskHoard's SQLite catalog database, invisible to the Hive Metastore and inaccessible to Databricks Unity Catalog.

HuskHoard solves this with husk export --format parquet. The command reads the entire catalog table and writes it out as a standard Apache Parquet file — the same columnar format Databricks uses natively. Sync that export to your data lake on any schedule you choose, and register it as an external table in Unity Catalog:

-- Register the HuskHoard catalog export as a Unity Catalog external table CREATE TABLE external_metadata.husk_catalog USING PARQUET LOCATION 's3://your-lake/husk/catalog-export/'; -- Now query your entire archive metadata with standard SQL SELECT file_path, version, ROUND(payload_size / 1e9, 2) AS size_gb, tape_uuid, blake3_hash, archived_at FROM external_metadata.husk_catalog WHERE archived_at > now() - INTERVAL '30 days' ORDER BY payload_size DESC;

This turns the HuskHoard catalog into a first-class data engineering asset. Your data scientists can query the complete metadata of every archived object — sizes, versions, physical tape locations, integrity hashes — using standard SQL inside a Databricks notebook, without touching the actual cold storage and without any specialized tooling. It also means you can build automated pipelines that decide which archived files to query based on their metadata, before committing any I/O against the cold backend.

What the schema looks like from Databricks

Because the Parquet export is a direct projection of the catalog table, its schema is immediately familiar to anyone who has worked with the Catalog . Each row represents one version of one archived file. The tape_uuid column is the join key that links a catalog entry to its physical volume — the same UUID that HuskHoard uses internally to route a StreamGate request to the correct backend device.

Without HuskHoard

Archive Discoverability

Cold data is invisible to Unity Catalog and the Hive Metastore
Engineers must track archive contents in spreadsheets or manually
No programmatic way to query metadata without a restore
Data governance and lineage tooling can't see archived assets
With HuskHoard

Archive Discoverability

Full catalog exported as Parquet, registered as an external table
Standard SQL over the entire archive history in any Databricks notebook
Metadata queries cost zero egress — the Parquet export lives in your lake
Unity Catalog lineage and tagging work on archived assets like any other table

StreamGate and Spark Predicate Pushdown

The Parquet catalog export handles discoverability. The actual query-time magic happens through StreamGate — HuskHoard's built-in HTTP Streaming Gateway, running by default on port 8080.

StreamGate speaks plain HTTP/1.1 and implements RFC 7233 Range requests: the same byte-range protocol used by every S3-compatible object store, every media player, and every Parquet reader in existence. When Databricks Photon or a standard Spark executor opens a Parquet file over HTTP, it doesn't read the entire file. It first sends a HEAD request to discover the file size, then sends a GET with a Range: bytes= header to fetch just the 32 KB Parquet footer — the index that tells it where each column chunk and row group lives inside the file. Only then does it issue targeted byte-range requests to fetch the specific columns and row groups that satisfy the query predicate.

This is Spark's predicate pushdown at the I/O level, and HuskHoard is designed to serve it. StreamGate parses the incoming Range header, consults the jump table in object_frames, locates the exact compressed frames that contain the requested bytes, pulls only those frames from the cold backend, and streams the decompressed result back with a 206 Partial Content response — all in a single synchronous roundtrip that Spark never distinguishes from a standard S3 read.

HuskHoard + Databricks — Zero Thaw

Spark HEAD request → file size
StreamGate: catalog lookup
206 response, instant
Spark GET Range: footer bytes
Jump table: find 16MB frame
Stream decompressed footer
Spark GET Range: row group N
Pull only required frames
Job completes. No thaw.

From Databricks' perspective, the archived Parquet file looks like any other HTTP object store endpoint. No custom connectors, no JVM agents, no special Spark configuration beyond pointing the reader at http://husk-host:8080/stream/path/to/file.parquet.

Inside the Zstd Jump Table

The reason StreamGate can serve a byte-range request without decompressing from the beginning of the file is the jump table — an index of 16 MB compressed frame boundaries that HuskHoard builds at archive time and stores in the object_frames database table.

When HuskHoard archives a large file, it passes the data through the Zstd compressor in 16 MB chunks. At each chunk boundary, it records three numbers: the byte offset in the uncompressed stream where this chunk begins, the byte offset in the compressed stream on tape where this chunk begins, and the compressed size of the chunk. These three numbers per frame are stored in object_frames, and a compact copy (just the compressed size of each frame, 4 bytes per frame) is also embedded directly in the on-tape object header as a TLV record — so the jump table survives even a full catalog rebuild from raw tape.

-- The object_frames table powers every StreamGate byte-range request SELECT uncompressed_offset, compressed_offset, compressed_size FROM object_frames WHERE file_path = '/lake/events/year=2024/month=11/part-00001.parquet' AND version = 3 ORDER BY uncompressed_offset ASC; -- Example result for a 4 GB Parquet file (256 frames at 16MB each): -- uncompressed_offset | compressed_offset | compressed_size -- ------------------ | ---------------- | --------------- -- 0 | 0 | 11,534,336 -- 16,777,216 | 11,534,336 | 10,485,760 -- 33,554,432 | 22,020,096 | 12,582,912 -- ...256 rows total...

When a Spark executor sends Range: bytes=3,145,728,000-3,145,760,000 — asking for 40 KB somewhere inside the third gigabyte of a file — StreamGate does a binary search over the object_frames rows to find which 16 MB compressed frame contains uncompressed offset 3,145,728,000. It calculates the exact compressed byte offset on the tape, fetches only that single frame from the cold backend (whether LTO, a flat image file, or an rclone cloud remote), decompresses forward to the requested starting byte, and writes exactly the requested length to the socket. The rest of the file is never touched.

Why 16 MB frames?

A 10 GB Parquet file produces roughly 640 frames. At 4 bytes per frame, the entire jump table fits in 2.5 KB — small enough to embed directly in the 4 KB on-tape object header and retrieve in a single I/O. Smaller frames would give finer-grained seeking at the cost of a larger index and more I/O round trips; larger frames reduce round trips but increase the amount of cold data fetched per request. 16 MB was chosen as the crossover point for typical Parquet row group sizes.

Decoupling Compute from Cold Storage

The combination of the Parquet metadata catalog and StreamGate's byte-range serving effectively decouples Databricks compute from the physical constraints of cold storage. You get the economics of deep archive with the access latency expectations of a modern data lake.

This matters beyond just cost. Consider a few practical scenarios that become trivially easy with HuskHoard in place:

Historical ML Training Sets

Large video or sensor datasets archived a year ago can feed PyTorch or TensorFlow dataloaders that sample small random subsets of massive files. Because StreamGate serves arbitrary byte-range requests in real time, a dataloader that requests 5,000 random 4 KB samples from a 200 GB archived video file will trigger 5,000 independent StreamGate requests — each fetching only the single 16 MB compressed frame that contains the requested bytes. The training job never waits for a restore and never pulls more data than it actually needs.

Regulatory Audit Queries

Compliance teams frequently need to run point-in-time queries against historical transaction logs that have long since been tiered to cold storage. With HuskHoard, those queries run directly against the archived Parquet partitions through StreamGate. There is no restore step, no temporary duplication cost, and no 12-hour SLA to explain to a regulator.

Cost-Controlled Egress

Cloud egress fees on full-file restores from Deep Archive are a significant hidden cost. Because StreamGate fetches only the compressed frames that contain the requested byte ranges, and Spark's predicate pushdown means most queries touch only a fraction of a Parquet file's columns, real-world egress savings are substantial compared to restoring entire files before querying them.

~0s
Thaw Time

StreamGate serves byte-range requests synchronously. Spark jobs don't pause, fail, or require manual intervention at cold partition boundaries.

16 MB
Max Frame Pull

Each StreamGate request fetches at most one 16 MB compressed frame from cold storage, regardless of file size. Most Spark reads touch fewer than five frames per file.

$1/TB
Storage Cost

LTO tape and Deep Archive storage costs, with full queryability. No mandatory hot-tier duplication. No restore fees for standard predicate-pushdown queries.

For the Databricks community, the goal of "everything in the Lakehouse" has always implied that data should be queryable regardless of where it lives. HuskHoard makes that true even at the bottom of the storage tier hierarchy — your deepest, oldest archives remain first-class queryable assets, not write-only graveyards. The Thaw Wall doesn't have to be the end of your data's useful life.