The Problem with Traditional Tiering (HSM)
Hierarchical Storage Management (HSM) has been around for decades. The logic is flawless: keep data that is actively being worked on in fast, expensive storage (Hot), and move older, finalized data to slower, cheaper storage (Cold).
But the execution has historically been a nightmare for end-users. When an IT script moves an old video project from /mnt/nvme_fast/ to /mnt/tape_archive/, the original file path vanishes. Premiere Pro projects break. Databases lose their asset links. Users panic because their folders look empty, resulting in panicked tickets to the sysadmin asking "Where did my files go?"
With HuskHoard, we threw away the concept of multiple mount points. The user only ever interacts with the Hot Tier. When a file is moved to a Warm HDD or a Cold Tape, HuskHoard leaves behind a sparse "stub" file that consumes a few bytes of physical SSD space, while perfectly preserving the file name, permissions, and metadata. The directory structure never changes.
Defining the Arsenal: Hot, Warm, and Cold
To build a bulletproof storage architecture, you have to align your hardware with the physics of data gravity. In the HuskHoard configuration (husk_config.toml), you map these physical realities to logical volumes.
The Workspace
Configured as hot_tier = "hot_tier". This is your frontend cache. It provides massive IOPS for active projects. However, it is expensive and low-capacity. HuskHoard ensures this tier never fills up.
The Online Archive
Configured via primary_volumes. This is cheap spinning rust or a local object store. Files here are compressed and packed, but crucially, they are online. When a user clicks a stubbed file, HuskHoard's StreamGate instantly seeks to the exact block on the HDD and streams it back in O(1) time.
Configured via replication_volumes. This is where you put data to survive a disaster. Files here are written sequentially to append-only media like LTO drives (/dev/nst0) or AWS S3 (rclone:my_aws). Why not use tape for everything? Because tape requires Hardware Wake-Ups and rewind times. If a user tries to open a file resting only on tape, HuskHoard issues an MTREW SCSI command and spins up the drive—a process that takes seconds to minutes, not milliseconds.
The Policy Engine: When to Move What
So, how does HuskHoard know when to push a file down the stack? In the enterprise world, you don't want a human making these decisions. You rely on the Janitor—HuskHoard's built-in background policy scanner.
The rules of engagement are defined entirely in your configuration file. Let's look at the three primary thresholds:
max_age_days, the Janitor assumes the active project is over. It copies the file to the Warm/Cold tiers, and replaces the hot file with a 0-byte stub..mov video files that users are just dropping off? By adding them to immediate_archive_extensions, HuskHoard sweeps them instantly. Furthermore, by using no_compress_extensions, we bypass CPU-heavy Zstd compression, copying them as raw block streams at maximum hardware speed.Architecting the Storage Flow
Putting it all together, HuskHoard manages the lifecycle of a file seamlessly behind the scenes using standard Linux syscalls like fanotify and fallocate.
What happens when a user comes back a year later and double-clicks that 50GB file?
HuskHoard's VFS interceptor pauses the read request for a fraction of a millisecond. It queries the local SQLite catalog (husk_catalog.db) and notices the file has a replica on the Warm Tier (HDD) and the Cold Tier (Tape). Because HuskHoard is smart, it skips the offline Tape drive entirely. It targets the Warm HDD, utilizes the StreamGate jump tables to find the exact byte offset the user requested, and pipes the data straight to the application. The user literally cannot tell the file was ever moved.
The Economics of the Tri-Tier Stack
Why go through the trouble of separating Warm HDDs from Cold Tapes? It comes down to cost, power consumption, and instant accessibility.
| Architecture Setup | Cost per 100TB | Power / Cooling | Retrieval Time |
|---|---|---|---|
| All-Flash SAN (No Tiering) | $8,000 - $12,000 | Very High | Instant (Overkill) |
| Disk-Only NAS (ZFS/RAID) | $2,500 - $4,000 | High (Spinning 24/7) | Instant |
| Husk Tri-Tier (NVMe + HDD + LTO-8) | ~$1,200 | Low (Tapes use 0W at rest) | Instant (Warm) / 2 Min (Cold) |
By defining your boundaries, you stop overpaying for inactive data. You don't need petabytes of NVMe storage. You just need a reasonably sized Hot Tier cache to absorb the day's ingest, a Warm array of high-density HDDs to provide instant recall for older projects, and a deep Cold stack of LTO tapes or Cloud buckets to guarantee you never lose a byte to bit-rot or ransomware.
Eliminate broken links. Users see one massive drive, completely ignorant of whether the file rests on an SSD or an offline Tape.
By skipping compression on pre-compressed media and avoiding tape wake-ups when warm disks are available, hardware lasts longer.
Set your thresholds once. The Janitor daemon works endlessly in the background to ensure your active drives never hit 100% capacity.
Final Thought
A smart file system isn't one that tries to hold everything in RAM or forces you to buy endless expansion shelves. A smart file system acts as an air-traffic controller, routing data to the exact hardware tier that balances cost and speed.
With HuskHoard, you get the enterprise architecture of a multi-million dollar HSM appliance, built entirely in open-source Rust, completely agnostic to your choice of hardware or cloud provider. Stop chasing symlinks, and start building tiers.