The Dilemma: Massive Data, Tiny Gateway
Let's set the scene. You have an aging 50TB Isilon NAS, a sprawling Windows Server share, or a massive Mac-based video production SAN. You need to migrate all of this data to a shiny new EMC hardware cluster, a TrueNAS appliance, or an S3 cloud bucket.
Because you can't install custom software directly on the aging proprietary NAS, you stand up a migration gateway—a small Linux VM, an EC2 instance, or a spare Intel NUC in the server closet. This gateway mounts the legacy share via SMB or NFS, and talks to the destination via API or a high-speed 10GbE network mount. The problem? Your migration box only has a 20GB local SSD.
You are trying to push an ocean of data through a very small needle. We call this the Small Buffer problem.
Why Rsync and Rclone Aren't Quite Enough
The traditional approach is to use standard CLI tools. You mount the SMB share on the gateway and run an rsync or rclone copy command pointed directly at the cloud remote.
These tools are incredible. But for a multi-terabyte, multi-day migration, they are fundamentally stateless pipes. When you use them for massive migrations, you run into three distinct pain points:
The Pipe Approach
The Laundromat Approach
The Solution: Transparent Tiered Migration
HuskHoard is usually discussed in the context of LTO tape. But fundamentally, Husk is a Hybrid User-Space Storage Kernel. Because it supports rclone: targets as native backend volumes, it is an incredibly powerful universal archiver.
HuskHoard solves the Small Buffer problem by treating the migration not as a "copy job", but as a Tiering Event.
Instead of mapping the destination directly, you set up HuskHoard on your tiny gateway machine. You define the 20GB SSD as the "Hot Tier", and your destination (a cloud bucket, or a mounted NFS share from your new on-prem hardware) as the "Archive Tier". Then, you expose that Hot Tier via an SMB or NFS share back to the legacy system.
To the legacy Isilon or Windows box, it looks like they are simply copying files into a small 20GB network drive. But that drive is actually a "Data Laundromat".
Under the Hood: Hole-Punching and Spillovers
How do you copy 50TB of data into a 20GB drive without it filling up? You use the HuskHoard Janitor and Linux Hole-Punching.
When you start copying data into the HuskHoard Hot Tier, here is exactly what happens at the kernel level:
fanotify detects the write.FALLOC_FL_PUNCH_HOLE. Local SSD space freed!The magic happens in step four. When HuskHoard finishes streaming a file to the archive, it doesn't delete the local file. Instead, it pairs the Linux system call fallocate with FALLOC_FL_PUNCH_HOLE and FALLOC_FL_KEEP_SIZE.
This transforms the file into an instantaneous sparse file: the underlying physical storage blocks are deallocated and handed back to the SSD's free pool, while the file's logical length remains intact. We then use utimensat to restore the original file timestamps.
The result? If you run ls -l or inspect the share in Windows Explorer, the file still reports its full 14.2 GB size. Permissions, paths, and creation dates remain identical. But du -h confirms it now consumes 0 bytes of physical disk space.
Your local SSD must be sized with the "faucet vs. drain" rule in mind. If your incoming LAN transfer pushes 1 GB/s and your outgoing cloud pipe only uploads at 100 MB/s, data will accumulate. Ensure your buffer is large enough to absorb this bandwidth delta and can comfortably hold your largest single file while the background Janitor works.
The "Data Laundromat" in Practice
Because the stubbed files remain visible in the directory tree, the migration process becomes self-documenting. If you drag 5,000 video files from your Mac to the HuskHoard SMB share, you can literally watch the migration happen.
As the files copy over, the gateway's disk space fills up. Then, HuskHoard processes them. The disk space drops back down, but the files remain visible. It's a "Laundromat" — data goes in dirty (untracked, local), gets processed through the tiny buffer, and comes out clean (stubbed, cryptographically hashed, and safely on the new hardware or cloud).
And what if an editor needs to verify a file that has already been stubbed? You have two choices: accessing the file directly on the SMB share triggers an automatic, full transparent recall via fanotify. Alternatively, users can point VLC or video players to HuskHoard's built-in HTTP Streaming Gateway, which performs zero-disk, byte-range streaming directly from the archive without pulling the whole file back to the SSD.
The Golden Audit Trail
Perhaps the biggest advantage of the Laundromat architecture is the audit trail. In a corporate environment, simply knowing a transfer "completed without errors" isn't enough. You need to prove what moved, when it moved, and that it wasn't corrupted in transit.
Every file that passes through the HuskHoard gateway generates a permanent receipt in the husk_catalog.db SQLite database.
Because HuskHoard streams the file through a high-speed Zstd compression frame (if enabled) and a BLAKE3 hashing function simultaneously, you get a cryptographic signature of the exact payload that landed in your cloud bucket. No third-party migration tool gives you this level of local, queryable confidence.
If a user modifies a file on the source and overwrites the copy in the Laundromat, HuskHoard doesn't delete the remote file. It creates a new version in the database, protecting you from mid-migration accidental deletions.
Commercial cloud-to-cloud migration services require you to hand over your API keys. HuskHoard runs entirely on your infrastructure. Your data and credentials never pass through a third party.
If the power dies or the network drops, you don't need to rescan 50TB of data. When the gateway boots back up, HuskHoard simply picks up the remaining un-stubbed files in the hot tier and continues.
Zero-Space Verification and Rollbacks
Let's say your migration finishes. 50TB of data has successfully passed through your 20GB gateway and resides on your new EMC cluster. Before you decommission the old Isilon NAS, your boss asks: "Are you absolutely sure the data at the new destination isn't corrupted?"
Normally, verifying remote data requires downloading it back to local disk and running a checksum. But your gateway only has 20GB of space. How do you verify 50TB?
You run the Husk Scrubber.
For on-prem image targets, the husk scrub command reads raw object frames into memory, verifies their embedded header CRCs and BLAKE3 payloads on the fly, and immediately discards the bytes. For cloud destinations, every uploaded object's BLAKE3 checksum is already recorded in husk_catalog.db, giving you a tamper-evident audit ledger without requiring hundreds of gigabytes of local verification cache.
Building Your Own Laundromat
To use HuskHoard as a migration gateway, you need a Linux machine, Docker or a native Rust toolchain, and Rclone installed and configured with your target cloud.
Here is the stripped-down husk_config.toml required to turn HuskHoard into an aggressive migration gateway:
Start the Husk daemon, expose /mnt/local_ssd/migration_buffer as an SMB share to your network, and start copying files from your legacy system. The Laundromat handles the rest.
Because the gateway relies on fanotify and fallocate, the Hot Tier must reside on a Linux filesystem that natively supports hole-punching (ext4, XFS, Btrfs). Do not use an exFAT, FAT32, or NTFS formatted drive as your buffer.
Migration is Now a Tiering Problem
Migrating enterprise data shouldn't mean losing visibility, wrestling with brittle sync scripts, or buying massively over-provisioned hardware just to act as temporary landing pads.
By treating migration as an active tiered architecture rather than a blind copy pipe, HuskHoard turns your limited SSD into a self-draining funnel. You get O(1) memory guarantees, an immutable cryptographic paper trail, and full user transparency from the first gigabyte to the fifty-thousandth.
Try it on your next hardware refresh or cloud transition—you may never write a migration shell script again.
The Art of the Silent Move
How the background Janitor moves data safely without breaking open file descriptors.
File Stubs: Under the Hood
Why hole-punched sparse files maintain application compatibility when symlinks fail.
Stop Murdering Your SMR Drives
Shingled Magnetic Recording is cheap, but requires sequential append discipline.