The Dilemma: Massive Data, Tiny Gateway
Let's set the scene. You have an aging 50TB Isilon NAS, a sprawling Windows Server share, or a massive Mac-based video production SAN. You need to migrate all of this data to a shiny new EMC hardware cluster, a TrueNAS appliance, or an S3 cloud bucket.
Because you can't install custom software directly on the aging proprietary NAS, you stand up a migration gateway—a small Linux VM, an EC2 instance, or a spare Intel NUC in the server closet. This gateway mounts the legacy share via SMB or NFS, and talks to the destination via API or a high-speed 10GbE network mount. The problem? Your migration box only has a 20GB local SSD.
You are trying to push an ocean of data through a very small needle. We call this the Small Buffer problem.
Why Rsync and Rclone Aren't Quite Enough
The traditional approach is to use standard CLI tools. You mount the SMB share on the gateway and run an rsync or rclone copy command pointed directly at the cloud remote.
These tools are incredible. But for a multi-terabyte, multi-day migration, they are fundamentally stateless pipes. When you use them for massive migrations, you run into three distinct pain points:
The Pipe Approach
The Laundromat Approach
The Solution: Transparent Tiered Migration
HuskHoard is usually discussed in the context of LTO tape. But fundamentally, Husk is a Hybrid User-Space Storage Kernel. Because it supports rclone: targets as native backend volumes, it is an incredibly powerful universal archiver.
HuskHoard solves the Small Buffer problem by treating the migration not as a "copy job", but as a Tiering Event.
Instead of mapping the destination directly, you set up HuskHoard on your tiny gateway machine. You define the 20GB SSD as the "Hot Tier", and your destination (a cloud bucket, or a mounted NFS share from your new on-prem hardware) as the "Archive Tier". Then, you expose that Hot Tier via an SMB or NFS share back to the legacy system.
To the legacy Isilon or Windows box, it looks like they are simply copying files into a small 20GB network drive. But that drive is actually a "Data Laundromat".
Under the Hood: Hole-Punching and Spillovers
How do you copy 50TB of data into a 20GB drive without it filling up? You use the HuskHoard Janitor and Linux Hole-Punching.
When you start copying data into the HuskHoard Hot Tier, here is exactly what happens at the kernel level:
fanotify detects the write.FALLOC_FL_PUNCH_HOLE. Local SSD space freed!The magic happens in step four. When HuskHoard finishes uploading a file to the cloud, it doesn't delete the local file. Instead, it uses a Linux system call called fallocate with the FALLOC_FL_PUNCH_HOLE flag.
This tells the filesystem: "Keep the file name, keep the creation date, keep the metadata, and tell the OS the file is exactly 14.2 GB. But delete all the actual data blocks from the SSD."
The result? The file remains perfectly visible in the folder. If you run ls -l or look at it in Windows Explorer, it looks like a normal 14.2 GB file. But it consumes 0 bytes of physical SSD space.
In your husk_config.toml, this behavior is governed by a single line: hot_tier_max_usage_percent = 80. If a massive copy job pushes the local disk past 80% capacity, the Janitor thread interrupts its normal schedule and triggers an "Emergency Spillover", rapidly streaming and stubbing the oldest files in the buffer to make room for the incoming data.
The "Data Laundromat" in Practice
Because the stubbed files remain visible in the directory tree, the migration process becomes self-documenting. If you drag 5,000 video files from your Mac to the HuskHoard SMB share, you can literally watch the migration happen.
As the files copy over, the gateway's disk space fills up. Then, HuskHoard processes them. The disk space drops back down, but the files remain visible. It's a "Laundromat" — data goes in dirty (untracked, local), gets processed through the tiny buffer, and comes out clean (stubbed, cryptographically hashed, and safely on the new hardware or cloud).
And because of HuskHoard's transparent fanotify demand-load, if a user accidentally double-clicks a stubbed video file on the SMB share, HuskHoard instantly intercepts the read, pulls the specific byte-range from the cloud, and serves it to the media player. The user never knows the file wasn't on the local disk.
The Golden Audit Trail
Perhaps the biggest advantage of the Laundromat architecture is the audit trail. In a corporate environment, simply knowing a transfer "completed without errors" isn't enough. You need to prove what moved, when it moved, and that it wasn't corrupted in transit.
Every file that passes through the HuskHoard gateway generates a permanent receipt in the husk_catalog.db SQLite database.
Because HuskHoard streams the file through a high-speed Zstd compression frame (if enabled) and a BLAKE3 hashing function simultaneously, you get a cryptographic signature of the exact payload that landed in your cloud bucket. No third-party migration tool gives you this level of local, queryable confidence.
If a user modifies a file on the source and overwrites the copy in the Laundromat, HuskHoard doesn't delete the remote file. It creates a new version in the database, protecting you from mid-migration accidental deletions.
Commercial cloud-to-cloud migration services require you to hand over your API keys. HuskHoard runs entirely on your infrastructure. Your data and credentials never pass through a third party.
If the power dies or the network drops, you don't need to rescan 50TB of data. When the gateway boots back up, HuskHoard simply picks up the remaining un-stubbed files in the hot tier and continues.
Zero-Space Verification and Rollbacks
Let's say your migration finishes. 50TB of data has successfully passed through your 20GB gateway and resides on your new EMC cluster. Before you decommission the old Isilon NAS, your boss asks: "Are you absolutely sure the data at the new destination isn't corrupted?"
Normally, verifying remote data requires downloading it back to local disk and running a checksum. But your gateway only has 20GB of space. How do you verify 50TB?
You run the Husk Scrubber.
The husk scrub command streams the data from the new storage directly into memory, passing it through the BLAKE3 hashing algorithm on the fly, and immediately discarding the bytes. It compares the result against the SQLite catalog. It requires exactly 0 bytes of local disk space, allowing your tiny gateway to cryptographically verify petabytes of migrated data.
Building Your Own Laundromat
To use HuskHoard as a migration gateway, you need a Linux machine, Docker or a native Rust toolchain, and Rclone installed and configured with your target cloud.
Here is the stripped-down husk_config.toml required to turn HuskHoard into an aggressive migration gateway:
Start the Husk daemon, expose /mnt/local_ssd/migration_buffer as an SMB share to your network, and start copying files from your legacy system. The Laundromat handles the rest.
Because the gateway relies on `fanotify` and `fallocate`, it must run on a Linux filesystem that supports hole-punching (ext4, XFS, Btrfs). Do not use an exFAT or NTFS formatted external drive as your Hot Tier.
Migrating data shouldn't mean losing visibility, dealing with brittle sync scripts, or purchasing massively over-provisioned local hardware. By turning migration into a tiered architecture, HuskHoard brings enterprise HSM (Hierarchical Storage Management) capabilities to a single, free Rust binary.
Try it out on your next cloud migration. You might never use rsync for a massive move again.