The Black Hole of Cold Storage
Standard cold storage is a black hole. When you dump 100 TB of video production files or audio logs onto an LTO tape or an SMR drive, you are entirely reliant on the filename and the folder structure to ever find that data again.
If you're looking for a specific clip five years later, and your file is named B_Roll_Final_v2.mp4, you are out of luck. Without manually logging timecodes into a separate Media Asset Management (MAM) system, that clip is practically gone.
Dumb Storage
Semantic Storage
We wanted to fix this. But we also didn't want to bloat HuskHoard. We are a high-performance Rust storage kernel. If we started compiling Python machine-learning libraries into our binary, we would ruin the stability of the system.
So, we turned to the Unix Philosophy: Do one thing well, and communicate via text streams.
The Unix Philosophy: The Sidecar Hook
In HuskHoard's architecture, there is an optional but incredibly powerful feature called the Sidecar Bridge. It is a simple Unix Domain Socket (/tmp/husk_sidecar.sock) that external scripts can listen to.
When the HuskHoard Janitor decides it's time to move a file from your blazing-fast NVMe to your cold storage, it pauses. It sends a PRE_ARCHIVE event over the socket to whatever program is listening, handing it the file path, and says: "I'm about to archive this. Do you have anything you want to add?"
HuskHoard gives your external script up to an hour to chew on the file. You can run OpenAI's Whisper model to generate a full audio transcript. You can run an open-source Vision-Language Model (VLM) like LLaVA to analyze the video and write descriptions of the scenes. You can run OCR on PDFs.
When your script is done, it hands a JSON payload back to HuskHoard. HuskHoard binds that JSON to the file forever.
Finding the Chicken: AI in the Loop
Let's look at a practical example. Imagine you have a 50 GB raw video file. Your local AI sidecar processes the video, sampling one frame every 5 seconds, and returns this JSON to HuskHoard:
HuskHoard takes this JSON and injects it directly into the custom_metadata column in the SQLite Catalog. Then, it moves the massive 50 GB video file to an LTO tape, punches a hole in the local file to turn it into a zero-byte stub, and ejects the tape.
Six months later, a video editor urgently needs a shot of a chicken driving a car. The video file is sitting in a fireproof safe down the hall. But the editor can query HuskHoard instantly:
This is revolutionary for media management. It doesn't matter if the file is on an SMR drive, in an S3 Glacier bucket, or on a physical tape on your desk. HuskHoard tells you exactly where the file is, and exactly what timestamp the chicken appears at.
Hot Index, Cold Payload
This entire system relies on the physical reality of data sizes. A 4K video file is 50 Gigabytes. An AI-generated JSON summary of that video is 50 Kilobytes.
The 50 GB payload is heavy, cold, and expensive to keep on an SSD. So we put it on tape (which costs $0.005 per GB). The 50 KB JSON summary is incredibly light. We keep millions of them in the local SQLite database on your blazing-fast NVMe drive.
Because the Sidecar runs locally, you can use local models like Ollama or Whisper.cpp. Your proprietary data never goes to the cloud, but you still get Google-level semantic search.
By mapping descriptions to timestamps in the JSON, you don't just find the 2-hour video file; you find the exact 5-second clip you need before you even bother fetching the tape.
The search works identical regardless of the backend. Whether the system relies on an Rclone cloud bucket or a SAS-attached tape drive, the metadata index is always hot.
Future-Proofing in the TLV Header
What if your primary SSD dies and you lose your SQLite Catalog? Is all that expensive, AI-generated metadata gone forever?
Absolutely not. When HuskHoard writes a file to tape, it creates a 4KB Object Header just before the payload. Inside that header is a TLV (Type-Length-Value) block. HuskHoard actually packs a compressed copy of your JSON metadata directly into the tape header.
The metadata physically travels with the file. If you take that tape to a completely different computer in a different country ten years from now, HuskHoard will read the 4KB header, reconstruct the SQLite database, and completely restore your AI-powered search index.
The Ultimate Archive
By decoupling the AI processing from the core storage daemon, HuskHoard gives you the ultimate active archive. We handle the brutal, low-level physics of byte-alignment, O_DIRECT I/O, and kernel-level fanotify interception. You write a 50-line Python script that asks an AI to watch your videos.
Together, it turns a dumb plastic tape cartridge into a fully semantic, offline search engine.