In this article
  1. The Client-Server Web Bias
  2. The Physics of Filesystem Latency
  3. Write-Ahead Logging: Concurrent Magic
  4. The Myth of Scale
  5. Future Proofing: The Parquet Hook

The Client-Server Web Bias

There is a pervasive bias in modern software engineering that assumes if an application is "serious," it must be distributed. If a database doesn't require a Docker container, an open TCP port, a dedicated connection pool, and a DBA to manage it, it must be a toy.

PostgreSQL and MySQL are phenomenal databases. They are designed to handle 10,000 concurrent network clients scattered across the globe hitting a web server simultaneously. But HuskHoard is not a web app. HuskHoard is an operating system level storage kernel.

When you double-click a file on your desktop, you expect it to open instantly. You do not expect your operating system to initiate a TCP handshake with a database server, authenticate a user, execute a query across a network socket, parse the JSON response, and then figure out where your video file is located.

The Physics of Filesystem Latency

Because HuskHoard uses fanotify to intercept file reads at the Linux kernel level, every microsecond matters. When the OS pauses an application to ask HuskHoard, "Where are the physical bytes for this stub?", we need an answer immediately.

Client-Server DB (PostgreSQL)

Network Dependency

✗Requires local loopback TCP/IP routing
✗Connection pooling overhead on every request
✗Requires running a separate background service daemon
✗Data passes through multiple kernel context switches
Embedded DB (SQLite)

In-Process Direct Access

✓Zero network routing; executes directly in RAM
✓No connection pools; just a file handle
✓Zero setup; it compiles directly into the Rust binary
✓Microsecond query latency

SQLite is an embedded database. It runs inside the same memory space as the HuskHoard Rust binary. There is no IPC (Inter-Process Communication). There is no network stack. A query is just a direct function call in memory to read a B-Tree from the local NVMe drive. For a filesystem driver, anything else is unacceptable overhead.

Write-Ahead Logging: Concurrent Magic

The most common (and outdated) criticism of SQLite is: "But the whole database locks when you write to it!"

This was true 15 years ago. It is no longer true today thanks to WAL (Write-Ahead Logging) mode. In the HuskHoard Rust architecture, we explicitly configure the SQLite engine on startup:

// From src/database.rs — This changes everything conn.execute_batch("PRAGMA journal_mode = WAL;")?;

WAL mode completely changes how SQLite handles concurrency. With WAL enabled, readers do not block writers, and writers do not block readers.

Why is this critical for HuskHoard? Because HuskHoard has two very different threads running at the same time:

Without WAL mode, the background archiver would lock the database, and the user's desktop would freeze when trying to open a file. With WAL mode, the Interceptor reads instantly from the main database file, while the Janitor seamlessly appends changes to the -wal file in the background. It is enterprise-grade concurrency in a single file.

The Myth of Scale

People often confuse the "size of their data" with the "size of their metadata".

Yes, HuskHoard is designed to manage 100 TB to Petabyte-scale tape archives. But the catalog doesn't hold the data. The catalog just holds the map. It stores strings (file paths), integers (offsets and sizes), and hex hashes (BLAKE3 checksums).

281 TB
Max SQLite Size

The theoretical maximum size of a single SQLite database is 281 Terabytes. Your archive catalog will likely never exceed a few gigabytes.

100M+
Row Capacity

SQLite can comfortably index and search tables with hundreds of millions of rows in milliseconds, provided you have proper B-Tree indexes (which HuskHoard generates automatically).

VACUUM
Atomic Backups

Backing up a running Postgres database requires specialized dump tools. Backing up HuskHoard's catalog is a single line of code that copies the DB into a safe, atomic state without halting the system.

A SQLite database tracking 10 million archived files is roughly 1.5 GB. Any modern NVMe drive can cache that entire B-Tree in RAM. It's incredibly fast.

Future Proofing: The Parquet Hook

Now, to be entirely fair to the data-engineering crowd: SQLite is an OLTP (Online Transaction Processing) database. It stores data in rows. If you want to do massive, archive-wide analytics—like "Show me the total bytes used by all `.mp4` files created between 2022 and 2024 grouped by the tape they live on"—row-based databases aren't the most efficient tool.

For data lakes and massive analytics, you want Columnar Data.

We know this. That’s why HuskHoard's architecture includes a post-commit trigger. While SQLite handles the "hot path" (the millisecond-sensitive reads required to keep your operating system happy), we built an event bridge for the big-data ecosystem.

The Sidecar Hook

Using the SidecarBridge over a Unix socket, HuskHoard broadcasts a JSON payload every time a file is archived, modified, or stubbed. Enterprise users can attach a lightweight listener to this socket to continuously transform the SQLite catalog into Apache Parquet files.

This gives you the ultimate hybrid system. Your Linux kernel gets its instantaneous, zero-latency, embedded SQLite lookups. Meanwhile, your data science team can query the exported Parquet files using AWS Athena, DuckDB, or Snowflake without ever putting read-pressure on your live storage server.

The Right Tool For The Job

SQLite is the most widely deployed database engine in the world. It runs the flight systems on Airbus planes. It runs inside every iPhone. It runs inside your web browser.

It is not a toy. It is a masterpiece of C engineering. By embracing it, HuskHoard remains a single, dependency-free Rust binary that you can deploy on any machine in seconds, rather than a fragile stack of containers. Engineering is about picking the right tool for the physics of your problem. And for localized, kernel-level file tracking, SQLite is undefeated.