Start

Concepts

The handful of ideas that explain everything else. These are roots and the index, how steward stays current without inotify, what offline means, classification, content ids, and the observations and events built on them.

Roots and the index

A root is a directory tree you ask steward to index, together with its policy: how often to rescan it, what to exclude, whether to classify it, which of its folders get content ids. All roots share one index and one namespace of absolute paths, so a query about /home/media/TV and one about ~/src go to the same place. If roots are nested, a path belongs to the deepest root that contains it.

For every path under a root, the index keeps what lstat reports:

Each directory also carries subtree totals: total size, total space on disk, and the number of files, directories and items beneath it. Items counts every entry: files, directories, symlinks, sockets and the rest. It is what uses up a filesystem’s inodes (or, on btrfs, its metadata space), so ranking by items finds where millions of small files pile up. They are kept current as the tree changes, so “how big is this folder” is a lookup, not a walk. A file with several hard links contributes its share of the bytes to each directory that holds a link, so totals don’t count the same bytes twice. Items count names, so each link counts once: like du --inodes -l, not plain du --inodes, which counts a shared inode only once.

The index is a single database file in SQLite format, by default ~/.local/state/steward/index.db. Its size grows with the number of paths, not the size of the files: about 130 bytes per path.

Staying current without inotify

steward does not watch the filesystem with inotify. Watching tens of millions of paths needs one kernel watch per directory, raised system limits, and a full rescan after every overflow or restart anyway. Instead, steward keeps up in four ways:

Periodic rescans

Each root is rescanned on its own schedule, once a day by default (interval_minutes). Writes are limited to what changed.

Trusting rescans

A directory whose modification and change times match the index has the same names in it as last time, so a trusting rescan does not read it again. It only descends into its subdirectories. This is the trick updatedb uses, and it makes a rescan of an unchanged /usr take milliseconds. It cannot see a file rewritten in place (the directory’s times don’t change), so every full_everyth rescan stats every file. With the default full_every = 1, every scheduled rescan is full.

Invalidation

An application that changed files can say so (invalidate). steward waits two seconds to gather related changes, then rescans the nearest directory that still exists.

Inspection

An application that needs the answer now asks steward to inspect specific paths. They are rescanned and, for files, hashed before the call returns.

A directory modified within a second of a scan is stored as “untrusted”, so the next trusting scan reads it again. A change landing in the same clock tick as the scan cannot be missed for good.

Filesystems and offline volumes

steward records each path’s filesystem id, which Linux derives from the filesystem’s UUID (and the subvolume, on btrfs). It does not use the device number, which can change between boots.

When a scan finds a known directory on a different filesystem than it was indexed on, that directory’s volume isn’t mounted: the scan sees the empty mount point instead. steward then reports the directory as offline, leaves everything under it exactly as indexed, and does not read or change it. Unplugging a disk is not mistaken for deleting everything on it, and plugging it back in brings everything back, still identified.

Offline status shows up in settings, in resolve results, in storage.offline and storage.online events, and in the desktop app.

By default (one_filesystem = true) a root does not descend into other filesystems mounted beneath it. Make another root for each one you want indexed.

Classification

Classification records what things are, using names and a little context and never contents. It runs after each scan of a root that has classify = true.

Directory classes are tags on the topmost directory they apply to, and every path beneath inherits them. They are prefixed classify:.

Names that would be ambiguous alone need a neighbour to confirm them: a folder called build is only build output if a build file sits next to it.

File categories come from the file name alone and are computed when asked, not stored: image, video, audio, document, source, archive, object, disk-image, torrent.

Content ids

A content id names a file’s bytes, not its path. steward uses the BitTorrent v2 format (BEP 52) and writes it as btv2: followed by 64 hex digits:

btv2:1d8e17666fc6c23760eb2d8ec0883756d1a0dd34b6cf71292c6f37b38567434f

It is the root of a SHA-256 Merkle tree over the file’s 16 KiB blocks: the same value BEP 52 calls the file’s pieces root. Any software that reads v2 torrents, or computes the same tree, gets the same id for the same bytes, and the id doesn’t depend on a torrent’s piece size. Empty files have no content id, as in BEP 52.

When an id is computed

• In the background, for every file under a root’s contentid folders, after each scan of that root.
• On demand, for any file, through inspect, content_id or verify, or hash_tree for a whole folder.

Hashing runs on its own thread pool (hash_threads, at most four by default) so it never slows scans, and saves its progress as it goes.

When an id stays and when it goes

An id is stored per inode, with the size and modification time it was computed under. It is reported only while both still match.

• Renames and moves keep it. They change neither size nor modification time, so the id moves with the file, and a scan that finds the file at its new path links it at once without reading it again. A move onto another filesystem creates a new inode, which has to be read again before it has an id (the same one), either by its folder’s policy or because an application asks.
• Hard links share it. All paths to one inode show the same id.
• Writes drop it. Any write changes the modification time, so the old id stops being reported, and the file is hashed again.
• One gap: bytes changed while size and modification time were restored (touch -d, some sync tools) go unnoticed until something reads the file. verify exists for exactly this. It rereads the file and records what it really holds.

The verification layer

Alongside each id of a file over 1 MiB, steward keeps the hash of every 1 MiB section of the file: 32 bytes per MiB, under 1 GB for a 28 TiB library. From it steward derives the tree’s layer for any power-of-two piece size of 1 MiB or more (piece_layer) without reading the file again. That is the same data a v2 torrent stores in its piece layers field.

Observations

steward’s catalog is a set of observations: “content X was seen at path P on filesystem F, inode I”. resolve turns a content id into its current observations and a state:

Copies with the same id are byte-identical, so “which copy” is a choice of convenience (reachable, nearby, on a fast disk), not of correctness.

Events

Applications can subscribe to changes in the catalog instead of polling:

Events are numbered, and the daemon keeps the most recent 4,096, so a client that reconnects can resume where it stopped. When that isn’t possible (the daemon restarted, or the client fell too far behind) the client is told so explicitly and should re-resolve what it tracks. Without inotify, an event arrives when steward notices the change: at the next scan, invalidation, inspection or verification.

Two sockets

steward listens on two Unix sockets in $XDG_RUNTIME_DIR/steward/, and the split is by capability:

• content.socket is for applications: catalog lookups, the content primitives (resolve, inspect, verify, piece_layer), and events. Nothing on it changes what is indexed or how.
• api.socket is administration: everything above, plus adding and removing roots, forcing scans, classification and hashing jobs, settings, and qdirstat exports.

The command line tool and the desktop app use api.socket. Most applications only need content.socket, and that is the one to grant to a sandboxed or less trusted program.