Index, label and browse an audio archive you no longer remember making.
Built for the case where the folder structure is a mess, the filenames are lies, and you can't search because you can't remember what to search for. It scores every file against a vocabulary you control, groups by sound, and renders the whole archive as a wall of spectrograms you can skim by eye and by ear.
Nothing in your archive is ever moved, renamed, modified or deleted. Every
byte sift writes goes to ~/.sift.
Tested on Apple silicon (M1–M4), macOS 13+. Needs Python 3.10 or newer.
./setup.sh # venv + dependencies + CLAP checkpoint (~2.5 GB total)
./setup.sh --no-clap # skip the model download and run on dev embeddingsbrew install ffmpeg if your archive contains mp3, m4a or opus. WAV, AIFF, FLAC
and OGG work without it.
./run.sh scan "/Volumes/Archive/everything" # index (the slow part, run it overnight)
./run.sh classify # score against the current axes
./run.sh cluster --min-size 5 # group by sound
./run.sh serve # → http://127.0.0.1:7373Or do all of it from the interface: the Index tab takes a folder path and runs the same pipeline with a progress bar.
Two axes, not a deep tree. Every file is scored against two independent
vocabularies — behaviour (drone, texture, impact, rhythm…) and character
(metallic, granular, dusty, hollow…). Crossing them gives drone / metallic,
impact / wooden, and so on. A file is filed under its top two matches per
axis rather than forced into one home, which roughly doubles the chance you
stumble onto it.
The vocabulary is yours and it's cheap to change. Edit the terms in the Axes tab and hit Save and re-score. Scoring runs against cached vectors, so re-cutting the entire taxonomy takes seconds, not another full pass. This is the point of never moving the files: the first vocabulary will be wrong, and you'll want to replace it once you've heard what's actually in there.
Clusters answer what the vocabulary can't. HDBSCAN over the audio embeddings finds the shapes this particular archive contains, with no vocabulary involved. Names are provisional (assembled from assigned terms and recurring folder words) and meant to be overwritten — listen to a cluster, rename it in the inspector.
Skim, don't read. Skim bucket stitches 1.5 seconds of every file in the current view into one clip, level-matched. Forty files in a minute is how you re-learn your own archive; it's worth more than any labelling accuracy improvement.
Every row is its own spectrogram. Drones are horizontal bands, impacts are
vertical spikes, granular material is speckle. Click to play; a playhead sweeps
the spectrogram. j/k move down and up the list, space pauses.
| Stage | What happens |
|---|---|
| Walk | Every audio file under the root, symlinks not followed |
| Decode | 48 kHz mono, capped at 10 minutes per file |
| Hash | blake2b over decoded PCM, so the same drone as 24-bit WAV and 16-bit AIFF collapses to one entry |
| Features | Duration, loudness, crest, onset rate, harmonic/percussive split, spectral centroid drift, flatness, pitch and its stability, tempo, stereo width |
| Embed | CLAP, 512-d. Files over 30 s are cut into 10 s windows; the file vector is the mean, and the windows are kept for segment-level work |
| Score | Cosine against each term prompt, softmax per axis, top 2 kept, weak matches routed to _unsorted |
| Cluster | HDBSCAN on the unit-norm vectors (falls back to k-means) |
| Materialise | Optional symlink tree under ~/.sift/views |
clap — laion/clap-htsat-unfused
through HuggingFace transformers, which is far less brittle to install than the
laion_clap package and does the same job. Audio and text land in one space, so
prompts and sounds are directly comparable. Runs on MPS.
dev — no torch required. Projects a mel summary through a fixed random matrix.
Clustering and nearest-neighbour work properly; term scores are meaningless
and the interface marks the backend in red when you're on it.
Speed. Roughly 0.5 s per file for decode plus features on a synthetic set,
plus CLAP inference. Reckon on a few hours for tens of thousands of files. Run
scan with --limit 500 first to see what you're in for. Re-running scan on
the same folder skips anything whose mtime hasn't changed, so it's resumable —
just run it again if you interrupt it.
The bottleneck is the harmonic/percussive split, which is single-threaded. If
you want it faster, the change is to wrap decode + features in a
ProcessPoolExecutor in scan.scan and keep the SQLite writes on the main
thread; it wasn't worth the complexity for an overnight job.
Accuracy. CLAP was trained on captions describing recognisable sources and events — dog barking, church bell, acoustic guitar. Abstract synthesis is exactly the material those captions don't cover. Expect the behaviour axis (drone / impact / rhythm) to work noticeably better than the character axis, and expect timbral adjectives like granular or dusty to be weakly grounded.
The scores are not calibrated probabilities. A softmax over your twelve
adjectives is only meaningful within that comparison — you can't compare a
score on one axis to a score on the other, and the MIN_SCORE threshold in
config.py is a knob to tune by looking at _unsorted, not a confidence level.
Nearest-neighbour by audio never touches the text tower, which is why it's the most reliable thing in here.
sift/config.py paths, thresholds, the default vocabulary
sift/db.py SQLite schema — the only place metadata lives
sift/audio.py decode, content hash, DSP features
sift/embed.py CLAP and dev backends, windowing
sift/scan.py ingest
sift/classify.py axis scoring, text search, nearest neighbours
sift/cluster.py HDBSCAN + provisional naming
sift/render.py spectrogram PNGs, bucket montages
sift/views.py symlink tree
sift/server.py local HTTP API (binds 127.0.0.1 only)
web/ interface, no build step
tools/ synthetic test archive generator
GET /api/export dumps the whole index as JSON if you want to do something else
with it.
MIT — see LICENSE. Third-party components and model weights are listed in NOTICE.md; all are permissive, and none are bundled here.
- The server binds to localhost only. It serves audio from anywhere on your disk by design, so don't expose it.
~/.sift/index.dbis the entire state. Delete it to start over; the archive is untouched either way.- Duplicates are detected but never removed — they're flagged in the index and hidden from browsing unless you ask for them.
- Files over 60 s are filed as
pieceand kept out of the sample buckets, since a handful of long bounces will otherwise dominate every category they land in.