dig

A searchable index of The Nibble - the timeless bits (tools, TILs, reads, quotes) pulled out of every edition of the newsletter, with the week's news left behind. Answers "when did we first and last talk about X" for any tag or search term.

Live at https://dig.nibbles.dev

Hosting

index.html is a single self-contained static file - the full index is inlined, all logic is vanilla JS, no build step or backend at runtime. It's served by GitHub Pages (see CNAME; .nojekyll disables Jekyll so models/ and vendor/ are served as-is). To ship: commit and push.

Even the optional smart-search model is vendored into the repo (models/ and vendor/transformers/), so the search stack pulls nothing from a third-party CDN at runtime. The two webfonts are the one exception - they still come from Google Fonts.

Regenerating the index

The data is produced from a Substack export by a heuristic parser (build-index.py) - deterministic regex, best-effort tagging. It is not a canonical system of record; a future canonical parser can overwrite the inlined data wholesale.

# 1. Substack -> Settings -> Exports -> download, unzip to ~/Downloads/nibble-archive/
#    (expects posts/ of {id}.{slug}.html + posts.csv)
# 2. inline the fresh index straight into index.html:
python3 build-index.py

# optional: also write the raw index as JSON
python3 build-index.py --json index.json

build-index.py rewrites the const BOOTSTRAP = ... line in index.html in place, so the page stays self-contained.

If the data changed, also regenerate the precomputed smart-search vectors (needs onnxruntime, tokenizers, numpy):

python3 build-vectors.py   # embeds the corpus with the vendored model -> vectors.f32

What it does / doesn't

• Editions #1-#100. News is excluded (temporal); tools, curiosity, TILs, reads and quotes are kept. Unknown section headings are parked in Curiosity and reported, never dropped.
• First/last trace works for any tag chip and any free-text query.
• URL reflects state (#q=..., #tag=...), so a first/last view is a shareable link.

Descriptions and edition links

Each entry links to its source, and its edition number links back to that edition on Substack (nibbles.dev/p/<slug>). This is edition-level, since Substack has no reliable per-line anchor.

Descriptions are extracted as the text after the first link, so build-index.py tidies them deterministically (strips stray leading punctuation and a dangling "and"/"but", capitalizes, adds a full stop). For an extra polish pass, an opencode sub-agent can rewrite them:

python3 build-index.py                          # also writes descriptions.todo.json
opencode run "$(cat rewrite-descriptions.md)"   # writes descriptions.json
python3 build-index.py                          # merges + re-inlines into index.html

The rewritten text goes to a separate descriptionClean field and the page prefers it; the deterministic description (ground truth) is never overwritten. descriptions.json is committed; descriptions.todo.json is an intermediate.

How search works

There are two search modes. The fast one is always on; the smart one is on by default and can be toggled off.

1. Keyword + synonym (default, instant, no download)

Token/prefix matching over each entry's title, description, heading, domain and tags, plus a hand-written concept-synonym layer. Typing rag expands to vector, embedding, langchain, etc., so it reaches those tools even when an entry never says "rag". Matching is token/prefix based, not raw substring, so rag does not match "leverage". Fully deterministic and offline.

2. Smart search (on by default, in-browser semantic / "RAG retrieval")

This is the retrieval half of a RAG pipeline - semantic search, no generation, no server. An embedding model runs in the browser via WebAssembly (ONNX Runtime through transformers.js). All of the heavy work happens in a Web Worker, so the page never freezes.

flowchart TD
  Q[User types a query] --> M{smart search on?}
  M -- no --> K["keyword + synonym match<br/>(instant, deterministic)"]
  M -- yes --> W[Web Worker]
  subgraph bg [Web Worker - background thread]
    W --> L["load all-MiniLM-L6-v2<br/>~23 MB, vendored in repo"]
    L --> E["embed 1862 entries<br/>(batched)"]
    W --> QE[embed the query]
  end
  E --> CACHE[(IndexedDB<br/>vector cache)]
  CACHE --> S[cosine similarity]
  QE --> S
  S --> R[ranked results + first/last trace]

Loading

Step by step:

1. Toggle on. The worker imports the vendored transformers.js and loads all-MiniLM-L6-v2 (quantized, ~23 MB) from the repo (vendor/ + models/), then the browser caches it. Status line: "Setting up smart search…". It warms up in the background on load; keyword search answers meanwhile.
2. Load the corpus vectors. The 1,862 doc vectors are precomputed at build time (build-vectors.py) and shipped as vectors.f32, so the browser just fetches them - no first-load embedding. They are cached in IndexedDB keyed to the corpus. If vectors.f32 is missing or stale, the worker falls back to embedding the corpus in-browser (reporting "Setting up smart search… N/1862").
3. Query. Each query is embedded by the same model (in the worker). The main thread computes cosine similarity against the cached vectors (normalized, so it's a dot product), then fuses that ranking with the keyword one (see below).
4. Fallback. While the model warms up, keyword search still answers. If the model fails to load, smart search turns itself off and keyword search remains.

3. How the two are combined

Smart search does not replace keyword search; the two rankings are fused with reciprocal rank fusion, and every entry the keyword matcher accepts is kept regardless of its cosine, so a semantic near-miss can never bury an exact match. A literal hit always outranks a synonym one - searching claude puts Anthropic entries above the OpenAI and Gemini entries that share its synonym group - and terms are weighted by inverse document frequency, so a common word in a long question counts for less than a distinctive one.

When there are no results. A cosine floor cannot tell a real query from a keyboard mash: python3 build-vectors.py --probe scores 30 gibberish strings against the corpus and their top match reaches 0.51, higher than the weakest of 20 genuine queries. So nonsense is caught the only way that works - a query none of whose words appear anywhere in the archive returns nothing, rather than the twenty least-dissimilar rows. The same applies to a real word the archive has never covered, which is the honest answer too.

Results are paginated, 20 to a page (or 50 or 100). The page is part of the URL alongside the query, so any page of any search is a link.

Cost / tradeoffs. One-time ~23 MB model download (then cached). First-query latency is the model load plus a one-time corpus embed; instant after that. The search stack has no third-party runtime dependency - the library, WASM runtime and model are all served from dig.nibbles.dev (they add ~42 MB to the repo). Only the webfonts are still fetched from Google.