nidus

nidus is a pure-Rust vector store with full-text search that runs anywhere Rust runs: in process as a library, behind nidus serve over HTTP, as an MCP server, or in a browser on wasm. Its bytes live on local disk or in object storage (S3, GCS), with an optional shared memory tier (Redis, Valkey). Hand it natural language and nidus embeds the text for you (optionally summarizing first) with the provider of your choice, or bring your own vectors: exact-by-default nearest-neighbour search (cosine, dot, or Euclidean), approximate (HNSW/IVF) when you opt in, with typed metadata filters and many logical collections sharing one embedding space, queryable through a typed API or a SQL-shaped syntax that compiles to it. No SQL engine, no query planner, no bundled C++ tree.

nidus (Latin, "nest"): a small place where things are kept safe.

Why it exists

nidus answers a query over your data: vector search (cosine, dot, or Euclidean, exact by default, approximate HNSW/IVF when you opt in), BM25 full-text search, and hybrid search that fuses the two by reciprocal rank fusion, all narrowed by typed metadata filters applied before scoring, and reachable through a typed API or a SQL-shaped read syntax that compiles to it. Ingestion, chunk some source, embed each chunk, store the vectors and metadata, is one way to get data in, not the point of the store. Either way, the obvious off-the-shelf options fail the build-and-ship test, not the functionality test:

• DuckDB (via libduckdb-sys) bundles a large C++ source tree and compiles it from scratch: a required C++ toolchain, a bloated binary, and FFI that can't run under Miri. A vector workload uses ~1% of it.
• LanceDB is "written in Rust" yet drags in Arrow + DataFusion (a full SQL engine) + a columnar format: hundreds of crates to do ORDER BY distance LIMIT k.

The workload is a vector store, not a database. nidus is that store, plus a memory layer built on top; --no-default-features gives the storage-and-search core alone, and either way it embeds as a normal Rust dependency.

See what that looks like end to end: codebase indexing, RAG over your documents, and agent memory.

The constraints are the product

The bar is build-and-ship speed, not zero-C absolutism. The enemy is the sprawling C/C++ tree (DuckDB) or hundred-crate graph (LanceDB), not a small, fast dependency.

• No bundled C++ tree: no bundled C++ source tree, no vendored OpenSSL, no aws-lc, no SQL engine, pure-Rust core. --no-default-features gives the storage-and-search core alone (four crates plus the backends), with ring (the TLS used by the S3/GCS backends and rediss://, a small C+asm compile) as the only native code; the Redis client itself is sync/pure-Rust (no tokio). The default build adds the CLI, server, MCP, memory, every embed/summarize/rerank provider, and code; the code feature's tree-sitter parsers are the only other native code either lane touches.
• Near-zero unsafe in our code (#![deny(unsafe_code)]). The one exception is the opt-in Config::mmap path: a single scoped mmap call for serving large stores from disk; every other unsafe is a hard compile error.
• Pure-Rust core: the local store and search path are pure Rust with no native library; the cloud backends are sans-IO clients (rusty-s3/tame-gcs) over a small blocking HTTP client.
• Miri-checkable: the lean library build (--no-default-features) runs all of nidus's own logic, including the local file IO, under Miri (only the network TLS paths are excluded).

Quick start

[dependencies]
nidus = "0.105"

use std::collections::BTreeMap;
use nidus::{Nidus, Config, Record, SearchOpts, Scope, Value, Filter, Predicate};

// Open (or create) a store. The directory is always the caller's choice;
// the dimension is pinned for the life of the store.
let mut db = Nidus::open(Config::new("/path/to/store", 4))?;
db.create_collection("code")?;

// Index some records: id + embedding + arbitrary typed metadata.
let mut attrs = BTreeMap::new();
attrs.insert("path".into(), Value::Str("src/auth/login.rs".into()));
db.upsert("code", &[Record::new("a", vec![1.0, 0.0, 0.0, 0.0], attrs)])?;

// Nearest neighbours (cosine), top-k.
let hits = db.search("code", &[1.0, 0.0, 0.0, 0.0], &SearchOpts { top_k: 5, ..Default::default() })?;
for h in &hits {
    println!("{:.3}  [{}] {}", h.score, h.collection, h.id);
}

// Search the whole store at once, with a metadata filter + score floor.
let opts = SearchOpts {
    top_k: 10,
    filter: Filter(vec![Predicate::Glob("path".into(), "src/auth/*".into())]),
    min_score: Some(0.5),
    ..Default::default()
};
let hits = db.search(Scope::All, &[1.0, 0.0, 0.0, 0.0], &opts)?;
# anyhow::Ok(())

See examples/demo.rs for an end-to-end run (cargo run --example demo).

What it does

• Remember text, recall the relevant bits: hand nidus natural language and it embeds it for you, optionally summarizing first, with a provider of your choice (Voyage, OpenAI, Ollama, Cohere, Gemini, Mistral, Jina, or any OpenAI-compatible endpoint), then answers queries by similarity. The raw Vec<f32> API is untouched: bring your own vectors and skip it entirely. --no-default-features gives the storage-and-search core alone, without this layer. See the remember & recall guide.
• Exact or approximate search: exact by default (100% recall; scan cost scales with the rows scanned). Score by cosine, dot, or Euclidean (cosine the default; cosine vectors are unit-normalized on insert, so a score is plain similarity in [-1, 1]). Opt into an approximate index (HNSW or IVF) or int8 quantization to trade some recall for speed at larger scale.
• Scoped search: query one collection, a subset, or the whole store in one call, merged into a single ranking. Sound because every collection shares one embedding space (one pinned dimension).
• Typed metadata + filters: attach Str/Int/Float/Bool/List/DateTime/ Null attributes and narrow results before they score: equality and sets, ranges, globs, list containment, fuzzy/token/phrase/regex text matching, and All/Any/Not boolean composition over any of them.
• Rank and shape the answer: layer a recency decay over the store's metric, weight the legs of a hybrid query, ORDER BY an attribute, cap hits per attribute value, project which attrs come back, paginate a documented total ordering, or ask a hit to explain itself with per-clause scores and highlighted fragments.
• Idempotent upserts by caller-supplied id; delete, delete_where, per- collection metadata.
• Crash-safe & durable: an append-only flat-f32 data segment plus a framed, CRC-checked op log (the commit record). A crash loses at most the in-flight batch; a torn tail is recovered on open. Cross-process readers get a consistent, lock-free snapshot (OpenMode::ReadOnly).
• Synchronous, runtime-agnostic: the hot path is CPU-bound, so there's no async core to lock you into a runtime. Arc<RwLock<Nidus>> gives concurrent searchers + one writer; async callers bridge with spawn_blocking.
• Runs in the browser: nidus compiles for wasm32-unknown-unknown, storing its data in the browser's Origin Private File System (opfs://) from a dedicated worker, no server round trip. See the browser guide.
• Code search: nidus code ingest/nidus code search chunk a repository per file (AST-aware for source, heading-aware for docs) into one corpus, and answer queries grouped by file and symbol, never a source body. Part of the default build (the code feature); --no-default-features gives the storage-and-search core alone, without tree-sitter. See the code search guide.

Command line & server

The same crate ships a nidus binary: a CLI for working with a store directly, and nidus serve, an HTTP server exposing the full store (create, upsert, search, inspect, maintain) over JSON. It is part of the default build, so cargo install nidus produces the whole binary; a library dependency that wants the storage-and-search core alone (no binary, no async stack) adds nidus with --no-default-features.

# Install: no Rust toolchain needed (prebuilt binary for your platform)
curl -fsSL https://raw.githubusercontent.com/duckedup/nidus/main/install.sh | sh
# …or, with cargo: `cargo binstall nidus` / `cargo install nidus`

# Use it on a store directory (records/queries are JSON). --dim is pinned at
# creation, then inferred from the store, so later commands don't repeat it.
nidus create  --dir ./store --dim 3 docs
echo '[{"id":"a","vector":[1,0,0],"attrs":{}}]' | nidus upsert --dir ./store docs
echo '[1,0,0]' | nidus search --dir ./store docs -k 5

# Snapshot the whole store to one portable .tar.gz (safe while a writer runs),
# and restore it, handy before an upgrade or as a cron job.
nidus backup  --dir ./store --out ./store.tar.gz
nidus restore --in ./store.tar.gz --dir ./restored

Or drive the same store over the network (no Rust toolchain on the client, just HTTP and JSON):

nidus serve --dir ./store --dim 3 --addr 127.0.0.1:7700

curl -s -X POST localhost:7700/collections/docs
curl -s localhost:7700/collections/docs/upsert -H 'content-type: application/json' \
  -d '{"records": [{"id": "a", "vector": [1,0,0], "attrs": {}}]}'
curl -s localhost:7700/search -H 'content-type: application/json' \
  -d '{"query": [1,0,0], "top_k": 5}'

The server shares the library's storage model, durability, and search semantics. See the command-line and HTTP server & API guides.

Performance

Every vector store ships a benchmark proving it's the fastest, on synthetic data that looks nothing like your workload. It's a genre. Here's ours, and yes, we win our own benchmark, that's how this works.

Exact brute-force cosine KNN, 100k vectors, single thread, measured against DuckDB (array_cosine_similarity) and LanceDB (bypass_vector_index), both pinned to the same exact search, so all three return the same neighbours. The harness computes its own independent ground truth and reports recall@k for every engine (including nidus), so none is trusted as the oracle. Numbers are query p50; lower is better.

n=100k	top_k	nidus	LanceDB	DuckDB	recall
dim=384	10	5.44 ms	12.29 ms	32.29 ms	100%
dim=384	100	5.53 ms	28.52 ms	30.59 ms	100%
dim=768	10	8.09 ms	24.78 ms	69.54 ms	100%
dim=768	100	8.57 ms	53.16 ms	64.99 ms	100%

All three are exact (recall 100%); nidus is the fastest in every cell while being the one with no bundled C++ tree. The kernel is plain safe Rust: an 8-lane chunked dot the optimizer can vectorize, an allocation-free top-k scan, and a storage-order (prefetcher-friendly) sweep of the matrix. Reproduce with just bench all (see benchmarks/; the heavy DuckDB/LanceDB deps are quarantined off nidus's own build path). Synthetic data on an Apple Silicon laptop: useless, like all benchmarks, but there it is.

On-disk layout

A store is a directory:

<dir>/
  data    append-only, fixed-stride, row-major f32 matrix (header pins dimension)
  log     append-only framed op stream: [len][bincode(Op)][crc32] (the commit record)
  lock    O_EXCL writer-exclusion lock file

open reads data into RAM and replays log into an in-RAM index (collection → { id → (row, attrs) }). Search never touches disk.

Configuration

use std::time::Duration;
use nidus::{Config, Fsync, OpenMode};

let cfg = Config::new("/path/to/store", 768)
    .fsync(Fsync::PerBatch)          // durability granularity (default)
    .open_mode(OpenMode::ReadWrite)  // ReadOnly = no lock, search-only
    .auto_compact(Some(0.5))         // compact on open above this dead-row ratio
    .lock_ttl(Duration::from_secs(60));

The store location is always the caller's choice: nidus contributes no path defaults, env vars, or hidden directories.

Development

just test     # all tests, lean library build
just ci       # fmt-check + clippy (-D warnings) + test, lean library build
just miri     # undefined-behavior check (nightly; the lean library build's own logic)
just demo     # the end-to-end example
just deps     # the dependency tree, default build (`just deps-lean` for the lean one)

just ci-cli   # the same gate, isolated to the `cli` feature (binary + server)
just serve ./store 768   # run `nidus serve` from the checkout

just test/just ci run the lean library build (--no-default-features): the storage-and-search core alone, which keeps the inner loop fast. The default build, which is what cargo install nidus ships, adds CLI, server, MCP, memory, every embed/summarize/rerank provider, and code; just ci-serve is the gate for it. Lanes that isolate one slice, like just ci-cli, pair --no-default-features with that slice's feature so a failure there points at one crate, not the whole tree. Miri runs against the lean library build: all of nidus's own logic, including the local file IO and the in-RAM object-store/memory-tier paths; only the network paths (S3/GCS TLS, the Redis socket) and the opt-in mmap syscall are outside its reach.

Rust 1.98+ (pinned via rust-toolchain.toml), edition 2024.

Design

The full design, covering the data model, on-disk format, durability/concurrency model, the opt-in modes (approximate ANN/HNSW + IVF, scalar/binary quantization, memory-mapped larger-than-RAM stores), and the remaining deferred seams, lives in SPEC.md. Each module also carries its own contract in src/<module>/SPEC.md.

License

MIT