Plan: four structural fixes — graph DSL tokens, per-row sequences, timestamp units, fused BBQ

[EnRaiha]

Summary

Four open issues share one shape: a contained fix would leave the failure
class alive. This post states the approach for each, its acceptance
criteria, and the order we propose. Feedback wanted before any code lands.

#296 — graph DSL ignores unconsumed tokens

Current: a mistyped keyword (GRAPH TRAVERSE FROM 1 DEPTS 3 IN g) parses
as "not that clause", so DEPTH defaults and the query runs. Value
vocabulary is already validated; keyword-level gaps stay silent.

Approach: track token consumption during the parse, not after it. A
cursor holds the token slice and a used-set; each clause reader marks the
spans it consumes. After the variant parser returns, any unconsumed word
token is an error carrying its position and text.

Steps

• add a cursor type in ddl_ast/graph_parse, move helpers to methods
• mark keyword + value + object spans in every helper and variant
• reject leftover tokens with the token text and offset
• one test per clause (typo keyword, stray word, correct forms)

Acceptance: DEPTS 3 errors as unexpected token 'DEPTS'; every
existing valid form parses unchanged.

Risk: typo'd statements start failing. That is the intent, but it may
surface latent typos in user queries.

#314 — per-row sequence accessors

Current: SELECT id, nextval('s') FROM t raises 0A000. Row scope has
no sequence state; the refusal replaced a silent NULL. Constant contexts
(from-less SELECT, VALUES cells) already evaluate at plan time.

Approach: make the allocation a plan value. The control plane allocates
a range, the plan carries it, the data plane stamps one value per row in
output order — the same slot can serve RETURNING. Row counts unknown at
plan time get lazy windowed allocation. currval follows session
semantics; gaps under retry are acceptable (PostgreSQL accepts them).

Acceptance: SELECT id, nextval('s') FROM t returns one fresh value per
row, in order; RETURNING nextval(...) matches; unknown row counts work.

Open question: is per-row support wanted, or does 0A000 stay?

#317 — timestamp units per path

Current: four per-path millisecond/microsecond converters. Recent fixes
covered the read path, joins that scan locally, and renames. The contract
lives in review, not in the type system: a new path can repeat the bug, and
a computed expression over an instant can still differ between a join and a
direct scan.

Approach: one type at one boundary. A Millis newtype crosses every
internal boundary; convert once at ingest and once at render. Joins, sorts,
and aggregations consume Millis only, so the join-predicate rule
(comparisons are milliseconds) becomes a compile error instead of a
convention. Converters are deleted, not added.

Acceptance: no raw i64 timestamps cross module boundaries; the
computed-expression divergence and time_bucket over joins get tests;
existing timestamp tests pass unchanged.

Size: broad but mechanical; needs a migration order to avoid a
big-bang.

#280 — fused BBQ kernels

Current: bbq_dequantize materializes an f32 vector per hit, then
computes distance. Scalar; the rerank oversample multiplies the cost.

Approach: fuse dequantize into the distance so the vector is never
materialized — bbq_distance(packed, residual_norm, centered_query) -> f32,
sign expansion in registers, one pass, no allocation. Tiered kernels
(scalar / AVX2 / AVX-512 / NEON) behind runtime feature detection, a
portable reference for tests, property tests for parity, benchmarks to
justify each tier.

Acceptance: result parity with the reference within tolerance; measured
speedup on the rerank path; no allocation per hit.

Order and dependencies

1. Timestamp unit conversion is per-path, so a join or a cross-node scan compares mismatched units #317 first: correctness with a crisp boundary, blocks nothing.
2. Graph DSL ignores unconsumed tokens — a typo'd clause keyword silently runs the statement with default semantics #296 second: self-contained to the parser module.
3. Evaluate sequence accessors per row in SELECT-list contexts (0A000 today) #314 third: reuses the per-row stamping machinery Timestamp unit conversion is per-path, so a join or a cross-node scan compares mismatched units #317 settles.
4. Fused BBQ dequantize-distance kernels with tiered SIMD dispatch #280 last: pure performance; schedule when rerank is on the hot path.

What we want from reviewers

• agree the token cursor is the right shape for Graph DSL ignores unconsumed tokens — a typo'd clause keyword silently runs the statement with default semantics #296
• decide whether per-row nextval is supported or stays refused
• agree on Millis as the single boundary type for Timestamp unit conversion is per-path, so a join or a cross-node scan compares mismatched units #317
• ranking of Fused BBQ dequantize-distance kernels with tiered SIMD dispatch #280 against other performance work