fix: complete quickstart search, chunking, and retry contracts

[ajroetker]

The real local quickstart exposed correctness gaps: default dynamic strings were missing from _all, empty separator sections could bypass chunk limits, and generation-capacity errors lost their public retry contract.

Include unmapped dynamic strings in _all, matching the schemaless default, and record that inclusion in observed metadata. Keep exact .keyword copies excluded and preserve explicit mappings. Default omitted text-query fields to _all in both direct and generated-schema parser paths, as allowed by the public schemas. Preserve explicit field selectors. The real Gemma plan submitted a legal fieldless match query that previously received UnsupportedQueryRequest; its query-string fallback found no evidence. Guide query planning toward concise subject terms while retaining normal _all search. There is no separate configurable query-default field in the current Zig full-text index configuration.

Preserve query-string AND semantics when converting boolean filters into scored text queries. Internal filters ignore their optional-clause minimum when the optional list is empty; the scorer correctly rejects a positive minimum over an empty list. Normalize that minimum to zero during conversion so implicit AND, explicit AND, repeated clauses and nested negation retain their meaning. Keep match OR semantics and actual disjunction minimums unchanged.

Skip empty and tokenizer-empty sections, including whitespace-only sections, and check span overflow whenever a current section exists. This prevents ,alpha from exceeding a one-token limit and alpha,beta, from emitting an empty terminal chunk. Independently re-tokenize source windows because continuation pieces can expand when encoded alone (for example, ord from unaffordable). Recursively split oversized windows with guaranteed byte progress and preserve validated counts for the assembly loop. When token offsets are unavailable or invalid, partition contiguous UTF-8 source spans instead of searching normalized decoded text; this preserves leading text in cases such as HELLOhelloЖ.

Use the existing inference-capacity envelope for retrieval JSON, query-builder JSON, and live/buffered SSE: error, message, reason, retryable, and retry_after_ms. HTTP responses also include Retry-After. Query-builder document-identity and transient-embedding failures use the existing typed query-unavailability response instead of minimal JSON or plain text. All three explicitly mapped query-builder 503 variants now conform to the declared union and carry retry headers. Declare agent 503 variants and SSE retry fields in OpenAPI, and regenerate Zig, Go, TypeScript, and Python clients. TypeScript agent HTTP calls throw the existing InferenceCapacityError; SSE delivers it through onErrorDetail while preserving the legacy string onError callback. Streaming chatAgent().messages rejects on terminal SSE errors, premature EOF, and abort, preserving typed capacity failures and forwarding caller callbacks. Completion settles the promise and removes the abort listener; a later abort cannot invalidate a completed turn. Completion also preserves the terminal generation and server-provided history for JSON fallback, done-only SSE, and streams whose final answer differs from partial deltas.

Validation:

• All 48 document-mapper tests pass. The strengthened default _all regression fails before the fix.
• All 16 chunker tests pass. Whitespace-only sections are explicitly covered. Custom-separator edge cases, continuation windows, and mixed-case/Unicode fallback source coverage are covered at token targets 1–3. The regressions reproduced before their fixes; the removed decoded-search helper test is replaced by full source-preservation coverage.
• 12 Zig HTTP/SSE tests pass, including the generated public capacity type and both SSE delivery modes.
• TypeScript: 279 tests pass, 1 skipped; type-checking and lint pass. Chat lifecycle coverage includes capacity and generic SSE errors, EOF, abort with empty callbacks, successful completion and later abort.
• Go SDK suite passes, including validation against both published agent 503 schemas, SSE schema, and generated response parsers.
• Python: 63 client tests pass.
• Full make generate, make fmt, and git diff --check pass. The embedded Antfarm bundle was rebuilt after the SDK changes and is current (index-CzrJXyaN.js).
• Added six portable quickstart E2E cases for unfielded query strings, match (including analyzer options), term and prefix across dynamic title/body fields, explicit narrowing, return projection, and evidence-bearing RAG SSE with exactly one terminal completion. Against the prior binary, query-string search passes and the other five regressions fail as expected. All six pass against 0c0a2eda8.
• Omitted-field parser regression passes across 12 text query forms. The broader query-contract run has 124 passing tests and two failures (hierarchy controls and filter-depth validation); both failures reproduce on the preceding commit without this parser change.
• The preceding b6dd12a9a ReleaseFast + Metal build passed 27/27 steps. Its fresh-data quickstart passed through image search but RAG completed with zero hits and an apology, exposing the omitted-field parser gap above. The exact 0c0a2eda8 ReleaseFast + Metal build passed 27/27 steps (20m41s), and the unchanged fresh-data local CLI/cURL quickstart passed end to end at the documented 8192 MiB budget. Both real Gemma RAG commands returned five evidence hits, nonempty generation, exactly one terminal completion and no SSE error (CLI 22.626s, cURL 19.770s). Optional query planning passed; the missing-generator command correctly exited 1. Both standalone processes stopped with exit 0.
• This final run used 10,000 articles and cached models cloned from the earlier cold-store validation. Text readiness took 86.331s (1,000 sources, 2,120 vectors); image readiness took 132.879s (250 artifacts). No scratch admission denial appeared. Image worker watchdog recovery and shutdown cancellation remain visible in retained logs. These are observed workflow timings on an M4 Max, not controlled benchmarks.
• Retrieval relevance remains mixed on partial indexes. The RAG answer summarizes the Korean-history textbook controversy and acknowledges that the evidence is insufficient for a comprehensive history overview. This is an execution/evidence smoke-test pass, not quantitative relevance qualification.
• Follow-up review identified and fixed the two omitted query-builder 503 variants, the pending chat messages promise on terminal failure, and whitespace-only chunk artifacts. The new chat, chunker and server-response regressions fail before their fixes and pass afterward. Both generated Python agent parsers accept all three 503 variants (six new cases); Go validates the schemas and generated response types; TypeScript preserves the typed dependency errors. Full make generate and make fmt pass again.
• Fixed the stale query-builder document-identity HTTP test that failed Zig CI on 7fee6e39b: it now parses the published retry schema and checks a positive Retry-After. Repaired surrounding query-builder tests to parse raw query JSON before inspecting fields. All 113 selected query-builder and capacity-response tests pass with zero leaks; Zig formatting and diff checks pass. This follow-up (90b90b365) changes tests only.
• Added twelve query-string/structured boolean controls before and after reopen to the HTTP storage regression and quickstart E2E suite. Both the retained-binary E2E case and the new HTTP regression reproduce the pre-fix AND failure. The 10da5e326 conversion fix passes all twelve controls in both phases (24 HTTP requests against real storage). The ReleaseFast regression build passes 11/11 steps; the test passes with zero leaks. Python lint/formatting, Zig formatting and diff checks pass. The new standalone E2E case reproduced the failure against the retained binary and will run against the new binary in CI; a new full real-model quickstart run is not claimed.
• The full real-model quickstart and six portable quickstart E2E passes above are for 0c0a2eda8. Follow-ups through 7fee6e39b have targeted failure-path/chunker/SDK validation (including four final-answer/history cases); a new full real-model quickstart run is not claimed for this follow-up. SDK CI passed on 0c0a2eda8; CI will rerun for the new head.

Previously indexed data needs its full-text index rebuilt or documents reindexed to populate missing _all postings; existing segments are not migrated. The rerun uses the unchanged guide and its 8192 MiB budget, with model artifacts downloaded and validated in the preceding cold-store run. That earlier run confirmed the #664 scratch failure no longer reproduced, but RAG returned zero hits and an apology before this default-field fix.

An independent user-provided replay on 7fee6e39b also passed the unchanged local CLI workflow with real Gemma, five RAG evidence hits, a terminal completion and no capacity error. It reproduced the separate query-string AND failure addressed by this follow-up. Retrieval relevance, agent query refinement and previously indexed-data migration qualification remain outside that execution pass.