diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md index ec1894b08..e4052b283 100644 --- a/RELEASE_NOTES.md +++ b/RELEASE_NOTES.md @@ -1,9 +1,136 @@ # Release notes -## Changes from 4.10.1 to 4.10.2 +## Changes from 4.10.1 to 4.11.0 XXX version-specific blurb XXX +### New features + +#### Mask-based nullable columns for CTable, and they are now the default + +A nullable CTable column keeps its nulls in a per-column **validity sidecar** — +Arrow's own model — instead of reserving a value from its own range. This is now +what a bare `nullable=True` resolves to, and what every nullable column inferred +from Arrow, Parquet or CSV gets: + +```python +blosc2.bool(nullable=True) # no reserved 255; dtype stays np.bool_ +blosc2.int8(nullable=True) # all 256 values usable, plus nulls +blosc2.utf8(nullable=True) # any string, including "" and "\x00" +blosc2.complex128(nullable=True) # nullable at all, for the first time +``` + +This is what makes nullability **lossless**: a sentinel steals a value from the +dtype, so a nullable `int8` could not hold `-128`, a free-text `utf8` column had +no safe sentinel at all, and Arrow columns whose type had no value to spare could +not be imported. `to_arrow(from_arrow(x))` now returns `x` for nullable `bool`, +full-range `int8`/`uint8`, `float64` containing `nan`/`±inf`/`-0.0` as values, +`utf8` containing `""` and `"__BLOSC2_NULL__"`, and `timestamp` with `int64.min` +as a value — none of which round-trip through a sentinel. + +Under mask storage `None` is how you write a null (`t.append((None,))`, +`t["price"][3] = None`), which a fixed-width sentinel column cannot accept at +all. `is_null()` is unchanged and remains the uniform API across every kind. + +**Nothing on disk changes.** The new default governs *creation* only: opening a +stored table never re-resolves anything, so every existing table keeps the +storage, dtype and sentinel it was written with, and every rewrite rule for the +reserved `255` stays permanently in place. Sentinel storage is supported +indefinitely and is one keyword away, per column (`null_storage="sentinel"`, or +any explicit `null_value=`) or globally through `NullPolicy`. Setting a type-wide +`NullPolicy` sentinel field still implies sentinel storage for the kinds it +covers, so existing `NullPolicy(float_value=...)` code is unaffected — with one +unavoidable exception: `255` is the only value a nullable bool may reserve, so it +is also `bool_value`'s default, and `NullPolicy(bool_value=255)` carries no +information to act on. A bool column that wants a sentinel has to say so with +`null_storage` or `column_null_values`. + +A table containing a mask column records **schema version 3**. Only such tables do: +a table with no nullable column still records version 1, exactly as before, and a +sentinel one does too. Readers older than 4.11.0 refuse a version-3 table rather +than misreading it, but their message is a bare `ValueError: Unsupported schema +version 3` — the hint naming `convert_nulls(to='sentinel')` ships in 4.11.0, so +only readers that can already open the file will print it. + +If some of your data has to stay readable by an earlier release, pin the storage +rather than discovering this downstream. Per column with +`null_storage="sentinel"` (or any explicit `null_value=`), or process-wide, +including for schemas inferred from Arrow, Parquet and CSV: + +```python +with blosc2.null_policy(blosc2.NullPolicy(null_storage="sentinel")): + t = blosc2.CTable.from_parquet("data.parquet") +``` + +That reinstates the sentinel's lossiness — a float column's nulls become `NaN` +again, and a type with no value to spare still cannot be imported — which is the +trade being made. + +`Column.null_storage` reports where a column keeps its nulls and `info` tags each +column (`int64 nullable[mask]`), so `CTable.convert_nulls()` can move columns +between the two in either direction — never implicitly, and refusing rather than +silently relabelling data when a sentinel is unavailable. + +One deliberate semantic difference: in a mask column `NaN` is a **value**, +following Arrow, and only the sidecar marks a null. Sentinel float columns keep +NaN-as-null. See "Where nulls are stored" in the CTable reference. + +#### Column indexes are null-aware + +Every index kind stores per-segment `min`/`max`, and those extrema are now taken +over the rows that carry a **value**: a column's nulls are read from its +validity channel and left out, and a segment with no value at all is flagged +rather than summarised. This applies to both storages — an `INT64_MIN` sentinel +is exactly as invisible to a summary as a mask column's fill. + +Two things follow. `Column.min`/`Column.max` answer from the index for a +nullable column (**236x** on a 20M-row `int64`, measured) where before every +nullable column but a NaN-sentinel float had to scan; and `where()` with an `OR` +over a nullable indexed column uses the index instead of falling back to a full +scan (**1.6x** on a 20M-row two-column probe). The `OR` fallback existed because +the only null filtering available was global, and a global filter drops a row +that is null in one branch but matches the other; the segment path never needed +it, because it *evaluates* the predicate, which has been null-aware per leaf +since the string-predicate fix below. + +Indexes written by an earlier release are read as not null-aware and keep the +old fallback, so nothing silently changes meaning; `rebuild_index()` promotes +them. Building an index over a nullable column that actually holds nulls now +costs one decompression pass (33 ms for a 20M-row `int64` column) because the +incremental per-block summaries folded during writes carry no validity; a +nullable column with no nulls keeps that fast path untouched. + +### Bug fixes + +- **`group_by` returned the wrong `min` for a `bool` value column.** The + per-group accumulator was seeded from the dtype's opposite identity, and `bool` + had none, so an all-`True` group reduced to `False`. Reachable with any plain + non-nullable bool column on the generic aggregation path. +- **Descending `sort_by` on a `bool` column raised**, and on a signed-integer + column holding its dtype's minimum (`-128` for `int8`) that row sorted as if it + were the largest. The descending key negated in the column's own dtype, where + `bool` has no unary minus and a narrow signed type wraps. +- **`add_column()` after `copy()` backfilled one row short**, and raised for a + variable-length column: the copy recorded its write watermark one below the + convention every other writer follows. +- **A string predicate over a nullable column returned its nulls as matches.** + `t.where("a > 10")` compared the stored sentinel, so any sentinel satisfying + the predicate (`null_value=999` against `> 10`) came back as a match. The + operator form (`t.where(t.a > 10)`) was always correct. Fixed for both storages. +- **A nullable `uint8` ndarray column came back as `bool`.** The `bool → uint8` + widening that sentinel storage needs was undone by dtype rather than by + whether it had been applied, so a column declared `uint8` was truncated to + flags. +- **`~` on a nullable bool column selected its nulls.** SQL `WHERE` semantics + say a null satisfies neither a predicate nor its negation; the mask path + inverted the stored `False` fill instead. (The sentinel path was already + correct, via its `== 0` rewrite.) +- **CSV import and export ignored a validity sidecar.** `to_csv` compared + against the sentinel to find nulls, so a mask column wrote its fill as if it + were data, and `from_csv` had nothing to put in an empty field and raised. + Both go through the sidecar now: an empty CSV field is a null in either + direction. Sentinel columns keep writing their sentinel, unchanged. + ## Changes from 4.10.0 to 4.10.1 diff --git a/doc/reference/ctable.rst b/doc/reference/ctable.rst index acc764086..6cb4b2c54 100644 --- a/doc/reference/ctable.rst +++ b/doc/reference/ctable.rst @@ -118,13 +118,100 @@ the original unnamed root-list grouping, row groups, encoding choices, or file metadata exactly. +Where nulls are stored +---------------------- + +A nullable column keeps its nulls in one of two places, and +:attr:`Column.null_storage` says which:: + + t["price"].null_storage # 'sentinel', 'mask', 'code', 'native' or 'none' + t.info # tags every column: int64 nullable[mask] + +``"sentinel"`` + A value reserved from the column's own range — ``255`` for a nullable + ``bool`` (which is therefore physically ``uint8``), ``INT64_MIN`` for an + ``int64``, ``NaN`` for a float, the literal ``"__BLOSC2_NULL__"`` for a + string. Nothing extra is stored, and every reader understands it. + +``"mask"`` + A per-column ``.notnull`` validity array, one byte per row, ``True`` where + the value is present — Arrow's own model. Nullity lives outside the value + range, so the column keeps every value its dtype can hold, and a nullable + ``bool`` stays a real ``np.bool_``. + +The two other values are the only representation their kind has: ``"code"`` for +a dictionary column, which reserves ``-1``; ``"native"`` for the +variable-length container kinds, whose cells simply hold ``None``. + +**Mask storage is the default** since 4.11.0, because it is what makes +nullability lossless. A bare ``nullable=True`` — and every nullable column +inferred from Arrow, Parquet or CSV — keeps its nulls in a sidecar, so: + +.. code-block:: python + + blosc2.bool(nullable=True) # no reserved 255; dtype stays np.bool_ + blosc2.int8(nullable=True) # all 256 values usable, plus nulls + blosc2.utf8(nullable=True) # any string, including "" and "\x00" + blosc2.complex128(nullable=True) # nullable at all, for the first time + +Sentinel storage is supported indefinitely and is one keyword away, per column +(``null_storage="sentinel"``, or any explicit ``null_value=``) or globally +through :class:`NullPolicy`. It is the right choice when a column has to stay +readable by a Blosc2 release older than 4.11.0: a table containing a mask column +records **schema version 3**, which earlier readers refuse — with a bare +``ValueError: Unsupported schema version 3``, since the hint naming +:meth:`CTable.convert_nulls` ships in 4.11.0 — rather than misreading. Only a +table that actually contains a mask column is affected; one with no nullable +column still records version 1. + +Nothing on disk changes. The flip governs *creation* only — opening a stored +table never re-resolves anything, so every existing table keeps the storage, +dtype and sentinel it was written with. + +Under mask storage ``None`` is the way to write a null — ``t.append((None,))``, +``t["price"][3] = None`` — which a fixed-width sentinel column cannot accept at +all (there you write the sentinel yourself). Reads are unchanged: ``col[:]`` +returns values with a deterministic fill in the null slots, and +:meth:`Column.is_null` is what tells you which those are. + +One semantic difference is deliberate: in a **mask** column ``NaN`` is a +*value*, following Arrow, and only ``mask=False`` is a null. A sentinel float +column keeps NaN-as-null. So ``dropna``, ``group_by`` and ``min``/``max`` can +differ between the two for float columns holding a real NaN. + +Indexes understand both storages: a column index summarises only the rows that +carry a value, so ``min``/``max`` and ``where`` are as fast on a nullable +column as on a plain one. See `Indexes`_. + +Converting between them +~~~~~~~~~~~~~~~~~~~~~~~ + +Nothing migrates on its own: opening, copying and saving a table all preserve +each column's storage. :meth:`CTable.convert_nulls` is the only thing that +changes it:: + + lossless = t.convert_nulls() # every convertible column -> mask + lossless.copy(urlpath="out.b2d") # land it back on disk + t.convert_nulls("flag", to="mask", inplace=True) + +Going back to ``"sentinel"`` refuses what it cannot represent — a column whose +data already contains the proposed sentinel, or one with no value left to +reserve — and says which value is in the way, rather than silently relabelling +a real row as null. + +.. autosummary:: + + CTable.convert_nulls + +.. automethod:: CTable.convert_nulls + + Null policy ----------- -Nullable scalar CTable columns are represented with per-column sentinel values, -not native validity bitmaps. When CTable has to infer those sentinels, the -selection can be customized with :class:`NullPolicy` and scoped with -:func:`null_policy`:: +:class:`NullPolicy` decides what a bare ``nullable=True`` resolves to when the +schema does not say: whether to use a mask at all, and if not, which sentinel to +reserve. Scope it with :func:`null_policy`:: policy = blosc2.NullPolicy( signed_int_strategy="max", @@ -148,10 +235,27 @@ The same policy is used by explicit nullable schema specs when no with blosc2.null_policy(policy): table = blosc2.CTable(Row) -Sentinels are resolved in this order: explicit ``null_value`` in the schema, -``NullPolicy.column_null_values`` for a matching column, then the type-wide -``NullPolicy`` default. Columns without ``nullable=True`` or an explicit -``null_value`` are not nullable. +Storage is resolved in this order, first match winning: an explicit +``null_storage=`` on the spec; an explicit ``null_value=`` (which *is* a request +for in-band storage); a ``NullPolicy.column_null_values`` entry for the column; +a type-wide ``NullPolicy`` sentinel field covering the column's kind; and +finally ``NullPolicy.null_storage``. Once sentinel storage is chosen, the +sentinel itself comes from ``column_null_values`` if listed and from the +type-wide default otherwise. + +Setting any type-wide sentinel field therefore *implies* sentinel storage for +the kinds it covers, so ``NullPolicy(float_value=-1.0)`` keeps working exactly as +it did before the default flipped. Passing ``null_storage="mask"`` alongside one +is the one combination that raises, because it contradicts itself. + +``bool_value`` is the exception, and unavoidably so: ``255`` is the only value a +nullable bool may reserve, so it is also the field's default, and +``NullPolicy(bool_value=255)`` carries no information to act on. A bool column +that wants sentinel storage has to say so with ``null_storage`` or +``column_null_values``. + +Columns without ``nullable=True``, an explicit ``null_value`` or an explicit +``null_storage`` are not nullable. .. autosummary:: @@ -198,9 +302,15 @@ expression: To exclude them, write the complementary comparison instead (``t.price <= 0``), which never matches nulls. +All of this applies to mask-storage columns unchanged: propagation reads an +opaque boolean "is null" predicate, which a sidecar supplies as directly as a +sentinel comparison does. + Kleene three-valued logic (where ``null > 0`` evaluates to null rather than -``False``) is intentionally out of scope — it needs a validity channel on -boolean intermediates, i.e. masks, which CTable does not use. +``False``) remains intentionally out of scope. Mask storage supplies the +validity channel it needs, but it also requires comparison *results* to carry +one, so that ``~`` can invert three states rather than two — a change to the +expression layer rather than to storage. Reductions on derived expressions skip nulls too: arithmetic involving a nullable column returns a ``NullableExpr`` — a thin wrapper that remembers @@ -594,6 +704,23 @@ Choosing an index kind than 50 % of candidate segments, the planner skips the index and falls back to a full scan to avoid per‑segment evaluation overhead. +Indexes on nullable columns + Every index kind stores per‑segment ``min``/``max``, and since 4.11.0 those + extrema are taken over the rows that carry a **value**: a column's nulls are + read from its validity channel — the ``.notnull`` sidecar of a mask column, + the reserved value of a sentinel one — and left out. A segment with no + value at all is flagged rather than summarised. + + Two things follow. :meth:`Column.min` and :meth:`Column.max` answer from + the summaries for a nullable column instead of scanning it (~240x on a + 20M‑row column), where before any nullable column except a NaN‑sentinel + float had to fall back; and segment pruning is tighter, because a block + whose only large values are nulls no longer looks like a candidate. + + Indexes built by an earlier release are read as *not* null‑aware and keep + the old fallback, so nothing silently changes meaning; + :meth:`CTable.rebuild_index` promotes them. + .. autosummary:: CTable.create_index @@ -737,10 +864,12 @@ Attributes Column.dtype Column.null_value + Column.null_storage Column.row_transformer .. autoproperty:: Column.dtype .. autoproperty:: Column.null_value +.. autoproperty:: Column.null_storage .. autoproperty:: Column.row_transformer diff --git a/plans/mask-based-nulls.md b/plans/mask-based-nulls.md new file mode 100644 index 000000000..51ca78ad7 --- /dev/null +++ b/plans/mask-based-nulls.md @@ -0,0 +1,1022 @@ +# Mask-based nullable columns for CTable + +> **Status: complete — Phases 0–10 landed 2026-08-08.** Lossless Arrow/Parquet +> round-trip works for every V1 kind, sort/groupby/query honour a sidecar, `convert_nulls` migrates +> columns in either direction, **mask storage is now the default** — a bare `nullable=True` +> resolves to it — and column indexes summarise only the rows that carry a value, so `min`/`max` +> and indexed `OR` no longer fall back on a nullable column. Six premises were disproven during +> implementation and are corrected in place, each in a blockquote beside the text it corrects: the +> index path cannot be fixed by a null-aware expression (§Expression layer), the bool dtype-flip +> cannot move out of `__init__` (§Schema layer), ndarray columns do not get lazy null propagation +> for free (§Expression layer), the "free" summary min/max fast path is unsound (§Reductions), +> `np.packbits` is not needed and avoiding it is safer (§Arrow/Parquet), and `.equals()` cannot +> express the round-trip contract (§Arrow/Parquet). Phase 7 added a seventh — **Phase 1 left the +> mask half of the query path undone**, and nobody noticed until sort work went looking +> (§Expression layer, "Addendum 2") — and Phase 8 an eighth: the in-place migration ordering +> recorded below is **wrong at its middle step**, and moving that step last makes every +> intermediate state correct rather than merely recoverable (§Migration). Phase 10 added a ninth, +> which is a partial retraction of the fourth: the summary fast path is unsound only for a *genuine* +> NaN, not for a whole mask float column, and `null_order` — planned beside `null_aware` — records +> a promise no index kind keeps (§Reductions, §Sort and indexes). It also narrows the first: an +> ordered index cannot be fixed by a null-aware expression, but the *segment* index was never +> ordered and needed no fixing, which is what lets indexed `OR` work +> (§Expression layer, "Addendum 3"). Drafted 2026-08-08. +> Revised 2026-08-08 after review: lazy sidecar materialization (decision 9), `NullPolicy` +> inference instead of raising, staged default flip (Phase 9), null-predicate rewrite pulled +> forward to Phase 1, sidecar suffix renamed `.notnull`. +> +> **This is the reference document for how CTable stores null values from now on.** It +> supersedes the sentinel-only decisions recorded in `plans/ctable-nulls.md` (whose non-goals +> list *"add separate validity bitmap storage for scalar CTable columns"*), +> `plans/enhancing-ctable.md` §Gap C, `plans/enhancing-ctable-phase2.md` §P5, and +> `plans/enhancing-ctable-phase3.md` §"Scope decision". Those remain accurate records of what +> was decided and shipped at the time and are not being amended; read them as history, and +> read this document for intent. + +## Context + +CTable represents nulls three different ways today: an **in-band sentinel** (`null_value`) for +numeric/bool/timestamp/string/bytes/utf8/ndarray, a **reserved code** `-1` for dictionary +columns, and **native `None`** for vlstring/vlbytes/list/struct/object. The sentinel model is +the default and it is lossy by construction — a sentinel steals a value from the dtype's range: + +- `bool(nullable=True)` silently becomes physical `uint8` with `0/1/255`, leaking raw `255` + through `col[:]` and row tuples, and requiring a whole layer of filter rewrites + (`flag == True → raw == 1`) so nulls don't leak into predicates. +- `int8`/`uint8` cannot use their full range alongside nulls. +- Free-text `utf8`/`string` has no safe sentinel — any value is legal. `blosc2.utf8(null_value="\x00")` + is *rejected outright* because NumPy 2.4 won't match a lone `"\x00"` against `StringDType`, + which would silently stop marking anything as null. +- Parquet import already carries a workaround artifact in schema metadata: + `"conversion": "nullable_scalar_wrapped_as_singleton_list"` — a nullable **bool** wrapped as a + one-element list because the sentinel couldn't represent it. +- `_compiled_columns_from_arrow` (`ctable.py:7457`) outright **raises** when a nullable Arrow + column has no available sentinel, so those types can't be imported at all. + +The goal is lossless Arrow/Parquet round-trip for every scalar type, achieved by moving nullity +into a **sidecar validity array** per column — Arrow's own model — while keeping the sentinel +path fully supported for API compatibility and giving existing tables an explicit migration. + +This design is already recorded and parked in `plans/enhancing-ctable-phase2.md` §P5 +("Mask-based nullable columns: PARKED — design recorded, do not build yet"). Its first unpark +criterion, *"a user asks for nullable bool without the 255 reservation"*, is what this work +answers. + +## Decisions (settled during planning — do not re-litigate) + +1. **Mask becomes the default** for `nullable=True` on newly created tables — in two steps: + the capability ships opt-in first (Phase 6), and the default flips no earlier than one + release later (Phase 9), so version-3-capable readers are in circulation before + default-created tables require them. *(The two-step staging was not kept — both shipped in + 4.11.0; see the deviation note under §Phasing.)* Sentinel remains fully supported and readable forever, + selectable per column (`null_value=...`, `null_storage="sentinel"`) or globally via + `NullPolicy`. Existing on-disk tables keep working unchanged. +2. **V1 scope** = fixed-width scalars + utf8: numeric (incl. **complex**, which gains nullability + for the first time), bool, timestamp, `string` (`U*`), `bytes` (`S*`), `utf8`, `ndarray`. + Dictionary keeps `null_code=-1`; vlstring/vlbytes/list/struct/object keep native `None`. + `is_null()` remains the uniform user-facing API across all kinds. +3. **Layout** = plain `np.bool_` NDArray, 1 byte/row, `True = valid` (Arrow polarity), mirroring + `/_valid_rows`. Not bit-packed; `np.packbits` only at the Arrow boundary. +4. **Read semantics unchanged**: `col[:]` returns raw values with a deterministic fill in null + slots; `is_null()`/`notnull()` read the mask. Adds opt-in `col.to_numpy(masked=True)`. +5. **Fill is loud**: `float → NaN`, `timestamp → int64.min` (decodes to `NaT` for free via + `_maybe_decode_timestamp_values`), everything else dtype zero / `""` / `b""` / `0j`. + The fill is **not** part of the format contract and is **not** recorded in the schema — + recording it would recreate sentinel collisions at the metadata layer. +6. **NaN is a value, not a null**, in a mask-backed float column. Only `mask=False` is null. + This matches Arrow and is the point of a side channel. Sentinel float columns keep + NaN-as-null forever. Consequence: `dropna`/`groupby`/`min`/`max` differ between the two + storages for float — must be documented and covered by the equivalence tests. Frame the + docs positively — "mask columns follow Arrow semantics for NaN" — not as a changelog caveat. +7. **Nothing auto-migrates.** Not on `open()`, not on `copy()`, and — importantly — + not on `save()`/`to_cframe()`, which must *preserve* each column's `null_storage`. +8. **Kleene three-valued logic stays out of scope.** Masks make it possible (they supply the + validity channel `plans/enhancing-ctable.md` §Gap C named as the blocker), but + `_null_aware_compare` deliberately collapses null → `False` (SQL `WHERE` semantics). + Named deferred follow-up. +9. **An absent sidecar means all-valid.** The `.notnull` array is materialized lazily, on the + first write that actually contains a null; a mask-storage column with no `.notnull` key on + disk is a valid — and expected common — state meaning "no nulls so far". Null-free nullable + columns therefore cost zero bytes on disk and zero read-path work: `null_pred()` returns + `None`, which the expression layer already treats as never-null. It also makes + `convert_nulls(to="mask")` on a null-free sentinel column a pure schema update. + +## Architecture + +### Invariants to encode in the code, not just here + +- **Chunk pinning**: the mask array's `chunks[0]`/`blocks[0]` must equal the value column's, not + whatever `compute_chunks_blocks` picks for a bool dtype. Otherwise `_nonnull_chunks`, the + chunk-aligned writers, and index-segment alignment all re-align on every read. +- **Values under `mask=False` are unobservable** through the `Column` API. +- **Crash safety is inherited, not added**: `extend()` flips `self._valid_rows[start:end] = True` + at `ctable.py:12923`, *after* every column write. Mask writes inserted before that line are + invisible until the row goes live. Put a comment there so nobody reorders it. + +### New module: `src/blosc2/ctable_nulls.py` + +`ctable.py` is already 13.5k lines and already delegates to `ctable_storage.py` / +`ctable_indexing.py`; a fourth sibling fits. + +```text +NULL_NONE, NULL_MASK, NULL_SENTINEL, NULL_CODE, NULL_NATIVE = "none", "mask", "sentinel", "code", "native" + +class NullChannel: + """Uniform read/write accessor for one column's validity channel. + + Subsumes all four representations CTable uses -- sidecar validity array, + in-band sentinel, dictionary null code, native ``None`` -- so callers ask + *what is null* without knowing which one a column uses. + """ + __slots__ = ("_table", "_name", "_spec", "kind", "fill_value") + + def valid_array(self) # physical NDArray; mask kind only, else None + def null_pred(self) # physical LazyExpr, True where null; None if never null + def valid_pred(self) # physical, True where valid + def null_mask(self, key) # logical numpy bool; key = slice | positions | None + def valid_slice(self, a, b) # physical numpy bool for [a, b) -- Arrow export + def null_count(self) + def coerce_batch(self, values, n) # -> (values_with_fill, valid_np | None) + def coerce_scalar(self, value) # -> (storage_value, is_valid) + def set_valid(self, key, valid); def resize(self, n); def gather(self, positions) +``` + +Owned by `CTable` as `self._null_channels: dict[str, NullChannel]`, invalidated whenever +`_schema`/`col_names` change. The mask NDArrays live in `CTable._null_masks`, opened **lazily** +via `storage.open_null_mask(name)` — mirror `_LazyColumnDict` (`ctable.py:3658-3746`) rather +than opening a sidecar per column on `open()`. Creation is lazy too (decision 9): `NullChannel` +materializes the sidecar the first time `coerce_batch`/`coerce_scalar` actually reports a null; +until then `valid_array()`/`null_pred()` return `None` and the column reads as never-null. + +`Column` gets `self._nulls` and a public `Column.null_storage` property. + +This module is what keeps the work from becoming a 50-site grep, so **it lands first, +sentinel-only, with zero behavior change** (Phase 0). + +> **As built (2026-08-08).** Three departures from the sketch above, all in the same direction — +> away from snapshotting state that can go stale: +> +> - **Bound to a `Column`, not to `(table, name)`.** Logical reads (`is_null()`, `null_count()`) +> have to honour the column's view — sorted order, row filter — and `Column` already carries +> that. `Column._nulls` builds one on first use and keeps it; a table-level accessor can be +> added in Phase 3 for the physical write paths, which have no `Column`. +> - **`kind` and `fill_value` are properties, not `__slots__` entries.** `_resolve_nullable_specs` +> mutates `spec.null_value` *in place* (`ctable.py:4536`), so a channel that snapshotted at +> construction would report the wrong kind afterwards. Everything reads through to the live +> schema. Pinned by `test_channel_reads_through_to_live_schema`. +> - **`null_mask()` takes no `key` yet.** Dictionary columns answer through `_dictionary_eq`, +> which is whole-column, so a `key` argument would have been silently ignored for them. It +> arrives in Phase 4, when masks make it meaningful. +> +> The `NULL_*` constants and `fill_value_for` ended up **defined in `schema.py`** and re-exported +> here, because this module imports the spec classes from `schema.py` — that is the lower layer. +> Import them from either place. +> +> Also landed here, not in the sketch: `sentinel_mask`, `is_nan_sentinel`, `is_null_value`, +> `kind_of_spec`, `sentinel_guard_expr` and `rewrite_null_predicates` (Phase 1). Unifying the +> "is this sentinel a NaN" test incidentally fixed several sites that spelled it +> `isinstance(nv, float)` and so missed a `float32` NaN sentinel. + +### Schema layer — `src/blosc2/schema.py`, `schema_compiler.py` + +Every spec re-declares `nullable`/`null_value` today (`_NumericSpec:94`, `timestamp:251`, +`bool:285`, `string:335`, `bytes:384`, `UTF8Spec:622`, `NDArraySpec:784`). Add a +`_NullableSpecMixin` above `_NumericSpec` with `_init_nulls(...)`, `uses_mask`/`uses_sentinel` +properties, and `_null_metadata()`; each spec's `__init__` gains one `null_storage=None` kwarg +and one call, and each `to_metadata_dict` swaps its two nullability blocks for +`**self._null_metadata()`. `_null_metadata` **never emits `"sentinel"`**, so sentinel tables +serialize byte-identically to today. + +Do *not* dodge the signature edits by post-setting `null_storage` in `spec_from_metadata_dict` — +`spec_cls(**data)` (`schema_compiler.py:447`) is the clean-fail mechanism for old readers. + +Two spec fixes fall out: +- **`bool`** (`schema.py:275-300`): drop the `null_value != 255` rejection under mask storage, and + move the `bool_ → uint8` dtype flip **out of `__init__`** into `_resolve_nullable_specs`, which + already flips there (`ctable.py:4543`). Same for `NDArraySpec` bool (`ctable.py:4510-4516`, `4545-4551`). +- **`complex64`/`complex128`** (`schema.py:208-238`): gain nullability via the mixin, fill `0j`. + +> **Correction (implemented 2026-08-08).** The dtype-flip relocation above is wrong as stated and +> **was not done**. `_resolve_nullable_specs` runs only on the *creation* paths; **opening a stored +> table never calls it** — `open()` rebuilds each spec through `spec_cls(**data)` +> (`schema_compiler.py:447`) and that is the whole of it. Moving the flip out of `__init__` would +> therefore bring every persisted nullable-bool column back as `np.bool_` while its bytes are +> `uint8`, silently misreading `255` as `True`. Verified by instrumenting the resolver across a +> save/open cycle. +> +> What shipped instead splits the responsibility by what each site can know. `__init__` resolves as +> far as metadata alone allows — `nullable and not uses_mask → uint8` — which is exactly the +> information a reopened table carries, so persisted columns come back correct with no resolver +> involved. `_resolve_nullable_specs` then corrects it **in both directions** once the policy has +> spoken, via `_unflip_mask_bool_dtype`: a bare `nullable=True` that resolves to mask gets its +> `uint8` undone. Same split for `NDArraySpec` bool. Regression-pinned by +> `test_stored_uint8_bool_reopens_as_uint8_without_the_resolver`. +> +> Also implemented, beyond what this section specified: **complex is mask-only**. There is no +> complex value safe to reserve, so `complex64(null_value=...)` raises and `nullable=True` resolves +> to mask regardless of the policy default. The spec classes carry a `supports_sentinel` class flag +> for this, which also replaced a `hasattr(spec_cls(), "null_value")` duck-type test in the Arrow +> importer (`ctable.py:7256`) that the mixin would otherwise have made answer True for complex. + +**Version gating** — `schema_to_dict` (`schema_compiler.py:487`) computes the version as an +explicit feature max: + +```python +uses_mask = any(getattr(c.spec, "null_storage", None) == "mask" for c in schema.columns) +schema_version = ( + 3 if uses_mask else (2 if schema.metadata.get("nested") is not None else 1) +) +``` + +and `schema_from_dict` accepts `(1, 2, 3)`. This deviates from P5's "no global version bump" but +preserves its actual goal — the bump is *conditional on a mask column existing*, so only +mask-using tables are unreadable by old readers. What it buys is the failure message: a readable +`ValueError: Unsupported schema version 3` instead of a `TypeError` from deep inside +`spec_cls(**data)`. Include a hint naming `convert_nulls(to="sentinel")`. + +**`NullPolicy`** (`ctable.py:117-197`) gains `null_storage: Literal["mask", "sentinel"]` +(default `"sentinel"` until Phase 9 flips it). +`CTable._resolve_nullable_specs` (`ctable.py:4496-4553`) stays the single decision point, resolving +in order: explicit `spec.null_storage` → explicit `spec.null_value` (⇒ sentinel) → +`policy.column_null_values[name]` (⇒ sentinel) → a set type-wide sentinel field matching the +column's kind (⇒ sentinel) → `policy.null_storage`. Under `"mask"` it skips sentinel selection +entirely — **no `max_length` widening, no bool dtype flip**. + +Note: once the default flips, `NullPolicy`'s type-wide sentinel fields (`string_value`, +`float_value`, `signed_int_strategy`, …) would become silently inert for plain `nullable=True`. +Do **not** have `__post_init__` raise on that combination — that would break existing working +code (`NullPolicy(float_value=...)`) on the very release that flips the default. Instead, +setting any type-wide sentinel field **implies `null_storage="sentinel"`** for the types it +covers (the resolution-order entry above); `__post_init__` raises only when +`null_storage="mask"` is passed *explicitly* alongside them, which is a genuine contradiction +the user wrote. `column_null_values` stays meaningful — it forces sentinel per column. + +Add one function `schema.fill_value_for(spec)` implementing decision 5. + +### Storage layer — `src/blosc2/ctable_storage.py` + +New key suffix beside `_UTF8_DATA_SUFFIX` (`:410`), collision-free by the same documented +argument (no column name can map to a key containing a literal `.`): + +```python +_NOTNULL_SUFFIX = ".notnull" # -> /_cols/.notnull, extension .b2nd +``` + +Named `.notnull` — deliberately **not** `.valid` — because `/_valid_rows` already exists and +means something different (row liveness vs. per-column null validity); two bool arrays with +near-identical names would invite conflation. The name also states its polarity: `True` = not +null, matching Arrow. + +Five new `TableStorage` methods (`create_null_mask` / `install_null_mask` / `open_null_mask` / +`has_null_mask` / `delete_null_mask`), implemented across all four backends exactly as +`create_valid_rows`/`open_valid_rows` (`:132-141`) already are. Per decision 9, +`has_null_mask(name) is False` is the common case and means all rows valid; `create_null_mask` +is called by `NullChannel` on the first null actually written, never at column creation: + +| backend | notes | +|---|---| +| `InMemoryTableStorage:215` | `blosc2.zeros(shape, np.bool_, ...)`; `open_*` raises as `open_column` does | +| `FileTableStorage:560` | `store[key + _NOTNULL_SUFFIX] = arr` | +| `TreeStoreTableStorage:1081` | copy the utf8-companion block at `:1304-1319` (`_dest_path`, `map_tree`, `_modified`) | +| `EmbedStoreTableStorage:413` | read-only: `open_null_mask` only; creates go in the `_not_supported` list at `:527-537` | + +The mask is a TreeStore *key* like the utf8 offsets array, so `.b2z`/`mmap_mode` are handled by +`_open_store()[key]` — no `store.offsets` special-casing (`open_varlen_scalar_column:737-741` +is the precedent). + +`delete_column` / `rename_column` already carry a bespoke `+_UTF8_DATA_SUFFIX` branch in both +`FileTableStorage` (`:901-933`) and `TreeStoreTableStorage` (`:1496-1540`). **Generalize to a loop +over `_COMPANION_KEY_SUFFIXES = (_UTF8_DATA_SUFFIX, _NOTNULL_SUFFIX)`** rather than adding a third copy. + +Capacity/persistence sites in `ctable.py`: `_grow` (`:4902`), `trim_capacity` (`:4875`), +`compact` (`:11391`), `_save_to_storage` (`:5827` — **both** the `install_column` reblock fast path +at `:5952` and the `create_column` path at `:5964`), `to_cframe` (`:5754`, per-column loop `:5809-5823`), +and **`CTable.load` (`:6099`), a second parallel open path that is easy to miss**. All of these +operate only on masks that exist — an absent sidecar needs no growing, trimming, or copying, +which is most of the lazy-materialization payoff. + +> **As built (2026-08-08).** All five storage methods, four backends, both companion-suffix loops, +> and every capacity/persistence site above. Six notes, the first two of which are corrections: +> +> - **The site list was incomplete.** `copy()`'s in-memory path (`ctable.py:12563`) builds its +> result through `_empty_copy` + a per-column gather and never touches `_save_to_storage`, so a +> copied table would have silently dropped its sidecars. It gathers them by the same `live_pos` +> now. `to_b2z`/`to_b2d` need nothing: their physical-pack fast paths zip the TreeStore leaves +> as-is, and `.notnull` is one of them. +> - **`resize()` zero-fills, i.e. *invalid*.** `_grow` must write `True` over the new tail +> explicitly, or every appended row past the old capacity reads as null. Same for the freed tail +> in `compact`. This is the one place where "absent means all-valid" and "present means read the +> bytes" have to be reconciled by hand. Pinned by `test_grow_extends_the_sidecar_as_all_valid`. +> - **Iteration must consult storage, not just the cache.** `_existing_null_masks()` asks the +> storage backend per mask-storage column rather than iterating what happens to be open; +> `_grow`/`trim_capacity` on a *freshly reopened* table have opened nothing yet, and skipping an +> unopened sidecar would leave it out of step with its column. Pinned by +> `test_capacity_paths_find_an_unopened_sidecar`. +> - **Chunk pinning has a documented fallback.** `_null_mask_grid` pins to the value column's +> `chunks[0]`/`blocks[0]`, dictionary columns to their `codes`, and anything whose payload is not +> a plain row-indexed NDArray — utf8, whose offsets array carries `n + 1` entries — to the +> table-wide `_valid_rows` grid, which is the same shared grid the fixed-width columns use. +> `_save_to_storage` records the grid each column *actually landed on* (`dest_grids`) rather than +> re-deriving it, because `chunks_override` and the reblock fast path can both change it. +> - **`_null_masks` is a property that resolves through `base`**, so the six view-construction +> sites that build a `CTable` via `__new__` needed no edits: a view shares its base's sidecars +> exactly as it already shares its `_cols` NDArrays. +> - **Rename drops the cached handle on disk, carries it in memory.** `storage.rename_column` +> re-keys the sidecar, so a cached handle points at a key that no longer exists; in-memory +> storage re-keys nothing, so there the handle *is* the sidecar and must be carried. + +### Read / write paths + +Today fixed-width scalar columns cannot accept `None` at all — `_coerce_row_to_storage`'s +else-branch (`ctable.py:4823`) is `np.array(val, dtype=col.dtype).item()`, so users write the +sentinel literally. Under masks **`None` becomes the canonical way to write a null**, which is +new capability, not re-plumbing. `NullChannel.coerce_batch` null-detection rules: + +- Python object sequence: `v is None` → null; `pandas.NA`/`pyarrow.NA` → null (duck-typed, **not** `float('nan')`). +- NumPy float array input: NaN is a **value** (decision 6). +- `np.ma.MaskedArray` input: `~arr.mask` is the validity verbatim. +- Arrow/pandas nullable input: the source's own validity is authoritative. +- Returns `valid_np is None` when nothing was null, so callers skip the mask write entirely. + +Sites: `extend` (`:12718-12927` — coercion loop `:12845-12886`, write loop `:12898-12920`, and the +timestamp `None`-substitution at `:12855-12873`); `append` (`:12628`) plus +`_coerce_row_to_storage` returning `(dict, null_names)`; `Column.assign` (`:2503`); +`_write_arrow_batch` (`:7759`) gaining a parallel `_ChunkAlignedWriter` per masked column. + +**`Column.__getitem__` / `_values_from_key` (`:1219-1316`) do not change** — that's what makes the +read side cheap. + +**`Column.__setitem__` (`:1394-1527`) is the highest-risk function in the plan**: four key +branches × three fast paths plus two chunked loops (`:1476-1490`, `:1511-1518`). For V1, route +mask columns through the single unified `else` path (`:1491-1520`) — compute `phys_indices` +explicitly, coerce once, write values and mask together — and skip the fast paths. Measure, then +re-add them in a follow-up. Threading mask writes through all six paths at once is where this +project would break. + +> **As built (2026-08-08).** `__setitem__` was not the hard part; the reroute was one added +> condition on the NDArray fast path plus one `_assign_validity` call per branch. Four things that +> were not in this section turned out to matter more: +> +> - **`NullChannel` must not be cached on its `Column`.** The pair was a two-object reference cycle +> (`Column._null_channel → NullChannel._col → Column`), and since a `Column` also holds its +> `CTable`, the moment `extend` started building a channel per column *every* write pinned a whole +> table until the next gc pass. Caught by `test_persistent_releases_without_gc`, which had been +> passing since long before this work. A weakref back to the column is the wrong fix — the channel +> is often the only owner (`table._null_channel(name)` builds a throwaway `Column`) — so the cache +> is gone instead and `_nulls` builds a fresh one-slot object per access. Pinned by +> `test_a_table_is_freed_without_a_gc_pass`. +> - **Validation runs before coercion**, so `schema_vectorized` and its ndarray branch had to learn +> that a bare `None` in the batch is a null for a mask column: a null cell has no value to +> constrain. Without this, `extend` raised out of `_validate_string_lengths` before reaching any +> of the new code. +> - **An overwrite replaces validity, it does not merge it.** `assign`/`__setitem__` must write +> `True` back over rows that were null before, which means `coerce_batch` returning `valid=None` +> ("nothing null in this batch") cannot simply be skipped there the way it can on append. +> `Column._assign_validity` is that distinction, and the one exception it keeps is a column with +> no sidecar at all, which is already all-valid. +> - **Every materialization path had to be found, not just the write paths.** `sort_by` (all three +> forms), `take`, `slice`, and `_sorted_small_copy_from_live_positions` each rebuild a table from +> gathered rows and silently dropped the sidecar. Factored into `_permute_null_masks` (in-place, +> shared with `compact`) and `_gather_null_masks_into` (copy). The latter skips columns whose +> gathered selection happens to be all-valid, so a copy that contains no null stays sidecar-free. +> +> Also landed here rather than in Phase 5: `null_pred`/`valid_pred` return the sidecar for mask +> columns. It is six lines inside `NullChannel`, and `_lazy_nonnull_mask` already consumes them, so +> leaving it out would have shipped aggregates that silently counted fill values. Fixed-shape +> ndarray columns still return `None` (an N-D values array does not broadcast against a +> one-flag-per-row sidecar) — that part stays Phase 5. + +### Null API — `ctable.py:2587-2721` + +`is_null()` (`:2627`) becomes `~channel.null_mask(...)`, i.e. O(1 byte/row) instead of +O(itemsize/row). `null_count()` (`:2645`) on a hole-free base table is +`n - blosc2.count_nonzero(mask[:n])` — bool NDArrays compress to near nothing, so effectively +O(chunks). `fillna` (`:2659`) becomes correct even when the fill collides with real data +(impossible under sentinels). `_nonnull_chunks` (`:2673`) zips value and mask chunks — this is +what the chunk-pinning invariant is for. New `Column.to_numpy(masked=False)` returns +`np.ma.MaskedArray` when `masked=True` and **must work for sentinel columns too** (derive the +mask from the sentinel) so the API is uniform. `_is_nullable_column` (`:13265`) becomes +`channel.kind != NULL_NONE`; `info_items` (`:1738`) gains `null_storage`. + +Struct/object columns keep the per-row Python loop at `:2638` — not made worse, worth a docs note. + +### Expression layer + +This gets *simpler*, because Phase 0 already centralized it. `Column._raw_null_pred()` +(`:1863-1881`) becomes `return self._nulls.null_pred()`, which for masks is `~mask_ndarray` +(a `LazyExpr` that composes with `&`). **`_combined_null_pred` (`:1883`), `_null_aware_arith` +(`:1899`), `_null_aware_compare` (`:1913`) and `NullableExpr` (`:859-1060`) are unchanged** — +they already consume an opaque boolean predicate. Keep null-pred (not validity) polarity so the +diff stays at zero; the single `~` fuses into the lazy expression. + +Two gains fall out: `_raw_null_pred` currently returns `None` for `is_ndarray` columns because a +per-item sentinel mask doesn't align 1:1 with rows — a mask *is* row-level, so **ndarray columns +gain null propagation in expressions for free**. And `_lazy_nonnull_mask` (`:2775-2802`) reads a +stored array instead of synthesizing a comparison, keeping the miniexpr reduction fast path with +one fewer computed operand. + +> **Correction (measured 2026-08-08).** The second gain is real and landed. **The first is not, +> and was not done.** Row-level is necessary but not sufficient: the values array is +> `(n, *item_shape)` and the sidecar is `(n,)`, and reshaping it to `(n, 1, …)` — the obvious fix — +> fails twice. `blosc2.where(pred_(n,1), nan, values_(n,3))` returns shape `(n, 1)`, silently +> **dropping the item dimension from the values** rather than broadcasting; and `NullableExpr`'s +> reduction mask combines the predicate with the `(n,)` `_valid_rows`, where `(n,) & (n,1)` explodes +> to `(n, n)`. Row-level null propagation for ndarray columns needs broadcasting support in the lazy +> layer, not a reshape at this call site. +> +> What ndarray columns *do* gain in this phase is null-aware **reductions**, which are NumPy-based +> and row-level throughout: `min`/`max`/`sum`/`mean`/`argmin`/`argmax` now skip null rows instead of +> reducing over the fill item. Pinned by `test_ndarray_columns_still_get_no_lazy_null_predicate`, +> which asserts both halves — no predicate, working reduction — so the gap stays deliberate. + +`_is_nullable_bool` (`:1824-1831`) becomes `kind == "bool" and channel.kind == NULL_SENTINEL`; +the `raw_col == 1` rewrites (`:2039`, `:2049`, `:2767`, `:13392`) go dead for mask bools and stay +alive forever for sentinel ones. + +### Arrow / Parquet + +**Export** — `iter_arrow_batches` (`:6981-7122`): hoist `valid = channel.valid_slice(start, stop)` +to the top of the per-column body, then per branch pass `mask=~valid` to `pa.array` (pyarrow packs +the numpy bool itself). The bool branch (`:7103-7107`) loses its `arr == 1`; the `U`/`S` branch +(`:7093-7102`) loses its `[None if null_mask[i] else v ...]` Python list comprehension; the ndarray +branch (`:7070-7087`) stops requiring *every element* to equal the sentinel (a lossy, surprising +rule) and uses row-level validity. Dictionary (`:7043`) and varlen/list/struct (`:7038`) are unchanged. + +`UTF8Array.arrow_slice` (`_utf8_array.py:961-984`) is the **one** place `np.packbits` is genuinely +needed, because it builds Arrow buffers directly: + +```python +validity = pa.py_buffer(np.packbits(valid, bitorder="little")) # bitorder is MANDATORY +``` + +Arrow validity bitmaps are LSB-first. This is the easiest bug in the plan to introduce and the +hardest to catch — a round-trip test passes with **either** bit order if import unpacks the same +way. Pin it with a test asserting the literal packed bytes for a known pattern. + +> **As built (2026-08-08): `np.packbits` is not needed at all, and avoiding it is strictly safer.** +> `arrow_slice` already had the answer for its sentinel path — `pa.array(~mask).buffers()[1]`. +> Arrow packs booleans and validity bitmaps identically (LSB-first), so handing pyarrow the +> booleans and taking the resulting array's *data* buffer borrows its packing and makes the bit +> order unrepresentable-as-wrong rather than merely tested. The mask path does the same with +> `pa.array(valid).buffers()[1]`. The literal-bytes test was still worth writing and is the useful +> half of the advice above: `test_utf8_validity_bitmap_is_lsb_first` pins +> `[valid, null, valid, valid, valid, null, valid, valid]` to `0xDD`, not the MSB-first `0xBB`. + +**Import** — in `_compiled_columns_from_arrow` (`:7369-7482`), when the resolved storage is +`"mask"` the entire sentinel-selection block is skipped and **the `"no null_value sentinel is +available"` error at `:7457` never fires**. That single deletion is what makes nullable bool, +full-range `int8`/`uint8`, and free-text utf8 importable at all. Add a `null_storage=` parameter +to `from_arrow`/`from_parquet` beside the nullable knob at `:8349`, defaulting to the policy. + +New `_arrow_column_to_numpy_masked(arrow_col, col) -> (values, valid)` beside +`_arrow_column_to_numpy` (`:7794-7828`): `arrow_col.fill_null(fill_value_for(spec)).to_numpy(zero_copy_only=False)` +(no Python loop, and the fill is exactly ours), with `valid = arrow_col.is_valid().to_numpy(...)` +for correctness first — optimize to `np.unpackbits(..., bitorder="little", count=n)` off the raw +validity buffer later, honoring `arrow_col.offset` and chunking. + +When extending an **existing** table from Arrow, the stored schema's `null_storage` wins; only +inferred schemas consult the policy. + +**Round-trip contract** (stated and tested): `to_arrow(from_arrow(x)).equals(x)` exactly, for +nullable `bool`; `int8`/`uint8` using **all 256 values** plus nulls; `float64` containing `nan`, +`±inf`, `0.0`, `-0.0` **as values** plus separate nulls; `utf8` containing `""`, `"\x00"`, +`"__BLOSC2_NULL__"`, 4-byte UTF-8 plus nulls; `timestamp` with `int64.min` as a value plus +separate nulls; `string(max_length=4)`/`bytes(max_length=4)` fully occupying the width. **None of +these round-trip under sentinels.** Same list for Parquet. + +> **Correction (2026-08-08): `.equals()` cannot express this contract.** `pyarrow.Array.equals` +> compares floats with IEEE semantics, so two *identical* arrays containing NaN compare unequal — +> the float case of the list above fails by construction, whatever the implementation does. The +> tests use `assert_same_logical` instead, which compares what is actually observable: the validity +> bitmap, and the values under valid rows, with NaN equal to NaN and signed zeros kept distinct. +> Values under `valid=False` are deliberately **not** compared — the fill is explicitly not part of +> the format contract (decision 5), so asserting on it would pin something the design says may +> change. +> +> **The "none of these round-trip under sentinels" claim is understated, and now measured.** The +> sentinel path does not fail — it *silently returns different data*. `pa.array([-128, None, 127], +> int8)` imports and re-exports as `[None, None, 127]`, because `-128` is the sentinel `int8` +> picks; `["", "__BLOSC2_NULL__", None]` comes back as `["", None, None]`, because that literal is +> the utf8 sentinel. Both are pinned side by side against the mask result in +> `test_sentinel_storage_is_lossy_where_mask_storage_is_not`, which is the single most direct +> statement of why this project exists. + +### Reductions and summary indexes + +Mechanical: `_ndarray_values_for_reduction` (`:2843-2867`) and `argmin`/`argmax` (`:3218-3271`) +swap `_null_mask_for` for `channel.null_mask`. + +**`_summary_minmax_source` (`:3004-3068`) is NOT fixed by masks** — verified. The bail at `:3046` +(`if nullable and not is_nan_float`) exists because a non-NaN sentinel pollutes per-block extrema, +and the **fill value pollutes them identically**: the summary builder reads the physical array and +never consults a side channel. Two honest routes: + +1. *Free, partial*: with the NaN float fill (decision 5), mask-backed float columns qualify under + the existing `is_nan_float` escape hatch at `:3045` with a one-condition change. Floats only — + `int64.min` **is** the block minimum, so timestamps get nothing free. + + > **Correction (measured 2026-08-08). Route 1 is unsound and was not taken.** It is defeated by + > decision 6, three sections up: the summary builder drops NaNs, and under mask storage a NaN is + > a *value*, so it drops real data too. On `[1.0, nan, 5.0, null, 3.0]` the scan gives `nan` + > (NumPy semantics, NaN participates) while the summaries would answer `1.0`/`5.0` — the same + > query returning different answers depending on whether an index happens to exist. Contrast the + > sentinel-NaN column the hatch was written for, where NaN *is* the null, so dropping it is + > exactly right and the two paths agree. Mask columns keep the bail; the reasoning is now a + > comment at the bail site so nobody re-derives the one-liner. Both halves pinned by + > `test_summary_minmax_shortcut_stays_disabled_for_mask_columns` and + > `test_sentinel_nan_float_keeps_its_summary_shortcut`. +2. *Real fix (Phase 10)*: make the summary builder in `ctable_indexing.py` mask-aware — extrema over + `values[valid]`, a per-segment `all_null` flag, `"null_aware": true` in the descriptor. This + retroactively enables the fast path for **sentinel** columns too. + +Until (2) lands, mask columns take the same bail as sentinel columns. **Do not ship a fast path +that is silently wrong.** + +> **As built (2026-08-08, Phase 10).** Route 2, and it lands in `indexing.py` rather than +> `ctable_indexing.py` — the summary builder lives there; what `ctable_indexing.py` contributes is +> the validity channel to build with. Five notes: +> +> - **The builder takes a callable, not an array.** `validity(values, start, stop) -> valid | None` +> is threaded through `_build_levels_descriptor{,_ooc}` and called per chunk. That shape is what +> lets a *sentinel* column answer for free from the values the builder already decompressed +> (`~sentinel_mask(values, nv)`) while a mask column reads one byte per row off its sidecar — +> the same split every other phase made, arriving here unchanged. +> - **`FLAG_ALL_NULL` is set together with `FLAG_ALL_NAN`, deliberately.** The established meaning +> of `FLAG_ALL_NAN` at every consumer is "the extrema are placeholders, skip this segment", which +> is exactly what an all-null segment needs; riding along with it means the pruning path +> (`_candidate_units_from_summary`) and the min/max path both got it right with no edit. +> - **The prediction that this is not a mask feature is confirmed, and pinned by a test that was +> already there.** `test_minmax_matches_reference`'s `k` column — an `INT64_MIN`-sentinel `int64` +> — was the suite's canonical *fallback* case, on the grounds that the sentinel **is** the block +> minimum. It takes the shortcut now, and the parametrization flipped from `False` to `True`. +> - **`FLAG_HAS_NAN` computed over the valid rows is what makes decision 6 expressible.** Phase 5 +> disabled the shortcut for the whole mask-float column because a NaN fill and a NaN value were +> indistinguishable to the summary. They are distinguishable to a *null-aware* summary: the flag +> marks only a NaN among valid rows, so a mask float column qualifies as a source and declines +> only when it actually holds a NaN — where the scan poisons to NaN and the summaries must not +> contradict it. `test_summary_minmax_shortcut_stays_disabled_for_mask_columns` was rewritten in +> place as `test_a_genuine_nan_still_keeps_a_mask_float_column_off_the_shortcut`, with its +> converse beside it. +> - **Measured: 236.89x** for `min()` on a 20M-row nullable `int64` (39.32 ms → 0.17 ms), which is +> the same order as the ~240x the non-nullable path already claimed. The cost is on the build +> side: the per-block summaries folded incrementally during writes carry no validity, so a +> nullable column holding a null cannot use them and pays one decompression pass at `close()` +> (1.8 ms → 33.2 ms for that column). A nullable column with **no** nulls keeps the fast path +> untouched — it needs no validity provider, and is marked `null_aware` anyway. Threading +> validity into `_ColumnSummaryAccumulator` is the named follow-up; it was left out because it +> reaches into `extend`'s write loop and one of its two feed sites (the Arrow writer's +> `on_write=` callback) has no validity to give. + +### Sort and indexes + +`_build_lex_keys` (`:11653-11730`): the null-indicator key becomes `(~valid[live_pos]).astype(np.intp)` — +cheaper than a sentinel compare, and no string comparison for `U`/`S`. Semantics unchanged. + +`_sorted_positions_from_full_index` (`:11519-11651`): line `:11633` currently reads the **entire raw +column** just to compute `null_phys`; with a mask that becomes `~np.asarray(mask[:])`, an 8×–64× +I/O reduction on exactly the path whose comment at `:11631` apologizes for its temporaries. +**Highest value-per-line change in the plan.** + +`_utf8_rank_arrays` (`ctable_indexing.py:99-143`) — verified landmine. Line `:120` is +`is_null = uniques == null_value`; under masks there's no sentinel in the vocabulary and the `""` +fill factorizes as an ordinary entry with **rank 0, so nulls would sort first and collide with +genuine empty strings**. New signature `_utf8_rank_arrays(col, n_phys, null_value=None, *, valid=None)` +with `ranks[~valid] = null_rank` after the `code_to_rank[codes]` gather, and `"null_aware": True` +in the returned meta. Comment the hazard in place. `_DictRankWrapper` (`:167-209`) is unchanged. + +Index descriptors gain `{"null_aware": true, "null_order": "last"}` (anticipated by +`plans/ctable-nulls.md:614-623`, present in neither `plans/ctable-indexes-opsi.md` nor the code). +Read with `.get("null_aware", False)`; bump the build token so stale indexes rebuild. + +> **As built (2026-08-08, Phase 10): `null_aware` yes, `null_order` no, and no token bump.** +> `null_order` would have recorded something untrue. No index kind *reorders* nulls: a FULL index +> sorts them wherever their sentinel or fill lands, which is why `_sorted_slice_positions` bails +> (Phase 7) rather than locating a null run. The nulls-last contract lives in `_build_lex_keys`, +> not in a descriptor. Recording `"null_order": "last"` beside it would have read as a promise the +> index does not keep — see the follow-up on a null-run FULL index below for what would earn it. +> +> The token bump is not needed either, and skipping it is the safer choice: `.get("null_aware", +> False)` already makes an older index take the old bail, so it is *correct* rather than stale, and +> a bump would rebuild every stored index in the wild to buy a shortcut. This is the same +> distinction Phase 7 drew for `_utf8_rank_arrays`, whose pre-mask indexes were genuinely *wrong* +> and did have to be invalidated. `rebuild_index()` promotes an old index on request. + +> **As built (2026-08-08).** All four items, plus one more site and two pre-existing sort bugs that +> only mask storage can reach. +> +> - **`_build_lex_keys`: the indicator key is not a refinement here, it is the whole of nulls-last.** +> For a sentinel the value key already groups the nulls together and the indicator only decides +> *where* that group goes; for a mask column the fill sorts wherever an ordinary `0` or `""` would, +> so without the key nulls came out **first** ascending and last descending — the contract exactly +> inverted, and silently. `valid[live_pos]` is one byte per row and needs no string comparison for +> `U`/`S`. +> - **`_sorted_positions_from_full_index`: real, and smaller than advertised.** The three +> partition branches (dict rank / mask / sentinel) collapsed into one shared body. Measured on 2M +> rows: **3.0x** faster for a `U16` string column (17.1 ms → 5.7 ms) but only **1.2x** for `int64` +> (8.5 → 6.9 ms). The 8x–64x is real as *bytes read*; both arrays compress well, so wall time +> tracks it only where the itemsize gap is wide. Still the best line-for-line change in the phase, +> just not by the margin the paragraph above claims. +> - **`_sorted_slice_positions` had to be found, and it bails.** Not in this section's list. The +> window read locates the null block by bisecting the *sorted values* sidecar for the null's stored +> value; under mask storage that value is the fill, which genuine rows share, and the validity +> sidecar is indexed by *physical* position where the window is indexed by sorted position — so the +> window cannot tell them apart at all. It declines for a mask column that has a sidecar and falls +> back to the full sorted view, which is now mask-aware. A null-free mask column keeps the window +> path: there is no null block to locate. +> - **`_utf8_rank_arrays` is the landmine this section promised, and the staleness rule matters more +> than the fix.** The fix is three lines (`ranks[~valid] = null_rank`, after the gather — the fill +> is a legitimate vocabulary entry other rows may share, so it has to be per row). The subtlety is +> that an index built *before* those three lines is wrong and **neither O(1) staleness signal can +> see it**: the column's row count and byte size are exactly what they were. `null_aware` absent +> from the meta is what marks it, and only for a column that has a sidecar — a sentinel column's +> nulls have always carried `null_rank`, so invalidating those would rebuild every stored utf8 +> index in the wild for nothing. +> - **Two pre-existing sort bugs, neither about nulls, both newly reachable.** The descending value +> key is built by negating the values, which breaks on the two dtypes a *nullable* column could not +> previously be: `bool` has no unary minus (`TypeError`, and this fires for a plain non-nullable +> bool column today), and a narrow signed dtype wraps on its own minimum — `-(-128)` is `-128` in +> int8 — so that row sorts as if it were the largest. A sentinel had to reserve int8's `-128` and a +> nullable bool was physically `uint8`, which is why the combination never came up. Fixed by +> negating in int64 for `b`/`u`/`i` alike. + +**The indexed-OR bail (`ctable_indexing.py:1459-1461`) is not fixed by masks either** — the problem +is that global post-filtering (`_exclude_null_positions:1498-1509`) drops rows that legitimately +match via the *other* branch. The right fix is per-leaf and **independent of storage**: add +`_rewrite_null_predicates(expr, operands)` alongside `_rewrite_dictionary_predicates` (`:12953`) +and `_rewrite_utf8_predicates` (`:13158`), rewriting each nullable comparison leaf `a > 10` into +`(a > 10) & _valid_a` (`~mask` for mask columns, `a != nv` for sentinel ones). Because this helps +sentinel columns that exist today and is independent of storage, **it is pulled forward: it is +Phase 1**, landing right after the `NullChannel` refactor and before any mask work. + +> **Correction (implemented 2026-08-08).** This paragraph originally claimed that "AND and OR both +> become correct with no post-filter", so `_exclude_null_positions` and the +> `nullable_indexed`/`nullable_needs_exclude` bookkeeping at `:1443-1457` would disappear. +> **That is wrong, and Phase 1 shipped with the post-filter retained.** An ordered index does not +> *evaluate* the predicate: it answers `a > 90` by taking a **range of the sorted column**, and the +> sentinel lies inside that range whether or not it would satisfy the comparison. Measured on 1M +> rows with a NaN sentinel: the scan matches 83 146 rows, while the FULL index returns 160 070 +> positions, 76 924 of them NaN. Making the *expression* null-aware therefore cannot make the +> *index result* correct — the post-filter is load-bearing, and the indexed-OR bail stays too. +> Genuinely indexed OR over a nullable column needs the index itself to know about nulls, which is +> Phase 10 (`null_aware`/`null_order` descriptors), not something the expression layer can deliver. +> +> What Phase 1 did land is larger than this paragraph anticipated: **string predicates were not +> null-aware at all**. Only the operator form (`Column._null_aware_compare`) forced nulls to False; +> `t.where("a > 10")` compared the raw sentinel, so any sentinel that satisfies the predicate +> (`null_value=999` against `> 10`) returned its nulls as matches — on the scan path, indexed or +> not. That is the bug the rewrite fixes, and it makes the scan fallback correct, which is what +> finally makes the *bailing* OR path return the right answer. +> +> Two refinements the original text did not have: +> +> - **Guards are emitted inline, not as an injected operand.** `(a > 90) & (a != 999)` keeps the +> guard a predicate on the same column; an opaque extra boolean operand pushed the planner off +> the index onto a full scan (measured ~13 % slower on the OR probe). +> - **A guard is emitted only when the sentinel could actually satisfy the leaf.** `-1` cannot +> match `> 10`, and NaN cannot match anything but `!=`, so those leaves are left alone. Without +> this, every nullable-column query loses its index. Never applied inside a negation: a sentinel +> that *fails* `a > 10` *passes* `not (a > 10)`. +> +> **Negation caveat, resolved.** The original text proposed either bailing on `~`/`not` or pushing +> validity to the outermost conjunction. Bailing was rejected: the index path is SQL-correct for +> negation today while the scan path is not, so bailing would have regressed the index. Validity is +> pushed to the negation point — `(~(a > 10)) & valid_a` — which is exact for every tested form and +> conservative in one three-valued corner: `not (a > 10 and b == 999)` with a false second term +> makes SQL's `NULL AND FALSE` collapse to `FALSE`, so the row should survive, while the guard drops +> it. Rows are only ever dropped, never wrongly returned. +> +> **Addendum (2026-08-08): the *operator* form has its own negation leak — a pre-existing bug, not +> a Phase 1 artifact.** `_null_aware_compare` collapses null → False at the comparison leaf and +> returns a plain `LazyExpr`, so `~(t.a > 10)` inverts the collapsed False and **wrongly returns +> null rows** — the failure direction the guarantee above rules out for the string form. Measured: +> `[0, 2]` where SQL and the string form both give `[0]`. It also means the operator form's SQL- +> exact answer in the three-valued corner above is accidental (plain booleans at `~`, not Kleene +> logic). Both behaviors are pinned in `tests/ctable/test_null_predicate_rewrite.py`: +> `test_negation_over_and_corner_is_conservative` (the intentional string/operator divergence) and +> a `strict=True` xfail, `test_operator_form_negation_drops_nulls` (the leak). A real fix needs +> comparison results to carry their null predicate through `~` — a boolean analogue of +> `NullableExpr` with `__invert__` — and folds naturally into decision 8's deferred Kleene +> follow-up rather than Phase 1. +> +> **Addendum 2 (2026-08-08): Phase 1 only did the sentinel half, and Phase 6 shipped the gap.** +> `_rewrite_null_predicates` tested `kind_of_spec(spec) != NULL_SENTINEL` and skipped everything +> else, so a mask column got no guard at all — and the same one-kind test in +> `ctable_indexing.py:1453` left it out of `nullable_indexed`, so no post-filter either. Both were +> measured on a 2000-row table with 1 % nulls, and both leaked: +> +> - **Scan.** `t.where("a < 500")` returned **every null**, because the int fill is `0`. Same shape +> of bug Phase 1 fixed for sentinels, read the other way round: there the stored value was a +> sentinel that happened to satisfy the leaf, here it is a fill that does. +> - **Index.** `t.where("f > 0.5")` over a float column returned **every null** even after the scan +> was fixed, because an ordered index answers by taking a range of the sorted column and the NaN +> fill sorts to the end of it. This is precisely the correction above — a null-aware expression +> cannot fix an index that never evaluates the expression — arriving a second time for a second +> storage. +> +> Fixed by giving mask storage the same two mechanisms: +> +> - the guard is `valid_pred()` injected as a `__nv{i}` operand, since there is no in-band literal +> to compare against, and **the fill stands in for the sentinel in `_sentinel_can_match`** — which +> is exactly right, being what a null row actually holds. `0` cannot satisfy `a > 10` and NaN +> cannot satisfy any ordered comparison, so those leaves stay unguarded and stay on their index; +> - a mask column with a sidecar joins `nullable_indexed`, so `_exclude_null_positions` filters it — +> reading **one byte per candidate off the sidecar**, without touching the values at all, which is +> the cheaper half of what masks buy. It stays out of `nullable_needs_exclude` for the same reason +> a NaN sentinel does: that fall-back exists for the mask-direct path, which evaluates the +> (guarded) predicate through miniexpr. +> +> Two more sites the one-kind test had hidden, both utf8, both fixed in place rather than through +> the expression layer: `_utf8_scalar_mask`/`_utf8_compare_column` compared away the sentinel string +> and so let the `""` fill through, and `utf8_span_eval` derived its per-span nulls by looking for +> the sentinel in the values — it now takes a `valids=` dict alongside `sentinels=`. A **string** +> result keeps the fill rather than a sentinel: there is nothing to write back, and a bare +> `UTF8Array` has nowhere to carry a validity channel. +> +> One drive-by fix falls out of the `partial_exact_positions` refinement block, which narrowed +> `pos` column by column while trimming the prefetched primary values only for the primary column's +> own filter: with two nullable indexed columns and nulls in the non-primary one, `prefetched` came +> out **misaligned with `candidates`**. Rewritten as one combined keep-mask applied once, which is +> both correct and shorter. +> +> Verified against a NumPy SQL oracle over 32 combinations — {mask, sentinel} × {indexed, +> unindexed} × 8 expression shapes including `|` and `~` — all agreeing exactly. +> +> **Addendum 3 (2026-08-08, Phase 10): the indexed-OR bail is lifted for the path that evaluates, +> and kept for the paths that do not.** The correction above is right that a null-aware expression +> cannot fix an index which never evaluates it — but it over-generalized from there to "OR over a +> nullable indexed column must fall back to the scan". Only *some* index paths answer without +> evaluating. The segment/candidate-unit path prunes blocks by their summaries and then runs the +> predicate through miniexpr over the survivors, so its result is exact and +> `_exclude_null_positions` there was not merely wrong for OR — it was unnecessary. That path now +> serves OR with the filter skipped. The exact-position paths (FULL/PARTIAL/BUCKET), which answer +> by slicing the sorted column, still bail. +> +> Two details: +> +> - **Those three bails are unreachable today, and are kept as a guard.** `_plan_exact_conjunction` +> declines any expression containing an OR, so an exact plan cannot coexist with one. They exist +> so a planner that learns to build one cannot silently reintroduce the bug. +> - **The OR test now parses.** `"|" in expression` also fires on a string literal that contains +> one (`name == 'a|b'`), which took the index away from a query with no OR in it at all; +> `_expression_has_or` walks the AST for `ast.Or`/`ast.BitOr` instead. +> +> Measured on a 20M-row two-column probe, `(a > 4000) | (b > 6000)` with SUMMARY indexes on both: +> **1.61x** (12.35 ms → 7.65 ms). Modest, and honestly so — the miniexpr scan it was falling back +> to is already fast; the win is proportional to how much the summaries prune. + +### Groupby + +`groupby.py:_null_mask` (`:2116-2137`) is already the single central helper, but it receives an +already-gathered `values` chunk, so validity has to be gathered at each of ~8 independent read +sites: `:534`, `:583-589`, `:624-637`, `:716-718`, `:1005-1017`, `:1394`, `:1430`, `:1757`, plus +`_null_value_for` (`:2589`) and `_null_output_value` (`:2239`), which for a mask-backed output +column become "write fill + mask=False". Signature grows `*, valid=None`. **Messiest integration +after `__setitem__`** — budget accordingly. + +One semantic improvement follows from decision 6: NaN in a float *value* column is no longer +missing. Keep the `is_key` NaN coercion at `:2131-2134` for keys (dropna semantics). + +> **As built (2026-08-08).** It was not the messiest integration — that was the *key* side, which +> this section does not mention at all. +> +> - **A mask key column needs recoding, not a threaded flag.** Threading `valid=` fixes value +> columns, and that part is as described (`_null_mask` grows the kwarg; the generic path and the +> dense single-key path gather validity by the same `live_mask` as the values). But a *key* column +> has a second problem no validity flag solves: with `dropna=False` the null rows have to form a +> group of their own, and their fill is a value a genuine row may hold, so they would merge into +> the `0` group of an int key or the `""` group of a string one. The fix is to give them a reserved +> code — exactly what a dictionary column gets for free from `null_code`. `_Utf8KeyChunk` was +> already the right shape for that, so it became `_CodedKeyChunk` with a `null_code` field, and +> `_coded_chunk_with_nulls` recodes any mask key chunk into one. `uniques[null_code] is None`, so +> the null group comes out keyed **`None`** and a mask-storage output column writes it back as a +> real null — where a sentinel column can only offer its sentinel (`group_by(dropna=False)` over +> one returns a group keyed `-1`). +> - **`_null_output_value` returns `None` for a mask output spec**, which is the same point one layer +> up: a group with no non-null input no longer has to come back as `0` and hope nobody reads it as +> data. Note `sum`'s output spec is a fresh non-nullable `float64`/`int64`, so *that* aggregate +> still spells missing as `NaN` — unchanged, and the same for both storages. +> - **The fast paths bail, but not all of them, and the difference is 4.5x.** Every path reads +> nullity out of the values; a Cython kernel is handed a `skip_nan` flag, not a validity array, so +> the four Cython paths defer to the generic path whenever a mask column in play holds a null. The +> dense single-int-key path is plain NumPy and already routes value columns through `_null_mask`, +> so it only needed the sidecar — and keeping it matters: 2M-row `sum` grouped by an int key went +> 91.8 ms → 20.4 ms once it stayed, against 10.8 ms for the sentinel equivalent. A mask *key* +> column still has to leave it, since a `_CodedKeyChunk` is not the array of dense non-negative +> ints that path indexes with. Threading validity into the kernels is the named follow-up. +> - **One pre-existing bug, storage-independent.** `min`/`max` seed a per-group accumulator with the +> dtype's opposite identity, and `_max_identity`/`_min_identity` had no `bool` case — so +> `np.full(n, None, dtype=bool)` gave `False`, a min accumulator could never rise above it, and +> every all-`True` group reduced to `False`. Reachable today with a plain non-nullable bool column +> on any generic-path aggregation (a string key is enough); a nullable one was `uint8`, whose +> identities are fine, which is why mask storage is what surfaced it. + +### Nullable-bool cleanup + +Under masks, `bool(nullable=True)` yields physical `np.bool_` — no `uint8`, no reserved `255`. +`t.flag[:]` returns booleans; `t.where(t.flag)` works directly; export is +`pa.array(arr, mask=~valid, type=pa.bool_())`. + +**No deprecation cycle is needed, and that's the point of the design**: the change is scoped by +construction to tables created *after* the default flips. Existing tables carry `null_value: 255` +in their schema and stay uint8 forever; `blosc2.bool(nullable=True, null_value=255)` and +`null_storage="sentinel"` keep producing uint8 explicitly; `_is_nullable_bool` and its rewrite +sites stay permanently. What is needed: the two dtype-flip relocations, a +`doc/reference/ctable.rst` note, a release-note entry, and a test that opens a checked-in fixture +table with a uint8 nullable-bool column and asserts nothing changed. + +### Migration + +```python +def convert_nulls( + self, columns=None, *, to="mask", null_value=None, inplace=False +) -> CTable: + """Convert nullable columns between sentinel and validity-mask storage. + + Never called implicitly: opening, copying, and saving a table all preserve + each column's existing null storage. + """ +``` + +Per kind for `to="mask"`: a null-free sentinel column (no slot holds the sentinel) converts as +a **pure schema update** — no sidecar is written, per decision 9. Otherwise +numeric/timestamp/string/bytes convert chunkwise in one pass +(`valid = ~sentinel_mask(chunk)`; overwrite sentinel slots with the fill), dtype unchanged, +in-place possible — but `string`/`bytes` keep their widened `max_length` (shrinking is a dtype +change; document `copy()` as the way to reclaim it, don't shrink silently). `bool` needs +uint8 → `np.bool_`, i.e. a new array. `ndarray` derives the row mask under the old all-elements +rule first. `utf8` rewrites null rows to zero-length spans (effectively a column rebuild). +Dictionary and varlen/list/struct/object are **no-ops** — raise if named explicitly, skip silently +if implicit. + +`to="sentinel"` is the inverse and must **reject** what it cannot represent: full-range +`int8`/`uint8`, and utf8/string whose data already contains the proposed sentinel. Check before +writing; raise naming the offending value. + +`inplace=True` ordering **is** the crash-safety argument, so put it in the docstring: (1) write +the complete `.notnull` sidecar, (2) rewrite value slots to fill, (3) update `/_meta` schema last. +A crash after (1) or (2) leaves the schema saying `sentinel`, the orphan `.notnull` key unread, and +the table intact. `inplace=False` (default, recommended) builds a new table via `copy()`. + +Detection is `Column.null_storage` plus an `info()` column — no separate report function. + +> **Correction and as-built (2026-08-08).** The ordering above is **wrong at step 2, and the fix is +> to move it rather than to accept the window.** After (2) the table is *not* intact: the null slots +> now hold the fill, and a schema still saying `sentinel` reads a fill `0` as the value `0`. What +> shipped is **(1) sidecar, (2) schema, (3) fill** — and every intermediate state is then correct, +> not merely recoverable. A crash before (2) leaves an orphan `.notnull` key that a sentinel column +> never opens; a crash before (3) leaves a correct mask column whose null slots happen to still hold +> the old sentinel, which is unobservable through the `Column` API and which decision 5 explicitly +> excludes from the format contract. `to="sentinel"` runs the same argument backwards: sentinel into +> the null slots first (harmless while the sidecar is still authoritative), then the schema, then +> drop the sidecar. Both orderings are asserted in `test_null_migration.py`. +> +> Four further departures: +> +> - **A dtype change is refused for a persistent in-place conversion**, and this is the one real +> capability limit. `bool` (`uint8` ↔ `np.bool_`) and a `string`/`bytes` column too narrow for its +> sentinel need the stored array *replaced*, and there is no ordering of that write and the schema +> update a crash cannot land between. `inplace=False` has no such window — it builds a new +> table — so those columns raise, naming themselves and pointing at it. In-memory `inplace=True` is +> allowed: there is nothing to crash into. The check runs in `_convert_null_targets`, before any +> write, alongside the sentinel-availability checks, so a refusal never leaves litter. +> - **`copy()` shares its schema object with the source**, which conversion — the one operation that +> mutates a spec in place — cannot live with: a converted copy relabelled its *source's* columns +> too, leaving that table reporting a storage its data does not use. `_detach_schema()` deep-copies +> the specs first, and `_convert_nulls_inplace` always calls it, so `copy()` followed by an +> in-place conversion is safe as well. +> - **The all-elements ndarray rule needs no special handling.** `sentinel_mask(item_ndim=)` already +> implements it, and the sentinel direction writes the scalar sentinel across every element of a +> null row, which is the same rule read backwards. +> - **`Column.null_storage` and the `info()` rows already existed**, from Phases 0 and 4. What Phase +> 8 added is the *table-level* tag (`int64 nullable[mask]` in `info`'s per-column summary), which is +> what you actually read to decide whether a table needs converting. +> +> Also fixed here, storage-independent and pre-existing: `copy()` recorded its write watermark as +> `n - 1` where `_resolve_last_pos()` and every other writer mean an exclusive bound, so +> `add_column()` on a copied table backfilled one row short — and *raised* for a variable-length +> column. And `_unflip_mask_bool_dtype` keyed off the dtype rather than off whether the flip had +> happened, so a nullable **`uint8` ndarray** column under mask storage came back as `bool_`, every +> byte truncated to a flag; the specs now record `bool_widened_to_uint8`. + +## Verification + +**Differential oracle — the single highest-value test.** New +`tests/ctable/test_null_storage_equivalence.py`: build the same logical data twice +(sentinel-backed and mask-backed), assert every public API agrees — `is_null`, `notnull`, +`null_count`, `fillna`, `dropna`, `sort_by` (both directions, single and multi-key), `group_by` +(as key and as value, `dropna=True/False`), `where` (indexed and unindexed, AND and OR), all +reductions, `argmin`/`argmax`, `unique`, `value_counts`, `to_arrow`, `to_pandas` — parametrized +over every V1 kind, with float NaN cases marked as intentionally divergent per decision 6. This +converts "mask is a second implementation" from a permanent liability into a checked invariant. + +Additions to existing suites: +- `tests/ctable/test_nullable.py` (756 → ~1150): parametrize over `null_storage`; mask-only cases + for nullable bool `dtype == np.bool_`, full-range `int8`, utf8 containing `"__BLOSC2_NULL__"`/`""`/`"\x00"`, + `string(max_length=4, nullable=True)` staying `U4` (regression against the widening at `:4538`), + fill determinism, `to_numpy(masked=True)` for both storages, `null_count` with deletions, + `is_null()` on sorted and `where()` views, `fillna` with a value equal to real data, and `None` + accepted by `append`/`extend`/`__setitem__`/`assign` for every V1 kind. Plus the decision-9 + invariant: a mask column written with no nulls has **no** `.notnull` key on disk and answers + `is_null()` all-`False`; the key appears exactly when the first null is written. And the + `NullPolicy` resolution rules: type-wide sentinel fields imply sentinel storage for the types + they cover; explicit `null_storage="mask"` combined with them raises. +- `test_null_expressions.py` (284 → ~440): mask/sentinel differential for arithmetic and comparison + propagation; ndarray-column propagation (new capability); NaN-is-a-value assertions. +- `test_arrow_interop.py` (630 → ~840) and `test_parquet_interop.py` (1323 → ~1500): the round-trip + contract list exhaustively, plus the literal-bytes bit-order test. + +New files: +- `tests/ctable/test_null_persistence.py`: mask survives `.b2d`, `.b2z`, `to_cframe`/`ctable_from_cframe`, + inline TreeStore save/load, `mmap_mode="r"`, `CTable.load()`, `delete_column`, `rename_column`, + `compact()`, `trim_capacity()`, and repeated `_grow()` cycles. +- `tests/ctable/test_null_migration.py`: `convert_nulls` per kind both directions, in-place and copy; + the null-free schema-only fast path; crash-ordering (an orphan `.notnull` key with an unmodified + schema still reads as sentinel); and the two guarantees — old tables open unchanged, `save()` + preserves storage under a mask-default policy. +- A version-gate test: hand-build a `version: 3` schema dict and assert a simulated old accept-list + `(1, 2)` raises a clear `ValueError` naming the version. +- `tests/ctable/test_null_aware_indexes.py` (Phase 10): the summaries at unit level (extrema over + the valid rows, `FLAG_ALL_NULL`, `FLAG_HAS_NAN` marking a value and not a fill); the `null_aware` + claim, including the null-free mask column that earns it without a sidecar and the older index + that must not be trusted with it; `min`/`max` against the scan for every V1 kind and both + storages, plus full-range `int8`, the straddling tail block, an all-null column, and a null + written *after* the build; and indexed `OR`, asserted to use the index rather than only to be + right. + +End-to-end smoke, run manually: import a nullable-bool + full-range-`int8` + free-text-utf8 Parquet +file, round-trip it, and assert `pq.read_table(out).equals(pq.read_table(in))` — the case that is +impossible today. Also re-run the OFF importer round-trip (`plans/ctable-nulls.md` §Tests) and +confirm the `nullable_scalar_wrapped_as_singleton_list` workaround can be deleted. + +> **Correction (2026-08-08).** The smoke test is now a real test rather than a manual one — +> `test_parquet_round_trip_is_lossless`, parametrized over the whole contract list. +> +> The `nullable_scalar_wrapped_as_singleton_list` workaround **cannot be deleted, and does not need +> to be.** `src/blosc2/cli/parquet_to_blosc2.py` stopped *producing* it before this work began — the +> conversion table at `:410-496` emits `nullable_scalar_sentinel` for that case now. The two +> surviving references (`:1405-1406`) are in the *export* path, which reads the tag out of archive +> metadata to unwrap what an older version wrote. That is a permanent backward-compatibility reader, +> in the same category as `_is_nullable_bool` and its rewrite sites: it stays forever. + +## Phasing + +Each phase is independently landable. **The default does not flip until Phase 9**, which is a +deliberate one-release-minimum lag behind Phase 6 so version-3-capable readers circulate before +default-created tables require them. + +| # | Phase | Size | Risk | +|---|---|---|---| +| 0 | ✅ **`NullChannel` refactor, sentinel-only.** New `ctable_nulls.py`; route the ~40 `getattr(spec, "null_value")` sites through it across `ctable.py`, `groupby.py`, `ctable_indexing.py`, `schema_validation.py`, `schema_vectorized.py`. Test suite must pass **unmodified**. | M | Low | +| 1 | ✅ **Per-leaf null-predicate rewrite** *(storage-independent — pulled forward because it fixes sentinel tables that exist today)*. `_rewrite_null_predicates`, guards emitted inline and only where the sentinel could satisfy the leaf; validity pushed to the negation point. **`_exclude_null_positions` and the indexed-OR bail are retained** — see the correction above; an ordered index never evaluates the predicate, so a null-aware expression cannot fix it. Real payoff: string predicates become null-aware at all. | M | Med | +| 2 | ✅ **Schema plumbing.** `_NullableSpecMixin`, `null_storage` kwarg on ~9 specs, conditional version 3, `NullPolicy.null_storage` (**still defaulting to `"sentinel"`**) with sentinel-field inference, `_resolved_null_storage` as the single decision point, `fill_value_for`, complex nullable (mask-only). Dtype-flip relocation **not** done — see the correction above. | S | Low | +| 3 | ✅ **Storage sidecar.** 5 methods × 4 backends, lazy creation (absent key = all valid); `_grow`/`trim_capacity`/`compact`/`_save_to_storage`/`to_cframe`/`load`, **plus `copy()`'s in-memory path**, which the section above had missed; companion-suffix loop in delete/rename. `tests/ctable/test_null_persistence.py` (38 tests) drives it with a hand-built mask. | M | Low | +| 4 | ✅ **Read/write + null API.** `extend`/`append`/`_coerce_row_to_storage`/`__setitem__`/`assign`; `is_null`/`notnull`/`null_count`/`fillna`/`_nonnull_chunks`/`to_numpy(masked=)`/`dropna`; **plus every gather-and-rebuild path** (`sort_by` ×3, `take`, `slice`), which this section had not listed. Mask columns fully usable. `tests/ctable/test_null_mask_api.py` (90 tests). | **L** | **High** (turned out to be the reference cycle, not `__setitem__`) | +| 5 | ✅ **Expressions + reductions.** `_ndarray_values_for_reduction`, argmin/argmax (both were reducing over the *fill*), `_reduction_null_mask` as the one storage-agnostic entry point. `_raw_null_pred`/`_lazy_nonnull_mask`/`_is_nullable_bool` needed nothing — Phases 0–4 had already made them storage-agnostic. **The ndarray-propagation gain is not real and was not done**, and the free summary fast path is unsound; both corrections are above. `tests/ctable/test_null_mask_expressions.py` (39 tests). | S (was M) | Low (was Med) | +| 6 | ✅ **Arrow/Parquet.** Import + export for all V1 kinds; `arrow_slice(valid=)`; `null_storage=` on `from_arrow`/`from_parquet`; the "no sentinel available" import error now names the way out instead of being deleted (it still fires for sentinel storage, which still cannot represent those types). No `packbits` — pyarrow's own packing is borrowed instead. Ships **opt-in**; the default stays `"sentinel"`. `tests/ctable/test_null_mask_arrow.py` (42 tests). | M | Med | +| 7 | ✅ **Sort + groupby.** `_build_lex_keys` (the indicator key is nulls-last *entirely*, not a refinement), `_sorted_positions_from_full_index` (3.0x for `U16`, 1.2x for `int64` — the I/O win is in bytes, not proportionally in time), `_utf8_rank_arrays(valid=)` plus a `null_aware` staleness rule no O(1) signal could replace, `_sorted_slice_positions` bails, groupby `_null_mask(valid=)` **plus `_CodedKeyChunk`**, which this section had not anticipated: a mask *key* column needs a reserved null code, not a threaded flag. **Plus the mask half of Phase 1**, which had never been done — `where()` leaked nulls on both the scan and the index (see §Expression layer, Addendum 2). Three pre-existing storage-independent bugs fixed on the way: descending sort of `bool` (raised) and of full-range signed ints (wrong order), and groupby `min`/`max` over `bool` (always `False`). `tests/ctable/test_null_mask_sort_groupby.py` (75 tests). | **L** (was M) | Med-High | +| 8 | ✅ **Migration + docs.** `convert_nulls` both directions for every V1 kind, refusing what a sentinel cannot represent; the crash ordering **corrected** (fill after the schema flip, not before) and asserted; a persistent in-place dtype change refused with a reason; `_detach_schema` so a converted copy stops relabelling its source; the table-level `info` null tag (`Column.null_storage` and the per-column `info` rows already existed). `doc/reference/ctable.rst` gains a "Where nulls are stored" section and a rewritten null-policy resolution order; release notes. Three storage-independent bugs fixed on the way: `copy()`'s off-by-one write watermark (which *raised* in `add_column`), and a nullable `uint8` ndarray column coming back as `bool_`. `tests/ctable/test_null_migration.py` (50 tests). | M (was S–M) | Low | +| 9 | ✅ **Default flips to `"mask"`.** Not a one-line change: `null_storage` had to become tri-state (`None` = unspecified) so that a type-wide sentinel field can still imply sentinel storage without contradicting the new default, and ~65 tests that wrote a sentinel literally to mean "null" had to say `null_storage="sentinel"` and mean it. Three real gaps the flip exposed, all fixed: **CSV import/export was sentinel-only** (`from_csv` raised on an empty field, `to_csv` wrote the fill as data), **`~` on a mask bool column selected its nulls**, and **the Arrow importer ignored the type-wide sentinel inference**, so `NullPolicy(signed_int_strategy="max")` meant one thing for a declared schema and another for an inferred one. Landed in the **same** session as Phase 6, not a release later — see the note below. | M (was S) | Med (was Low) | +| 10 | ✅ **Index null-awareness remainder** *(independent)*. Summary builder takes a per-column validity provider (one callable, both storages); `FLAG_ALL_NULL` riding on `FLAG_ALL_NAN`; `null_aware` in the descriptor, **`null_order` deliberately not recorded** and **no token bump** — see the corrections above. `_summary_minmax_source` re-enabled for mask *and* sentinel columns (**236.89x** on a 20M-row `int64` `min()`), with `FLAG_HAS_NAN` over the valid rows narrowing Phase 5's whole-column bail down to the one genuine-NaN case it was really about. Plus the **indexed-OR lift**, which this row did not anticipate: the segment path evaluates the predicate, so it never needed the global post-filter that forced the bail (1.61x on a 20M-row probe). One cost, recorded: an index over a nullable column holding nulls can no longer use the incremental write-time summaries and pays a decompression pass at `close()`. `tests/ctable/test_null_aware_indexes.py` (22 tests). | M (was L) | Med (was High) | + +The riskiest, most-coupled work is isolated into Phases 4, 7 and 10, each of which can slip +without blocking the others. Phase 9 is a policy change, not code — its only prerequisite is +that Phases 2–8 have soaked for a release. + +Phase 10 came in smaller than budgeted for the same reason Phase 5 did: Phases 0–7 had already +made every consumer read nullity through one channel, so the remaining work was to *supply* that +channel to one more builder rather than to teach a new subsystem about nulls. + +> **Deviation from decision 1 (2026-08-08), recorded deliberately.** Phase 9 landed in the same +> session as Phase 6, not a release later. The staging existed so that version-3-capable readers +> would be in circulation before default-created tables required them; shipping both at once means +> the first release carrying the flip is also the first release able to read what it writes. The +> mitigation is unchanged and was always the real safety net: the version bump is **conditional on a +> mask column existing**, so an older reader meets a clear `ValueError: Unsupported schema version 3` +> rather than misreading anything, and `null_storage="sentinel"` remains one keyword away for data +> that has to stay readable by them. Flagged to the maintainer at the time as a release-scheduling +> call rather than a technical one. + +## Named follow-ups (not blocking any phase) + +- **Validity through the Cython groupby kernels.** A mask column holding a null costs ~1.9x on a + 2M-row grouped `sum` against the sentinel equivalent, because the four Cython paths bail; the + kernels take a `skip_nan` flag where they would need a `values_valid` array. Two of them + (`groupby_hash_i64x2_f64`, `groupby_dense_int_count_checked`) already accept one, so this is + partly a matter of using what is there. +- **A mask *key* column back on the dense single-key path.** It leaves because + `_CodedKeyChunk.codes` are chunk-local, not the dense global ints that path indexes with. +- **`__setitem__`'s fast paths** for mask columns (deferred in Phase 4, still unmeasured). +- **Kleene three-valued logic**, decision 8 — which now also owns the operator-form negation leak + pinned as a `strict=True` xfail in `tests/ctable/test_null_predicate_rewrite.py`. +- **Validity through `_ColumnSummaryAccumulator`** (Phase 10). The per-block summaries folded + during writes carry no validity, so a nullable column holding a null cannot use them and pays a + decompression pass at `close()` — 1.8 ms → 33.2 ms for a 20M-row `int64`. `extend`'s feed site + already has the validity in hand (`batch_valid`); the Arrow writer's `on_write=` callback does + not, so the accumulator would need a "nullable columns must be fed validity or invalidate" rule + and that path would keep today's cost. +- **A null-run FULL index**, which is what would earn a truthful `"null_order": "last"`. Sorting + nulls last *within ties* makes them a contiguous range of the sorted array, so a range query + could subtract it in sorted space — exact, per leaf, no I/O. That would let + `_exclude_null_positions` go away entirely, put `OR` on the exact-position paths, and give + `_sorted_slice_positions` (which bails since Phase 7) its window read back. It needs the sort key + and the external-merge builder to carry validity, in every index kind. diff --git a/src/blosc2/_utf8_array.py b/src/blosc2/_utf8_array.py index e7ad86182..a0d4b4017 100644 --- a/src/blosc2/_utf8_array.py +++ b/src/blosc2/_utf8_array.py @@ -126,6 +126,19 @@ def utf8_spans(arrays: dict, n_logical: int, span_rows: int, budget: int): yield a, min(a + rows, stop) +def _span_null_mask(raw: np.ndarray, sentinel, valid, start: int, stop: int) -> np.ndarray | None: + """Which rows of one span are null, or ``None`` when the operand has no nulls. + + Whichever channel the operand uses: a sentinel found among *raw*'s values, + or a slice of a mask-storage column's validity sidecar. + """ + if sentinel is not None: + return raw == sentinel + if valid is not None: + return ~np.asarray(valid[start:stop], dtype=bool) + return None + + def utf8_span_eval( expr: str, operands: dict, @@ -133,6 +146,7 @@ def utf8_span_eval( sentinels: dict, n_phys: int, *, + valids: dict | None = None, strict: bool = False, span_rows: int = UTF8_EXPR_SPAN, budget: int = UTF8_EXPR_BUDGET, @@ -155,6 +169,14 @@ def utf8_span_eval( result therefore needs the operands to agree on one sentinel, and raises ``ValueError`` when they do not. + A **mask-storage** operand has no sentinel to find in its values -- its null + rows already hold the ``""`` fill, which other rows may legitimately share -- + so it names its validity sidecar in *valids* instead (physical, ``True`` = + not null). Either channel feeds the same per-span *nulls*, so a boolean + result is forced ``False`` for those rows just the same. A **string** result + keeps the fill: there is no sentinel to write back, and a bare + :class:`UTF8Array` has nowhere to carry a validity channel of its own. + Span operands are handed over as blosc2 arrays rather than NumPy ones: the NumPy route evaluates through ``slices_eval``, which never reaches miniexpr, so the string kernels would be bypassed for correct-looking @@ -179,18 +201,18 @@ def utf8_span_eval( nulls = None for name, arr in arrays.items(): raw = np.asarray(arr[start:stop]) - nv = sentinels[name] - if nv is not None: - mask = raw == nv - if mask.any(): - nulls = mask if nulls is None else (nulls | mask) - raw = np.where(mask, "", raw) + mask = _span_null_mask( + raw, sentinels[name], None if valids is None else valids.get(name), start, stop + ) + if mask is not None and mask.any(): + nulls = mask if nulls is None else (nulls | mask) + raw = np.where(mask, "", raw) span[name] = blosc2.asarray(raw.astype(utf8_span_dtype(raw))) res = np.asarray(blosc2.lazyexpr(expr, span).compute(**compute_kwargs)) if nulls is not None: if res.dtype.kind == "b": res = res & ~nulls - elif res.dtype.kind == "U": + elif res.dtype.kind == "U" and null_value is not None: # The sentinel may be wider than the computed values. res = np.where( nulls, @@ -452,8 +474,16 @@ def _bytes_used(self) -> int: return self._bytes_used_cache def _coerce(self, value: Any) -> str: - """Coerce *value* to ``str``, mapping ``None`` to the null sentinel.""" + """Coerce *value* to ``str``, mapping ``None`` to whatever fills a null slot. + + Under sentinel storage that is the sentinel string itself. Under mask + storage it is the zero-length fill, and what actually records the null + is the column's validity sidecar — written by the caller alongside this + value, not here. + """ if value is None: + if getattr(self._spec, "uses_mask", False): + return "" null_value = getattr(self._spec, "null_value", None) if null_value is None: raise TypeError("Column of utf8 strings is not nullable; received None.") @@ -958,14 +988,25 @@ def order_masks_span(self, value: str, a: int, b: int) -> tuple[np.ndarray, np.n gt[rows] = row_gt return lt, gt - def arrow_slice(self, pa, a: int, b: int, null_value: str | None = None): + def arrow_slice(self, pa, a: int, b: int, null_value: str | None = None, *, valid=None): """Persisted rows ``[a, b)`` as a ``pyarrow.LargeStringArray``. The storage layout (int64 offsets + UTF-8 byte blob) is exactly Arrow's ``large_string`` layout, so the array is built directly from the raw buffers with no per-row decode or Python string objects. - When *null_value* is given, rows equal to the sentinel become Arrow - nulls (matched on raw bytes, still without decoding). + + Nullity comes from whichever channel the column uses. Pass *valid* -- + a boolean array over rows ``[a, b)``, ``True`` where the row is not + null -- for mask storage; pass *null_value* for sentinel storage, where + rows equal to the sentinel become Arrow nulls, matched on raw bytes and + still without decoding. + + The validity bitmap is produced by handing the booleans to pyarrow and + taking the resulting array's data buffer, rather than packing the bits + here. Arrow packs booleans and validity bitmaps the same way (LSB + first), so this borrows pyarrow's own packing and removes any chance of + emitting the bitmap in the wrong bit order -- a bug a round-trip test + cannot catch, because import would unpack it the same wrong way. """ n = b - a offs = np.ascontiguousarray(self._offsets[a : b + 1], dtype=np.int64) @@ -974,11 +1015,17 @@ def arrow_slice(self, pa, a: int, b: int, null_value: str | None = None): data = np.ascontiguousarray(self._data[start:end]) if end > start else np.empty(0, dtype=np.uint8) validity = None null_count = 0 - if null_value is not None and n > 0: - mask = self.equal_mask_span(null_value, a, b) - null_count = int(mask.sum()) - if null_count: - validity = pa.array(~mask).buffers()[1] + if n > 0: + if valid is not None: + valid = np.ascontiguousarray(valid, dtype=np.bool_) + null_count = n - int(np.count_nonzero(valid)) + if null_count: + validity = pa.array(valid).buffers()[1] + elif null_value is not None: + mask = self.equal_mask_span(null_value, a, b) + null_count = int(mask.sum()) + if null_count: + validity = pa.array(~mask).buffers()[1] return pa.LargeStringArray.from_buffers( n, pa.py_buffer(rel), pa.py_buffer(data), validity, null_count ) diff --git a/src/blosc2/ctable.py b/src/blosc2/ctable.py index 9f0eef5b8..db347234b 100644 --- a/src/blosc2/ctable.py +++ b/src/blosc2/ctable.py @@ -34,6 +34,21 @@ import blosc2 from blosc2 import compute_chunks_blocks from blosc2.ctable_indexing import _CTableIndexingMixin +from blosc2.ctable_nulls import ( + NULL_MASK, + NULL_NONE, + NULL_SENTINEL, + NullChannel, + fill_item_for, + fill_value_for, + is_na_marker, + is_nan_sentinel, + kind_of_spec, + rewrite_null_predicates, + sentinel_guard_expr, + sentinel_mask, + split_batch_validity, +) from blosc2.ctable_storage import ( FileTableStorage, InMemoryTableStorage, @@ -116,13 +131,18 @@ def _is_arrow_binary_type(pa, pa_type) -> bool: @dataclass(frozen=True) class NullPolicy: - """Default sentinels for inferred CTable scalar nulls. + """Where a nullable CTable column keeps its nulls, and which sentinel if in band. + + Consulted whenever a schema does not say -- a bare ``nullable=True``, or a + nullable Arrow/Parquet/CSV column being inferred. Whatever it decides is + written into the resulting schema, so a stored table stays self-describing + and is never re-resolved on open. - CTable nullable scalar columns are represented with per-column sentinel - values. This policy is used when CTable has to infer those sentinels, such - as when importing nullable scalar Arrow or Parquet columns without an - explicit column-level null sentinel. The selected sentinel is stored in the - resulting CTable schema, so existing tables remain self-describing. + Since 4.11.0 the default is a **validity sidecar** (``null_storage="mask"``), + which is what makes nullability lossless. Setting any type-wide sentinel + field below asks for in-band storage for the kinds that field covers, so + existing ``NullPolicy(float_value=...)`` code keeps its sentinels; passing + ``null_storage="mask"`` alongside one is a contradiction and raises. Examples -------- @@ -151,6 +171,12 @@ class Row: ``column_null_values`` takes precedence over the type-wide defaults in the policy. This is useful when a particular column needs a sentinel that is known not to collide with its real values. + + ``bool_value`` cannot imply sentinel storage the way the other fields do: + ``255`` is the only value a nullable bool may reserve, so it is also this + field's default, and ``NullPolicy(bool_value=255)`` states the default rather + than choosing anything. A bool column that wants a sentinel says so with + ``null_storage="sentinel"`` or ``column_null_values``. """ string_value: str = "__BLOSC2_NULL__" @@ -161,6 +187,75 @@ class Row: unsigned_int_strategy: Literal["min", "max"] = "max" timestamp_value: int = int(np.iinfo(np.int64).min) column_null_values: Mapping[str, Any] = dataclass_field(default_factory=dict) + #: Where a bare ``nullable=True`` keeps its nulls. ``None`` means *not + #: specified*, which resolves to :attr:`DEFAULT_NULL_STORAGE` -- and which + #: has to stay distinct from an explicit ``"mask"``, so that setting a + #: type-wide sentinel field can imply sentinel storage for the kinds it + #: covers without contradicting anything the caller wrote. + null_storage: Literal["mask", "sentinel"] | None = None + + #: What an unspecified ``null_storage`` resolves to. ``"mask"`` since + #: 4.11.0: lossless nullability is why the sidecar exists, so it is what a + #: newly created nullable column should get. Sentinel storage stays fully + #: supported and is one kwarg away. + DEFAULT_NULL_STORAGE: ClassVar[str] = NULL_MASK + + #: Type-wide sentinel fields, paired with the spec attribute that says + #: which columns each one covers. Setting any of them is what makes a + #: policy imply sentinel storage for those types (see __post_init__). + _SENTINEL_FIELDS: ClassVar[tuple[str, ...]] = ( + "string_value", + "bytes_value", + "float_value", + "bool_value", + "signed_int_strategy", + "unsigned_int_strategy", + "timestamp_value", + ) + + def resolve_null_storage(self) -> str: + """Where a bare ``nullable=True`` puts its nulls under this policy. + + An unspecified :attr:`null_storage` resolves to + :attr:`DEFAULT_NULL_STORAGE` here rather than at construction, so that + ``NullPolicy()`` keeps meaning "whatever this version's default is" + however the policy is copied around. + """ + return self.DEFAULT_NULL_STORAGE if self.null_storage is None else self.null_storage + + def __post_init__(self): + if self.null_storage is not None and self.null_storage not in ("mask", "sentinel"): + raise ValueError(f"null_storage must be 'mask', 'sentinel' or None, got {self.null_storage!r}") + # Setting a type-wide sentinel field alongside an explicit + # null_storage="mask" is a contradiction the caller wrote down, so say + # so. Setting one *without* an explicit null_storage is not: it simply + # means "use sentinels for the types I named", which is what + # _resolve_nullable_specs does with it. Raising on that instead would + # break existing NullPolicy(float_value=...) code on the very release + # that flipped the default -- which is exactly why an unspecified + # null_storage stays None rather than being resolved here. + if self.null_storage == "mask": + explicit = [f for f in self._SENTINEL_FIELDS if self._sentinel_field_is_set(f)] + if explicit: + raise ValueError( + f"null_storage='mask' contradicts the type-wide sentinel field(s) " + f"{', '.join(explicit)}: a mask column has no sentinel to choose. " + f"Drop the sentinel field(s), or use column_null_values to force " + f"sentinel storage on specific columns." + ) + + def _sentinel_field_is_set(self, field_name: str) -> bool: + """True when *field_name* was given a non-default value. + + Used by :meth:`CTable._resolve_nullable_specs` to infer sentinel + storage for the types a field covers. + """ + current = getattr(self, field_name) + default = _POLICY_DEFAULTS[field_name] + if is_nan_sentinel(current) and is_nan_sentinel(default): + # float_value defaults to NaN, which never equals itself. + return False + return current != default def sentinel_for_arrow_type(self, pa, pa_type): """Return the default sentinel for *pa_type*, or ``None`` if unsupported.""" @@ -197,6 +292,13 @@ def sentinel_for_arrow_type(self, pa, pa_type): return None +#: The as-declared defaults, so a policy can tell "left alone" from +#: "explicitly set to the same value". Read from the dataclass fields rather +#: than duplicated, so it cannot drift from the declarations above. +_POLICY_DEFAULTS = { + f.name: f.default for f in dataclasses.fields(NullPolicy) if f.default is not dataclasses.MISSING +} + DEFAULT_NULL_POLICY = NullPolicy() _NULL_POLICY = contextvars.ContextVar("blosc2_null_policy", default=DEFAULT_NULL_POLICY) # Sentinel for set_printoptions params whose valid value includes ``None`` @@ -541,6 +643,15 @@ def __repr__(self) -> str: # --------------------------------------------------------------------------- +def _refresh_ndarray_dtype_fields(spec) -> None: + """Re-derive the dtype mirrors an :class:`NDArraySpec` caches beside ``dtype``.""" + spec.itemsize = spec.dtype.itemsize + spec.kind = spec.dtype.kind + spec.type = spec.dtype.type + spec.str = spec.dtype.str + spec.name = spec.dtype.name + + def _rank_index_row_lookup(values_path: str, positions_path: str, table, null_rank: int): """Build ``rows_for_ranks(lo, hi)`` over a rank index's sorted sidecars. @@ -1068,6 +1179,18 @@ def __init__(self, table: CTable, col_name: str, mask=None): self._col_name = col_name self._mask = mask + @property + def _nulls(self) -> NullChannel: + """This column's validity channel (see :mod:`blosc2.ctable_nulls`). + + Built fresh on each access and deliberately **not** cached: the channel + holds this ``Column``, and caching it here would close a reference + cycle that keeps the whole table alive until a gc pass. The channel is + a one-slot object and reads through to the live schema, so building one + is both cheap and always current. + """ + return NullChannel(self) + @property def _raw_col(self): cc = self._table._computed_cols.get(self._col_name) @@ -1187,6 +1310,20 @@ def _resolve_live_positions(self) -> np.ndarray: return slp return np.where(self._valid_rows[:])[0] + def _physical_index(self, key: int) -> int: + """Physical position of logical row *key*, honouring the view's order.""" + n_rows = len(self) + if key < 0: + key += n_rows + if not (0 <= key < n_rows): + raise IndexError(f"index {key} is out of bounds for column with size {n_rows}") + # A sorted view holds its positions in sorted, not physical-ascending, + # order, so the key-th entry is the answer directly. + cached = getattr(self._table, "_cached_live_positions", None) + if cached is not None and self._table.base is not None: + return int(cached[key]) + return int(_find_physical_index(self._valid_rows, key)) + def _has_identity_positions(self) -> bool: """True when logical row ``k`` maps to physical row ``k`` for every row. @@ -1417,10 +1554,13 @@ def __setitem__(self, key: int | slice | list | np.ndarray, value): # noqa: C90 if not (0 <= key < n_rows): raise IndexError(f"index {key} is out of bounds for column with size {n_rows}") pos_true = _find_physical_index(self._valid_rows, key) + value, is_valid = self._nulls.coerce_scalar(value) if self.is_ndarray: spec = self._table._schema.columns_by_name[self._col_name].spec value = CTable._coerce_ndarray_value(self._col_name, spec, value) self._raw_col[int(pos_true)] = value + if self._nulls.uses_mask and not (is_valid and self._nulls.valid_array() is None): + self._nulls.set_valid(int(pos_true), is_valid) elif isinstance(key, np.ndarray) and key.dtype == np.bool_: n_live = len(self) @@ -1430,6 +1570,7 @@ def __setitem__(self, key: int | slice | list | np.ndarray, value): # noqa: C90 ) all_pos = np.where(self._valid_rows[:])[0] phys_indices = all_pos[key] + value, valid = self._nulls.coerce_batch(value, len(phys_indices)) if self.is_list or self.is_varlen_scalar: if len(value) != len(phys_indices): raise ValueError("Length mismatch in list-column assignment") @@ -1442,6 +1583,7 @@ def __setitem__(self, key: int | slice | list | np.ndarray, value): # noqa: C90 elif isinstance(value, (list, tuple)): value = np.array(value, dtype=self._raw_col.dtype) self._raw_col[phys_indices] = value + self._assign_validity(phys_indices, valid) elif isinstance(key, (slice, list, tuple, np.ndarray)): # Fast path: slice of a blosc2.NDArray into a scalar or ndarray column when @@ -1456,6 +1598,11 @@ def __setitem__(self, key: int | slice | list | np.ndarray, value): # noqa: C90 and isinstance(value, blosc2.NDArray) and not self.is_list and not self.is_varlen_scalar + # Mask columns take the unified path below, which writes values + # and validity from one explicit phys_indices. Threading the + # sidecar through all the fast paths as well is a follow-up; + # correctness first. + and not self._nulls.uses_mask and _tbl.base is None and _tbl._resolve_last_pos() == _tbl._n_rows ): @@ -1496,6 +1643,7 @@ def _coerce(v, n): else: phys_indices = np.array([real_pos[i] for i in key], dtype=np.int64) + value, valid = self._nulls.coerce_batch(value, len(phys_indices)) if self.is_list or self.is_varlen_scalar: if len(value) != len(phys_indices): raise ValueError("Length mismatch in list-column assignment") @@ -1518,6 +1666,7 @@ def _coerce(v, n): self._raw_col[phys_indices[c:c_end]] = chunk else: self._raw_col[phys_indices] = value + self._assign_validity(phys_indices, valid) else: raise TypeError(f"Invalid index type: {type(key)}") @@ -1680,6 +1829,23 @@ def info(self) -> _CTableInfoReporter: """ return _CTableInfoReporter(self) + def _null_info_items(self, spec) -> list[tuple[str, object]]: + """The nullability rows of :attr:`info_items`. + + Beyond the plain flag, a nullable column reports *where* it keeps its + nulls, and a mask column also reports whether it has actually had to + write a sidecar yet — the visible difference between "nullable" and + "has ever held a null". + """ + nullable = self.null_value is not None or getattr(spec, "nullable", False) + items: list[tuple[str, object]] = [("nullable", nullable)] + if nullable: + channel = self._nulls + items.append(("null_storage", channel.kind)) + if channel.uses_mask: + items.append(("null_sidecar", channel.valid_array() is not None)) + return items + @property def info_items(self) -> list[tuple[str, object]]: """Structured summary items used by :attr:`info`.""" @@ -1735,7 +1901,7 @@ def info_items(self) -> list[tuple[str, object]]: if cratio is not None: items.append(("cratio", f"{cratio:.2f}x")) - items.append(("nullable", self.null_value is not None or getattr(spec, "nullable", False))) + items.extend(self._null_info_items(spec)) if self.is_dictionary: items.append(("dictionary_size", len(raw.dictionary))) @@ -1823,11 +1989,17 @@ def _unwrap_operand(other): @property def _is_nullable_bool(self) -> bool: - col = self._table._schema.columns_by_name.get(self._col_name) + """True for a bool column whose nulls are the in-band ``255`` sentinel. + + Such a column is physically ``uint8``, so predicates over it must be + rewritten (``flag == True`` -> ``raw == 1``) to keep the sentinel from + reading as truthy. + """ + spec = self._nulls.spec return ( - col is not None - and col.spec.to_metadata_dict().get("kind") == "bool" - and getattr(col.spec, "null_value", None) is not None + spec is not None + and spec.to_metadata_dict().get("kind") == "bool" + and self._nulls.kind == NULL_SENTINEL ) @property @@ -1862,23 +2034,12 @@ def _coerce_timestamp_operand(self, other): def _raw_null_pred(self): """Boolean lazy predicate over the raw physical array, True where the - value is this column's null sentinel. + value is null. - Returns ``None`` when there is nothing to propagate: no ``null_value`` - configured, or a fixed-shape ndarray column (whose per-item sentinel - mask does not align 1:1 with the row-level predicates built here; - see ``Column.is_null()`` for those instead). Dictionary and - variable-length scalar columns never reach here because - ``_ensure_queryable`` already rejects them for arithmetic/comparisons. + Returns ``None`` when there is nothing to propagate; see + :meth:`NullChannel.null_pred` for exactly when. """ - if self.is_ndarray: - return None - nv = self.null_value - if nv is None: - return None - if isinstance(nv, (float, np.floating)) and np.isnan(nv): - return blosc2.isnan(self._raw_col) - return self._raw_col == nv + return self._nulls.null_pred() def _combined_null_pred(self, other): """OR of self's and other's raw null predicates; ``None`` if neither @@ -2016,7 +2177,18 @@ def __invert__(self): self._ensure_queryable() if self._is_nullable_bool: return self._raw_col == 0 - return ~self._raw_col + inverted = ~self._raw_col + if self.dtype == np.dtype(np.bool_): + valid = self._nulls.valid_pred() + if valid is not None: + # A mask-backed nullable bool holds the ``False`` fill in its + # null rows, and ``~False`` is True -- so without this, negating + # a nullable flag *selects* its nulls. SQL WHERE semantics: a + # null satisfies neither the predicate nor its negation. The + # sentinel branch above gets this from ``== 0``, which excludes + # the reserved 255 as well as the true rows. + return inverted & valid + return inverted def __lt__(self, other): if self.is_utf8: @@ -2116,6 +2288,11 @@ def _utf8_compare_column(self, numpy_op, other: Column): other_arr = other._raw_col nv = self.null_value other_nv = other.null_value + # Mask storage keeps nullity out of the values, so the side channel is + # the only thing that can exclude a null here; the fill it left behind + # ("" for utf8) is an ordinary string that compares like any other. + valid = self._nulls.valid_array() + other_valid = other._nulls.valid_array() def fn(chunk, start, stop): rhs = other_arr[start:stop] @@ -2124,6 +2301,10 @@ def fn(chunk, start, stop): res &= chunk != nv if other_nv is not None: res &= rhs != other_nv + if valid is not None: + res &= np.asarray(valid[start:stop], dtype=bool) + if other_valid is not None: + res &= np.asarray(other_valid[start:stop], dtype=bool) return res raw = self._utf8_chunked_bool(fn) @@ -2139,6 +2320,11 @@ def _utf8_scalar_mask(self, numpy_op, value: str) -> np.ndarray: see :meth:`_utf8_compare_scalar` for that. """ nv = self.null_value + # Under mask storage there is no sentinel string to compare away: a null + # row holds the empty-string fill, which is a perfectly ordinary value + # here (and one other rows may legitimately share). The sidecar is the + # only thing that can tell the two apart. + valid = self._nulls.valid_array() indexed = self._utf8_index_mask(numpy_op, value) if indexed is not None: @@ -2152,6 +2338,8 @@ def fn(arr, start, stop): res = ~res if nv is not None: res &= ~arr.equal_mask_span(nv, start, stop) + if valid is not None: + res &= np.asarray(valid[start:stop], dtype=bool) return res else: @@ -2167,6 +2355,8 @@ def fn(arr, start, stop): res = ~lt if nv is not None: res = res & ~arr.equal_mask_span(nv, start, stop) + if valid is not None: + res = res & np.asarray(valid[start:stop], dtype=bool) return res return self._utf8_chunked_bytes(fn) @@ -2318,6 +2508,19 @@ def _dictionary_eq(self, other, *, negate: bool = False): def isin(self, values) -> np.ndarray: """Return a boolean array True where the live value is in *values*. + A **null row matches nothing**, whichever way this column stores its + nulls. Testing membership against the raw values would instead match + whatever stands in for a null -- the fill under mask storage, the + sentinel under a sentinel one -- and those are not the row's value: + the fill is explicitly not part of the format contract, and a sentinel + is a reserved value the column promises never to mean literally. + + To select the nulls, put ``None`` in *values*: it matches exactly the + rows :meth:`is_null` reports, and nothing else. ``pandas.NA`` and a + ``NaT`` are accepted as spellings of the same request. A float + ``NaN`` is **not** one of them -- under mask storage NaN is an ordinary + value (decision 6 of the mask-nulls design), so it is matched as data. + For dictionary columns this performs efficient integer-code membership testing (no decoding of all values). Values absent from the dictionary are treated as not-present. @@ -2326,12 +2529,21 @@ def isin(self, values) -> np.ndarray: membership in a set. """ if self.is_dictionary: + # Its own path already answers None with the reserved null code. return self._dictionary_isin(values) + values = list(values) + wants_null = any(is_na_marker(v) for v in values) + test_set = {v for v in values if not is_na_marker(v)} + live_values = self[:] - test_set = set(values) if isinstance(live_values, np.ndarray): - return np.array([v in test_set for v in live_values.tolist()], dtype=bool) - return np.array([v in test_set for v in live_values], dtype=bool) + live_values = live_values.tolist() + found = np.array([v in test_set for v in live_values], dtype=bool) + + nulls = self._nulls.null_mask() + if nulls.any(): + found[nulls] = wants_null + return found def _dictionary_isin(self, values) -> np.ndarray: """Return a boolean array for in-membership tests against a dictionary column.""" @@ -2547,6 +2759,9 @@ def assign(self, data) -> None: root._mark_all_indexes_stale() return n_live = len(self) + # Split nullity off before the astype below, which has no way to + # represent a None in a typed array. + data, valid = self._nulls.coerce_batch(data, n_live) arr = np.asarray(data) if len(arr) != n_live: raise ValueError(f"assign() requires {n_live} values (live rows), got {len(arr)}.") @@ -2556,10 +2771,31 @@ def assign(self, data) -> None: raise TypeError(f"Cannot coerce data to column dtype {self.dtype!r}: {exc}") from exc live_pos = np.where(self._valid_rows[:])[0] self._raw_col[live_pos] = arr + # assign() replaces every live value, so validity is replaced wholesale + # too: rows this batch did not mark null must come back valid even if a + # previous assign had nulled them. + self._assign_validity(live_pos, valid) root = self._table._root_table root._mark_generated_columns_stale(self._col_name) root._mark_all_indexes_stale() + def _assign_validity(self, positions: np.ndarray, valid: np.ndarray | None) -> None: + """Overwrite the validity of *positions* after a wholesale value write. + + *valid* is what ``coerce_batch`` returned: ``None`` when nothing in the + batch was null. That still has to be *written* here rather than + skipped, because the rows may have been null before -- unlike an + append, an assign replaces what was there. A column with no sidecar is + the one exception: it is already all-valid. + """ + if not self._nulls.uses_mask: + return + if valid is None: + if self._nulls.valid_array() is None: + return + valid = True + self._nulls.set_valid(positions, valid) + def _assign_varlen_scalar(self, data) -> None: """``assign()`` for utf8/vlstring/vlbytes/struct/object columns. @@ -2568,8 +2804,9 @@ def _assign_varlen_scalar(self, data) -> None: varlen column rewrites its whole batch, so the loop would be quadratic. Dead slots keep their current contents. """ - values = list(data) n_live = len(self) + cells, valid = self._nulls.coerce_batch(list(data), n_live) + values = list(cells) if len(values) != n_live: raise ValueError(f"assign() requires {n_live} values (live rows), got {len(values)}.") raw = self._raw_col @@ -2577,12 +2814,14 @@ def _assign_varlen_scalar(self, data) -> None: n_phys = len(raw) if n_live == n_phys: raw.set_all(values) + self._assign_validity(np.arange(n_phys, dtype=np.intp), valid) return current = list(raw[:]) live_pos = np.flatnonzero(self._valid_rows[:n_phys]) for pos, value in zip(live_pos, values, strict=True): current[int(pos)] = value raw.set_all(current) + self._assign_validity(live_pos, valid) # ------------------------------------------------------------------ # Null sentinel support @@ -2591,10 +2830,19 @@ def _assign_varlen_scalar(self, data) -> None: @property def null_value(self): """The sentinel value that represents NULL for this column, or ``None``.""" - col_info = self._table._schema.columns_by_name.get(self._col_name) - if col_info is None: - return None - return getattr(col_info.spec, "null_value", None) + return self._nulls.sentinel + + @property + def null_storage(self) -> str: + """How this column represents its nulls. + + One of ``"none"``, ``"mask"`` (a ``.notnull`` sidecar validity array, + Arrow's model), ``"sentinel"`` (a reserved in-band value), ``"code"`` + (dictionary) or ``"native"`` (variable-length containers holding ``None`` + cells). The first two are the ones :meth:`CTable.convert_nulls` moves + between; the last two are the only representation their kind has. + """ + return self._nulls.kind def _null_mask_for(self, arr: np.ndarray) -> np.ndarray: """Return a bool array True where *arr* contains the null sentinel. @@ -2602,94 +2850,75 @@ def _null_mask_for(self, arr: np.ndarray) -> np.ndarray: Always returns an array of the same length as *arr*; all False when no null_value is configured. """ - nv = self.null_value - if nv is None: - return np.zeros(len(arr), dtype=np.bool_) - arr = np.asarray(arr) - if self.is_ndarray: - if arr.ndim <= self.item_ndim: - arr = arr.reshape((1, *arr.shape)) - if isinstance(nv, (float, np.floating)) and np.isnan(nv): - elem_mask = np.isnan(arr) - else: - elem_mask = arr == nv - inner_axes = tuple(range(1, elem_mask.ndim)) - return elem_mask.all(axis=inner_axes) if inner_axes else elem_mask.astype(np.bool_) - if np.issubdtype(arr.dtype, np.datetime64): - # Timestamp columns materialize with the int64 sentinel already - # decoded into np.datetime64('NaT') (they share the same bit - # pattern), so the sentinel value itself never appears in arr. - return np.isnat(arr) - if isinstance(nv, (float, np.floating)) and np.isnan(nv): - return np.isnan(arr) - return arr == nv + return self._nulls.mask_for_values(arr) def is_null(self) -> np.ndarray: - """Return a boolean array True where the live value is the null sentinel. + """Return a boolean array True where the live value is null. For varlen scalar columns (vlstring/vlbytes) nullability is represented as native ``None`` values, so this returns True wherever the value is ``None``. For dictionary columns, returns True where the code equals the null_code (``-1`` by default). """ - if self.is_dictionary: - return self._dictionary_eq(None) - if self.is_varlen_scalar and not self.is_utf8: - return np.array([v is None for v in self], dtype=np.bool_) - return self._null_mask_for(self[:]) + return self._nulls.null_mask() def notnull(self) -> np.ndarray: - """Return a boolean array True where the live value is *not* the null sentinel.""" + """Return a boolean array True where the live value is *not* null.""" return ~self.is_null() def null_count(self) -> int: - """Return the number of live rows whose value equals the null sentinel. + """Return the number of live rows whose value is null. - Returns ``0`` in O(1) if no ``null_value`` is configured for this column - and the column is not a varlen scalar column. + Returns ``0`` in O(1) if this column has no null channel at all. """ - if self.is_dictionary: - return int(self.is_null().sum()) - if self.is_varlen_scalar and not self.is_utf8: - return sum(1 for v in self if v is None) - if self.null_value is None: - return 0 - return int(self.is_null().sum()) + return self._nulls.null_count() def fillna(self, value): - """Return live values with null sentinels replaced by *value*. + """Return live values with nulls replaced by *value*. Dictionary and variable-length scalar columns (whose nulls are native ``None`` cells) return a list; other columns return a NumPy array. + + Under mask storage this is correct even when *value* equals data the + column really holds, because which rows are null is read from the + sidecar rather than inferred from the values. A sentinel column + cannot make that distinction, by construction. """ if (self.is_dictionary or self.is_varlen_scalar) and not self.is_utf8: return [value if v is None else v for v in self[:]] arr = np.array(self[:], copy=True) - if self.null_value is not None: - arr[self._null_mask_for(arr)] = value + kind = self._nulls.kind + if kind == NULL_SENTINEL: + arr[self._nulls.mask_for_values(arr)] = value + elif kind == NULL_MASK: + arr[self._nulls.null_mask()] = value return arr + def to_numpy(self, *, masked: bool = False): + """Return this column's live values as a NumPy array. + + Parameters + ---------- + masked: + When ``True``, return a :class:`numpy.ma.MaskedArray` whose mask + marks the null rows. Works for **both** null storages -- a + sentinel column derives the mask from its sentinel -- so callers + get one uniform way to ask for values-plus-validity regardless of + how the column happens to store its nulls. + """ + arr = np.asarray(self[:]) + if not masked: + return arr + return np.ma.MaskedArray(arr, mask=self.is_null()) + def _nonnull_chunks(self): """Yield chunks of live, non-null values. - Each yielded array has the null sentinel values removed. If no - null_value is configured this behaves identically to - :meth:`iter_chunks`. + Each yielded array has the null values removed. If this column has no + null channel this behaves identically to :meth:`iter_chunks`. """ - nv = self.null_value - if nv is None: - yield from self.iter_chunks() - return - is_nan_nv = isinstance(nv, float) and np.isnan(nv) - for chunk in self.iter_chunks(): - if is_nan_nv: - mask = ~np.isnan(chunk) - else: - mask = chunk != nv - filtered = chunk[mask] - if len(filtered) > 0: - yield filtered + yield from self._nulls.nonnull_chunks() def unique(self) -> np.ndarray: """Return sorted array of unique live, non-null values. @@ -2791,12 +3020,8 @@ def _lazy_nonnull_mask(self, where=None): mask = None if all_rows_visible else self._lazy_valid_rows() if where is not None: mask = where if mask is None else mask & where - nv = self.null_value - if nv is not None: - if isinstance(nv, (float, np.floating)) and np.isnan(nv): - nonnull = ~blosc2.isnan(raw) - else: - nonnull = raw != nv + nonnull = self._nulls.valid_pred() + if nonnull is not None: mask = nonnull if mask is None else mask & nonnull return mask @@ -2840,9 +3065,25 @@ def _sum_lazy_fastpath(self, acc_dtype, where=None, *, jit=None, jit_backend=Non except Exception: return NotImplemented + def _reduction_null_mask(self, arr: np.ndarray) -> np.ndarray | None: + """Per-live-row null flags for already-materialized values *arr*. + + ``None`` when this column can hold no nulls at all, which lets the + caller skip the filtering entirely. A sentinel column answers from + *arr* itself; a mask column reads its sidecar, which is why this cannot + simply be :meth:`_null_mask_for`. Both answers are one flag per row, + including for a fixed-shape ndarray column. + """ + channel = self._nulls + if channel.uses_mask: + return channel.null_mask() + if self.null_value is None: + return None + return channel.mask_for_values(arr) + def _ndarray_values_for_reduction(self, where=None) -> np.ndarray: arr = np.asarray(self[:]) - null_mask = self._null_mask_for(arr) if self.null_value is not None else None + null_mask = self._reduction_null_mask(arr) if null_mask is not None and null_mask.any(): arr = arr[~null_mask] if where is None: @@ -3002,15 +3243,16 @@ def _lazy_aggregate_fastpath(self, op: str, *, where=None, dtype=None, ddof: int return NotImplemented def _summary_minmax_source(self): - """Return ``(sidecar_path, dtype, nullable, segment_len)`` for a - summary-readable ``min``/``max``, or ``None`` when the index shortcut is - not provably correct. + """Return ``(sidecar_path, dtype, nullable, nan_is_data, segment_len)`` + for a summary-readable ``min``/``max``, or ``None`` when the index + shortcut is not provably correct. Excluded: a view (its summary describes the base table); a column kind - without numeric/string block extrema; a leaky null sentinel (only a - non-nullable column, or a NaN-sentinel float — whose NaNs the summary - drops — match the nulls-skipped contract of ``min()``); and a stale, - absent, or in-memory-only index. + without numeric/string block extrema; a nullable column whose index was + built without a validity channel — its per-segment extrema cover the + sentinel or the fill as if it were data, and only a NaN-sentinel float + (whose nulls the summary drops as NaNs anyway) escapes that; and a + stale, absent, or in-memory-only index. Appends mark the index stale, so they are covered. Deletions are *not* (``delete()`` tombstones in place and leaves the index usable for @@ -3042,13 +3284,27 @@ def _summary_minmax_source(self): spec = col.spec if col is not None else None nullable = getattr(spec, "nullable", False) null_value = getattr(spec, "null_value", None) - is_nan_float = dtype.kind == "f" and isinstance(null_value, float) and np.isnan(null_value) - if nullable and not is_nan_float: - return None # non-NaN sentinel leaks into the block extrema + is_nan_float = dtype.kind == "f" and is_nan_sentinel(null_value) root = table._root_table desc = root._get_index_catalog().get(self._col_name) if not desc or desc.get("stale", False): return None + if nullable and not is_nan_float and not desc.get("null_aware", False): + # Both a non-NaN sentinel and a mask column's fill leak into the + # block extrema of an index built without a validity channel — the + # summary builder reads the physical array and cannot tell either + # from data. ``null_aware`` says the extrema were taken over the + # valid rows only, which is what makes them agree with a scan that + # skips nulls. Indexes built before Phase 10 carry no such key, so + # they keep the old bail and rebuild into the shortcut. + return None + # Whether a NaN in this column is a *value* rather than a null. It is + # for a non-nullable float and for a mask column (Arrow semantics, + # decision 6 of plans/mask-based-nulls.md), and it is not for a + # NaN-sentinel float. Where NaN is data it poisons a scanned + # min()/max() to NaN while the summaries drop it, so the caller must + # decline rather than answer with a real extremum. + nan_is_data = dtype.kind == "f" and not is_nan_float # A tombstoned row still sits in its block and still contributes to that # block's extrema, and the summaries index physical slots while min() # reads logical rows. Both only line up while every slot below the @@ -3065,7 +3321,38 @@ def _summary_minmax_source(self): segment_len = levels[level].get("segment_len") if not segment_len: return None - return path, dtype, nullable, int(segment_len) + return path, dtype, nullable, nan_is_data, int(segment_len) + + def _summary_minmax_tail(self, tail_start, n_live, dtype, nullable, nan_is_data, op): + """Reduce by hand the one block the summaries could not cover. + + Its tail is capacity padding, so it has no usable summary entry. The + rescan has to apply exactly the rules the summaries applied: skip the + nulls, and decline where a NaN is data and would poison the answer the + summaries gave. + + Returns the block's extremum, ``None`` when it holds nothing usable, or + ``NotImplemented`` to abandon the shortcut altogether. + """ + try: + seg = np.asarray(self[tail_start:n_live]) + except Exception: + return NotImplemented + if nullable: + null = self._nulls.null_mask_slice(seg, tail_start, n_live) + if null is not None: + seg = seg[~null] + if dtype.kind == "f" and seg.shape[0]: + nan = np.isnan(seg) + if nan.any(): + if nan_is_data: + return NotImplemented + seg = seg[~nan] + if not seg.shape[0]: + return None + if dtype.kind in "US": + return min(seg) if op == "min" else max(seg) + return seg.min() if op == "min" else seg.max() def _index_summary_minmax(self, op: str): """Exact ``min``/``max`` from the column index's block summaries, or @@ -3085,7 +3372,7 @@ def _index_summary_minmax(self, op: str): source = self._summary_minmax_source() if source is None: return NotImplemented - path, dtype, nullable, segment_len = source + path, dtype, nullable, nan_is_data, segment_len = source n_live = self._table._root_table._n_rows if n_live is None or n_live == 0: return NotImplemented @@ -3107,30 +3394,26 @@ def _index_summary_minmax(self, op: str): # Drop the padded tail: keep only blocks lying wholly below n_rows. flags = flags[:n_full] vals = vals[:n_full] - # A non-nullable float with NaN *data* makes numpy min/max return NaN, + # A float column where NaN is *data* makes numpy min/max return NaN, # but the summary dropped those NaNs — they would disagree, so bail. - if dtype.kind == "f" and not nullable and bool((flags & (FLAG_HAS_NAN | FLAG_ALL_NAN)).any()): + # On a null-aware index FLAG_HAS_NAN is raised over the valid rows + # only, so a mask column's NaN fill does not trip this; a genuine NaN + # value does, which is exactly the distinction decision 6 draws. + if nan_is_data and bool((flags & FLAG_HAS_NAN).any()): return NotImplemented - valid = (flags & FLAG_ALL_NAN) == 0 + valid = (flags & FLAG_ALL_NAN) == 0 # also clear on all-null segments vals = vals[valid] # The straddling block is not summarisable (its tail is padding), so read # just its live rows. This is also the whole answer when the column is # shorter than one block, in which case no summary entry is usable. - tail = n_live - n_full * segment_len - if tail: - try: - seg = np.asarray(self[n_full * segment_len : n_live]) - except Exception: + if n_live - n_full * segment_len: + seg_val = self._summary_minmax_tail( + n_full * segment_len, n_live, dtype, nullable, nan_is_data, op + ) + if seg_val is NotImplemented: return NotImplemented - if dtype.kind == "f": - seg = seg[~np.isnan(seg)] - if not nullable and seg.shape[0] != tail: - return NotImplemented # NaN data: see above - if seg.shape[0]: - seg_val = min(seg) if dtype.kind in "US" else seg.min() - if op == "max": - seg_val = max(seg) if dtype.kind in "US" else seg.max() + if seg_val is not None: vals = np.concatenate([vals, np.asarray([seg_val], dtype=dtype)]) if vals.shape[0] == 0: @@ -3234,9 +3517,9 @@ def argmin(self, axis=None, *, where=None): arr = np.asarray(self[:]) if arr.size == 0: raise ValueError("argmin() called on an empty column.") - mask = ( - self._null_mask_for(arr) if self.null_value is not None else np.zeros(len(arr), dtype=np.bool_) - ) + mask = self._reduction_null_mask(arr) + if mask is None: + mask = np.zeros(len(arr), dtype=np.bool_) if mask.all(): raise ValueError("argmin() called on a column where all values are null.") positions = np.where(~mask)[0] @@ -3261,9 +3544,9 @@ def argmax(self, axis=None, *, where=None): arr = np.asarray(self[:]) if arr.size == 0: raise ValueError("argmax() called on an empty column.") - mask = ( - self._null_mask_for(arr) if self.null_value is not None else np.zeros(len(arr), dtype=np.bool_) - ) + mask = self._reduction_null_mask(arr) + if mask is None: + mask = np.zeros(len(arr), dtype=np.bool_) if mask.all(): raise ValueError("argmax() called on a column where all values are null.") positions = np.where(~mask)[0] @@ -3440,14 +3723,10 @@ def __init__(self, table: CTable, prefix: str, leaves: list[str]): def _leaf_is_null_at_logical(self, leaf: str, idx: int) -> bool: col = self._table[leaf] - v = col[idx] - nv = col.null_value - if nv is None: - return v is None try: - return bool(col._null_mask_for(np.asarray([v]))[0]) + return col._nulls.is_null_at(idx) except Exception: - return v is None + return col[idx] is None def _row_value_at_logical(self, idx: int): # If every descendant leaf is null at this row, represent the struct as None. @@ -3590,9 +3869,10 @@ def info_items(self) -> list[tuple[str, object]]: ) else: col_meta = table._schema.columns_by_name.get(name) + spec = col_meta.spec if col_meta else None dtype_label = table._dtype_info_label( - getattr(table._cols[name], "dtype", None), col_meta.spec if col_meta else None - ) + getattr(table._cols[name], "dtype", None), spec + ) + table._null_info_tag(spec) cbytes = getattr(table._cols[name], "cbytes", None) if cbytes is not None: nbytes = getattr(table._cols[name], "nbytes", None) @@ -3744,6 +4024,63 @@ def __delitem__(self, name: str) -> None: dict.__delitem__(self, name) +class _NullMaskCache: + """Open-on-demand cache of the per-column ``.notnull`` validity sidecars. + + Mirrors :class:`_LazyColumnDict`: a wide table should not pay one + ``storage.open_null_mask()`` per nullable column just to be opened. + + A miss is a normal state, not an error. A mask-storage column that has + never held a null has no sidecar at all -- an absent one means *every row + is valid* -- so :meth:`get` answers ``None`` there and callers read the + column as never-null. Negative answers are remembered too, so a null-free + column costs one membership test for the life of the table rather than one + per access. + """ + + __slots__ = ("_absent", "_masks", "_storage") + + def __init__(self, storage: TableStorage) -> None: + self._storage = storage + self._masks: dict[str, blosc2.NDArray] = {} + self._absent: set[str] = set() + + def get(self, name: str) -> blosc2.NDArray | None: + """The sidecar for column *name*, or ``None`` when it has none.""" + mask = self._masks.get(name) + if mask is not None or name in self._absent: + return mask + try: + present = self._storage.has_null_mask(name) + except (NotImplementedError, RuntimeError): + present = False + if not present: + self._absent.add(name) + return None + mask = self._storage.open_null_mask(name) + self._masks[name] = mask + return mask + + def set(self, name: str, mask: blosc2.NDArray) -> None: + self._masks[name] = mask + self._absent.discard(name) + + def pop(self, name: str) -> None: + self._masks.pop(name, None) + self._absent.discard(name) + + def rename(self, old: str, new: str) -> None: + mask = self._masks.pop(old, None) + self._absent.discard(old) + self._absent.discard(new) + if mask is not None: + self._masks[new] = mask + + def materialized(self) -> dict[str, blosc2.NDArray]: + """The sidecars opened so far, without opening any more.""" + return dict(self._masks) + + class _ChunkAlignedWriter: """Buffer writes to a fixed-size NDArray and flush them chunk-aligned. @@ -3996,6 +4333,46 @@ def col(name: str) -> ColExpr: return ColExpr(lambda t: t[name], name) +# CSV text escapes: an empty field is CSV for "missing", which leaves a valid "" +# in a mask-backed text column with nothing to write. \N marks a null and \E a +# valid "", and a value already shaped like one grows a backslash (\N -> \\N). +_CSV_TEXT_TOKEN = re.compile(r"^\\+[NE]$") + + +def _csv_text_escape(value, is_null: bool) -> str: + """Encode one text cell (and its validity) as an unambiguous CSV field.""" + if is_null: + return "\\N" + if isinstance(value, bytes): + value = value.decode() + else: + value = str(value) + if value == "": + return "\\E" + return "\\" + value if _CSV_TEXT_TOKEN.match(value) else value + + +def _csv_text_unescape(field: str) -> str | None: + """Inverse of :func:`_csv_text_escape`; ``None`` means the cell is missing.""" + if field in ("", "\\N"): + return None + if field == "\\E": + return "" + return field[1:] if _CSV_TEXT_TOKEN.match(field) else field + + +def _csv_is_text_column(col) -> bool: + """True when *col* holds text, so an empty CSV field would be ambiguous. + + Takes a :class:`Column`, and must ask ``is_utf8`` rather than only reading + the dtype: a utf8 column reports ``StringDType()``, whose kind is ``"T"`` + and not the ``"U"``/``"S"`` a fixed-width text column reports. Testing the + kind alone left utf8 -- the one text kind with no safe sentinel, and so the + kind that most needs the escapes -- writing ``""`` and a null identically. + """ + return col.is_utf8 or col.dtype.kind in ("U", "S") + + class CTable(_CTableIndexingMixin, Generic[RowT]): """Columnar compressed table with typed columns and row-oriented access.""" @@ -4349,6 +4726,13 @@ def _utf8_rank_index_stale(self, name: str, utf8_rank_meta: dict) -> bool: col = self._root_table._cols.get(name) if col is None: return True + if not utf8_rank_meta.get("null_aware", False) and self._root_table._null_mask(name) is not None: + # Built before mask nulls were stamped with null_rank, so its ranks + # place them wherever the empty-string fill factorized -- rank 0, + # where a genuine "" is indistinguishable from a null. The column's + # bytes need not have changed for that to be wrong, so neither + # signal below can catch it: rebuild rather than trust it. + return True return len(col) != utf8_rank_meta.get("n_rows") or int(col._bytes_used) != utf8_rank_meta.get( "nbytes" ) @@ -4493,6 +4877,87 @@ def _validate_null_value_for_spec(name: str, spec: SchemaSpec, null_value) -> No if isinstance(spec, b2_bytes) and not isinstance(null_value, bytes): raise TypeError(f"Null sentinel for bytes column {name!r} must be bytes") + #: Which policy sentinel field governs which spec kinds. Setting one of + #: these implies sentinel storage for the kinds it covers, so existing + #: ``NullPolicy(float_value=...)`` code keeps working once the default + #: flips to mask. + _POLICY_SENTINEL_FIELD_KINDS: ClassVar[tuple[tuple[str, tuple], ...]] = ( + ("string_value", (string, UTF8Spec)), + ("bytes_value", (b2_bytes,)), + ("bool_value", (b2_bool,)), + ("timestamp_value", (timestamp,)), + ) + + @classmethod + def _policy_implies_sentinel(cls, spec, policy) -> bool: + """True when a type-wide policy sentinel field covers *spec*'s kind.""" + for field_name, kinds in cls._POLICY_SENTINEL_FIELD_KINDS: + if isinstance(spec, kinds) and policy._sentinel_field_is_set(field_name): + return True + dtype = getattr(spec, "dtype", None) + kind = np.dtype(dtype).kind if dtype is not None else "" + if kind == "f" and policy._sentinel_field_is_set("float_value"): + return True + if kind == "i" and policy._sentinel_field_is_set("signed_int_strategy"): + return True + return bool(kind == "u" and policy._sentinel_field_is_set("unsigned_int_strategy")) + + @classmethod + def _resolved_null_storage(cls, name: str, spec, policy) -> str: + """Decide where *spec*'s nulls live. The single decision point. + + In order: an explicit ``spec.null_storage`` wins; then an explicit + ``null_value`` (which *is* a request for in-band storage); then a + per-column ``policy.column_null_values`` entry; then a type-wide + policy sentinel field covering this kind; and finally the policy's + own ``null_storage`` default. + """ + if spec.null_storage is not None: + return spec.null_storage + if getattr(spec, "null_value", None) is not None: + return NULL_SENTINEL + if name in policy.column_null_values: + return NULL_SENTINEL + if not getattr(spec, "supports_sentinel", True): + # Complex has no representable sentinel, so it is mask or nothing. + return NULL_MASK + if cls._policy_implies_sentinel(spec, policy): + return NULL_SENTINEL + return policy.resolve_null_storage() + + @staticmethod + def _unflip_mask_bool_dtype(spec) -> None: + """Undo the conservative uint8 flip that ``__init__`` applies to bools. + + ``bool.__init__`` cannot know whether a bare ``nullable=True`` will + resolve to mask or sentinel, and it has to assume sentinel so that + *opening* a stored table -- which rebuilds specs without running this + resolver -- brings a persisted uint8 column back as uint8. Once the + policy has spoken for mask, the column is a real ``np.bool_`` again. + + Keyed off ``bool_widened_to_uint8``, not off the dtype: an **ndarray** + column may be ``uint8`` because that is what the user declared, and + turning that into ``np.bool_`` would truncate every byte to a flag. + """ + if not getattr(spec, "bool_widened_to_uint8", False): + return + spec.bool_widened_to_uint8 = False + spec.dtype = np.dtype(np.bool_) + if isinstance(spec, NDArraySpec): + _refresh_ndarray_dtype_fields(spec) + + @staticmethod + def _widen_bool_dtype_to_uint8(spec) -> None: + """Make room for the reserved ``255`` a sentinel-backed bool needs. + + The inverse of :meth:`_unflip_mask_bool_dtype`, and it records the flip + so that inverse knows it may be undone. + """ + spec.bool_widened_to_uint8 = True + spec.dtype = np.dtype(np.uint8) + if isinstance(spec, NDArraySpec): + _refresh_ndarray_dtype_fields(spec) + @classmethod def _resolve_nullable_specs( cls, schema: CompiledSchema, *, validate_column_null_values: bool = True @@ -4508,12 +4973,7 @@ def _resolve_nullable_specs( if isinstance(spec, NDArraySpec) and getattr(spec, "null_value", None) is not None: cls._validate_null_value_for_spec(col.name, spec, spec.null_value) if spec.dtype == np.dtype(np.bool_): - spec.dtype = np.dtype(np.uint8) - spec.itemsize = spec.dtype.itemsize - spec.kind = spec.dtype.kind - spec.type = spec.dtype.type - spec.str = spec.dtype.str - spec.name = spec.dtype.name + cls._widen_bool_dtype_to_uint8(spec) col.dtype = getattr(spec, "dtype", None) col.display_width = compute_display_width(spec) continue @@ -4527,6 +4987,16 @@ def _resolve_nullable_specs( continue if not getattr(spec, "nullable", False): continue + if cls._resolved_null_storage(col.name, spec, policy) == NULL_MASK: + # Mask storage keeps nullity out of band, so none of the + # sentinel machinery below applies: no sentinel is chosen, no + # max_length widening for string/bytes, no bool -> uint8 flip. + spec.null_storage = NULL_MASK + cls._unflip_mask_bool_dtype(spec) + col.dtype = getattr(spec, "dtype", None) + col.display_width = compute_display_width(spec) + continue + spec.null_storage = NULL_SENTINEL null_value = policy.column_null_values.get(col.name) if null_value is None: null_value = cls._policy_null_value_for_spec(spec, policy) @@ -4540,15 +5010,10 @@ def _resolve_nullable_specs( elif isinstance(spec, b2_bytes): spec.max_length = max(spec.max_length, len(null_value), 1) spec.dtype = np.dtype(f"S{spec.max_length}") - elif isinstance(spec, b2_bool): - spec.dtype = np.dtype(np.uint8) - elif isinstance(spec, NDArraySpec) and spec.dtype == np.dtype(np.bool_): - spec.dtype = np.dtype(np.uint8) - spec.itemsize = spec.dtype.itemsize - spec.kind = spec.dtype.kind - spec.type = spec.dtype.type - spec.str = spec.dtype.str - spec.name = spec.dtype.name + elif isinstance(spec, b2_bool) or ( + isinstance(spec, NDArraySpec) and spec.dtype == np.dtype(np.bool_) + ): + cls._widen_bool_dtype_to_uint8(spec) col.dtype = getattr(spec, "dtype", None) col.display_width = compute_display_width(spec) @@ -4795,11 +5260,23 @@ def _normalize_row_input(self, data: Any) -> dict[str, Any]: # Fallback: try positional indexing return {name: data[i] for i, name in enumerate(stored)} - def _coerce_row_to_storage(self, row: dict[str, Any]) -> dict[str, Any]: - """Coerce each value in *row* to the column's storage representation.""" + def _coerce_row_to_storage(self, row: dict[str, Any]) -> tuple[dict[str, Any], set[str]]: + """Coerce each value in *row* to the column's storage representation. + + Returns ``(storage_row, null_names)``, where *null_names* is the set of + mask-storage columns the caller must mark invalid. Their entry in + *storage_row* is this column's fill, so the write itself is an ordinary + typed one -- ``None`` never reaches the value array. + """ result = {} + null_names: set[str] = set() for col in self._schema.columns: val = row[col.name] + channel = self._null_channel(col.name) + if channel.uses_mask: + val, is_valid = channel.coerce_scalar(val) + if not is_valid: + null_names.add(col.name) if self._is_list_column(col): result[col.name] = coerce_list_cell(col.spec, val) elif self._is_varlen_scalar_column(col): @@ -4821,7 +5298,7 @@ def _coerce_row_to_storage(self, row: dict[str, Any]) -> dict[str, Any]: result[col.name] = np.array(val, dtype=col.dtype).item() else: result[col.name] = np.array(val, dtype=col.dtype).item() - return result + return result, null_names def _open_column_from_storage(self, storage: TableStorage, name: str): """Open one stored column from *storage*.""" @@ -4834,6 +5311,145 @@ def _open_column_from_storage(self, storage: TableStorage, name: str): return storage.open_dictionary_column(name, cc.spec) return storage.open_column(name) + # ------------------------------------------------------------------ + # Per-column validity sidecars + # ------------------------------------------------------------------ + + @property + def _null_masks(self) -> _NullMaskCache: + """Open-on-demand cache of this table's ``.notnull`` validity sidecars. + + A view shares its base table's cache: sidecars are physical, + row-indexed arrays, exactly like the ``_cols`` NDArrays a view already + shares with its base. + """ + base = self.base + if base is not None: + return base._null_masks + cache = self.__dict__.get("_null_mask_cache") + if cache is None: + cache = _NullMaskCache(self._storage) + self.__dict__["_null_mask_cache"] = cache + return cache + + def _null_channel(self, name: str) -> NullChannel: + """The null channel for column *name*. + + The physical write paths (``append``, ``extend``) have no ``Column`` of + their own; this gives them the same accessor the read paths use, over a + base-table column where logical and physical positions coincide. + """ + return self[name]._nulls + + @property + def _null_mask_names(self) -> list[str]: + """Columns whose schema says their nulls live in a sidecar. + + Being listed here says nothing about whether a sidecar *exists*: one + is written only once a null actually is, so most entries answer + ``None`` from :meth:`_null_mask`. + """ + return [c.name for c in self._schema.columns if getattr(c.spec, "uses_mask", False)] + + def _null_mask(self, name: str) -> blosc2.NDArray | None: + """Column *name*'s validity sidecar, or ``None`` when it has none. + + ``None`` means *every row is valid*, not "unknown" — see + :class:`_NullMaskCache`. + """ + return self._null_masks.get(name) + + def _null_mask_grid(self, name: str, capacity: int) -> tuple[tuple[int], tuple[int]]: + """Chunk/block grid for column *name*'s validity sidecar. + + **Pinned to the value column's row grid**, not to whatever + ``compute_chunks_blocks`` would pick for a one-byte bool dtype. A + sidecar on its own grid would make every paired read — the + chunk-aligned writers, ``_nonnull_chunks``, index-segment alignment — + re-align mask against values on each pass. + + Falls back to the table-wide ``_valid_rows`` grid (which is the same + shared grid the fixed-width columns use) for columns whose payload is + not a plain row-indexed NDArray, such as utf8, whose offsets array + carries ``n + 1`` entries. + """ + cc = self._schema.columns_by_name[name] + arr = self._cols[name] + if self._is_dictionary_column(cc): + arr = arr.codes + elif not isinstance(arr, blosc2.NDArray): + arr = self._valid_rows + chunks = (max(1, min(int(arr.chunks[0]), capacity)),) + blocks = (max(1, min(int(arr.blocks[0]), chunks[0])),) + return chunks, blocks + + def _ensure_null_mask(self, name: str) -> blosc2.NDArray: + """Return column *name*'s validity sidecar, materializing it if absent. + + This is the one place that turns a null-free mask column into one with + bytes on disk: creation is deferred to the first null actually + written, since until then the column's absent sidecar already says + exactly the right thing. + """ + if self.base is not None: + return self.base._ensure_null_mask(name) + mask = self._null_masks.get(name) + if mask is not None: + return mask + capacity = len(self._valid_rows) + chunks, blocks = self._null_mask_grid(name, capacity) + mask = self._storage.create_null_mask(name, shape=(capacity,), chunks=chunks, blocks=blocks) + self._null_masks.set(name, mask) + return mask + + def _existing_null_masks(self) -> dict[str, blosc2.NDArray]: + """Every validity sidecar this table actually has, opening as needed. + + Unlike ``_null_masks.materialized()`` this consults storage for each + mask-storage column, so it is the iterator the capacity and + persistence paths must use: they cannot miss a sidecar merely because + nothing has read that column yet. + """ + found = {} + for name in self._null_mask_names: + mask = self._null_masks.get(name) + if mask is not None: + found[name] = mask + return found + + def _permute_null_masks(self, positions: np.ndarray, n: int) -> None: + """Reorder every validity sidecar in place by *positions*, as the columns were. + + The tail past *n* becomes all-valid: those slots hold no live row, and + "valid" is what an absent sidecar would say of them. + """ + for mask in self._existing_null_masks().values(): + mask[:n] = mask[positions[:n]] + mask[n:] = True + + def _gather_null_masks_into(self, target: CTable, positions: np.ndarray, n: int) -> None: + """Gather this table's sidecars into *target*'s, under *positions*. + + A column with no sidecar here gets none there: nothing in it is null, + and materializing one in the copy would invent storage the source never + needed (decision 9). + """ + if n <= 0: + return + for name, mask in self._existing_null_masks().items(): + gathered = np.asarray(mask[positions]) + if gathered.all(): + # The selection happens to contain no null, so the copy needs + # no sidecar -- "absent" already says exactly this. + continue + target._ensure_null_mask(name)[:n] = gathered + + def _drop_null_mask(self, name: str) -> None: + """Forget and (for persistent tables) delete column *name*'s sidecar.""" + self._null_masks.pop(name) + with contextlib.suppress(NotImplementedError, RuntimeError, KeyError): + self._storage.delete_null_mask(name) + def _resolve_last_pos(self) -> int: """Return the physical index of the next write slot. @@ -4896,6 +5512,10 @@ def trim_capacity(self) -> None: col_arr.resize((target,)) continue col_arr.resize(self._column_physical_shape(cc, target)) + # Only sidecars that exist need trimming; an absent one has no bytes to + # reclaim, which is most of the payoff of materializing them lazily. + for mask in self._existing_null_masks().values(): + mask.resize((target,)) self._valid_rows.resize((target,)) self._last_pos = target @@ -4912,6 +5532,13 @@ def _grow(self) -> None: col_arr.resize((new_capacity,)) continue col_arr.resize(self._column_physical_shape(cc, new_capacity)) + # Grow the validity sidecars alongside their columns. resize() zero-fills, + # i.e. *invalid*, so the new tail must be marked valid explicitly to keep + # the "no null written yet" reading of those rows. + for mask in self._existing_null_masks().values(): + old_capacity = mask.shape[0] + mask.resize((new_capacity,)) + mask[old_capacity:new_capacity] = True self._valid_rows.resize((new_capacity,)) # ------------------------------------------------------------------ @@ -5785,6 +6412,7 @@ def to_cframe(self) -> bytes: from blosc2.ctable_storage import ( _DICT_SUFFIX, + _NOTNULL_SUFFIX, _UTF8_DATA_SUFFIX, _column_name_to_relpath, ) @@ -5822,6 +6450,11 @@ def to_cframe(self) -> bytes: # Scalar NDArray or ListArray — both serialize via to_cframe(). estore[key] = arr + # Validity sidecars travel beside their columns; a column without one + # simply contributes no entry, which reconstructs as all-valid. + for name, mask in src._existing_null_masks().items(): + estore[f"/_cols/{_column_name_to_relpath(name)}{_NOTNULL_SUFFIX}"] = mask + return estore.to_cframe() def _save_to_storage( # noqa: C901 @@ -5856,14 +6489,21 @@ def _save_to_storage( # noqa: C901 ) # --- valid_rows (all True, compacted) --- + valid_chunks = shared_chunks if shared_chunks is not None else default_chunks + valid_blocks = shared_blocks if shared_blocks is not None else default_blocks disk_valid = storage.create_valid_rows( shape=(capacity,), - chunks=shared_chunks if shared_chunks is not None else default_chunks, - blocks=shared_blocks if shared_blocks is not None else default_blocks, + chunks=valid_chunks, + blocks=valid_blocks, ) if n_live > 0: disk_valid[:n_live] = True + # Row grid each fixed-width column actually landed on, so its validity + # sidecar below can be pinned to that same grid rather than re-derived + # (chunk overrides and the reblock path can both change it). + dest_grids: dict[str, tuple[tuple[int], tuple[int]]] = {} + # --- columns --- for col in self._schema.columns: name = col.name @@ -5951,6 +6591,7 @@ def _save_to_storage( # noqa: C901 copy_kwargs["cparams"] = cparams_override new_arr = src_arr.copy(**copy_kwargs) storage.install_column(name, new_arr) + dest_grids[name] = ((int(new_arr.chunks[0]),), (int(new_arr.blocks[0]),)) else: eff_chunks = chunks_override if chunks_override is not None else col_storage["chunks"] if chunks_override is not None and blocks_override is None: @@ -5970,10 +6611,23 @@ def _save_to_storage( # noqa: C901 cparams=cparams_override if cparams_override is not None else col_storage.get("cparams"), dparams=col_storage.get("dparams"), ) + dest_grids[name] = ((int(disk_col.chunks[0]),), (int(disk_col.blocks[0]),)) if n_live > 0: # Slice is ~30x faster than fancy-index for sequential no-deletion access. disk_col[:n_live] = src_arr[:n_live] if no_deletions else src_arr[live_pos] + # --- validity sidecars (compacted alongside their columns) --- + # Written after the columns so each can be pinned to the grid its column + # actually landed on. A column whose sidecar is absent stays absent: + # no nulls, no bytes, and nothing to copy. + for name, mask in self._existing_null_masks().items(): + mask_chunks, mask_blocks = dest_grids.get(name, (valid_chunks, valid_blocks)) + disk_mask = storage.create_null_mask( + name, shape=(capacity,), chunks=mask_chunks, blocks=mask_blocks + ) + if n_live > 0: + disk_mask[:n_live] = mask[:n_live] if no_deletions else mask[live_pos] + storage.save_schema(self._schema_dict_with_computed()) def save(self, urlpath: str, *, overwrite: bool = False) -> None: @@ -6196,6 +6850,28 @@ def load(cls, urlpath: str) -> CTable: # noqa: C901 mem_col[:phys_size] = disk_cols[name][:] mem_cols[name] = mem_col + # Validity sidecars, pinned to the grid their in-memory column got. + # Columns with no stored sidecar are left without one — that already + # means all-valid, so there is nothing to load. + mem_masks: dict[str, blosc2.NDArray] = {} + for col in schema.columns: + name = col.name + if not getattr(col.spec, "uses_mask", False) or not file_storage.has_null_mask(name): + continue + src_mask = file_storage.open_null_mask(name) + payload = mem_cols[name] + grid_src = payload if isinstance(payload, blosc2.NDArray) else mem_valid + mask_chunks = max(1, min(int(grid_src.chunks[0]), capacity)) + mem_mask = mem_storage.create_null_mask( + name, + shape=(capacity,), + chunks=(mask_chunks,), + blocks=(max(1, min(int(grid_src.blocks[0]), mask_chunks)),), + ) + if phys_size > 0: + mem_mask[:phys_size] = src_mask[:phys_size] + mem_masks[name] = mem_mask + file_storage.close() obj = cls.__new__(cls) @@ -6214,6 +6890,8 @@ def load(cls, urlpath: str) -> CTable: # noqa: C901 obj._summary_indexes_built = schema_dict.get("summary_indexes_built", False) obj.base = None obj._valid_rows = mem_valid + for mask_name, mask in mem_masks.items(): + obj._null_masks.set(mask_name, mask) obj._n_rows = n_live obj._last_pos = None # resolve lazily on first write obj._computed_cols = {} @@ -6373,10 +7051,14 @@ def take(self, indices, /) -> CTable: else: result._cols[col_name][:n] = arr._take_numpy(physical_pos, axis=0) + self._gather_null_masks_into(result, physical_pos, n) result._valid_rows[:n] = True result._valid_rows[n:] = False result._n_rows = n - result._last_pos = n - 1 if n > 0 else None + # Exclusive bound, the convention _resolve_last_pos() documents and every + # other writer follows -- n - 1 here made add_column() on a copied table + # backfill one row short (and raise, for a varlen column). + result._last_pos = n if n > 0 else None return result def slice(self, start, stop=None, /, *, copy: bool = True) -> CTable: @@ -6776,6 +7458,13 @@ def cov(self) -> np.ndarray: cast to int (0/1) before computation. Complex columns raise :exc:`TypeError`. + Nulls are dropped **listwise**: a row that is null in any column is + excluded from every column, so all entries of the result are computed + over the same rows and the matrix stays consistent with itself. This + differs from ``pandas.DataFrame.cov()``, which drops pairwise. Rows + are dropped by what :meth:`Column.is_null` reports, never by the value + standing in for a null, so both null storages give the same answer. + Returns ------- numpy.ndarray @@ -6787,7 +7476,8 @@ def cov(self) -> np.ndarray: TypeError If any column has an unsupported dtype (complex, string, …). ValueError - If the table has fewer than 2 live rows (covariance undefined). + If the table has fewer than 2 live rows (covariance undefined), or + fewer than 2 rows survive the listwise null drop. """ for name in self.col_names: col_info = self._schema.columns_by_name.get(name) @@ -6816,7 +7506,18 @@ def cov(self) -> np.ndarray: for name in self.col_names: col = self[name] arr = col[:] - nm = col._null_mask_for(arr) + channel = col._nulls + if channel.kind == NULL_MASK: + # Off the sidecar. The in-band test below cannot see these: + # a mask column has no sentinel, so it would call every null + # row an ordinary value and average in the fill. + nm = channel.null_mask() + else: + # In band, answered from the values already read rather than + # by reading the column a second time. All-False for a column + # with no nulls at all; the dictionary and variable-length + # kinds cannot reach here, having been rejected on dtype. + nm = channel.mask_for_values(arr) if nm.any(): null_union = nm if null_union is None else (null_union | nm) raw_arrays.append(arr) @@ -7023,15 +7724,17 @@ def iter_arrow_batches( # noqa: C901 # Dense root table: logical rows == persisted rows, so # export straight from the offsets/bytes buffers with # no per-row decode (storage is already Arrow layout). - arrays.append(arr8.arrow_slice(pa, start, stop, nv)) + arrays.append( + arr8.arrow_slice(pa, start, stop, nv, valid=col._nulls.valid_slice(start, stop)) + ) continue - values = col[start:stop] # StringDType array with sentinel nulls - null_mask = col._null_mask_for(values) if nv is not None else None + values = col[start:stop] # StringDType array; nulls per this column's channel + null_mask = col._nulls.null_mask_slice(values, start, stop) arrays.append( pa.array( values.astype(object), type=self._pa_type_from_spec(pa, spec), - mask=null_mask if null_mask is not None and null_mask.any() else None, + mask=null_mask, ) ) continue @@ -7070,7 +7773,11 @@ def iter_arrow_batches( # noqa: C901 if col.is_ndarray: spec = self._schema.columns_by_name[name].spec values = np.asarray(col[start:stop]) - null_mask = col._null_mask_for(values) if col.null_value is not None else None + # Row-level under mask storage. A sentinel ndarray column + # keeps the older, lossier rule -- a row is null only when + # *every* element equals the sentinel -- because that is the + # only thing its storage can express. + null_mask = col._nulls.null_mask_slice(values, start, stop) pa_type = self._pa_type_from_spec(pa, spec) flat_values = np.ascontiguousarray(values.reshape(-1)) pa_values = pa.array(flat_values, type=pa_type.value_type) @@ -7078,33 +7785,28 @@ def iter_arrow_batches( # noqa: C901 pa.FixedSizeListArray.from_arrays( pa_values, type=pa_type, - mask=( - pa.array(null_mask, type=pa.bool_()) - if null_mask is not None and null_mask.any() - else None - ), + mask=(pa.array(null_mask, type=pa.bool_()) if null_mask is not None else None), ) ) continue arr = np.asarray(col[start:stop]) - nv = col.null_value - null_mask = col._null_mask_for(arr) if nv is not None else None - has_nulls = null_mask is not None and bool(null_mask.any()) - if arr.dtype.kind == "U": - values = arr.tolist() - if has_nulls: - values = [None if null_mask[i] else v for i, v in enumerate(values)] - arrays.append(pa.array(values, type=pa.string())) - elif arr.dtype.kind == "S": - values = arr.tolist() - if has_nulls: - values = [None if null_mask[i] else v for i, v in enumerate(values)] - arrays.append(pa.array(values, type=pa.large_binary())) + null_mask = col._nulls.null_mask_slice(arr, start, stop) + if arr.dtype.kind in "US": + # pyarrow reads the mask alongside the values, so the null + # slots need no substitution here — under mask storage they + # already hold the fill, and under a sentinel the mask says + # to ignore whatever is there. + pa_type = pa.string() if arr.dtype.kind == "U" else pa.large_binary() + arrays.append(pa.array(arr.tolist(), type=pa_type, mask=null_mask)) elif ( self._schema.columns_by_name.get(name) is not None and self._schema.columns_by_name[name].spec.to_metadata_dict().get("kind") == "bool" ): - arrays.append(pa.array(arr == 1, mask=null_mask if has_nulls else None, type=pa.bool_())) + # A sentinel bool is physically uint8 (0/1/255) and needs the + # compare; a mask bool is already np.bool_ and must not get + # one, since `arr == 1` would be a no-op at best. + values = arr == 1 if col._is_nullable_bool else arr + arrays.append(pa.array(values, mask=null_mask, type=pa.bool_())) elif self._schema.columns_by_name.get(name) is not None and isinstance( self._schema.columns_by_name[name].spec, timestamp ): @@ -7113,12 +7815,12 @@ def iter_arrow_batches( # noqa: C901 arrays.append( pa.array( values, - mask=null_mask if has_nulls else None, + mask=null_mask, type=pa.timestamp(spec.unit, tz=spec.timezone), ) ) else: - arrays.append(pa.array(arr, mask=null_mask if has_nulls else None)) + arrays.append(pa.array(arr, mask=null_mask)) yield pa.RecordBatch.from_arrays(arrays, names=arrow_names) def to_arrow(self): @@ -7189,6 +7891,7 @@ def _arrow_type_to_spec( # noqa: C901 field_metadata=None, string_max_length=None, null_value=None, + null_storage=None, nullable=False, object_fallback: bool = False, ): @@ -7217,7 +7920,13 @@ def _arrow_type_to_spec( # noqa: C901 f"Arrow fixed-size-list metadata shape {shape} has size {int(np.prod(shape))}, " f"but the Arrow list size is {pa_type.list_size}." ) - return b2s.ndarray(shape, dtype=value_dtype, nullable=nullable, null_value=null_value) + return b2s.ndarray( + shape, + dtype=value_dtype, + nullable=nullable, + null_value=null_value, + null_storage=null_storage, + ) if pa.types.is_dictionary(pa_type): vt = pa_type.value_type @@ -7274,15 +7983,19 @@ def _arrow_type_to_spec( # noqa: C901 ] if pa.types.is_timestamp(pa_type): return b2s.timestamp( - unit=pa_type.unit, timezone=pa_type.tz, nullable=nullable, null_value=null_value + unit=pa_type.unit, + timezone=pa_type.tz, + nullable=nullable, + null_value=null_value, + null_storage=null_storage, ) for arrow_t, spec_cls in mapping: if pa_type == arrow_t: - if null_value is not None and hasattr(spec_cls(), "null_value"): - return spec_cls(null_value=null_value) - if null_value is not None and spec_cls is b2s.bool: + if null_value is not None and getattr(spec_cls, "supports_sentinel", False): return spec_cls(null_value=null_value) + if null_storage is not None: + return spec_cls(null_storage=null_storage) return spec_cls() if pa.types.is_list(pa_type) or pa.types.is_large_list(pa_type): @@ -7338,16 +8051,16 @@ def _arrow_type_to_spec( # noqa: C901 return b2s.vlstring(nullable=nullable) # No fixed-width threshold given: store as a variable-length # utf8 column (offsets + bytes, StringDType reads). - return b2s.utf8(nullable=nullable, null_value=null_value) + return b2s.utf8(nullable=nullable, null_value=null_value, null_storage=null_storage) max_length = max(string_max_length, len(null_value) if null_value is not None else 1, 1) - return b2s.string(max_length=max_length, null_value=null_value) + return b2s.string(max_length=max_length, null_value=null_value, null_storage=null_storage) if _is_arrow_binary_type(pa, pa_type): if string_max_length is None: # No fixed-width threshold given: store as variable-length scalar bytes. return b2s.vlbytes(nullable=nullable) max_length = max(string_max_length, len(null_value) if null_value is not None else 1, 1) - return b2s.bytes(max_length=max_length, null_value=null_value) + return b2s.bytes(max_length=max_length, null_value=null_value, null_storage=null_storage) if object_fallback: return b2s.object(nullable=nullable) @@ -7375,8 +8088,12 @@ def _compiled_columns_from_arrow( *, auto_null_sentinels: bool, object_fallback: bool = False, + null_storage: str | None = None, ): null_policy = get_null_policy() + # Only inferred schemas consult the policy; extending an existing table + # never reaches here, so its stored null_storage always wins. + storage_pref = null_storage if null_storage is not None else null_policy.resolve_null_storage() column_null_values = null_policy.column_null_values schema_names = set(schema.names) unknown_null_values = set(column_null_values) - schema_names @@ -7429,36 +8146,60 @@ def _compiled_columns_from_arrow( f"column_null_values is not supported for vlbytes/vlstring column {name!r}; " "these columns represent nulls as native None." ) - if has_null_value_override: - null_value = column_null_values[name] - elif ( - auto_null_sentinels + # Kinds that carry their own nullity (native None, a reserved code, + # a nested layout) and so never take either scalar null channel. + handles_own_nulls = ( + field_is_list + or field_is_struct + or field_is_dictionary + or field_is_varlen_scalar + or field_is_object_fallback + ) + # Mask storage needs no sentinel, so none of the selection below + # applies — and neither does its failure mode. This is what makes + # nullable bool, full-range int8/uint8 and free-text utf8 + # importable at all. + use_mask = ( + storage_pref == NULL_MASK and field.nullable - and not ( - field_is_list - or field_is_struct - or field_is_dictionary - or field_is_varlen_scalar - or field_is_object_fallback + and not handles_own_nulls + and not has_null_value_override + ) + if use_mask and null_storage is None: + # A type-wide policy sentinel field is a request for in-band + # storage for the kinds it covers, and has to be honoured here + # too -- otherwise NullPolicy(signed_int_strategy="max") would + # mean one thing for a declared schema and another for an + # inferred one. Asking needs a spec, so build a throwaway with + # no null decision in it; an explicit null_storage= argument + # has already won by this point. + probe = cls._arrow_type_to_spec( + pa, + field.type, + arrow_col, + field_metadata=field.metadata, + string_max_length=column_string_max_length, + nullable=field.nullable, + object_fallback=object_fallback, ) - ): + use_mask = not cls._policy_implies_sentinel(probe, null_policy) + if has_null_value_override: + null_value = column_null_values[name] + elif not use_mask and auto_null_sentinels and field.nullable and not handles_own_nulls: arrow_type_for_null = field.type.value_type if field_is_ndarray else field.type null_value = cls._auto_null_sentinel(pa, arrow_type_for_null, null_policy=null_policy) if ( arrow_col is not None and arrow_col.null_count - and not ( - field_is_list - or field_is_struct - or field_is_dictionary - or field_is_varlen_scalar - or field_is_object_fallback - ) + and not handles_own_nulls and null_value is None + and not use_mask ): raise TypeError( - f"Column {name!r} contains Parquet nulls. Provide a CTable schema with a " - "null_value sentinel for this column." + f"Column {name!r} contains Parquet nulls, and no null_value sentinel is " + f"available for its type. Provide a CTable schema with a null_value sentinel " + f"for this column, or import with null_storage='mask' to keep nullity in a " + f"sidecar validity array (which every type supports)." ) spec = cls._arrow_type_to_spec( pa, @@ -7467,6 +8208,7 @@ def _compiled_columns_from_arrow( field_metadata=field.metadata, string_max_length=column_string_max_length, null_value=null_value, + null_storage=NULL_MASK if use_mask else None, nullable=field.nullable, object_fallback=object_fallback, ) @@ -7738,7 +8480,9 @@ def _write_arrow_batches(cls, obj, batches, columns, new_cols, new_valid) -> Non while end > len(new_valid): obj._grow() new_valid = obj._valid_rows - pos = cls._write_arrow_batch(batch, columns, new_cols, new_valid, pos, list_normalizers, writers) + pos = cls._write_arrow_batch( + obj, batch, columns, new_cols, new_valid, pos, list_normalizers, writers + ) for writer in writers.values(): writer.flush() # All imported rows are valid; mark them in a single aligned write. @@ -7757,7 +8501,7 @@ def _write_arrow_batches(cls, obj, batches, columns, new_cols, new_valid) -> Non @classmethod def _write_arrow_batch( - cls, batch, columns, new_cols, new_valid, pos: int, list_normalizers, writers + cls, obj, batch, columns, new_cols, new_valid, pos: int, list_normalizers, writers ) -> int: m = len(batch) if m == 0: @@ -7777,6 +8521,12 @@ def _write_arrow_batch( values = [normalizer(value) for value in values] new_cols[col.name].extend(values, validate=False) elif cls._is_varlen_scalar_column(col): + # utf8 is a varlen kind that can still use a sidecar; the other + # varlen kinds keep their native None cells and need nothing. + if getattr(col.spec, "uses_mask", False) and arrow_col.null_count: + obj._ensure_null_mask(col.name)[pos : pos + m] = arrow_col.is_valid().to_numpy( + zero_copy_only=False + ) new_cols[col.name].extend(arrow_col.to_pylist()) elif cls._is_dictionary_column(col): import pyarrow as _pa @@ -7787,6 +8537,19 @@ def _write_arrow_batch( else: # Plain string array: encode values into the dictionary. new_cols[col.name][pos : pos + m] = arrow_col.to_pylist() + elif getattr(col.spec, "uses_mask", False): + values, valid = cls._arrow_column_to_numpy_masked(arrow_col, col) + if valid is not None: + # A batch with no nulls writes nothing, so a nullable-but + # -null-free column still ends up with no sidecar at all. + # A fresh sidecar is created all-True, so the rows already + # written before this first null need no back-fill. + # + # No chunk-aligned writer here, unlike the values: the + # sidecar is one byte per row and compresses to almost + # nothing, so a straddling write is not worth avoiding. + obj._ensure_null_mask(col.name)[pos : pos + m] = valid + writers[col.name].append(values) else: writers[col.name].append(cls._arrow_column_to_numpy(arrow_col, col)) return pos + m @@ -7794,10 +8557,18 @@ def _write_arrow_batch( @staticmethod def _arrow_column_to_numpy(arrow_col, col: CompiledColumn) -> np.ndarray: nv = getattr(col.spec, "null_value", None) + uses_mask = getattr(col.spec, "uses_mask", False) + # What a null slot ends up holding in storage. Under a sentinel that + # value *is* the null marker; under mask storage it is an arbitrary, + # unobservable fill and the sidecar carries the nullity instead. + fill = fill_value_for(col.spec) if uses_mask else nv if col.spec.to_metadata_dict().get("kind") == "bool" and col.dtype == np.dtype(np.uint8): - return np.array([nv if v is None else int(v) for v in arrow_col.to_pylist()], dtype=np.uint8) + return np.array([fill if v is None else int(v) for v in arrow_col.to_pylist()], dtype=np.uint8) if isinstance(col.spec, NDArraySpec): values = arrow_col.to_pylist() + if uses_mask and arrow_col.null_count: + item = np.full(col.spec.item_shape, fill, dtype=col.spec.dtype) + values = [item if v is None else v for v in values] arr = CTable._coerce_ndarray_batch(col.name, col.spec, values, len(values)) return arr.reshape((len(values), *col.spec.item_shape)) if isinstance(col.spec, timestamp): @@ -7806,27 +8577,44 @@ def _arrow_column_to_numpy(arrow_col, col: CompiledColumn) -> np.ndarray: .astype(f"datetime64[{col.spec.unit}]") .astype(np.int64) ) + # NaT already decodes to int64.min, which is exactly the mask fill, + # so only a sentinel that differs from it needs remapping. if arrow_col.null_count and nv is not None and int(nv) != int(np.iinfo(np.int64).min): arr[arr == np.iinfo(np.int64).min] = int(nv) return arr.astype(col.dtype, copy=False) if col.dtype.kind in "US": values = arrow_col.to_pylist() - if nv is not None: - values = [nv if v is None else v for v in values] + if fill is not None: + values = [fill if v is None else v for v in values] max_len = col.spec.max_length too_long = [v for v in values if v is not None and len(v) > max_len] if too_long: raise ValueError(f"Column {col.name!r} contains values longer than max_length={max_len}.") return np.array(values, dtype=col.dtype) if arrow_col.null_count: - if nv is None: + if fill is None: raise TypeError( f"Column {col.name!r} contains Arrow/Parquet nulls. Provide a CTable schema " - "with a null_value sentinel for this column." + "with a null_value sentinel for this column, or import with " + "null_storage='mask'." ) - arrow_col = arrow_col.fill_null(nv) + arrow_col = arrow_col.fill_null(fill) return arrow_col.to_numpy(zero_copy_only=False).astype(col.dtype) + @staticmethod + def _arrow_column_to_numpy_masked(arrow_col, col: CompiledColumn): + """Split an Arrow column into ``(values, valid)`` for a mask column. + + The source's own validity is authoritative — no value is inspected to + decide what is null, which is precisely why a mask import can accept + types no sentinel could represent. *valid* is ``None`` when the batch + contained no nulls, so the caller writes no sidecar for it. + """ + values = CTable._arrow_column_to_numpy(arrow_col, col) + if not arrow_col.null_count: + return values, None + return values, arrow_col.is_valid().to_numpy(zero_copy_only=False) + @staticmethod def _arrow_schema_metadata(schema) -> dict[str, Any]: import base64 @@ -8005,6 +8793,7 @@ def from_arrow( # noqa: C901 capacity_hint: int | None = None, string_max_length: int | Mapping[str, int] | None = None, auto_null_sentinels: bool = True, + null_storage: Literal["mask", "sentinel"] | None = None, blosc2_batch_size: int | None = _BATCH_SIZE_DEFAULT, blosc2_items_per_block: int | None = None, list_serializer: Literal["msgpack", "arrow"] = "msgpack", @@ -8156,6 +8945,7 @@ def from_arrow( # noqa: C901 string_max_length, auto_null_sentinels=auto_null_sentinels, object_fallback=object_fallback, + null_storage=null_storage, ) cls._apply_arrow_column_cparams(columns, column_cparams) for col in columns: @@ -8275,6 +9065,7 @@ def from_parquet( # noqa: C901 dparams=None, validate: bool = False, auto_null_sentinels: bool = True, + null_storage: Literal["mask", "sentinel"] | None = None, blosc2_batch_size: int | None = _BATCH_SIZE_DEFAULT, blosc2_items_per_block: int | None = None, list_serializer: Literal["msgpack", "arrow"] = "arrow", @@ -8528,6 +9319,7 @@ def _limited_batches(batch_iter, limit: int): capacity_hint=max_rows, string_max_length=string_max_length, auto_null_sentinels=auto_null_sentinels, + null_storage=null_storage, blosc2_batch_size=blosc2_batch_size, blosc2_items_per_block=blosc2_items_per_block, list_serializer=list_serializer, @@ -8564,6 +9356,7 @@ def _limited_batches(batch_iter, limit: int): capacity_hint=_capacity_hint, string_max_length=string_max_length, auto_null_sentinels=auto_null_sentinels, + null_storage=null_storage, blosc2_batch_size=blosc2_batch_size, blosc2_items_per_block=blosc2_items_per_block, list_serializer=list_serializer, @@ -8581,6 +9374,12 @@ def to_csv(self, path: str | None = None, *, header: bool = True, sep: str = "," Fixed-shape ndarray column cells are serialised as JSON arrays for readability and shape safety (e.g. ``"[1.0, 2.0, 3.0]"``). + Nulls are written as empty fields, except in a mask-backed **text** + column, where an empty field cannot mean both ``""`` and missing: there + a null is written as ``\\N`` and a valid ``""`` as ``\\E`` (a value that + already looks like either grows a backslash). :meth:`from_csv` reads + both back, so ``""`` and ``None`` survive a round trip. + Parameters ---------- path: @@ -8606,7 +9405,9 @@ def to_csv(self, path: str | None = None, *, header: bool = True, sep: str = "," col = self[name] if col.is_ndarray: arr = col[:] - null_mask = col._null_mask_for(arr) + # is_null(), not a sentinel comparison: a mask column's nulls are + # not in its values at all, and its fill is an ordinary item. + null_mask = col.is_null() json_strings: list[str] = [] for i in range(n): if null_mask[i]: @@ -8614,6 +9415,21 @@ def to_csv(self, path: str | None = None, *, header: bool = True, sep: str = "," else: json_strings.append(json.dumps(arr[i].tolist())) arrays.append(json_strings) + elif col.null_storage == NULL_MASK and _csv_is_text_column(col): + # An empty field cannot mean both "" and missing, so a mask-backed + # text column writes \N for a null and \E for a valid "", escaping + # any value that would collide (\N -> \\N). from_csv undoes both. + arrays.append( + [_csv_text_escape(v, is_null) for v, is_null in zip(col[:], col.is_null(), strict=True)] + ) + elif col.null_storage == NULL_MASK and col.null_count(): + # An empty field is CSV for missing, and for a mask column it is + # the *only* way to say it -- writing the fill would come back as + # a real 0 or "". A sentinel column keeps writing its sentinel, + # which from_csv maps back, so its output is unchanged. + arrays.append( + ["" if is_null else v for v, is_null in zip(col[:], col.is_null(), strict=True)] + ) else: arrays.append(col[:]) @@ -8633,20 +9449,33 @@ def _write(f) -> None: return None @staticmethod - def _csv_ndarray_col_to_array(raw: list[str], col) -> np.ndarray: - """Convert a list of JSON-array CSV strings to a stacked ndarray for an ndarray column.""" + def _csv_ndarray_col_to_array(raw: list[str], col) -> tuple[np.ndarray, np.ndarray | None]: + """Convert JSON-array CSV strings to ``(values, valid)`` for an ndarray column. + + Same split as :meth:`_csv_col_to_array`: an empty field is missing, and a + mask column reports that in *valid* rather than in the item it stores. + """ spec = col.spec null_value = getattr(spec, "null_value", None) + uses_mask = getattr(spec, "uses_mask", False) item_shape = spec.item_shape dtype = spec.dtype + fill = None if not uses_mask else np.full(item_shape, fill_value_for(spec), dtype=dtype) + valid = None rows = [] - for val in raw: + for i, val in enumerate(raw): stripped = val.strip() if stripped == "": if null_value is not None: rows.append(np.full(item_shape, null_value, dtype=dtype)) continue + if fill is not None: + if valid is None: + valid = np.ones(len(raw), dtype=np.bool_) + valid[i] = False + rows.append(fill) + continue raise ValueError(f"Column {col.name!r}: non-nullable column got empty cell") try: @@ -8658,25 +9487,78 @@ def _csv_ndarray_col_to_array(raw: list[str], col) -> np.ndarray: raise ValueError(f"Column {col.name!r}: expected item shape {item_shape}, got {arr.shape}") rows.append(arr) - return np.ascontiguousarray(rows, dtype=dtype) + return np.ascontiguousarray(rows, dtype=dtype), valid @staticmethod - def _csv_col_to_array(raw: list[str], col, nv) -> np.ndarray: - """Convert a list of raw CSV strings to a numpy array for *col*.""" + def _csv_col_to_array(raw: list[str], col, nv) -> tuple[np.ndarray, np.ndarray | None]: + """Convert raw CSV strings to ``(values, valid)`` for *col*. + + An empty field is CSV for "missing". A sentinel column stores *nv* + there and is done; a **mask** column has no in-band value to store, so + it takes the column's fill and reports the validity separately -- and + without that a bare ``""`` would be cast straight to the column's dtype + and raise. *valid* is ``None`` when nothing was missing, or when the + column is not nullable at all. + """ + uses_mask = getattr(col.spec, "uses_mask", False) + if col.dtype.kind in ("U", "S"): + raw, missing = CTable._csv_decode_text(raw, uses_mask) + else: + missing = np.array([v.strip() == "" for v in raw], dtype=np.bool_) + placeholder = nv + valid = None + if nv is None and uses_mask and missing.any(): + placeholder = fill_value_for(col.spec) + valid = ~missing + + def _fill_missing(values): + if placeholder is None: + return values + return [placeholder if m else v for v, m in zip(values, missing, strict=True)] + if col.dtype == np.bool_: + parsed = [v.strip() in ("True", "true", "1") for v in raw] + return np.array(_fill_missing(parsed), dtype=np.bool_), valid + if col.dtype.kind == "S": + return np.array(_fill_missing([v.encode() for v in raw]), dtype=col.dtype), valid + return np.array(_fill_missing(raw), dtype=col.dtype), valid - def _parse(v, _nv=nv): - stripped = v.strip() - if stripped == "" and _nv is not None: - return _nv - return stripped in ("True", "true", "1") + @staticmethod + def _csv_decode_text(raw: list[str], uses_mask: bool) -> tuple[list[str], np.ndarray]: + """Split raw text fields into ``(texts, missing)``. - return np.array([_parse(v) for v in raw], dtype=np.bool_) - if col.dtype.kind == "S": - prepared: list = [nv if (v.strip() == "" and nv is not None) else v.encode() for v in raw] - return np.array(prepared, dtype=col.dtype) - prepared2 = [nv if (v.strip() == "" and nv is not None) else v for v in raw] - return np.array(prepared2, dtype=col.dtype) + Only an *exactly* empty field is missing -- whitespace-only text stays a + real value -- plus the ``\\N``/``\\E`` escapes :meth:`to_csv` writes for a + mask column, where ``""`` and missing would otherwise look the same. + + *texts* carries ``""`` wherever *missing* is True; what a null slot ends + up holding is the caller's decision, since it differs per storage. + """ + decoded = [_csv_text_unescape(v) if uses_mask else (None if v == "" else v) for v in raw] + missing = np.array([v is None for v in decoded], dtype=np.bool_) + return ["" if v is None else v for v in decoded], missing + + @staticmethod + def _csv_text_col_to_cells(raw: list[str], col, nv) -> tuple[list[str], np.ndarray | None]: + """Convert raw CSV fields to ``(cells, valid)`` for a **utf8** column. + + The same missing/fill/validity split as :meth:`_csv_col_to_array`, but + utf8 is fed a list of Python strings instead of a typed array: it has no + fixed element dtype to cast through, which is exactly why it cannot go + through that method at all. + """ + uses_mask = getattr(col.spec, "uses_mask", False) + texts, missing = CTable._csv_decode_text(raw, uses_mask) + if not missing.any(): + return texts, None + if nv is not None: + # Sentinel storage: the null travels in band, as it does on export. + return [nv if m else v for v, m in zip(texts, missing, strict=True)], None + if not uses_mask: + # Not nullable at all, so an empty field is a genuine empty string. + return texts, None + fill = fill_value_for(col.spec) + return [fill if m else v for v, m in zip(texts, missing, strict=True)], ~missing @classmethod def from_csv( @@ -8694,6 +9576,10 @@ def from_csv( each column is bulk-written into a pre-allocated NDArray (one slice assignment per column, no ``extend()``). + Supported column kinds are the fixed-width scalars, :func:`~blosc2.utf8`, + and fixed-shape :func:`~blosc2.ndarray`; anything else raises rather than + being read as something it is not. + Parameters ---------- path: @@ -8717,7 +9603,8 @@ def from_csv( TypeError If *row_cls* is not a dataclass. ValueError - If a row has a different number of fields than the schema. + If a row has a different number of fields than the schema, or if + *row_cls* declares a column kind CSV cannot represent. """ import csv @@ -8753,6 +9640,23 @@ def from_csv( ) new_cols: dict[str, blosc2.NDArray] = {} for col in schema.columns: + if cls._is_utf8_column(col): + # utf8 has no fixed element dtype, so it cannot be created (or + # written) as a plain NDArray the way every other CSV-supported + # kind is. Its own storage handles both. + new_cols[col.name] = mem_storage.create_varlen_scalar_column( + col.name, spec=col.spec, cparams=None, dparams=None + ) + continue + if col.dtype is None: + # Everything below indexes off a fixed element dtype; a kind + # without one would otherwise fail deep in the conversion with + # an AttributeError on None. + raise ValueError( + f"Column {col.name!r}: from_csv() does not support " + f"{type(col.spec).__name__} columns. Supported kinds are the " + f"fixed-width scalars, utf8, and fixed-shape ndarray." + ) shape = cls._column_physical_shape(col, capacity) if col.name in aligned_names: chunks, blocks = shared_chunks, shared_blocks @@ -8792,12 +9696,23 @@ def from_csv( if n > 0: for i, col in enumerate(schema.columns): + nv = getattr(col.spec, "null_value", None) if isinstance(col.spec, NDArraySpec): - arr = cls._csv_ndarray_col_to_array(col_data[i], col) + arr, valid = cls._csv_ndarray_col_to_array(col_data[i], col) + new_cols[col.name][:n] = arr + elif cls._is_utf8_column(col): + cells, valid = cls._csv_text_col_to_cells(col_data[i], col, nv) + # Appended, not slice-assigned: a utf8 column grows to its + # logical length rather than being pre-sized to capacity. + new_cols[col.name].extend(cells) + new_cols[col.name].flush() else: - nv = getattr(col.spec, "null_value", None) - arr = cls._csv_col_to_array(col_data[i], col, nv) - new_cols[col.name][:n] = arr + arr, valid = cls._csv_col_to_array(col_data[i], col, nv) + new_cols[col.name][:n] = arr + if valid is not None: + # A mask column's empty cells: the values above hold the + # fill, and this is the only record that they were missing. + obj._ensure_null_mask(col.name)[:n] = valid new_valid[:n] = True obj._n_rows = n obj._last_pos = n @@ -8815,6 +9730,22 @@ def to_pandas(self): columns become ``object``-dtype columns whose cells hold NumPy arrays of per-row shape *item_shape*. + A **null becomes pandas NA**, never the value that stands in for it in + storage. A column holding one is therefore given a dtype that can say + so -- ``Int64``/``UInt8``/… for integers, ``boolean`` for bools, + ``NaN``/``NaT`` where the NumPy dtype already has one, and ``object`` + with ``None`` for text, bytes, complex and ndarray cells. A column + with **no** null keeps the dtype it has always had here. + + .. note:: + + A **float** column is the one lossy case, in one direction: pandas + spells missing as ``NaN`` in every float dtype it has (even + ``Float64`` folds ``NaN`` into ``NA``), so a mask-backed float + column, where ``NaN`` is an ordinary value, cannot distinguish its + nulls from its NaNs once converted. :meth:`to_arrow` has a + validity bitmap and is exact for this case. + Returns ------- pandas.DataFrame @@ -8844,13 +9775,124 @@ def to_pandas(self): data = {} for name in self.col_names: col = self[name] - if col.is_ndarray: - data[name] = list(col) - else: - data[name] = col[:] + values = list(col) if col.is_ndarray else col[:] + data[name] = self._pandas_values(pd, col, values) return pd.DataFrame(data) + @staticmethod + def _pandas_scalar_series_values(series, col): + """One pandas Series as something :meth:`extend` can write to *col*. + + The plain ``to_numpy(dtype=...)`` is kept as the fast path, so a + DataFrame that converted before converts identically now. It cannot + represent a missing value, though, and raises on any of pandas' NA + carrying extension dtypes -- which is exactly what :meth:`to_pandas` + emits for a nullable column that holds a null, so without the fallback + below a table would not survive its own round trip. + + The fallback hands the cells over one by one and lets the write path + split nullity out, as it already does for a list of rows containing + ``None``. A sentinel column takes its sentinel there, having nowhere + else to put a null. + + A float ``NaN`` stays a **value** (decision 6), not a null: pandas + spells missing in a float column as NaN and cannot tell the two apart, + so the round trip through a float column is lossy under mask storage + in that one direction. Reading it as a value is what keeps every + NaN-free DataFrame converting exactly as it did before. + """ + kind = np.dtype(col.dtype).kind if col.dtype is not None else "O" + # A float column is exempt from the missing-value check: pandas spells + # missing as NaN there and cannot tell it from a NaN value, and + # decision 6 says value. Everywhere else a missing cell has to leave + # the fast path, which does not merely fail on it -- ``to_numpy`` on a + # text column coerces it to the *string* "None"/"nan". + forces_cells = kind != "f" and bool(series.isna().any()) + if not forces_cells: + try: + return series.to_numpy(dtype=col.dtype) + except (ValueError, TypeError): + pass + + def missing(value): + if is_na_marker(value): + return True + # A float NaN is pandas' missing marker in a text or object column, + # and such a column cannot hold a float as data anyway. Decision + # 6's "NaN is a value" is about *float* columns, which took the + # fast path above and never reach here. + return isinstance(value, float) and value != value + + cells = [None if missing(value) else value for value in series.tolist()] + null_value = getattr(col.spec, "null_value", None) + if null_value is not None: + cells = [null_value if value is None else value for value in cells] + return cells + + @staticmethod + def _pandas_values(pd, col, values): + """*values*, with this column's nulls turned into something pandas reads as NA. + + A null slot holds the fill under mask storage and the sentinel under a + sentinel one, and neither is the row's value, so writing either into a + DataFrame states something the column does not say. Each becomes the + NA of a dtype that has one. + + A column with **no null is returned untouched**, so the dtype a caller + already gets never moves under them. Only a column that actually holds + a null can change, and only to a dtype able to express it -- the same + data-dependent widening ``pyarrow.Table.to_pandas`` does, and for the + same reason: NumPy has no missing value for most kinds. + """ + channel = col._nulls + kind = channel.kind + if kind == NULL_MASK: + nulls = channel.null_mask() # one byte per row, off the sidecar + elif kind == NULL_SENTINEL: + nulls = channel.mask_for_values(values) # in band, from what we just read + else: + # NULL_NONE has no nulls at all, and the dictionary and + # native-None kinds already materialize theirs as None, which is + # exactly what pandas wants. + return values + if not nulls.any(): + return values + + dtype = getattr(values, "dtype", None) + dtype_kind = "O" if dtype is None else dtype.kind + if dtype_kind == "f": + out = np.asarray(values).copy() + out[nulls] = np.nan + return out + if dtype_kind == "M": + out = np.asarray(values).copy() + # With the column's own unit: a bare np.datetime64("NaT") carries + # the generic unit, which NumPy deprecates. + out[nulls] = np.datetime64("NaT", np.datetime_data(out.dtype)[0]) + return out + if dtype_kind == "b" or getattr(channel.spec, "bool_widened_to_uint8", False): + # A sentinel bool is physically uint8 to make room for its 255, so + # it arrives here as an integer; either way the column is logically + # bool, and pandas' "boolean" is the dtype that admits NA. + out = pd.array(np.asarray(values).astype(np.bool_), dtype="boolean") + out[nulls] = pd.NA + return out + if dtype_kind in "iu": + # pandas' nullable integer dtypes keep the width, where letting the + # column widen to float would silently round a large int64. + prefix = "Int" if dtype_kind == "i" else "UInt" + out = pd.array(np.asarray(values), dtype=f"{prefix}{dtype.itemsize * 8}") + out[nulls] = pd.NA + return out + # Text, bytes, complex and ndarray cells have no NA-capable NumPy + # dtype, so they go to object with None -- which is what the dictionary + # and variable-length kinds already produce. + out = list(values) + for i in np.flatnonzero(nulls): + out[i] = None + return out + @classmethod def from_pandas(cls, df, row_cls) -> CTable: # noqa: C901 """Build a :class:`CTable` from a pandas DataFrame. @@ -8999,7 +10041,7 @@ def normalize_pandas_missing(value): ): raw_columns[col.name] = [normalize_pandas_missing(value) for value in series.tolist()] else: - raw_columns[col.name] = series.to_numpy(dtype=col.dtype) + raw_columns[col.name] = cls._pandas_scalar_series_values(series, col) obj.extend(raw_columns, validate=True) return obj @@ -9068,6 +10110,13 @@ def add_column( # noqa: C901 A declared default is still honoured for rows appended later, so *values* and ``blosc2.field(..., default=...)`` can be combined. + For a column whose nulls live in a validity sidecar + (``null_storage="mask"``, the default for ``nullable=True``), a + ``None`` entry writes a real null. Declaring ``default=None`` + likewise backfills every existing row as null, which is the only + way to say "this column has no value for the rows that predate + it" -- a sentinel column has to spell that with its sentinel. + Raises ------ ValueError @@ -9113,8 +10162,20 @@ def add_column( # noqa: C901 raise TypeError( "add_column() does not support dictionary columns; use the constructor with a full schema." ) + # Validity for the live rows, from either the values= batch or a + # default=None backfill. Written once at the end, when the column and + # its schema entry are both in place -- _ensure_null_mask needs both to + # pin the sidecar to the column's own row grid. + live_valid = None + uses_mask = getattr(spec, "uses_mask", False) if values is not None: - values = self._add_column_values(name, compiled_col, values, n_live) + values, live_valid = self._add_column_values(name, compiled_col, values, n_live) + # A declared default of None means "no value for the rows that predate + # this column", which only a sidecar can record; a sentinel column falls + # through to the coercion below and reports it cannot hold a None. + default_is_null = uses_mask and default is not MISSING and is_na_marker(default) + if default_is_null and values is None and n_live > 0: + live_valid = np.zeros(n_live, dtype=np.bool_) if self._is_varlen_scalar_column(compiled_col): # Varlen scalar columns don't use fixed-width NDArray storage, but the @@ -9142,13 +10203,22 @@ def add_column( # noqa: C901 new_col.extend(padded) new_col.flush() else: - if default is not MISSING: + if default_is_null: + # The stored cell is the fill; live_valid above is what records + # that nothing is there to read. + default_val = fill_item_for(spec) + elif default is not MISSING: try: if self._is_ndarray_column(compiled_col): default_val = self._coerce_ndarray_value(name, spec, default) + elif isinstance(spec, timestamp): + # Through the unit, as the values= path above does. + default_val = self._timestamps_to_stored_int64( + spec, [default], np.dtype(spec.dtype) + )[0] else: default_val = spec.dtype.type(default) - except (ValueError, OverflowError) as exc: + except (ValueError, OverflowError, TypeError) as exc: raise TypeError( f"Cannot coerce default {default!r} to dtype {spec.dtype!r}: {exc}" ) from exc @@ -9192,6 +10262,12 @@ def add_column( # noqa: C901 columns=new_columns, columns_by_name={**self._schema.columns_by_name, name: compiled_col}, ) + # Last, because _ensure_null_mask reads both self._cols[name] and the + # schema entry to pin the sidecar to the column's own row grid. A batch + # that held no null leaves live_valid None and writes nothing, so a + # null-free column stays sidecar-free (decision 9). + if live_valid is not None: + self._ensure_null_mask(name)[np.flatnonzero(self._valid_rows[:])] = live_valid if isinstance(self._storage, FileTableStorage): self._storage.save_schema(self._schema_dict_with_computed()) @@ -9203,6 +10279,11 @@ def _varlen_filler(spec, default): just has to be something the spec accepts. """ if default is not MISSING: + # A None default is the caller asking for nulls, not for a literal + # None cell: under mask storage the stored cell is the fill, and the + # sidecar is what says the row is missing. + if getattr(spec, "uses_mask", False) and is_na_marker(default): + return fill_item_for(spec) return default null_value = getattr(spec, "null_value", None) if null_value is not None: @@ -9213,11 +10294,72 @@ def _varlen_filler(spec, default): return "" return None + @staticmethod + def _timestamps_to_stored_int64(spec, values, target_dtype): + """Convert a batch of timestamps to the int64 the column actually stores. + + The conversion has to go **through the spec's unit**: a bare + ``astype(int64)`` on a ``datetime64[s]`` array yields seconds, which a + microsecond column then reads back as a date in 1970. Shared by + ``extend`` and ``add_column``, which both have to store the same + encoding and only one of which used to. + + Values that are already integers pass through, which is what lets a + mask column's fill (``int64.min``, decoded as ``NaT``) survive the + object branch untouched. + """ + arr = np.asarray(values) + if np.issubdtype(arr.dtype, np.datetime64): + return arr.astype(f"datetime64[{spec.unit}]").astype(np.int64) + if arr.dtype.kind in "OUS": + return np.array( + [ + spec.null_value + if v is None + else np.datetime64(v).astype(f"datetime64[{spec.unit}]").astype(np.int64) + if isinstance(v, (np.datetime64, str)) or hasattr(v, "isoformat") + else v + for v in arr + ], + dtype=target_dtype, + ) + return arr + + @staticmethod + def _reject_nulls_this_column_cannot_store(name: str, col: CompiledColumn, values) -> None: + """Say why a ``None`` in ``values=`` cannot be stored, and what to do instead. + + Only a fixed-width column whose nulls live in a sidecar can take a bare + ``None``. Called from the ``astype`` failure path, where NumPy's own + ``int() argument must be ...`` names neither the column nor the way + out; returns quietly when the batch held no null, leaving the caller to + report whatever else went wrong. + """ + _, invalid = split_batch_validity(values, None) + if invalid is None: + return + sentinel = getattr(col.spec, "null_value", None) + remedy = ( + f"write its null_value ({sentinel!r}) instead" + if sentinel is not None + else 'declare the column nullable, e.g. nullable=True (or null_storage="mask")' + ) + raise TypeError( + f"add_column() values= for {name!r} contains a null, which this column cannot store: {remedy}." + ) + def _add_column_values(self, name: str, col: CompiledColumn, values, n_live: int): """Validate and coerce the ``values=`` argument of :meth:`add_column`. - Returns a list for varlen scalar columns (which are fed row by row) and - a dtype-coerced ndarray for the fixed-width ones. + Returns ``(values, valid)``: a list for varlen scalar columns (which are + fed row by row) or a dtype-coerced ndarray for the fixed-width ones, + plus the validity of a mask-storage column whose batch held a null + (``None`` when nothing was, exactly as ``coerce_batch`` reports it). + + Nullity is split out **before** validation and the ``astype`` below, for + the same reason ``extend`` does it: a null cell has no value to + constrain, and ``np.asarray([1, None, 3]).astype(int64)`` raises rather + than yielding anything to store. Constraints declared on the spec are checked here, *before* the ``astype`` below: coercing to a fixed-width dtype truncates a too-long @@ -9226,6 +10368,8 @@ def _add_column_values(self, name: str, col: CompiledColumn, values, n_live: int """ from blosc2.schema_vectorized import validate_column_values + uses_mask = getattr(col.spec, "uses_mask", False) + if self._is_varlen_scalar_column(col): values = list(values) if len(values) != n_live: @@ -9233,8 +10377,20 @@ def _add_column_values(self, name: str, col: CompiledColumn, values, n_live: int f"add_column() values= for {name!r} requires {n_live} entries " f"(live rows), got {len(values)}." ) + valid = None + if uses_mask: + values, valid = split_batch_validity(values, fill_item_for(col.spec)) + values = list(values) + # No null check for the other varlen kinds: the container ones hold + # a native None, and a sentinel utf8 column maps None onto its + # sentinel in _ScalarVarLenArray._coerce. Only the fixed-width + # branch below has nowhere to put one. validate_column_values(col, values) - return values + return values, valid + + valid = None + if uses_mask: + values, valid = split_batch_validity(values, fill_item_for(col.spec)) arr = values[:] if isinstance(values, blosc2.NDArray) else np.asarray(values) if len(arr) != n_live: @@ -9247,9 +10403,17 @@ def _add_column_values(self, name: str, col: CompiledColumn, values, n_live: int f"add_column() values= for {name!r} must have shape {expected}, got {arr.shape}." ) validate_column_values(col, arr) + if isinstance(col.spec, timestamp): + # After validation, which sees the datetimes the caller passed, and + # before the astype, which would otherwise drop the unit. + arr = self._timestamps_to_stored_int64(col.spec, arr, np.dtype(col.spec.dtype)) try: - return arr.astype(col.spec.dtype) - except (ValueError, OverflowError) as exc: + return arr.astype(col.spec.dtype), valid + except (ValueError, OverflowError, TypeError) as exc: + # Only on the failure path, so a well-formed batch never pays for + # the scan: a null is the likeliest reason a batch will not cast, + # and NumPy's own message names neither the column nor the way out. + self._reject_nulls_this_column_cannot_store(name, col, values) raise TypeError( f"Cannot coerce values= for {name!r} to dtype {col.spec.dtype!r}: {exc}" ) from exc @@ -9297,7 +10461,10 @@ def drop_column(self, name: str) -> None: self._invalidate_index_catalog_cache() if isinstance(self._storage, FileTableStorage): + # delete_column already removes the column's companion keys, the + # validity sidecar among them; drop the cached handle to match. self._storage.delete_column(name) + self._null_masks.pop(name) self._materialized_cols.pop(name, None) del self._cols[name] @@ -9377,8 +10544,15 @@ def rename_column(self, old: str, new: str) -> None: if isinstance(self._storage, FileTableStorage): self._cols[new] = self._rename_stored_column(old, new) + # storage.rename_column re-keyed the validity sidecar too, leaving + # any cached handle pointing at a key that no longer exists. Drop + # it and let the cache reopen under the new name. + self._null_masks.pop(old) + self._null_masks.pop(new) else: self._cols[new] = self._cols[old] + # Nothing was re-keyed on disk (there is no disk): carry the handle. + self._null_masks.rename(old, new) del self._cols[old] idx = self.col_names.index(old) @@ -9421,6 +10595,434 @@ def _rename_stored_column(self, old: str, new: str): renamed_col = self._storage.open_varlen_scalar_column(new, old_compiled_col.spec) return renamed_col + # ------------------------------------------------------------------ + # Null-storage migration + # ------------------------------------------------------------------ + + #: Rows scanned per pass by :meth:`convert_nulls`. Bounds the transient + #: value and validity buffers without changing the result. + _CONVERT_NULLS_SPAN: ClassVar[int] = 1 << 20 + + def convert_nulls( + self, + columns: str | list[str] | None = None, + *, + to: str = "mask", + null_value: Any = None, + inplace: bool = False, + ) -> CTable: + """Convert nullable columns between sentinel and validity-mask storage. + + **Never called implicitly.** Opening, copying and saving a table all + preserve each column's existing null storage, so this is the only way a + column changes where it keeps its nulls. + + Parameters + ---------- + columns: + Column name, or list of names, to convert. ``None`` (default) + converts every column that *can* be converted and is not already in + the target storage. Naming a column that cannot be converted is an + error; an implicit sweep skips it silently. + to: + ``"mask"`` (default) moves nullity into a ``.notnull`` sidecar -- + Arrow's model, and the only one that is lossless for every type. + ``"sentinel"`` moves it back in band, and **refuses what it cannot + represent**: a column whose data already contains the proposed + sentinel, or one with no value left to reserve. + null_value: + The sentinel to reserve, for ``to="sentinel"``. Only accepted when + converting exactly one column, since a single value cannot be right + for several kinds at once; otherwise the active + :class:`NullPolicy`'s type-wide default is used. + inplace: + If ``True``, rewrite this table and return ``self``. If ``False`` + (default, and recommended) return a new **in-memory** table, leaving + this one untouched -- pass the result to ``copy(urlpath=...)`` to + land it back on disk. + + ``inplace=True`` on a *persistent* table cannot change a column's + dtype, which rules out ``bool`` (``uint8`` <-> ``np.bool_``) and a + ``string``/``bytes`` column too narrow for the sentinel it is being + given. Replacing a stored array means there is a moment when the + values and the schema disagree, and no ordering of the two survives + a crash in between; ``inplace=False`` has no such moment because it + builds a new table. Those columns raise, naming themselves. + + Notes + ----- + Everything else *is* crash-safe, and the ordering is the argument: + + ``to="mask"`` writes the complete sidecar first, then flips the schema, + then normalizes the null slots to the fill. A crash before the flip + leaves an orphan ``.notnull`` key that a sentinel column never reads; a + crash after it leaves a correct mask column whose null slots happen to + still hold the old sentinel -- which is unobservable except as raw + values, and the fill is explicitly not part of the format contract. + + ``to="sentinel"`` runs the same argument backwards: it writes the + sentinel into the null slots *first* (harmless while the sidecar is + still authoritative), then flips the schema, then drops the sidecar. + + A **null-free** column converts as a pure schema update in either + direction: nothing is scanned twice, and to="mask" writes no sidecar at + all, since an absent one already means "every row is valid". + + ``string``/``bytes`` columns keep the ``max_length`` a sentinel forced on + them -- shrinking it is a dtype change, so use ``copy()`` to reclaim the + width. Dictionary columns (reserved code) and the variable-length + container kinds (native ``None`` cells) have only one representation and + are never converted. + + Examples + -------- + >>> lossless = t.convert_nulls() # doctest: +SKIP + >>> lossless.copy(urlpath="out.b2d") # doctest: +SKIP + >>> t.convert_nulls("flag", to="mask", inplace=True) # doctest: +SKIP + """ + if to not in (NULL_MASK, NULL_SENTINEL): + raise ValueError(f"to must be 'mask' or 'sentinel', got {to!r}") + if null_value is not None and to != NULL_SENTINEL: + raise ValueError("null_value only applies to to='sentinel'") + if isinstance(columns, str): + columns = [columns] + # A dtype change means replacing an array, which only an in-memory table + # can do safely; inplace=False lands there by construction, since it + # converts the copy. + targets = self._convert_null_targets( + columns, + to, + null_value, + allow_recast=not (inplace and isinstance(self._storage, FileTableStorage)), + ) + + if not inplace: + # copy() preserves each column's null storage (decision 7), so the + # conversion happens exactly once, on a table nobody else holds -- + # and in memory, where replacing an array costs nothing. + result = self.copy() + result._convert_nulls_inplace(targets, to) + return result + if self._read_only: + raise ValueError("Table is read-only (opened with mode='r').") + if self.base is not None: + raise ValueError( + "Cannot convert a view's null storage inplace (it shares the base table's " + "columns). Use convert_nulls(inplace=False) to get a converted copy." + ) + self._convert_nulls_inplace(targets, to) + return self + + def _convert_null_targets( + self, columns: list[str] | None, to: str, null_value: Any, *, allow_recast: bool + ) -> list[tuple[str, Any]]: + """Resolve the columns to convert into ``[(name, sentinel)]``. + + *sentinel* is the value to reserve (``to="sentinel"``) or ``None``. + Everything that can make a conversion impossible is decided here, before + a single byte is written -- an unavailable sentinel, one the data already + uses, a dtype change a persistent table cannot make -- so a refusal + leaves the table exactly as it was rather than half converted. + """ + if columns is None: + names = [ + col.name + for col in self._schema.columns + if kind_of_spec(col.spec) in (NULL_MASK, NULL_SENTINEL) and kind_of_spec(col.spec) != to + ] + else: + names = [] + for name in columns: + col_info = self._schema.columns_by_name.get(name) + if col_info is None: + raise KeyError(f"Column {name!r} not found.") + kind = kind_of_spec(col_info.spec) + if kind == NULL_NONE: + raise ValueError(f"Column {name!r} is not nullable, so it has no nulls to convert.") + if kind not in (NULL_MASK, NULL_SENTINEL): + raise ValueError( + f"Column {name!r} stores its nulls as {kind!r}, which is the only " + f"representation its kind has -- a dictionary column reserves a code and " + f"the variable-length kinds hold native None. There is nothing to convert." + ) + if kind != to: + names.append(name) # already converted: skip, so this stays idempotent + if null_value is not None and len(names) != 1: + raise ValueError( + f"null_value applies to a single column, but {len(names)} would be converted; " + "name one column, or drop null_value to use the NullPolicy default for each kind." + ) + if to == NULL_MASK: + targets = [(name, None) for name in names] + else: + targets = [(name, self._sentinel_for_conversion(name, null_value)) for name in names] + if not allow_recast: + for name, sentinel in targets: + spec = self._schema.columns_by_name[name].spec + if self._convert_changes_dtype(spec, to, sentinel): + raise ValueError( + f"Converting column {name!r} changes its physical dtype (a nullable bool is " + f"{'uint8 under sentinel storage' if to == NULL_SENTINEL else 'np.bool_ under mask storage'}" + f"; a string or bytes column too narrow for its sentinel has to grow), which " + f"means replacing the stored array -- and there is no ordering of that write " + f"and the schema update that a crash cannot land between. Use " + f"convert_nulls(inplace=False), then copy(urlpath=...) to store the result." + ) + return targets + + def _sentinel_for_conversion(self, name: str, null_value: Any): + """Pick and vet the sentinel column *name* will reserve. + + Rejects, before any write, the two ways a sentinel can be unavailable: + a kind with no value to spare at all, and a value the column's own data + already uses -- which would silently relabel real rows as null, the very + loss mask storage exists to avoid. + """ + spec = self._schema.columns_by_name[name].spec + if not getattr(spec, "supports_sentinel", True): + raise ValueError( + f"Column {name!r} has dtype {spec.dtype}, for which no value can be reserved as a " + f"null sentinel. It can only use mask storage." + ) + if null_value is None: + null_value = self._policy_null_value_for_spec(spec, get_null_policy()) + if null_value is None: + raise ValueError( + f"No null policy sentinel is available for column {name!r}; pass " + f"null_value= explicitly, or keep the column on mask storage." + ) + self._validate_null_value_for_spec(name, spec, null_value) + col = self[name] + clash = self._convert_sentinel_clash(col, spec, null_value) + if clash is not None: + raise ValueError( + f"Column {name!r} already contains {null_value!r} at row {clash} as a real value, " + f"so reserving it as the null sentinel would relabel that row as null. Pass a " + f"different null_value=, or keep the column on mask storage." + ) + return null_value + + def _convert_sentinel_clash(self, col: Column, spec, null_value) -> int | None: + """The first non-null row already holding *null_value*, or ``None``. + + A NaN sentinel never clashes: under mask storage NaN is a value, but it + is one the sentinel model has always spelled "null", so folding the two + together is the documented semantic change rather than data loss. + """ + if is_nan_sentinel(null_value): + return None + valid_arr = self._null_mask(col._col_name) + item_ndim = col.item_ndim if col.is_ndarray else 0 + for start, stop, raw in self._iter_convert_spans(col._col_name): + hit = sentinel_mask(raw, null_value, item_ndim=item_ndim) + if valid_arr is not None: + hit &= np.asarray(valid_arr[start:stop], dtype=bool) + if hit.any(): + return start + int(np.flatnonzero(hit)[0]) + return None + + def _iter_convert_spans(self, name: str): + """Yield ``(start, stop, values)`` over column *name*'s written slots. + + Physical, and bounded to the write watermark rather than the capacity: + the padding past it holds no row, and a utf8 column is not even that + long. + """ + arr = self._cols[name] + n_phys = min(self._resolve_last_pos(), len(arr)) + span = self._CONVERT_NULLS_SPAN + for start in range(0, n_phys, span): + stop = min(start + span, n_phys) + yield start, stop, np.asarray(arr[start:stop]) + + def _detach_schema(self) -> None: + """Give this table its own copy of every column spec. + + ``_empty_copy`` hands the copy the *same* :class:`CompiledSchema` object, + which is right for everything that only reads it -- and wrong for the one + thing that does not. Conversion rewrites specs **in place**, and without + this a converted copy would reach back and relabel its source's columns + too, leaving that table reporting a storage its data does not use. + """ + columns = [dataclasses.replace(c, spec=copy.deepcopy(c.spec)) for c in self._schema.columns] + self._schema = CompiledSchema( + row_cls=self._schema.row_cls, + columns=columns, + columns_by_name={c.name: c for c in columns}, + ) + + def _convert_nulls_inplace(self, targets: list[tuple[str, Any]], to: str) -> None: + """Rewrite each target column's null channel, one column at a time. + + Per column rather than per phase, so an interrupted conversion leaves + every other column exactly as it was. + """ + self._detach_schema() + for name, sentinel in targets: + if to == NULL_MASK: + self._convert_column_to_mask(name) + else: + self._convert_column_to_sentinel(name, sentinel) + + def _convert_column_to_mask(self, name: str) -> None: + col_info = self._schema.columns_by_name[name] + spec = col_info.spec + sentinel = spec.null_value + item_ndim = self[name].item_ndim if self[name].is_ndarray else 0 + + # 1. The sidecar, complete, before the schema mentions it. + mask_arr = None + for start, stop, raw in self._iter_convert_spans(name): + null = sentinel_mask(raw, sentinel, item_ndim=item_ndim) + if not null.any(): + continue + if mask_arr is None: + mask_arr = self._ensure_null_mask(name) + # create_null_mask zero-fills, and zero is *invalid*; start from + # all-valid and punch the nulls out below. + mask_arr[:] = True + mask_arr[start:stop] = ~null + + # 2. The schema. A column with no null needs nothing else: an absent + # sidecar already says every row is valid (decision 9). + self._retype_converted_column(name, to=NULL_MASK, null_value=None) + if mask_arr is None: + return + + # 3. Normalize the null slots to the fill. Purely cosmetic -- what sits + # under valid=False is unobservable through the Column API -- so it + # runs last, where an interruption costs nothing. A bool column + # *needs* it: step 2 recast uint8 to np.bool_, turning the old 255 + # sentinel into True. + self._write_at_invalid(name, mask_arr, self[name]._nulls.fill_value) + + def _write_at_invalid(self, name: str, mask_arr, value) -> None: + """Overwrite the values under ``mask_arr == False`` with *value*.""" + col = self[name] + arr = self._cols[name] + n_phys = min(self._resolve_last_pos(), len(arr)) + span = self._CONVERT_NULLS_SPAN + for start in range(0, n_phys, span): + stop = min(start + span, n_phys) + invalid = ~np.asarray(mask_arr[start:stop], dtype=bool) + if not invalid.any(): + continue + positions = np.flatnonzero(invalid) + start + if col.is_utf8: + # A UTF8Array assigns one row at a time; the null rows are a + # small subset by construction. + for pos in positions: + arr[int(pos)] = value + elif isinstance(value, np.ndarray): + # An ndarray column's fill is a whole *item*; the fancy-index + # setitem wants one item per selected row rather than something + # to broadcast across them. + arr[positions] = np.broadcast_to(value, (len(positions), *value.shape)) + else: + arr[positions] = value + + def _convert_column_to_sentinel(self, name: str, sentinel) -> None: + spec = self._schema.columns_by_name[name].spec + mask_arr = self._null_mask(name) + # A bool column's 255 does not fit np.bool_, and a widened string's + # sentinel does not fit the old itemsize, so those have to be recast + # before the sentinel can be written at all. Both are in-memory-only + # (see _recast_converted_column), where the ordering below buys nothing. + recast_first = self._convert_changes_dtype(spec, NULL_SENTINEL, sentinel) + + if recast_first: + self._retype_converted_column(name, to=NULL_SENTINEL, null_value=sentinel) + if mask_arr is not None: + # The sentinel goes into the null slots while the sidecar is still + # what says which those are, so this write cannot lose anything. + self._write_at_invalid(name, mask_arr, sentinel) + if not recast_first: + # The schema last, which is what makes those slots mean "null". + self._retype_converted_column(name, to=NULL_SENTINEL, null_value=sentinel) + if mask_arr is not None: + # Now dead weight: a sentinel column never reads a sidecar. + self._drop_null_mask(name) + + @staticmethod + def _convert_changes_dtype(spec, to: str, null_value) -> bool: + """Whether this conversion also changes the column's physical dtype. + + Only three ways it can: a bool column loses or regains the ``uint8`` + widening, and a ``string``/``bytes`` column too narrow for its new + sentinel has to grow. + """ + if to == NULL_MASK: + return getattr(spec, "bool_widened_to_uint8", False) is True + if isinstance(spec, (string, b2_bytes)): + return len(null_value) > spec.max_length + if isinstance(spec, b2_bool): + return spec.dtype != np.dtype(np.uint8) + return isinstance(spec, NDArraySpec) and spec.dtype == np.dtype(np.bool_) + + def _retype_converted_column(self, name: str, *, to: str, null_value) -> None: + """Rewrite column *name*'s spec for its new null channel, and persist it. + + The dtype consequences are the same ones ``_resolve_nullable_specs`` + applies at creation, run in the other direction: mask storage undoes the + ``bool -> uint8`` flip, and sentinel storage reapplies it and widens a + ``string``/``bytes`` column that cannot hold its sentinel. A dtype + change means a new array, which is why it is refused for a persistent + in-place conversion. + """ + col_info = self._schema.columns_by_name[name] + spec = col_info.spec + before = spec.dtype + spec.nullable = True + spec.null_storage = to + spec.null_value = null_value + if to == NULL_MASK: + self._unflip_mask_bool_dtype(spec) + elif isinstance(spec, string): + spec.max_length = max(spec.max_length, len(null_value), 1) + spec.dtype = np.dtype(f"U{spec.max_length}") + elif isinstance(spec, b2_bytes): + spec.max_length = max(spec.max_length, len(null_value), 1) + spec.dtype = np.dtype(f"S{spec.max_length}") + elif isinstance(spec, b2_bool) or ( + isinstance(spec, NDArraySpec) and spec.dtype == np.dtype(np.bool_) + ): + self._widen_bool_dtype_to_uint8(spec) + if spec.dtype != before: + self._recast_converted_column(name, col_info, before) + col_info.dtype = getattr(spec, "dtype", None) + col_info.display_width = compute_display_width(spec) + self._col_widths[name] = max(len(name), col_info.display_width) + if isinstance(self._storage, FileTableStorage): + self._storage.save_schema(self._schema_dict_with_computed()) + + def _recast_converted_column(self, name: str, col_info, before: np.dtype) -> None: + """Replace column *name*'s array so it matches its new dtype. + + In-memory only, and :meth:`_convert_null_targets` has already refused the + persistent case: there the values and the schema would disagree for as + long as the write takes, and no ordering of the two survives a crash in + between. Here there is no crash window to protect, so the values are + simply cast across. + """ + spec = col_info.spec + old = self._cols[name] + shape = self._column_physical_shape(col_info, len(self._valid_rows)) + new = self._storage.create_column( + name, + dtype=spec.dtype, + shape=shape, + chunks=old.chunks, + blocks=old.blocks, + cparams=None, + dparams=None, + ) + n_phys = min(self._resolve_last_pos(), len(old)) + span = self._CONVERT_NULLS_SPAN + for start in range(0, n_phys, span): + stop = min(start + span, n_phys) + new[start:stop] = np.asarray(old[start:stop]).astype(spec.dtype, copy=False) + self._cols[name] = new + # ------------------------------------------------------------------ # Computed / virtual columns # ------------------------------------------------------------------ @@ -9733,7 +11335,13 @@ def _invalidate_all_summary_accumulators(self) -> None: def _precomputed_summary_for(self, name: str): """Return ``{"block": summaries}`` for *name* if a valid accumulator - fully covers the column's physical extent, else None.""" + fully covers the column's physical extent, else None. + + These are folded as rows are written, with no validity in hand, so the + index builder ignores them for a column that has nulls and decompresses + instead (see ``_build_levels_descriptor_ooc``). A nullable column with + no nulls keeps the fast path. + """ accs = self.__dict__.get("_summary_accumulators") if not accs: return None @@ -11447,6 +13055,10 @@ def compact(self): start += block_size end = min(end + block_size, self._n_rows) + # Shuffle the validity sidecars by the same gather, so mask and values + # stay row-aligned. + self._permute_null_masks(real_poss, self._n_rows) + self._valid_rows[: self._n_rows] = True self._valid_rows[self._n_rows :] = False self._last_pos = self._n_rows @@ -11529,6 +13141,7 @@ def _sorted_positions_from_full_index(self, name: str, ascending: bool) -> np.nd null_value = None null_code = None + valid_arr = None is_dict_rank = False if name in root._cols: col_info = root._schema.columns_by_name.get(name) @@ -11536,6 +13149,8 @@ def _sorted_positions_from_full_index(self, name: str, ascending: bool) -> np.nd null_value = getattr(col_info.spec, "null_value", None) if isinstance(col_info.spec, DictionarySpec): null_code = col_info.spec.null_code + elif getattr(col_info.spec, "uses_mask", False): + valid_arr = root._null_mask(name) descriptor = catalog.get(name) if descriptor is None or descriptor.get("kind") != "full" or descriptor.get("stale", False): descriptor = None @@ -11608,43 +13223,39 @@ def _sorted_positions_from_full_index(self, name: str, ascending: bool) -> np.nd current_valid = self._valid_rows[:] positions = positions[current_valid[positions]] + # The index sorts by stored value, but sort_by's contract is nulls-last. + # Partition explicitly so it holds for either order and for any way of + # spelling a null -- a NaN sentinel sorts last on its own, an integer + # sentinel like INT64_MIN sorts first, and a mask column's fill sorts + # wherever an ordinary 0 or "" would. Free each 24M-element temporary + # as soon as it is consumed to keep peak memory near the size of the + # permutation itself. + null_phys = None if is_dict_rank: - # Dict-rank index: positions sorted by rank (int32), nulls have sentinel null_rank. - # Partition null rows using codes (int32), not decoded strings. + # Positions are sorted by rank (int32), nulls carrying null_rank. + # Partition using the codes, not the decoded strings. codes = np.asarray(root._cols[name].codes[:], dtype=np.int32) null_phys = codes == null_code del codes - if null_phys.any(): - is_null = null_phys[positions] - del null_phys - nulls = positions[is_null] - nonnull = positions[~is_null] - del is_null, positions - if not ascending: - nonnull = nonnull[::-1] - return np.concatenate([nonnull, nulls]) - # No nulls: fall through to simple reverse + elif valid_arr is not None: + # Mask storage: one byte per row off the sidecar rather than the + # whole raw column, which is where this path used to spend most of + # its I/O -- 8x for int64, 64x for a U16 string column. + null_phys = ~np.asarray(valid_arr[:]) elif null_value is not None: - # The index sorts by raw value, but sort_by's contract is nulls-last. - # Partition explicitly so it holds for any sentinel (NaN sorts last, - # an integer sentinel like INT64_MIN sorts first) and either order. - # Free each 24M-element temporary as soon as it is consumed to keep - # peak memory near the size of the permutation itself. raw = np.asarray(root._cols[name][:]) - if isinstance(null_value, float) and np.isnan(null_value): - null_phys = np.isnan(raw) - else: - null_phys = raw == null_value + null_phys = sentinel_mask(raw, null_value) del raw - if null_phys.any(): - is_null = null_phys[positions] - del null_phys - nulls = positions[is_null] - nonnull = positions[~is_null] - del is_null, positions - if not ascending: - nonnull = nonnull[::-1] - return np.concatenate([nonnull, nulls]) + + if null_phys is not None and null_phys.any(): + is_null = null_phys[positions] + del null_phys + nulls = positions[is_null] + nonnull = positions[~is_null] + del is_null, positions + if not ascending: + nonnull = nonnull[::-1] + return np.concatenate([nonnull, nulls]) if not ascending: positions = positions[::-1] @@ -11702,6 +13313,11 @@ def _build_lex_keys( else: raw = gathered[name] if name in gathered else self._cols[name][live_pos] nv = getattr(col_info.spec, "null_value", None) if col_info else None + valid = ( + self._null_mask(name) + if col_info is not None and getattr(col_info.spec, "uses_mask", False) + else None + ) # Value key if not asc: @@ -11709,7 +13325,14 @@ def _build_lex_keys( # strings can't be negated — invert via rank rank = np.argsort(np.argsort(raw, kind="stable"), kind="stable") lex_keys.append((n - 1 - rank).astype(np.intp)) - elif np.issubdtype(raw.dtype, np.unsignedinteger): + elif raw.dtype.kind in "bui": + # Negate in int64, never in the column's own dtype. bool has + # no unary minus at all, and a narrow signed dtype wraps on + # its own minimum -- ``-(-128)`` is ``-128`` in int8 -- which + # silently leaves that row sorted as if it were the largest. + # Mask storage is what made both reachable in a *nullable* + # column: a sentinel had to reserve int8's -128, and a + # nullable bool was physically uint8. lex_keys.append(-raw.astype(np.int64)) else: lex_keys.append(-raw) @@ -11720,12 +13343,15 @@ def _build_lex_keys( # so nulls always sort last (0 before 1 → non-null before null). if is_dict_col and col_info.spec.nullable: lex_keys.append(is_null.astype(np.intp)) + elif valid is not None: + # Mask storage: the fill occupying a null slot is an ordinary + # value as far as lexsort is concerned (0 and "" both sort + # first), so this key is the whole of nulls-last here -- not + # just a refinement of the value key as it is for a sentinel. + # One byte per row, and no string comparison for U/S. + lex_keys.append((~np.asarray(valid[live_pos])).astype(np.intp)) elif nv is not None: - if isinstance(nv, float) and np.isnan(nv): - null_ind = np.isnan(raw).astype(np.intp) - else: - null_ind = (raw == nv).astype(np.intp) - lex_keys.append(null_ind) + lex_keys.append(sentinel_mask(raw, nv).astype(np.intp)) return lex_keys @@ -11876,6 +13502,20 @@ def _sorted_slice_positions(self, name: str, ascending: bool, key: slice) -> np. return None col_info = self._schema.columns_by_name.get(name) + if ( + col_info is not None + and getattr(col_info.spec, "uses_mask", False) + and self._null_mask(name) is not None + ): + # Mask storage with at least one null: the null rows hold this + # column's fill, which is an ordinary value in the sorted sidecar, + # so bisecting for it would sweep up genuine fill-valued rows as + # well. Only the sidecar can tell the two apart, and it is indexed + # by physical position, not by sorted position, so the window read + # cannot consult it. Fall back to the full sorted view, which is + # mask-aware. A mask column that has never held a null keeps this + # path: with no sidecar there is no null block to locate. + return None null_value = getattr(col_info.spec, "null_value", None) if col_info is not None else None # Rank index: the sidecar holds int32 ranks, so the null block is located # by null_rank, not by the column's own sentinel (a string, for utf8). @@ -11947,7 +13587,7 @@ def _null_block_bounds(self, full: dict, null_value, n: int) -> tuple[int, int]: from blosc2.indexing import _open_sidecar_file vnd = _open_sidecar_file(full["values_path"]) - if isinstance(null_value, float) and np.isnan(null_value): + if is_nan_sentinel(null_value): # NaN sorts last and breaks ordered comparisons, so count the trailing # block directly, one chunk at a time (peak memory = a single chunk). chunk = int(vnd.chunks[0]) if vnd.chunks else len(vnd) @@ -12005,6 +13645,7 @@ def _sorted_small_copy_from_live_positions( result._cols[col_name].codes[:n] = gathered[col_name][order] else: result._cols[col_name][:n] = gathered[col_name][order] + self._gather_null_masks_into(result, live_pos[order], n) result._valid_rows[:n] = True result._valid_rows[n:] = False result._n_rows = n @@ -12028,6 +13669,7 @@ def _sort_by_inplace(self, sorted_pos: np.ndarray, n: int) -> None: arr.set_all(arr[sorted_pos]) else: arr[:n] = arr[sorted_pos] + self._permute_null_masks(sorted_pos, n) self._valid_rows[:n] = True self._valid_rows[n:] = False self._n_rows = n @@ -12053,6 +13695,7 @@ def _sorted_copy_from_positions(self, sorted_pos: np.ndarray, n: int) -> CTable: result._cols[col_name].set_all(arr[sorted_pos]) else: result._cols[col_name][:n] = arr[sorted_pos] + self._gather_null_masks_into(result, sorted_pos, n) result._valid_rows[:n] = True result._valid_rows[n:] = False result._n_rows = n @@ -12223,10 +13866,22 @@ def copy( # noqa: C901 arr[:n_live] if is_dense else (arr[live_pos] if compact else arr[:n]) ) + # Validity sidecars follow their columns through the same gather. A + # column with no sidecar gets none: nothing in it is null, and copying + # must not invent storage the source never needed (decision 9). + if n > 0: + for col_name, mask in self._existing_null_masks().items(): + result._ensure_null_mask(col_name)[:n] = ( + mask[:n_live] if is_dense else (mask[live_pos] if compact else mask[:n]) + ) + if compact: result._valid_rows[:n] = True result._n_rows = n - result._last_pos = n - 1 if n > 0 else None + # Exclusive bound, the convention _resolve_last_pos() documents and + # every other writer follows -- n - 1 here made add_column() on a + # copied table backfill one row short (and raise, for a varlen one). + result._last_pos = n if n > 0 else None else: result._valid_rows[:n] = valid_np[:n] result._n_rows = n_live @@ -12488,9 +14143,10 @@ def info_items(self) -> list[tuple[str, object]]: ) else: col_meta = self._schema.columns_by_name.get(name) + spec = col_meta.spec if col_meta else None dtype_label = self._dtype_info_label( - getattr(self._cols[name], "dtype", None), col_meta.spec if col_meta else None - ) + getattr(self._cols[name], "dtype", None), spec + ) + self._null_info_tag(spec) cbytes = getattr(self._cols[name], "cbytes", None) if cbytes is not None: nbytes = getattr(self._cols[name], "nbytes", None) @@ -12556,6 +14212,17 @@ def _valid_rows_info_label(self) -> str: detail += f", cratio: {nbytes / cbytes:.2f}x" return f"{dtype_label} ({detail})" + @staticmethod + def _null_info_tag(spec: SchemaSpec | None) -> str: + """``" nullable[mask]"`` and friends, or ``""`` for a non-nullable column. + + Appended to a column's dtype in :attr:`info`'s per-column summary, which + is where a whole table's null storage becomes visible at a glance -- what + you read before deciding whether to run :meth:`convert_nulls`. + """ + kind = kind_of_spec(spec) + return "" if kind == NULL_NONE else f" nullable[{kind}]" + @staticmethod def _dtype_info_label(dtype: np.dtype | None, spec: SchemaSpec | None = None) -> str: """Return a compact dtype label for info reports.""" @@ -12660,7 +14327,7 @@ def append(self, data: list | np.void | np.ndarray) -> None: from blosc2.schema_validation import validate_row row = validate_row(self._schema, row) - row = self._coerce_row_to_storage(row) + row, null_names = self._coerce_row_to_storage(row) pos = self._resolve_last_pos() if pos >= len(self._valid_rows): @@ -12679,7 +14346,12 @@ def append(self, data: list | np.void | np.ndarray) -> None: if acc is not None and acc.valid: acc.feed(pos, np.asarray([row[name]], dtype=col_array.dtype)) + for name in null_names: + self._null_channel(name).set_valid(pos, False) + n_rows = self.nrows + # As in extend(): the row only becomes visible on the next line, so + # both the values and the validity above are written under its cover. self._valid_rows[pos] = True self._last_pos = pos + 1 self._n_rows = n_rows + 1 @@ -12715,6 +14387,49 @@ def delete(self, ind: int | slice | str | Iterable) -> None: self._last_pos = None # last live row deleted; recalculate on next write self._storage.bump_visibility_epoch() + def _batch_columns_from_table(self, source: CTable, names) -> tuple[dict, dict]: + """Read *source*'s live rows as ``(raw_columns, source_valid)`` for :meth:`extend`. + + Two things a plain ``source._cols[name][:n_rows]`` slice gets wrong, both + of which turn into silently altered data rather than an error: + + * **The rows.** A source with deleted rows -- or a sorted/filtered view -- + keeps its live rows scattered through the physical extent, so the + leading ``n_rows`` physical slots are not the rows being copied. The + dense case still takes the plain slice, which is the common one and + costs no position scan. + * **The nulls.** Under mask storage nullity lives beside the values, not + in them, so a raw slice copies a null's *fill* -- an ordinary-looking + ``0`` / ``""`` / ``NaT`` -- and drops the fact that it was missing. + Sentinel and native-``None`` sources need nothing here: their nulls + travel with the values. + + *source_valid* maps a column to its validity only where something was + actually null, matching what ``NullChannel.coerce_batch`` returns. + """ + # Dense means every live row sits at its own index below the write + # watermark, so the slice and the gather agree. Both terms are O(1). + dense = source.base is None and source._resolve_last_pos() == source._n_rows + positions = None if dense else source._live_positions_from_valid_rows_chunks() + + raw_columns: dict = {} + source_valid: dict = {} + for name in names: + if name not in source._cols: + continue + arr = source._cols[name] + raw_columns[name] = arr[: source._n_rows] if dense else arr[positions] + spec = source._schema.columns_by_name[name].spec + if not getattr(spec, "uses_mask", False): + continue + mask = source._null_mask(name) + if mask is None: + continue # no sidecar means the column has never held a null + valid = np.asarray(mask[: source._n_rows] if dense else mask[positions]) + if not valid.all(): + source_valid[name] = valid + return raw_columns, source_valid + def extend(self, data: list | CTable | Any, *, validate: bool | None = None) -> None: # noqa: C901 """Append multiple rows at once. @@ -12763,14 +14478,15 @@ def extend(self, data: list | CTable | Any, *, validate: bool | None = None) -> input_col_names = self._append_input_col_names new_nrows = 0 provided_names: set[str] = set() + # Validity carried over from a CTable source, whose nullity does not + # travel with its raw values under mask storage. Empty for every other + # input shape, which spells its nulls in band or as ``None`` cells. + source_valid: dict[str, np.ndarray] = {} if hasattr(data, "_cols") and hasattr(data, "_n_rows"): new_nrows = data._n_rows - raw_columns = {} - for name in current_col_names: - if name in data._cols: - raw_columns[name] = data._cols[name][: data._n_rows] - provided_names.add(name) + raw_columns, source_valid = self._batch_columns_from_table(data, current_col_names) + provided_names = set(raw_columns) else: if isinstance(data, dict): if any(isinstance(v, dict) for v in data.values()): @@ -12842,34 +14558,32 @@ def extend(self, data: list | CTable | Any, *, validate: bool | None = None) -> list_processed_cols: dict[str, list] = {} varlen_scalar_processed_cols: dict[str, list] = {} dict_processed_cols: dict[str, list] = {} + # Validity for the mask-storage columns whose batch actually contained a + # null. Absent means "nothing was null", which needs no sidecar write. + batch_valid: dict[str, np.ndarray] = {} for name in current_col_names: col_meta = self._schema.columns_by_name[name] if self._is_list_column(col_meta): list_processed_cols[name] = list(raw_columns[name]) elif self._is_varlen_scalar_column(col_meta): - varlen_scalar_processed_cols[name] = list(raw_columns[name]) + channel = self._null_channel(name) + cells, valid = channel.coerce_batch(raw_columns[name], new_nrows) + if valid is not None: + batch_valid[name] = valid + varlen_scalar_processed_cols[name] = list(cells) elif self._is_dictionary_column(col_meta): dict_processed_cols[name] = list(raw_columns[name]) else: target_dtype = self._cols[name].dtype + # Split nullity out *before* NumPy sees the batch: a None in a + # float column would otherwise become a NaN, which under mask + # storage is a value (decision 6), not a null. + channel = self._null_channel(name) + raw_columns[name], valid = channel.coerce_batch(raw_columns[name], new_nrows) + if valid is not None: + batch_valid[name] = valid if isinstance(col_meta.spec, timestamp): - values = np.asarray(raw_columns[name]) - if np.issubdtype(values.dtype, np.datetime64): - values = values.astype(f"datetime64[{col_meta.spec.unit}]").astype(np.int64) - elif values.dtype.kind in "OUS": - values = np.array( - [ - col_meta.spec.null_value - if v is None - else np.datetime64(v) - .astype(f"datetime64[{col_meta.spec.unit}]") - .astype(np.int64) - if isinstance(v, (np.datetime64, str)) or hasattr(v, "isoformat") - else v - for v in values - ], - dtype=target_dtype, - ) + values = self._timestamps_to_stored_int64(col_meta.spec, raw_columns[name], target_dtype) scalar_processed_cols[name] = np.ascontiguousarray(values, dtype=target_dtype) elif self._is_ndarray_column(col_meta): scalar_processed_cols[name] = self._coerce_ndarray_batch( @@ -12885,6 +14599,13 @@ def extend(self, data: list | CTable | Any, *, validate: bool | None = None) -> else: scalar_processed_cols[name] = np.ascontiguousarray(raw, dtype=target_dtype) + # A CTable source hands over its validity separately, because its raw + # values carry none: coerce_batch sees a fully typed array and rightly + # reports nothing null. setdefault, not update, so an input that *did* + # express its own nulls keeps them. + for name, valid in source_valid.items(): + batch_valid.setdefault(name, valid) + end_pos = start_pos + new_nrows if self.auto_compact and end_pos >= len(self._valid_rows): @@ -12919,7 +14640,16 @@ def extend(self, data: list | CTable | Any, *, validate: bool | None = None) -> self._cols[name][start_pos:end_pos] = values[:] self._feed_summary(name, start_pos, values) + # Validity sidecars, written before the rows go live below. Only the + # columns whose batch actually held a null appear here, so a null-free + # nullable column still never materializes a sidecar. + for name, valid in batch_valid.items(): + self._null_channel(name).set_valid(slice(start_pos, end_pos), valid) + n_rows = self.nrows + # Crash safety is inherited, not added: everything above -- values and + # validity alike -- is invisible until this line flips the rows live. + # Do not move any column write below it. self._valid_rows[start_pos:end_pos] = True self._last_pos = end_pos self._n_rows = n_rows + new_nrows @@ -13028,6 +14758,88 @@ def in_repl(match: re.Match, _dc=dc, _name=name) -> str: rewritten = new_expr return rewritten, new_operands + def _rewrite_null_predicates( + self, expr: str, operands: dict[str, blosc2.NDArray | blosc2.LazyExpr] + ) -> tuple[str, dict[str, blosc2.NDArray | blosc2.LazyExpr]]: + """Make a string predicate reject the nulls of the columns it reads. + + The operator form (``t.where(t.a > 10)``) has always been null-aware: + :meth:`Column._null_aware_compare` forces null rows to False. The + string form was not -- ``t.where("a > 10")`` compared the raw sentinel, + so a column whose sentinel happens to satisfy the predicate (say + ``null_value=999`` against ``> 10``) returned its nulls as matches. + + Each nullable column referenced by the expression contributes a + validity operand (``a != null_value``, or ``~isnan(a)`` for a NaN + sentinel; the ``.notnull`` sidecar for a mask-storage column), which + :func:`~blosc2.ctable_nulls.rewrite_null_predicates` conjoins onto every + comparison that reads it. The validity operand is a lazy expression + over the same raw array, so it fuses into the same pass rather than + materializing anything. + + A **mask-storage** column needs this just as much as a sentinel one, and + for the same reason read the other way round: its nulls hold the + column's fill, and a fill is a value like any other to miniexpr, so + ``a < 10`` over an int column matched every null (fill ``0``) before + this ran. + + This makes the *scan* path correct — including the scan that an + indexed OR over a nullable column bails to. It does not replace the + index path's null post-filtering: an ordered index answers by taking + a range of the sorted column without ever evaluating the predicate, + so ``_exclude_null_positions`` in ``ctable_indexing.py`` stays + load-bearing regardless of how null-aware the expression is. + + Run this *before* :meth:`_rewrite_nested_expression`, while names in + *expr* are still real column names; the injected operand names carry no + dot, so the nested rewrite passes over them. + """ + valid_exprs: dict[str, tuple[str, object]] = {} + new_operands = dict(operands) + for i, name in enumerate(operands): + if name in self._computed_cols: + continue + col_info = self._schema.columns_by_name.get(name) + if col_info is None: + continue + kind = kind_of_spec(col_info.spec) + if kind not in (NULL_SENTINEL, NULL_MASK): + continue + if not self._expression_references_name(expr, name): + continue + if kind == NULL_MASK: + # No in-band value to compare against, so the guard can only be + # the sidecar, injected as an operand. The *fill* stands in for + # the sentinel in the can-match test, and correctly so: it is + # what a null row actually holds, so a leaf the fill cannot + # satisfy cannot return a null either. That keeps ``a > 10`` on + # an int column (fill 0) and every ordered comparison on a float + # one (fill NaN) unguarded, and so still on their index. + valid_pred = self[name]._nulls.valid_pred() + if valid_pred is None: + continue # no sidecar: the column has never held a null + guard = f"__nv{i}" + new_operands[guard] = valid_pred + valid_exprs[name] = (guard, fill_value_for(col_info.spec)) + continue + guard = sentinel_guard_expr(name, col_info.spec.null_value) + if guard is None: + # A sentinel with no literal form: fall back to an injected + # boolean operand. Correct, but opaque to the index planner. + valid_pred = self[name]._nulls.valid_pred() + if valid_pred is None: + continue + guard = f"__nv{i}" + new_operands[guard] = valid_pred + valid_exprs[name] = (guard, col_info.spec.null_value) + + rewritten = rewrite_null_predicates(expr, valid_exprs) + if rewritten is None: + return expr, operands + return rewritten, { + k: v for k, v in new_operands.items() if not k.startswith("__nv") or k in rewritten + } + @staticmethod def _alias_dotted(expr: str, names: list[str], prefix: str) -> tuple[str, dict[str, str]]: """Replace each dotted name in *expr* with ``{prefix}{i}``. @@ -13257,14 +15069,16 @@ def _utf8_span_eval( {name: self._cols[col_of(name, name)] for name in utf8_names}, {name: self[col_of(name, name)].null_value for name in utf8_names}, len(self._valid_rows), + # A mask-storage operand carries no sentinel for the driver to find, + # so it hands over its validity sidecar instead. + valids={name: self._null_mask(col_of(name, name)) for name in utf8_names}, strict=strict, span_rows=self._UTF8_EXPR_SPAN, budget=self._UTF8_EXPR_BUDGET, ) def _is_nullable_column(self, name: str) -> bool: - col = self[name] - return col.null_value is not None or col.is_dictionary or col.is_varlen_scalar + return self[name]._nulls.is_nullable def dropna(self, subset: list[str] | None = None) -> CTable: """Return a view excluding rows where any column in *subset* is null. @@ -13383,6 +15197,7 @@ def where( # noqa: C901 utf8_names = self._utf8_names_in(expr_result) operands = self._where_expression_operands(expr_result) expr_result, operands = self._rewrite_dictionary_predicates(expr_result, operands) + expr_result, operands = self._rewrite_null_predicates(expr_result, operands) expr_result, operands = self._rewrite_nested_expression(expr_result, operands) expr_result = self._lazyexpr_over_cols(expr_result, operands, utf8_names) if isinstance(expr_result, np.ndarray) and expr_result.dtype == np.bool_: diff --git a/src/blosc2/ctable_indexing.py b/src/blosc2/ctable_indexing.py index 800f17dc2..6ba285dea 100644 --- a/src/blosc2/ctable_indexing.py +++ b/src/blosc2/ctable_indexing.py @@ -20,6 +20,7 @@ import blosc2 from blosc2 import compute_chunks_blocks +from blosc2.ctable_nulls import NULL_MASK, NULL_SENTINEL, is_nan_sentinel, kind_of_spec, sentinel_mask from blosc2.schema import ( DictionarySpec, ListSpec, @@ -96,7 +97,7 @@ def _dict_rank_hash(dictionary) -> str: _UTF8_RANK_SPAN = 1 << 20 -def _utf8_rank_arrays(col, n_phys: int, null_value: str | None): +def _utf8_rank_arrays(col, n_phys: int, null_value: str | None, *, valid=None): """Alphabetical rank per row for a utf8 column, plus its staleness metadata. Sorting by rank is sorting by decoded string, so an ``int32`` rank column @@ -105,9 +106,17 @@ def _utf8_rank_arrays(col, n_phys: int, null_value: str | None): there is no stored code array, so the column is factorized here; the factorizer hashes raw bytes and only ever decodes the distinct values. - Null rows carry a sentinel *string*, so the sentinel is just another - vocabulary entry; it is given the largest rank so nulls sort last, matching - both the dictionary index and ``_build_lex_keys``. + Under sentinel storage null rows carry a sentinel *string*, so the sentinel + is just another vocabulary entry; it is given the largest rank so nulls sort + last, matching both the dictionary index and ``_build_lex_keys``. + + Under mask storage there is **no sentinel in the vocabulary**: a null row + holds the empty-string fill, which factorizes as an ordinary entry with rank + 0 — so nulls would sort *first* and be indistinguishable from genuine ``""`` + rows in every rank comparison the index answers. *valid* (physical, one + flag per row) is what closes that hole: null rows are stamped with + ``null_rank`` after the rank gather, exactly where the sentinel path would + have put them. """ fact = col.factorizer() codes = np.empty(n_phys, dtype=np.int64) @@ -130,12 +139,18 @@ def _utf8_rank_arrays(col, n_phys: int, null_value: str | None): # turns a query literal into a rank (np.searchsorted) without touching data. sorted_vocab = uniques[order] ranks = code_to_rank[codes] if n_entries else np.zeros(n_phys, dtype=np.int32) + if valid is not None: + # Mask storage: after the gather, not before. The fill in a null row is + # a legitimate vocabulary entry that other rows may share, so the + # rewrite has to be per row rather than per vocabulary entry. + ranks = np.where(np.asarray(valid[:n_phys], dtype=bool), ranks, null_rank) # Staleness signals must be O(1) to check: re-deriving the vocabulary would # mean factorizing the column again on every query. Any write already marks # every index stale, so these only have to catch a rebuilt-but-changed # column, for which row count plus blob size is enough. meta = { "null_rank": int(null_rank), + "null_aware": null_value is not None or valid is not None, "vocab_len": int(n_entries), "n_rows": int(n_phys), "nbytes": int(col._bytes_used), @@ -158,12 +173,35 @@ def _persist_utf8_vocab(full: dict, meta: dict, sorted_vocab: np.ndarray) -> Non if values_path is None: # in-memory index meta["vocab"] = sorted_vocab.tolist() return - width = max(len(v) for v in sorted_vocab) + # max(1, ...): a mask-storage column's ``""`` fill is an ordinary vocabulary + # entry, so an all-null column factorizes to exactly ``[""]`` -- and `` bool: + """Whether *expression* really contains a boolean OR. + + A substring test for ``"|"`` also fires on a string literal that happens to + contain one (``name == 'a|b'``), and the answer decides whether a nullable + indexed column may keep its index -- so it is worth parsing for. + Unparseable input answers True, which is the conservative side. + """ + try: + tree = ast.parse(expression, mode="eval") + except SyntaxError: + return True + for node in ast.walk(tree): + if isinstance(node, ast.BoolOp) and isinstance(node.op, ast.Or): + return True + if isinstance(node, ast.BinOp) and isinstance(node.op, ast.BitOr): + return True + return False + + class _DictRankWrapper: """Wrap a dictionary column's codes NDArray, translating codes to alphabetical ranks on read. @@ -508,6 +546,54 @@ def _resolve_index_catalog_entry( return matches[0] raise TypeError("must specify col_name, expression, or name") + def _index_validity_provider(self, col_name: str): + """``(provider, null_aware)`` for *col_name*'s summary build. + + *provider* is a ``(values, start, stop) -> valid`` callable, or ``None`` + when every row is valid. It is what makes an index null-aware: the + segment-summary builder calls it per chunk and keeps null rows out of + the per-segment extrema. Without it a nullable column's summaries + describe its fill (or its sentinel) as if it were data, which is why + the ``min()``/``max()`` shortcut had to decline them. + + *null_aware* is the claim recorded in the descriptor, and it is true in + one case where *provider* is ``None``: a mask column that has never + held a null (no sidecar, per decision 9 of plans/mask-based-nulls.md). + Its summaries agree with a null-skipping scan for free, so it should + not pay the bail. + """ + col = self._schema.columns_by_name.get(col_name) + spec = col.spec if col is not None else None + if spec is None: + return None, False + kind = kind_of_spec(spec) + if kind == NULL_MASK: + mask = self._null_mask(col_name) + if mask is None: + return None, True + + def _valid_from_sidecar(values, start, stop, _mask=mask): + # One byte per row off the sidecar, and the values are not + # consulted at all. + valid = np.asarray(_mask[start:stop], dtype=bool) + if valid.shape[0] < stop - start: + # Capacity the sidecar has not been grown over yet: those + # rows hold no data, and _grow() writes True over new tail + # rows, so "valid" is the consistent answer for them. + pad = np.ones(stop - start - valid.shape[0], dtype=bool) + valid = np.concatenate([valid, pad]) + return valid + + return _valid_from_sidecar, True + if kind == NULL_SENTINEL: + null_value = spec.null_value + + def _valid_from_sentinel(values, start, stop, _nv=null_value): + return ~sentinel_mask(values, _nv) + + return _valid_from_sentinel, True + return None, False + def _build_index_persistent( self, col_name: str, @@ -523,6 +609,7 @@ def _build_index_persistent( opsi_max_cycles: int | None = None, summary_levels: tuple[str, ...] | None = None, precomputed_summaries: dict | None = None, + validity=None, ) -> dict: """Build index sidecar files for a persistent-table column; return the descriptor.""" import tempfile @@ -575,6 +662,7 @@ def _build_index_persistent( cparams_obj, summary_levels=summary_levels, precomputed_summaries=precomputed_summaries, + validity=validity, ) bucket = ( _build_bucket_descriptor_ooc( @@ -618,6 +706,7 @@ def _build_index_persistent( full, cparams_obj, opsi, + validity is not None, ) else: values = _values_for_target(proxy, target) @@ -631,6 +720,7 @@ def _build_index_persistent( persistent, cparams_obj, summary_levels=summary_levels, + validity=validity, ) bucket = ( _build_bucket_descriptor(proxy, token, kind, values, optlevel, persistent, cparams_obj) @@ -667,6 +757,7 @@ def _build_index_persistent( full, cparams_obj, opsi, + validity is not None, ) result = _copy_descriptor(descriptor) @@ -896,7 +987,7 @@ def create_index( # noqa: C901 # same length _utf8_rank_index_stale() compares the meta against. n_phys = len(col_arr) ranks_arr, utf8_rank_meta, utf8_vocab = _utf8_rank_arrays( - col_arr, n_phys, self[col_name].null_value + col_arr, n_phys, self[col_name].null_value, valid=self._null_mask(col_name) ) col_arr = blosc2.asarray(ranks_arr) @@ -920,6 +1011,15 @@ def create_index( # noqa: C901 col_arr = _DictRankWrapper( dict_col.codes, code_to_rank, null_rank, null_code, dict_col.spec.nullable, n_phys ) + # utf8 and dictionary columns are indexed by *rank*, and a null already + # has a rank of its own there (``null_rank``, sorting last), so their + # summaries are null-aware by construction and there is nothing for a + # validity channel to add -- nor would it line up, the ranks array being + # a different array from the column's payload. + validity, null_aware = ( + (None, False) if (is_utf8 or is_dictionary) else self._index_validity_provider(col_name) + ) + is_persistent = self._storage.index_anchor_path(col_name) is not None if is_persistent: @@ -936,6 +1036,7 @@ def create_index( # noqa: C901 opsi_max_cycles=opsi_max_cycles, summary_levels=summary_levels, precomputed_summaries=precomputed_summaries if kind_str == "summary" else None, + validity=validity, ) else: # In-memory path: materialise ranks as a proper NDArray (small tables only). @@ -958,6 +1059,7 @@ def create_index( # noqa: C901 opsi_max_cycles=opsi_max_cycles, summary_levels=summary_levels, precomputed_summaries=precomputed_summaries if kind_str == "summary" else None, + validity=validity, ) store = _IN_MEMORY_INDEXES[id(col_arr)] descriptor = _copy_descriptor(store["indexes"]["__self__"]) @@ -971,6 +1073,10 @@ def create_index( # noqa: C901 _persist_utf8_vocab(full, utf8_rank_meta, utf8_vocab) full["utf8_rank"] = utf8_rank_meta + # A null-free mask column builds with no validity provider (there is + # nothing to exclude) yet its summaries are null-aware all the same. + descriptor["null_aware"] = bool(descriptor.get("null_aware") or null_aware) + value_epoch, _ = self._storage.get_epoch_counters() descriptor["built_value_epoch"] = value_epoch @@ -1440,25 +1546,62 @@ def _try_index_where(self, expr_result: blosc2.LazyExpr) -> np.ndarray | None: return None primary_col_name, primary_col_arr, _ = indexed_columns[0] - nullable_indexed = [ + # Null exclusion still happens here, even though every predicate that + # reaches this point is null-aware (CTable._rewrite_null_predicates for + # the string form, Column._null_aware_compare for the operator one). + # An ordered index does not *evaluate* the predicate: it answers + # ``a > 90`` by taking a range of the sorted column, and the sentinel + # lives in that range whether or not it would satisfy the comparison -- + # a NaN sentinel sorts last, so every NaN row comes back for any + # ``>`` query. The expression being correct therefore does not make the + # index result correct, and these positions must still be filtered. + # Mask storage is in the same position as a sentinel here, with the fill + # playing the sentinel's part: the ordered range that answers ``f > 0.5`` + # contains the NaN fill of every null row, so a mask column's nulls come + # back too unless they are filtered out. A mask column with no sidecar + # has no nulls at all, so there is nothing for it to contribute. + null_kinds = {} + for name, _arr, _descriptor in indexed_columns: + kind = kind_of_spec(root._schema.columns_by_name[name].spec) + if kind == NULL_SENTINEL or (kind == NULL_MASK and root._null_mask(name) is not None): + null_kinds[name] = kind + nullable_indexed = list(null_kinds) + # Only non-NaN sentinels need the *positions* fall-back below; for a NaN + # sentinel the mask-direct path can stay, because that path evaluates the + # predicate through miniexpr rather than reading an ordered range. A + # mask column is in the same position: whatever its fill, the predicate + # miniexpr evaluates carries the sidecar guard that + # ``CTable._rewrite_null_predicates`` conjoined onto every leaf the fill + # could have satisfied. + nullable_needs_exclude = [ name - for name, _arr, _descriptor in indexed_columns - if getattr(root._schema.columns_by_name[name].spec, "null_value", None) is not None + for name, kind in null_kinds.items() + if kind == NULL_SENTINEL + and not is_nan_sentinel(root._schema.columns_by_name[name].spec.null_value) ] - # A NaN null sentinel can never satisfy a comparison (every comparison - # with NaN is False), so the predicate itself already drops those rows; - # only non-NaN sentinels (which *can* match a predicate) need explicit - # position-based exclusion. This lets the fast mask-direct path apply to - # NaN-nullable columns instead of falling back to the positions path. - nullable_needs_exclude = [] - for name in nullable_indexed: - nv = getattr(root._schema.columns_by_name[name].spec, "null_value", None) - if not (isinstance(nv, float) and np.isnan(nv)): - nullable_needs_exclude.append(name) - - # Global null post-filtering is not correct for OR expressions. - if nullable_indexed and ("|" in expr_result.expression or " or " in expr_result.expression): - return None + + # Global null post-filtering is not correct for OR expressions: it would + # drop a row that is null in one column but matches the other branch. + # Which paths that rules out depends on *how* each one answers. + # + # The segment/candidate-unit path evaluates the predicate through + # miniexpr over the surviving blocks, and the predicate is null-aware + # per leaf by the time it gets here (CTable._rewrite_null_predicates for + # the string form, Column._null_aware_compare for the operator one), so + # its result is already exact and the post-filter is not merely + # incorrect for OR but unnecessary. That path therefore serves OR, with + # the filter skipped. + # + # The exact-position paths (FULL/PARTIAL/BUCKET) answer by taking an + # ordered *range* of the sorted column and never evaluate anything, so + # the post-filter is load-bearing there and an OR over a nullable + # indexed column still falls back to the scan -- which is itself + # null-aware, and so returns the right answer. Those bails are + # belt-and-braces today: the planner declines an exact plan for any + # expression containing an OR, so such a plan cannot reach them. They + # are what stops a planner that learns to build one from silently + # reintroducing the bug. + skip_null_filter = bool(nullable_indexed) and _expression_has_or(expr_result.expression) # Inject every usable table-owned descriptor so plan_query can combine them. # In .b2z read mode all columns share the same urlpath, so _array_key() @@ -1497,21 +1640,27 @@ def _try_index_where(self, expr_result: blosc2.LazyExpr) -> np.ndarray | None: def _exclude_null_positions(positions): positions = np.asarray(positions, dtype=np.int64) - for name in nullable_indexed: - col = root._schema.columns_by_name[name] - raw = root._cols[name][positions] - nv = getattr(col.spec, "null_value", None) - if isinstance(nv, float) and np.isnan(nv): - keep = ~np.isnan(raw) + for name, kind in null_kinds.items(): + if positions.size == 0: + break + if kind == NULL_MASK: + # One byte per candidate off the sidecar, and the values are + # not read at all -- the cheaper half of what masks buy. + keep = np.asarray(root._null_mask(name)[positions], dtype=bool) else: - keep = raw != nv + nv = root._schema.columns_by_name[name].spec.null_value + keep = ~sentinel_mask(root._cols[name][positions], nv) positions = positions[keep] return positions if plan.exact_positions is not None: + if skip_null_filter: + return None # ordered range, no per-leaf null handling: see above return _exclude_null_positions(plan.exact_positions) if plan.partial_exact_positions is not None: + if skip_null_filter: + return None # ditto, and its refinement filter is global too # Cross-column refinement: the FULL index on one column gave us # exact positions, but the expression has additional predicates on # other columns. Refinement reads every operand column at those @@ -1540,27 +1689,23 @@ def _exclude_null_positions(positions): if nullable_indexed and primary_op_name is not None: raw = primary_col_arr[candidates] raw = np.asarray(raw) if hasattr(raw, "__array__") else raw - pos = candidates - for name in nullable_indexed: - if name == primary_col_name: - nv = getattr(root._schema.columns_by_name[name].spec, "null_value", None) - if isinstance(nv, float) and np.isnan(nv): - keep = ~np.isnan(raw) - else: - keep = raw != nv - pos = pos[keep] - raw = raw[keep] # already filtered for refinement reuse - else: - col = root._schema.columns_by_name[name] - vals = root._cols[name][pos] - nv = getattr(col.spec, "null_value", None) - if isinstance(nv, float) and np.isnan(nv): - keep = ~np.isnan(vals) - else: - keep = vals != nv - pos = pos[keep] - candidates = pos - prefetched = {primary_op_name: raw} + # One combined keep-mask over the full candidate set, so the + # prefetched primary values stay aligned with the positions + # however many columns contribute nulls. Narrowing the + # positions column by column while trimming ``raw`` only for the + # primary would silently misalign the two. + keep = np.ones(len(candidates), dtype=bool) + for name, kind in null_kinds.items(): + if kind == NULL_MASK: + keep &= np.asarray(root._null_mask(name)[candidates], dtype=bool) + continue + nv = root._schema.columns_by_name[name].spec.null_value + values = raw if name == primary_col_name else root._cols[name][candidates] + keep &= ~sentinel_mask(values, nv) + candidates = candidates[keep] + # Reuse the primary read for refinement, saving a second sparse + # gather -- but only when it survived as an aligned array. + prefetched = {primary_op_name: raw[keep]} if isinstance(raw, np.ndarray) else None else: candidates = _exclude_null_positions(candidates) @@ -1571,6 +1716,8 @@ def _exclude_null_positions(positions): # Fall through to full scan if refinement fails if plan.bucket_masks is not None: + if skip_null_filter: + return None # bucket masks are precomputed, not evaluated # When bucket pruning covers all units (100 % of chunks are # candidates), the per‑chunk evaluation overhead outweighs the # benefit over a plain scan. Fall back to the scan path. @@ -1617,7 +1764,7 @@ def _exclude_null_positions(positions): # lazily. Only columns whose null sentinel can satisfy the # predicate (non-NaN) require the positions fall-back; NaN # sentinels are already excluded by the predicate itself. - if not nullable_needs_exclude: + if skip_null_filter or not nullable_needs_exclude: try: mask = expr_result.compute(_candidate_blocks=bitmap) except Exception: @@ -1626,10 +1773,10 @@ def _exclude_null_positions(positions): return mask positions = self._block_pruned_positions(expr_result, bitmap, primary_col_arr) if positions is not None: - return _exclude_null_positions(positions) + return positions if skip_null_filter else _exclude_null_positions(positions) _, positions = evaluate_segment_query( expression, merged_operands, {}, where_dict, plan, return_positions=True ) - return _exclude_null_positions(positions) + return positions if skip_null_filter else _exclude_null_positions(positions) return None diff --git a/src/blosc2/ctable_nulls.py b/src/blosc2/ctable_nulls.py new file mode 100644 index 000000000..efdf01f4c --- /dev/null +++ b/src/blosc2/ctable_nulls.py @@ -0,0 +1,800 @@ +####################################################################### +# Copyright (c) 2019-present, Blosc Development Team +# All rights reserved. +# +# This source code is licensed under a BSD-style license (found in the +# LICENSE file in the root directory of this source tree) +####################################################################### + +"""Uniform access to a CTable column's validity (null) channel. + +CTable represents nulls in several different ways depending on the column +kind: an in-band **sentinel** value for fixed-width scalars and utf8, a +reserved **code** (``-1``) for dictionary columns, and native ``None`` cells +for the variable-length container kinds. A fourth representation -- a +sidecar validity array, Arrow's model -- is being added. + +:class:`NullChannel` hides that choice behind one accessor, so callers ask +*what is null* without knowing how the column stores it. Every site that +used to reach for ``getattr(spec, "null_value", None)`` and hand-roll a +comparison should go through here instead. +""" + +from __future__ import annotations + +import ast +import math +import operator +from builtins import bool as builtin_bool +from builtins import bytes as builtin_bytes + +import numpy as np + +import blosc2 +from blosc2.schema import ( + NULL_CODE, + NULL_MASK, + NULL_NATIVE, + NULL_NONE, + NULL_SENTINEL, + DictionarySpec, + NDArraySpec, + ObjectSpec, + StructSpec, + VLBytesSpec, + VLStringSpec, + fill_value_for, +) + +# The NULL_* constants and fill_value_for are defined in blosc2.schema -- the +# lower layer, since this module imports the spec classes from it -- and +# re-exported here, which is where the null machinery otherwise lives. +__all__ = [ + "NULL_CODE", + "NULL_MASK", + "NULL_NATIVE", + "NULL_NONE", + "NULL_SENTINEL", + "NullChannel", + "fill_item_for", + "fill_value_for", + "is_na_marker", + "is_nan_sentinel", + "is_null_value", + "kind_of_spec", + "rewrite_null_predicates", + "sentinel_guard_expr", + "sentinel_mask", + "split_batch_validity", +] + +# Specs whose cells can hold a native ``None``. ``ListSpec`` is deliberately +# absent: it matches ``Column.is_varlen_scalar``, which is what the null API +# has always keyed off, and list columns have never participated in +# ``dropna``'s default subset. +_NATIVE_NULL_SPECS = (VLStringSpec, VLBytesSpec, StructSpec, ObjectSpec) + + +def kind_of_spec(spec) -> str: + """Return the ``NULL_*`` representation *spec* uses for its nulls. + + Dictionary and native-``None`` kinds report a null channel regardless of + their ``nullable`` flag, because their storage can represent a null + either way; for the sentinel kinds it is the presence of a + ``null_value`` that decides. + """ + if spec is None: + return NULL_NONE + if isinstance(spec, DictionarySpec): + return NULL_CODE + if getattr(spec, "uses_mask", False): + # Checked before the native-None kinds so that a mask-storage utf8 + # column reports its sidecar rather than its container kind. + return NULL_MASK + if isinstance(spec, _NATIVE_NULL_SPECS): + return NULL_NATIVE + # UTF8Spec is a variable-length kind but represents nulls with a sentinel + # string, so it falls through to the sentinel test below. + if getattr(spec, "null_value", None) is not None: + return NULL_SENTINEL + return NULL_NONE + + +# Types whose instances mean "missing" without being ``None``. Matched by +# name so that neither pandas nor pyarrow has to be importable: ``pandas.NA`` +# is a ``NAType``, ``pyarrow.NA`` a ``NullScalar``, ``pandas.NaT`` a +# ``NaTType``. ``float('nan')`` is deliberately absent -- under mask storage +# NaN is a value, not a null (decision 6). +_NA_TYPE_NAMES = frozenset({"NAType", "NullScalar", "NaTType"}) + + +def is_na_marker(value) -> builtin_bool: + """True when *value* is a way of writing "this cell is null". + + ``None`` is the canonical spelling; the library NA singletons and a + ``datetime64`` ``NaT`` are accepted too, since each is unambiguously a + missing marker rather than a representable value. A float ``NaN`` is + **not** -- see :func:`~blosc2.schema.fill_value_for` and decision 6 of the + mask-storage design: keeping NaN a value is the point of a side channel. + """ + if value is None: + return True + if type(value).__name__ in _NA_TYPE_NAMES: + return True + return isinstance(value, np.datetime64) and builtin_bool(np.isnat(value)) + + +def fill_item_for(spec): + """The whole cell written into one of *spec*'s null slots. + + :func:`~blosc2.schema.fill_value_for` gives the scalar; a fixed-shape + ndarray column's null rows still have to hold something of the right + shape, so it is widened to a full item there. + """ + base = fill_value_for(spec) + if isinstance(spec, NDArraySpec): + return np.full(spec.item_shape, base, dtype=spec.dtype) + return base + + +def split_batch_validity(values, fill): + """Split an incoming batch into ``(storage_values, valid)``. + + *valid* is ``None`` when nothing in the batch was null -- the common case, + and the one that lets a caller skip the sidecar write entirely and so never + materialize one. Otherwise it is a bool array in Arrow polarity (``True`` + = not null), and *storage_values* has *fill* substituted into the null + slots, so what sits under ``valid=False`` is deterministic rather than + whatever NumPy made of a ``None``. + + Null detection follows what the input is able to express: + + * ``np.ma.MaskedArray`` -- ``~arr.mask`` is the validity, verbatim; + * an object array or Python sequence -- ``None`` and the library NA + singletons are null (:func:`is_na_marker`); + * a ``datetime64`` array -- ``NaT`` is null; + * a float array -- **NaN is a value, not a null** (decision 6). A mask + column's whole point is that nullity lives outside the value range, so + nothing in a typed numeric array reads as missing. + + Shared by :meth:`NullChannel.coerce_batch` and ``CTable.add_column``, which + have to agree on all of the above but reach it from different directions: + the channel has a live column to ask, ``add_column`` only has a spec. + """ + if isinstance(values, np.ma.MaskedArray): + valid = ~np.ma.getmaskarray(values) + if valid.ndim > 1: # ndarray column: a row is null only if wholly masked + valid = valid.any(axis=tuple(range(1, valid.ndim))) + filled = np.ma.filled(values, fill) + return filled, (None if valid.all() else valid) + + if isinstance(values, blosc2.NDArray): + # Already in typed storage; there is no way for it to carry a None. + return values, None + + arr = values if isinstance(values, np.ndarray) else np.asarray(values, dtype=object) + if arr.dtype.kind == "M": + invalid = np.isnat(arr) + elif arr.dtype.kind == "O": + # Row-level: for an ndarray column each entry is a whole item, and only + # a wholesale None makes the row null. + invalid = np.fromiter((is_na_marker(v) for v in arr), dtype=np.bool_, count=len(arr)) + else: + # Typed numeric/bool/U/S input has no in-band way to say "null". + return values, None + + if not invalid.any(): + return values, None + out = np.asarray(arr, dtype=object).copy() + if isinstance(fill, np.ndarray): + # An ndarray column's fill is a whole item: assigning it through a + # boolean mask would broadcast its elements across the selected slots + # instead of storing one item in each. + for i in np.flatnonzero(invalid): + out[i] = fill + else: + out[invalid] = fill + return out.tolist(), ~invalid + + +def is_nan_sentinel(value) -> bool: + """True when *value* is a NaN used as a null sentinel. + + Accepts any NumPy float width, not just Python ``float`` -- a ``float32`` + NaN sentinel compares unequal to itself just as a Python one does, so + treating it as an ordinary value would silently stop marking nulls. + """ + return isinstance(value, (float, np.floating)) and math.isnan(value) + + +# Internal short alias; ``is_nan_sentinel`` is the name other modules import. +_is_nan = is_nan_sentinel + + +def sentinel_mask(arr: np.ndarray, null_value, *, item_ndim: int = 0) -> np.ndarray: + """Return a boolean array, True where *arr* holds *null_value*. + + The result always has one entry per *row*: for a fixed-shape ndarray + column (*item_ndim* > 0) a row counts as null only when every element of + its item equals the sentinel. + + Returns an all-False array when *null_value* is ``None``. + """ + if null_value is None: + # Before np.asarray: a ragged list column would not survive the + # conversion, and it has no sentinel to compare against anyway. + return np.zeros(len(arr), dtype=np.bool_) + arr = np.asarray(arr) + if item_ndim: + if arr.ndim <= item_ndim: + arr = arr.reshape((1, *arr.shape)) + elem_mask = np.isnan(arr) if _is_nan(null_value) else arr == null_value + inner_axes = tuple(range(1, elem_mask.ndim)) + return elem_mask.all(axis=inner_axes) if inner_axes else elem_mask.astype(np.bool_) + if np.issubdtype(arr.dtype, np.datetime64): + # Timestamp columns materialize with the int64 sentinel already decoded + # into np.datetime64('NaT') (they share the same bit pattern), so the + # sentinel value itself never appears in arr. + return np.isnat(arr) + if _is_nan(null_value): + return np.isnan(arr) + return arr == null_value + + +def is_null_value(val, null_value) -> bool: + """Scalar counterpart of :func:`sentinel_mask` for a single Python value. + + A column with no sentinel has no in-band null, so this is ``False`` there + -- native ``None`` cells are the other kinds' business, not this one's. + """ + if null_value is None: + return False + try: + if _is_nan(null_value): + return isinstance(val, (float, np.floating)) and math.isnan(val) + except TypeError: + pass + return val == null_value + + +# --------------------------------------------------------------------------- +# Null-aware predicate rewriting +# --------------------------------------------------------------------------- + + +def _collect_names(node: ast.AST) -> set[str]: + """The operand names *node* references.""" + out: set[str] = set() + _predicate_names(node, out) + return out + + +def _predicate_names(node: ast.AST, out: set[str]) -> None: + """Collect the operand names *node* references, keeping dotted paths whole.""" + if isinstance(node, (ast.Name, ast.Attribute)): + # Stop here: descending into an Attribute would also yield the bare + # prefix (``trip`` for ``trip.begin.lon``), which is not an operand. + out.add(ast.unparse(node)) + return + for child in ast.iter_child_nodes(node): + _predicate_names(child, out) + + +_COMPARE_OPS = { + ast.Lt: operator.lt, + ast.LtE: operator.le, + ast.Gt: operator.gt, + ast.GtE: operator.ge, + ast.Eq: operator.eq, + ast.NotEq: operator.ne, +} + + +def _sentinel_can_match(node: ast.Compare, name: str, sentinel) -> bool: + """Whether *sentinel* could satisfy this comparison, so a guard is needed. + + A guard only changes the answer if the sentinel would otherwise pass the + leaf. ``score > 100`` with ``null_value=-1`` needs none: ``-1`` fails the + comparison already. Skipping it there matters for more than tidiness -- + an unguarded single-predicate expression is the shape the index planner + recognizes, so a needless guard would push a query that is correct today + off its index and onto a full scan. + + Answers True (guard) whenever the comparison is not a simple + ``column literal``, since anything else cannot be settled here. + """ + if len(node.ops) != 1 or len(node.comparators) != 1: + return True + op = _COMPARE_OPS.get(type(node.ops[0])) + if op is None: + return True + left, right = node.left, node.comparators[0] + if isinstance(left, (ast.Name, ast.Attribute)) and ast.unparse(left) == name: + column_first, other = True, right + elif isinstance(right, (ast.Name, ast.Attribute)) and ast.unparse(right) == name: + column_first, other = False, left + else: + return True + try: + # literal_eval, not Constant: a negative literal such as ``-1`` is a + # UnaryOp(USub) in the tree, not a plain constant. + literal = ast.literal_eval(other) + return builtin_bool(op(sentinel, literal) if column_first else op(literal, sentinel)) + except Exception: + return True + + +class _NullPredicateRewriter(ast.NodeTransformer): + """Conjoin a validity guard onto every predicate over a nullable column.""" + + def __init__(self, guards: dict[str, tuple[str, object]]) -> None: + self._guards = guards + self._in_negation = 0 + self.changed = False + + def _guard(self, node: ast.AST, names: set[str]) -> ast.AST: + for guard in sorted({self._guards[n][0] for n in names if n in self._guards}): + # Re-parse per insertion: the same guard can appear at several + # leaves, and an AST node must not be shared between them. + right = ast.parse(guard, mode="eval").body + node = ast.BinOp(left=node, op=ast.BitAnd(), right=right) + self.changed = True + return node + + def visit_Compare(self, node: ast.Compare) -> ast.AST: + if self._in_negation: + # The enclosing negation already conjoins validity for every column + # in its subtree, and that outer conjunct dominates whatever this + # leaf evaluates to for a null row. Guarding here as well would be + # redundant, and it would put the guard on the wrong side of the + # ``~`` (see the note in :func:`rewrite_null_predicates`). It would + # also be wrong to skip it via _sentinel_can_match: a sentinel that + # fails ``a > 10`` *passes* ``not (a > 10)``. + return node + names = { + n + for n in _collect_names(node) + if n in self._guards and _sentinel_can_match(node, n, self._guards[n][1]) + } + return self._guard(node, names) + + def visit_UnaryOp(self, node: ast.UnaryOp) -> ast.AST: + if not isinstance(node.op, (ast.Not, ast.Invert)): + return self.generic_visit(node) + names = _collect_names(node.operand) + self._in_negation += 1 + node = self.generic_visit(node) + self._in_negation -= 1 + return self._guard(node, names) + + def visit_BoolOp(self, node: ast.BoolOp) -> ast.AST: + """Normalize ``and``/``or`` to ``&``/``|``. + + The rewrite emits ``&`` conjuncts, and leaving a mix of Python boolean + operators and bitwise ones in one expression makes the result depend on + how the downstream parser reconciles their very different precedences. + Emitting one form removes the question. + """ + node = self.generic_visit(node) + op = ast.BitAnd() if isinstance(node.op, ast.And) else ast.BitOr() + combined = node.values[0] + for value in node.values[1:]: + combined = ast.BinOp(left=combined, op=op, right=value) + return combined + + +def sentinel_guard_expr(name: str, null_value) -> str | None: + """Source text for a predicate that is True where *name* is not null. + + Emitted inline rather than injected as a precomputed boolean operand, so + the guard stays a predicate *on the same column* -- an index that can serve + ``a > 90`` can serve ``(a > 90) & (a != 999)`` too, where an opaque extra + operand would push the planner off the index and onto a full scan. + + Returns ``None`` for a sentinel that has no literal form, leaving the + caller to fall back to an injected operand. + """ + if is_nan_sentinel(null_value): + # NaN is the only value that compares unequal to itself, so this needs + # no isnan() -- and no ``~``, which would fight the negation rule below. + return f"({name} == {name})" + if isinstance(null_value, np.generic): + null_value = null_value.item() + if isinstance(null_value, (builtin_bool, int, float, str, builtin_bytes)): + return f"({name} != {null_value!r})" + return None + + +def rewrite_null_predicates(expr: str, guards: dict[str, tuple[str, object]]) -> str | None: + """Make each predicate over a nullable column reject that column's nulls. + + *guards* maps a column name, as it appears in *expr*, to a + ``(guard_text, sentinel)`` pair: the source of a predicate that is True + where that column is not null (see :func:`sentinel_guard_expr`), and the + sentinel value itself. Every comparison referencing such a column is + rewritten from ``a > 10`` into ``(a > 10) & (a != 999)``, which is SQL + ``WHERE`` semantics: a null operand satisfies no comparison. A leaf the + sentinel could never have satisfied is left alone -- see + :func:`_sentinel_can_match`. + + Doing this **per leaf** rather than once over the whole expression is what + makes ``OR`` correct. A global ``result & valid_a`` would drop a row that + is null in ``a`` but matches the other branch of ``(a > 10) | (b == 0)``, + where SQL says the row qualifies. + + Under a negation the guard is attached at the negation itself, not inside + it: ``not (a > 10)`` must be False for a null ``a``, but ``~((a > 10) & + guard)`` would yield True. Guarding outside -- ``(~(a > 10)) & guard`` -- + gives False. This is exact for every form the tests cover, and + conservative in one three-valued corner: ``not (a > 10 and b == 999)`` + where the second term is False makes SQL's ``NULL AND FALSE`` collapse to + ``FALSE``, so the row should survive the negation, while the guard drops + it. Rows are only ever dropped, never wrongly returned. + + Returns the rewritten expression, or ``None`` when nothing was rewritten -- + including when *expr* does not parse, which leaves the caller's original + text untouched for the downstream parser to report on. + """ + if not guards: + return None + try: + tree = ast.parse(expr, mode="eval") + except SyntaxError: + return None + rewriter = _NullPredicateRewriter(guards) + tree = rewriter.visit(tree) + if not rewriter.changed: + return None + ast.fix_missing_locations(tree) + return ast.unparse(tree) + + +class NullChannel: + """Uniform read accessor for one column's validity channel. + + Subsumes the four representations CTable uses -- sidecar validity array, + in-band sentinel, dictionary null code, native ``None`` -- so callers ask + *what is null* without knowing which one a column uses. + + Bound to a :class:`~blosc2.ctable.Column`, so it sees that column's view + (sorted order, row filter) the same way the column itself does. Nothing + is snapshotted: every property reads through to the live schema, which + keeps a channel correct across in-place spec mutation. + + A channel holds its ``Column`` strongly and the ``Column`` does **not** + cache the channel back: the pair would otherwise be a reference cycle that + refcounting alone can never break, and since a ``Column`` also holds its + ``CTable``, every channel built on a write path would pin a whole table + until the next gc pass. Construction is one slot assignment, so rebuilding + a channel per access costs nothing worth caching. + """ + + __slots__ = ("_col",) + + def __init__(self, column) -> None: + self._col = column + + def __repr__(self) -> str: + return f"NullChannel({self._col._col_name!r}, kind={self.kind!r})" + + # ------------------------------------------------------------------ + # Identity + # ------------------------------------------------------------------ + + @property + def spec(self): + """This column's schema spec, or ``None`` if it has no schema entry.""" + col_info = self._col._table._schema.columns_by_name.get(self._col._col_name) + return None if col_info is None else col_info.spec + + @property + def kind(self) -> str: + """Which ``NULL_*`` representation this column uses.""" + return kind_of_spec(self.spec) + + @property + def is_nullable(self) -> bool: + """True when this column has a null channel at all.""" + return self.kind != NULL_NONE + + @property + def sentinel(self): + """The in-band sentinel value, or ``None`` for the other kinds.""" + return getattr(self.spec, "null_value", None) + + @property + def null_code(self): + """The reserved dictionary code, or ``None`` for the other kinds.""" + return getattr(self.spec, "null_code", None) + + @property + def uses_mask(self) -> builtin_bool: + """True when nullity lives in a sidecar validity array.""" + return self.kind == NULL_MASK + + @property + def fill_value(self): + """The value occupying this column's null slots under mask storage. + + Unobservable through the ``Column`` API and not part of the format + contract -- see :func:`~blosc2.schema.fill_value_for`. Widened to a + whole item for a fixed-shape ndarray column, whose null rows still + have to hold something of the right shape. + """ + return fill_item_for(self.spec) + + # ------------------------------------------------------------------ + # The sidecar (mask kind only) + # ------------------------------------------------------------------ + + def valid_array(self): + """The physical validity sidecar, or ``None`` when there is none. + + ``None`` for every non-mask kind, and also for a mask column that has + never been given a null: an absent sidecar means *all rows valid*, so + callers read that as never-null rather than as unknown. + """ + if self.kind != NULL_MASK: + return None + return self._col._table._null_mask(self._col._col_name) + + def _ensure_valid_array(self): + """The sidecar, materializing it if this is the column's first null.""" + return self._col._table._ensure_null_mask(self._col._col_name) + + def set_valid(self, key, valid) -> None: + """Record validity at *physical* positions *key*. + + A no-op for the non-mask kinds, whose nullity travels in band with the + values the caller has already written. + + Marking rows *valid* in a column with no sidecar is skipped rather + than made to materialize one: that is already what the column says. + """ + if self.kind != NULL_MASK: + return + arr = self.valid_array() + if arr is None: + if valid is True or (valid is not False and np.all(valid)): + return + arr = self._ensure_valid_array() + arr[key] = valid + + # ------------------------------------------------------------------ + # Reads + # ------------------------------------------------------------------ + + def mask_for_values(self, arr: np.ndarray) -> np.ndarray: + """True where an already-materialized *arr* of this column's values is null. + + Always returns one flag per row, all False when the column has no + sentinel. This is the vectorized in-band test only -- it is the right + entry point for callers that already hold the values. + """ + col = self._col + return sentinel_mask(arr, self.sentinel, item_ndim=col.item_ndim if col.is_ndarray else 0) + + def null_mask(self) -> np.ndarray: + """True where this column's live values are null, one flag per live row.""" + col = self._col + kind = self.kind + if kind == NULL_MASK: + valid = self.valid_array() + if valid is None: + return np.zeros(len(col), dtype=np.bool_) + # Gathering at the live positions is what makes this honour the + # column's view -- sorted order and row filters alike. + return ~np.asarray(valid[col._resolve_live_positions()]) + if kind == NULL_CODE: + return col._dictionary_eq(None) + if kind == NULL_NATIVE: + return np.array([v is None for v in col], dtype=np.bool_) + return self.mask_for_values(col[:]) + + def valid_slice(self, start: int, stop: int): + """Physical validity for rows ``[start, stop)``, or ``None`` if all valid. + + Physical, not logical: this serves the export paths that read straight + from the storage buffers, which only run on a dense root table where + the two coincide. Use :meth:`null_mask_slice` everywhere else. + """ + valid = self.valid_array() + return None if valid is None else np.asarray(valid[start:stop]) + + def null_mask_slice(self, values, start: int, stop: int): + """Null flags for the logical rows ``[start, stop)``, or ``None``. + + ``None`` -- rather than an all-False array -- is the answer when + nothing in the range is null, because that is what pyarrow wants for + "this array needs no validity buffer". + + *values* is whatever ``col[start:stop]`` already returned, so the + sentinel kinds answer from it instead of reading the column a second + time; the mask kind ignores it and reads its sidecar at the same rows. + """ + kind = self.kind + col = self._col + if kind == NULL_MASK: + valid = self.valid_array() + if valid is None: + return None + if col._has_identity_positions(): + null = ~np.asarray(valid[start:stop]) + else: + null = ~np.asarray(valid[col._resolve_live_positions()[start:stop]]) + elif kind == NULL_SENTINEL: + null = sentinel_mask(values, self.sentinel, item_ndim=col.item_ndim if col.is_ndarray else 0) + else: + return None + return null if null.any() else None + + def is_null_at(self, index: int) -> builtin_bool: + """Whether the value at *logical* row *index* is null. + + Single-row form of :meth:`null_mask`, kept separate so the mask and + sentinel kinds can answer without materializing a whole column. + """ + col = self._col + kind = self.kind + if kind == NULL_NONE: + return False + if kind == NULL_MASK: + valid = self.valid_array() + if valid is None: + return False + return not builtin_bool(valid[int(col._physical_index(index))]) + if kind == NULL_SENTINEL: + return builtin_bool(self.mask_for_values(np.asarray([col[index]]))[0]) + return col[index] is None + + def null_count(self) -> int: + """Number of live rows that are null; ``0`` in O(1) when never null.""" + kind = self.kind + if kind == NULL_NONE: + return 0 + if kind == NULL_MASK: + valid = self.valid_array() + if valid is None: + return 0 + col = self._col + if col._has_identity_positions(): + # Hole-free base table: count straight off the compressed + # sidecar. Bool NDArrays compress to almost nothing, so this + # is effectively O(chunks) rather than O(rows). + n = len(col) + return n - int(blosc2.count_nonzero(valid if n == valid.shape[0] else valid[:n])) + return int(self.null_mask().sum()) + if kind == NULL_NATIVE: + return sum(1 for v in self._col if v is None) + return int(self.null_mask().sum()) + + def nonnull_chunks(self): + """Yield chunks of live values with the null ones removed.""" + col = self._col + if self.kind == NULL_MASK: + valid = self.valid_array() + if valid is None: + yield from col.iter_chunks() + return + # Zip values against validity chunk for chunk. The sidecar shares + # the column's row grid (see CTable._null_mask_grid), so the two + # streams stay aligned without any re-chunking. + null = self.null_mask() + offset = 0 + for chunk in col.iter_chunks(): + keep = ~null[offset : offset + len(chunk)] + offset += len(chunk) + filtered = chunk[keep] + if len(filtered) > 0: + yield filtered + return + sentinel = self.sentinel + if sentinel is None: + yield from col.iter_chunks() + return + is_nan_sentinel = _is_nan(sentinel) + for chunk in col.iter_chunks(): + mask = ~np.isnan(chunk) if is_nan_sentinel else chunk != sentinel + filtered = chunk[mask] + if len(filtered) > 0: + yield filtered + + # ------------------------------------------------------------------ + # Writes + # ------------------------------------------------------------------ + + def coerce_scalar(self, value): + """Split one incoming cell into ``(storage_value, is_valid)``. + + Under mask storage ``None`` becomes the canonical way to write a null. + Fixed-width scalar columns could not accept it at all before -- users + had to write the sentinel literally -- so this is new capability, not + re-plumbing. + """ + if self.kind != NULL_MASK or not is_na_marker(value): + return value, True + return self.fill_value, False + + def coerce_batch(self, values, n: int): + """Split an incoming batch into ``(storage_values, valid)`` for this column. + + Thin wrapper over :func:`split_batch_validity`, which carries the null + detection rules and the reasoning behind them; all this adds is the + column's own fill, and the short-circuit for a column whose nulls do + not live in a sidecar at all. + """ + if self.kind != NULL_MASK: + return values, None + return split_batch_validity(values, self.fill_value) + + # ------------------------------------------------------------------ + # Lazy predicates over the raw physical array + # ------------------------------------------------------------------ + + def null_pred(self): + """Lazy predicate over the raw physical array, True where the value is null. + + Returns ``None`` when there is nothing to propagate -- the expression + layer reads that as "never null" and skips the operand entirely. + + Fixed-shape ndarray columns return ``None`` as well: their per-item + sentinel mask does not align 1:1 with the row-level predicates built + here. Use :meth:`null_mask` for those instead. Dictionary and + variable-length scalar columns never reach here, because + ``Column._ensure_queryable`` rejects them for arithmetic and + comparisons before any predicate is built. + """ + valid = self._mask_pred() + if valid is not None: + return ~valid + col = self._col + if col.is_ndarray: + return None + sentinel = self.sentinel + if sentinel is None: + return None + if _is_nan(sentinel): + return blosc2.isnan(col._raw_col) + return col._raw_col == sentinel + + def valid_pred(self): + """Lazy predicate over the raw physical array, True where the value is *not* null. + + Returns ``None`` under the same conditions as :meth:`null_pred`. + """ + valid = self._mask_pred() + if valid is not None: + return valid + col = self._col + if col.is_ndarray: + return None + sentinel = self.sentinel + if sentinel is None: + return None + if _is_nan(sentinel): + return ~blosc2.isnan(col._raw_col) + return col._raw_col != sentinel + + def _mask_pred(self): + """The sidecar as a physical validity operand, or ``None``. + + ``None`` covers both "not a mask column" and "a mask column with no + sidecar" -- the latter meaning never-null, which the expression layer + already handles by skipping the operand. + + **Fixed-shape ndarray columns are excluded**, and not merely because a + row-level flag is a different shape from an ``(n, *item_shape)`` values + array. Reshaping the sidecar to ``(n, 1, ...)`` looks like it should + fix that, and does not: ``blosc2.where`` returns the *predicate's* + shape rather than broadcasting, so the item dimension is silently + dropped from the values, and combining an ``(n, 1)`` predicate with the + ``(n,)`` row mask that :class:`NullableExpr` reductions use explodes + into ``(n, n)``. Row-level null propagation for ndarray columns needs + broadcasting support in the lazy layer; until then those columns get + their null handling from the NumPy-based reduction paths + (``Column._reduction_null_mask``), which are row-level throughout. + """ + if self.kind != NULL_MASK or self._col.is_ndarray: + return None + valid = self.valid_array() + return None if valid is None else blosc2.asarray(valid) diff --git a/src/blosc2/ctable_storage.py b/src/blosc2/ctable_storage.py index 0d01b4d44..c8813f1e0 100644 --- a/src/blosc2/ctable_storage.py +++ b/src/blosc2/ctable_storage.py @@ -141,6 +141,46 @@ def create_valid_rows( def open_valid_rows(self) -> blosc2.NDArray: raise NotImplementedError + # -- Per-column validity sidecars ---------------------------------------- + # + # A mask-storage nullable column keeps its nullity in a bool NDArray beside + # the values (``True`` = not null). The sidecar is materialized on the + # first null actually written, never at column creation, so + # ``has_null_mask(name) is False`` is the common case and means *all rows + # valid* -- a null-free nullable column costs nothing on disk and nothing + # on the read path. + + def create_null_mask( + self, + name: str, + *, + shape: tuple[int, ...], + chunks: tuple[int, ...], + blocks: tuple[int, ...], + ) -> blosc2.NDArray: + """Create column *name*'s validity sidecar, initialized all-valid. + + Filled with ``True`` rather than zeros so that a freshly materialized + sidecar says exactly what its absence said. + """ + raise NotImplementedError + + def install_null_mask(self, name: str, ndarray: blosc2.NDArray) -> blosc2.NDArray: + """Store a pre-built bool NDArray as column *name*'s validity sidecar.""" + raise NotImplementedError + + def open_null_mask(self, name: str) -> blosc2.NDArray: + """Open column *name*'s validity sidecar. Call only if it exists.""" + raise NotImplementedError + + def has_null_mask(self, name: str) -> bool: + """Whether column *name* has a stored validity sidecar.""" + raise NotImplementedError + + def delete_null_mask(self, name: str) -> None: + """Remove column *name*'s validity sidecar if it has one.""" + raise NotImplementedError + def save_schema(self, schema_dict: dict[str, Any]) -> None: raise NotImplementedError @@ -285,6 +325,23 @@ def create_valid_rows(self, *, shape, chunks, blocks): def open_valid_rows(self): raise RuntimeError("In-memory tables have no on-disk representation to open.") + def create_null_mask(self, name, *, shape, chunks, blocks): + return blosc2.full(shape, True, dtype=np.bool_, chunks=chunks, blocks=blocks) + + def install_null_mask(self, name, ndarray: blosc2.NDArray) -> blosc2.NDArray: + return ndarray + + def open_null_mask(self, name): + raise RuntimeError("In-memory tables have no on-disk representation to open.") + + def has_null_mask(self, name) -> bool: + # In-memory sidecars are held by the CTable, not by this storage, so + # there is never one to discover here — mirroring open_column. + return False + + def delete_null_mask(self, name) -> None: + pass # nothing persisted to remove + def save_schema(self, schema_dict): pass # nothing to persist @@ -409,6 +466,23 @@ def _column_name_to_relpath(name: str) -> str: # dots are percent-encoded, so no column name maps to a key containing ".". _UTF8_DATA_SUFFIX = ".utf8" +# Key suffix for the validity sidecar of a mask-storage nullable column: a +# plain bool NDArray, one byte per physical row, ``True`` where the value is +# *not* null (Arrow's polarity, which the name states). Deliberately not +# ``.valid``: ``/_valid_rows`` already exists and means row liveness, not +# per-column nullity, and two bool arrays with near-identical names would +# invite conflating them. Collision-free by the same argument as +# ``_UTF8_DATA_SUFFIX`` above. +# +# The sidecar is created lazily, on the first write that actually contains a +# null, so its *absence* is a valid -- and, for a nullable column that has +# never seen a null, the expected -- state meaning "every row is valid". +_NOTNULL_SUFFIX = ".notnull" + +# Per-column companion keys that live beside a column's own key and must be +# carried along by any operation that moves or removes the column. +_COMPANION_KEY_SUFFIXES = (_UTF8_DATA_SUFFIX, _NOTNULL_SUFFIX) + class EmbedStoreTableStorage(TableStorage): """Read-only :class:`CTable` storage backed by an in-memory :class:`blosc2.EmbedStore`. @@ -505,6 +579,14 @@ def open_dictionary_column(self, name: str, spec) -> DictionaryColumn: def open_valid_rows(self) -> blosc2.NDArray: return self._estore[_VALID_ROWS_KEY] + # -- per-column validity sidecars -------------------------------------- + + def open_null_mask(self, name: str) -> blosc2.NDArray: + return self._estore[self._col_key(name) + _NOTNULL_SUFFIX] + + def has_null_mask(self, name: str) -> bool: + return (self._col_key(name) + _NOTNULL_SUFFIX) in self._estore + # -- status ----------------------------------------------------------- def table_exists(self) -> bool: @@ -531,6 +613,9 @@ def _not_supported(self, *args, **kwargs): create_varlen_scalar_column = _not_supported create_dictionary_column = _not_supported create_valid_rows = _not_supported + create_null_mask = _not_supported + install_null_mask = _not_supported + delete_null_mask = _not_supported save_schema = _not_supported save_vlmeta = _not_supported delete_column = _not_supported @@ -820,6 +905,32 @@ def create_valid_rows(self, *, shape, chunks, blocks): def open_valid_rows(self) -> blosc2.NDArray: return self._open_store()[_VALID_ROWS_KEY] + def _notnull_key(self, name: str) -> str: + return self._col_key(name) + _NOTNULL_SUFFIX + + def create_null_mask(self, name, *, shape, chunks, blocks) -> blosc2.NDArray: + mask = blosc2.full(shape, True, dtype=np.bool_, chunks=chunks, blocks=blocks) + store = self._open_store() + store[self._notnull_key(name)] = mask + return store[self._notnull_key(name)] + + def install_null_mask(self, name, ndarray: blosc2.NDArray) -> blosc2.NDArray: + store = self._open_store() + store[self._notnull_key(name)] = ndarray + return store[self._notnull_key(name)] + + def open_null_mask(self, name: str) -> blosc2.NDArray: + return self._open_store()[self._notnull_key(name)] + + def has_null_mask(self, name: str) -> bool: + return self._notnull_key(name) in self._open_store() + + def delete_null_mask(self, name: str) -> None: + store = self._open_store() + key = self._notnull_key(name) + if key in store: + del store[key] + def save_schema(self, schema_dict: dict[str, Any]) -> None: """Write *schema_dict* (plus kind/version markers) to ``/_meta``.""" meta = blosc2.SChunk() @@ -899,12 +1010,13 @@ def column_names_from_schema(self) -> list[str]: return [c["name"] for c in d["columns"]] def delete_column(self, name: str) -> None: + store = self._open_store() key = self._col_key(name) - if key in self._open_store(): - del self._open_store()[key] - data_key = key + _UTF8_DATA_SUFFIX - if data_key in self._open_store(): - del self._open_store()[data_key] + if key in store: + del store[key] + for suffix in _COMPANION_KEY_SUFFIXES: + if key + suffix in store: + del store[key + suffix] return list_path = self._list_col_path(name) if os.path.exists(list_path): @@ -919,10 +1031,10 @@ def rename_column(self, old: str, new: str): if old_key in store: store[new_key] = store[old_key] del store[old_key] - old_data_key = old_key + _UTF8_DATA_SUFFIX - if old_data_key in store: - store[new_key + _UTF8_DATA_SUFFIX] = store[old_data_key] - del store[old_data_key] + for suffix in _COMPANION_KEY_SUFFIXES: + if old_key + suffix in store: + store[new_key + suffix] = store[old_key + suffix] + del store[old_key + suffix] return store[new_key] old_path = self._list_col_path(old) new_path = self._list_col_path(new) @@ -1426,6 +1538,58 @@ def create_valid_rows( def open_valid_rows(self) -> blosc2.NDArray: return self._open_leaf("/_valid_rows") + # -- per-column validity sidecars -------------------------------------- + + def _notnull_logical_key(self, name: str) -> str: + return self._col_logical_key(name) + _NOTNULL_SUFFIX + + def _register_leaf(self, logical_key: str, dest_path: str) -> None: + """Point the outer store's map_tree at an external leaf file.""" + rel_path = os.path.relpath(dest_path, self._working_dir()).replace(os.sep, "/") + self._store.map_tree[self._table_key(logical_key)] = rel_path + self._store._modified = True + + def create_null_mask(self, name, *, shape, chunks, blocks) -> blosc2.NDArray: + logical_key = self._notnull_logical_key(name) + dest_path = self._dest_path(logical_key, ".b2nd") + os.makedirs(os.path.dirname(dest_path), exist_ok=True) + mask = blosc2.full( + shape, + True, + dtype=np.bool_, + chunks=chunks, + blocks=blocks, + urlpath=dest_path, + mode="w", + ) + self._register_leaf(logical_key, dest_path) + return mask + + def install_null_mask(self, name, ndarray: blosc2.NDArray) -> blosc2.NDArray: + logical_key = self._notnull_logical_key(name) + dest_path = self._dest_path(logical_key, ".b2nd") + os.makedirs(os.path.dirname(dest_path), exist_ok=True) + saved = ndarray.copy(urlpath=dest_path) + self._register_leaf(logical_key, dest_path) + return saved + + def open_null_mask(self, name: str) -> blosc2.NDArray: + return self._open_leaf(self._notnull_logical_key(name)) + + def has_null_mask(self, name: str) -> bool: + full_key = self._table_key(self._notnull_logical_key(name)) + return full_key in self._store.map_tree or full_key in self._store._estore + + def delete_null_mask(self, name: str) -> None: + full_key = self._table_key(self._notnull_logical_key(name)) + if full_key not in self._store.map_tree: + return + filepath = self._store.map_tree.pop(full_key) + full_path = os.path.join(self._working_dir(), filepath) + if os.path.exists(full_path): + os.remove(full_path) + self._store._modified = True + # ------------------------------------------------------------------ # TableStorage interface — schema and manifest # ------------------------------------------------------------------ @@ -1497,9 +1661,10 @@ def delete_column(self, name: str) -> None: full_key = self._table_key(self._col_logical_key(name)) if full_key in self._store.map_tree: keys = [full_key] - data_key = self._table_key(self._col_logical_key(name) + _UTF8_DATA_SUFFIX) - if data_key in self._store.map_tree: - keys.append(data_key) + for suffix in _COMPANION_KEY_SUFFIXES: + companion = self._table_key(self._col_logical_key(name) + suffix) + if companion in self._store.map_tree: + keys.append(companion) for key in keys: filepath = self._store.map_tree.pop(key) full_path = os.path.join(self._working_dir(), filepath) @@ -1516,9 +1681,10 @@ def rename_column(self, old: str, new: str) -> blosc2.NDArray: old_key = self._table_key(self._col_logical_key(old)) if old_key in self._store.map_tree: moves = [(old_key, self._col_logical_key(new))] - old_data_key = self._table_key(self._col_logical_key(old) + _UTF8_DATA_SUFFIX) - if old_data_key in self._store.map_tree: - moves.append((old_data_key, self._col_logical_key(new) + _UTF8_DATA_SUFFIX)) + for suffix in _COMPANION_KEY_SUFFIXES: + companion = self._table_key(self._col_logical_key(old) + suffix) + if companion in self._store.map_tree: + moves.append((companion, self._col_logical_key(new) + suffix)) new_dest = None for src_key, dst_logical in moves: dst_dest = self._dest_path(dst_logical, ".b2nd") diff --git a/src/blosc2/groupby.py b/src/blosc2/groupby.py index 4eb1646dd..f9fa8e31e 100644 --- a/src/blosc2/groupby.py +++ b/src/blosc2/groupby.py @@ -22,6 +22,7 @@ import numpy as np +from blosc2.ctable_nulls import is_nan_sentinel, sentinel_mask from blosc2.dsl_kernel import DSLKernel from blosc2.schema import DictionarySpec, NDArraySpec, SchemaSpec, float64, int64 from blosc2.schema import bool as b2_bool @@ -54,24 +55,31 @@ class _AggState: @dataclasses.dataclass -class _Utf8KeyChunk: - """A utf8 key-column chunk, factorized to chunk-local integer codes. - - ``codes[i]`` indexes ``uniques`` (a ``StringDType`` array sorted - ascending), so null detection, live-row masking, and per-chunk - ``np.unique`` all run on int64 codes; only the (few) distinct strings are - ever decoded. Produced by :meth:`CTableGroupBy._read_key_chunk` via - ``UTF8Array.factorizer``. +class _CodedKeyChunk: + """A key-column chunk factorized to chunk-local integer codes. + + ``codes[i]`` indexes ``uniques``, so null detection, live-row masking, and + per-chunk ``np.unique`` all run on int64 codes; for a utf8 column, only the + (few) distinct strings are ever decoded. Produced by + :meth:`CTableGroupBy._read_key_chunk` -- for utf8 via + ``UTF8Array.factorizer``, whose ``uniques`` are ``StringDType`` sorted + ascending, and for a mask-storage column via :func:`_coded_chunk_with_nulls`. + + *null_code* is the code reserved for null rows, or ``-1`` when this chunk's + nulls are in band (a sentinel string among the ``uniques``). When it is set, + ``uniques[null_code] is None``, so the null group is displayed and written + as a null rather than as whatever value the storage left in those slots. """ codes: np.ndarray uniques: np.ndarray + null_code: int = -1 def __len__(self) -> int: return len(self.codes) - def take(self, mask: np.ndarray) -> _Utf8KeyChunk: - return _Utf8KeyChunk(self.codes[mask], self.uniques) + def take(self, mask: np.ndarray) -> _CodedKeyChunk: + return _CodedKeyChunk(self.codes[mask], self.uniques, self.null_code) def code_of(self, value: str) -> int: """Code of *value* in this chunk, or -1 when absent (uniques are sorted).""" @@ -81,6 +89,37 @@ def code_of(self, value: str) -> int: return -1 +def _coded_chunk_with_nulls(chunk, valid: np.ndarray) -> _CodedKeyChunk: + """Recode a mask-storage key chunk so its nulls get a code of their own. + + A mask-storage column has no value that *means* null: its null rows hold the + column's fill, and genuine rows may hold the same thing, so grouping on the + values alone silently merges the two -- every null row landing in the ``0`` + group of an int key, or the ``""`` group of a string one. Recoding to + chunk-local integers with one reserved code is what a dictionary column gets + for free from its ``null_code``; this gives it to the other kinds. + + The reserved code maps to ``None`` in the vocabulary, so with + ``dropna=False`` the null group comes out keyed ``None`` -- which a + mask-storage output column can write back as a real null, where a sentinel + column can only offer its sentinel. + """ + if isinstance(chunk, _CodedKeyChunk): + # Already coded by the utf8 factorizer; the null rows only need to be + # moved off their fill's code and onto one of their own. + codes, uniques = chunk.codes, chunk.uniques + else: + uniques, inverse = np.unique(chunk[valid], return_inverse=True) + codes = np.empty(len(chunk), dtype=np.int64) + codes[valid] = inverse + null_code = len(uniques) + vocab = np.empty(null_code + 1, dtype=object) + vocab[:null_code] = list(uniques) + vocab[null_code] = None + codes = np.where(valid, codes, null_code) + return _CodedKeyChunk(codes, vocab, null_code) + + def _is_column_like(value: Any) -> bool: return isinstance(getattr(value, "_col_name", None), str) @@ -480,10 +519,25 @@ def _try_fast_paths(self, specs: list[_AggSpec], use_arg_positions: bool): UDF aggregations always fall through to the generic chunked path below, which is the only one that accumulates raw per-group values instead of a mergeable scalar state. + + The **Cython** paths do too whenever a mask-storage column in play holds + a null: each reads nullity out of the values -- a kernel is told a + ``skip_nan`` flag, not given a validity array -- and a mask column's + values say nothing about which rows are null. Threading a sidecar + through six kernels is a performance follow-up; getting the answer right + is not. The dense single-key path is plain NumPy and already routes its + value columns through :meth:`_null_mask`, so a mask *value* column need + only hand over its sidecar there; only a mask *key* column has to leave, + since its recoded chunk is a :class:`_CodedKeyChunk` rather than the + array of dense non-negative ints that path indexes with. + + A mask column with no sidecar has never held a null and stays on every + fast path. """ if any(s.op == "udf" for s in specs): return None - if not use_arg_positions: + mask_null = self._mask_null_columns(specs) + if not use_arg_positions and not mask_null: for attempt in ( self._try_execute_cython_dense_int_key, self._try_execute_cython_two_int_key_hash, @@ -498,10 +552,11 @@ def _try_fast_paths(self, specs: list[_AggSpec], use_arg_positions: bool): # before the generic float hash path, which is markedly slower. Unlike # the Cython kernels above, it tracks row positions, so it also serves # argmin/argmax — keeping them off the slow generic hash+merge path. - fast = self._try_execute_dense_single_int_key(specs) - if fast is not None: - return fast - if not use_arg_positions: + if not any(name in self.keys for name in mask_null): + fast = self._try_execute_dense_single_int_key(specs) + if fast is not None: + return fast + if not use_arg_positions and not mask_null: return self._try_execute_cython_float_hash(specs) return None @@ -537,7 +592,7 @@ def _execute_with_result_target(self, specs: list[_AggSpec]): keys_live = [ values.take(live_mask) - if isinstance(values, _Utf8KeyChunk) + if isinstance(values, _CodedKeyChunk) else np.asarray(values)[live_mask] for values in raw_keys ] @@ -549,9 +604,16 @@ def _execute_with_result_target(self, specs: list[_AggSpec]): value_chunks = { name: np.asarray(self.table._cols[name][start:stop])[live_mask] for name in value_cols } + # Gathered by the same live_mask as the values, so a validity flag + # stays beside the row it belongs to. + valid_chunks = { + name: valid_slice[live_mask] + for name in value_cols + if (valid_slice := self._valid_chunk(name, start, stop)) is not None + } partials = self._compute_partials( - specs, unique_keys, inverse, value_chunks, logical_positions[live_mask] + specs, unique_keys, inverse, value_chunks, logical_positions[live_mask], valid_chunks ) display_keys = self._display_keys(unique_keys) normalized_keys = self._normalized_keys(display_keys) @@ -598,7 +660,7 @@ def _try_execute_cython_two_int_key_hash(self, specs: list[_AggSpec]): # noqa: if value_dtype is None or np.dtype(value_dtype).kind != "f": return None null_value = getattr(value_info.spec, "null_value", None) - if null_value is not None and not (isinstance(null_value, float) and math.isnan(null_value)): + if null_value is not None and not is_nan_sentinel(null_value): return None try: @@ -749,7 +811,7 @@ def _try_execute_cython_dense_int_key(self, specs: list[_AggSpec]): # noqa: C90 desc.update({"kernel": kernel, "state_kind": "counts", "value_dtype": value_dtype}) elif spec.op in {"sum", "mean", "min", "max"}: if value_dtype.kind == "f": - skip_nan = isinstance(null_value, float) and math.isnan(null_value) + skip_nan = is_nan_sentinel(null_value) if null_value is not None and not skip_nan: return None suffix = "sum" if spec.op == "sum" else spec.op @@ -1108,7 +1170,7 @@ def _try_execute_cython_float_hash(self, specs: list[_AggSpec]): # noqa: C901 if value_dtype is None or np.dtype(value_dtype).kind != "f": return None null_value = getattr(value_info.spec, "null_value", None) - nullable_nan_value = isinstance(null_value, float) and math.isnan(null_value) + nullable_nan_value = is_nan_sentinel(null_value) if null_value is not None and not nullable_nan_value: return None @@ -1419,6 +1481,13 @@ def ensure_size(size: int) -> bool: value_chunks = { name: np.asarray(self.table._cols[name][start:stop])[live_mask] for name in value_cols } + # Gathered by the same live_mask as the values, so a validity flag + # stays beside the row it belongs to. + valid_chunks = { + name: valid_slice[live_mask] + for name in value_cols + if (valid_slice := self._valid_chunk(name, start, stop)) is not None + } row_positions = logical_chunk[live_mask] if need_positions else None for spec in specs: @@ -1427,7 +1496,12 @@ def ensure_size(size: int) -> bool: continue assert spec.input_col is not None values = value_chunks[spec.input_col] - non_null = ~self._null_mask(spec.input_col, values, is_key=False) + non_null = ~self._null_mask( + spec.input_col, + values, + is_key=False, + valid=valid_chunks.get(spec.input_col), + ) if spec.op == "count": states[spec.output_col] += np.bincount( keys, weights=non_null.astype(np.int64), minlength=len(present) @@ -1558,7 +1632,48 @@ def _chunk_size(self) -> int: # column). return base * -(-target // base) + def _mask_null_columns(self, specs: list[_AggSpec]) -> list[str]: + """The key and value columns in play that have a validity sidecar. + + Non-empty means this aggregation has to take the generic chunked path; + see :meth:`_try_fast_paths`. Only mask-storage columns are probed, and + only those the schema names -- an absent sidecar is the common case and + means the column has never held a null. + """ + names = [*self.keys, *(s.input_col for s in specs if s.input_col is not None)] + candidates = set(self.table._null_mask_names) + return [ + name + for name in dict.fromkeys(names) + if name in candidates and self.table._null_mask(name) is not None + ] + + def _valid_chunk(self, name: str, start: int, stop: int) -> np.ndarray | None: + """Physical validity for rows ``[start, stop)`` of *name*, or ``None``. + + ``None`` is the answer for every column whose nulls travel in band with + its values, and equally for a mask-storage column that has never held a + null -- both mean "nothing here to look up on the side". Groupby reads + physical windows throughout, so the sidecar slices straight across. + """ + mask = self.table._null_mask(name) + return None if mask is None else np.asarray(mask[start:stop], dtype=bool) + def _read_key_chunk(self, name: str, start: int, stop: int) -> np.ndarray: + chunk = self._read_raw_key_chunk(name, start, stop) + valid = self._valid_chunk(name, start, stop) + if valid is None: + return chunk + if getattr(chunk, "dtype", None) is not None and chunk.dtype.kind == "f": + # A float *key* keeps NaN-as-missing so dropna stays predictable, + # as it does for a sentinel column -- see _null_mask. Folding it in + # here puts NaN keys in the same group as the nulls rather than + # leaving _null_mask to re-derive it from recoded values it can no + # longer see. + valid = valid & ~np.isnan(chunk) + return _coded_chunk_with_nulls(chunk, valid) + + def _read_raw_key_chunk(self, name: str, start: int, stop: int) -> np.ndarray: col_info = self.table._schema.columns_by_name[name] if self.table._is_dictionary_column(col_info): return np.asarray(self.table._cols[name].codes[start:stop], dtype=np.int32) @@ -1586,7 +1701,7 @@ def _read_key_chunk(self, name: str, start: int, stop: int) -> np.ndarray: codes = rank[codes] if len(order) else codes if stop > n: codes = np.concatenate([codes, np.zeros(stop - max(start, n), dtype=np.int64)]) - return _Utf8KeyChunk(codes, uniques[order]) + return _CodedKeyChunk(codes, uniques[order]) return np.asarray(self.table._cols[name][start:stop]) def _factorize_keys( @@ -1594,7 +1709,7 @@ def _factorize_keys( ) -> tuple[np.ndarray | list[np.ndarray], np.ndarray]: if len(keys_live) == 1: arr = keys_live[0] - if isinstance(arr, _Utf8KeyChunk): + if isinstance(arr, _CodedKeyChunk): # The chunk is already factorized to dense string-rank codes; # dedupe them with an O(n) bincount instead of a sort. The # np.unique contract — uniques ascending by string — holds @@ -1612,7 +1727,7 @@ def _factorize_keys( # order); the codes in the deduped rows are mapped back to strings # below, into object fields (StringDType cannot be a structured-array # field). - pack_arrs = [arr.codes if isinstance(arr, _Utf8KeyChunk) else arr for arr in keys_live] + pack_arrs = [arr.codes if isinstance(arr, _CodedKeyChunk) else arr for arr in keys_live] composite = self._composite_int_factorize(pack_arrs) if composite is not None: unique_fields, inverse = composite @@ -1624,13 +1739,13 @@ def _factorize_keys( packed_unique, inverse = np.unique(packed, return_inverse=True) unique_fields = [packed_unique[f"k{i}"] for i in range(len(pack_arrs))] out_dtype = [ - (f"k{i}", object if isinstance(arr, _Utf8KeyChunk) else pack_arrs[i].dtype) + (f"k{i}", object if isinstance(arr, _CodedKeyChunk) else pack_arrs[i].dtype) for i, arr in enumerate(keys_live) ] unique = np.empty(len(unique_fields[0]), dtype=out_dtype) for i, arr in enumerate(keys_live): field = unique_fields[i] - unique[f"k{i}"] = arr.uniques[field] if isinstance(arr, _Utf8KeyChunk) else field + unique[f"k{i}"] = arr.uniques[field] if isinstance(arr, _CodedKeyChunk) else field return unique, inverse @staticmethod @@ -1744,6 +1859,7 @@ def _compute_partials( inverse: np.ndarray, value_chunks: dict[str, np.ndarray], row_positions: np.ndarray, + valid_chunks: dict[str, np.ndarray] | None = None, ) -> dict[str, Any]: n_groups = len(unique_keys) partials: dict[str, Any] = {} @@ -1754,7 +1870,12 @@ def _compute_partials( assert spec.input_col is not None values = value_chunks[spec.input_col] - non_null = ~self._null_mask(spec.input_col, values, is_key=False) + non_null = ~self._null_mask( + spec.input_col, + values, + is_key=False, + valid=None if valid_chunks is None else valid_chunks.get(spec.input_col), + ) if spec.op == "count": partials[spec.output_col] = np.bincount( @@ -2113,27 +2234,43 @@ def _result_spec_for_agg(self, spec: _AggSpec) -> SchemaSpec: return float64() return copy.deepcopy(input_spec) - def _null_mask(self, name: str, values: np.ndarray, *, is_key: bool) -> np.ndarray: + def _null_mask( + self, name: str, values: np.ndarray, *, is_key: bool, valid: np.ndarray | None = None + ) -> np.ndarray: + """True where this chunk of *name* is null, one flag per row. + + *values* is the chunk the caller already read, which is enough for the + in-band kinds. A mask-storage column's nullity is not in its values at + all, so it arrives separately: as *valid* for a value column, or already + folded into a reserved code for a key column (:meth:`_read_key_chunk`). + """ col_info = self.table._schema.columns_by_name[name] spec = col_info.spec - if isinstance(values, _Utf8KeyChunk): - null_value = getattr(spec, "null_value", None) + null_value = getattr(spec, "null_value", None) + if valid is not None: + # The fill under a null must not be tested as a value: a 0 fill + # would look non-null and a NaN one doubly null. The sidecar is the + # whole answer here, and for a *value* column that includes leaving + # a genuine NaN alone -- in a mask column NaN is a value. + return ~np.asarray(valid, dtype=bool) + if isinstance(values, _CodedKeyChunk): + if values.null_code >= 0: + return values.codes == values.null_code if null_value is None: return np.zeros(len(values), dtype=bool) return values.codes == values.code_of(null_value) if isinstance(spec, DictionarySpec): mask = values == np.int32(spec.null_code) return mask if is_key or getattr(spec, "nullable", False) else np.zeros(len(values), dtype=bool) - null_value = getattr(spec, "null_value", None) mask = np.zeros(len(values), dtype=bool) # For keys, treat all NaNs as missing so dropna behaves predictably. # For values, only nullable NaN sentinels are skipped. - if values.dtype.kind == "f" and ( - is_key or (isinstance(null_value, float) and math.isnan(null_value)) - ): + nan_sentinel = is_nan_sentinel(null_value) + if values.dtype.kind == "f" and (is_key or nan_sentinel): mask |= np.isnan(values) - if null_value is not None and not (isinstance(null_value, float) and math.isnan(null_value)): - mask |= values == null_value + if null_value is not None and not nan_sentinel: + # A float key column with a non-NaN sentinel gets both tests. + mask |= sentinel_mask(values, null_value) return mask @@ -2214,6 +2351,12 @@ def _python_type_for_spec(spec: SchemaSpec): def _max_identity(dtype: np.dtype): dtype = np.dtype(dtype) + if dtype.kind == "b": + # bool needs its own case: np.full(n, None, dtype=bool) is *False*, so a + # min accumulator seeded with it can never rise above False and every + # all-True group reduced to False. (Storage-independent -- reachable + # with any plain bool column on the generic chunked path.) + return True if dtype.kind in "iu": return np.iinfo(dtype).max if dtype.kind == "f": @@ -2225,6 +2368,8 @@ def _max_identity(dtype: np.dtype): def _min_identity(dtype: np.dtype): dtype = np.dtype(dtype) + if dtype.kind == "b": + return False # see _max_identity if dtype.kind in "iu": return np.iinfo(dtype).min if dtype.kind == "f": @@ -2235,7 +2380,16 @@ def _min_identity(dtype: np.dtype): def _null_output_value(spec: SchemaSpec): + """The value to write for a group whose aggregate is missing. + + A mask-storage output column answers ``None``: nullity goes in its sidecar, + so the result is a *real* null rather than a value standing in for one -- + which is the whole reason a group with no non-null input no longer has to + come back as ``0`` or ``NaN`` and hope nobody reads it as data. + """ dtype = getattr(spec, "dtype", None) + if getattr(spec, "uses_mask", False): + return None null_value = getattr(spec, "null_value", None) if null_value is not None: return null_value diff --git a/src/blosc2/indexing.py b/src/blosc2/indexing.py index 6826cf086..f244641de 100644 --- a/src/blosc2/indexing.py +++ b/src/blosc2/indexing.py @@ -18,7 +18,7 @@ import tempfile import warnings import weakref -from collections.abc import Mapping +from collections.abc import Callable, Mapping from concurrent.futures import ThreadPoolExecutor from dataclasses import asdict, dataclass from pathlib import Path @@ -40,6 +40,11 @@ FLAG_ALL_NAN = np.uint8(1 << 0) FLAG_HAS_NAN = np.uint8(1 << 1) +# No row in the segment carries a value: every one of them is null. Set +# together with FLAG_ALL_NAN, whose established meaning -- "min/max are +# placeholders, skip this segment" -- is exactly what a reader that predates +# null-aware summaries needs to do with it. +FLAG_ALL_NULL = np.uint8(1 << 2) SEGMENT_LEVELS_BY_KIND = { # SUMMARY stores per-segment min/max. Block granularity prunes far more @@ -1154,8 +1159,48 @@ def _boundary_dtype(dtype: np.dtype) -> np.dtype: return np.dtype([("start", dtype), ("end", dtype)]) -def _segment_summary(segment: np.ndarray, dtype: np.dtype): +def _rowwise_str_minmax(data_2d: np.ndarray, dtype: np.dtype, valid_2d: np.ndarray | None): + """Per-row min/max for ``U``/``S`` rows; numpy's ufuncs lack a loop for them. + + Rows with no valid entry get the dtype's zero and are flagged by the + caller, which is the same convention the float path uses for all-NaN rows. + """ + n = data_2d.shape[0] + mins = np.empty(n, dtype=dtype) + maxs = np.empty(n, dtype=dtype) + empty = np.zeros((), dtype=dtype)[()] + for i in range(n): + row = data_2d[i] + if valid_2d is not None: + row = row[valid_2d[i]] + if row.shape[0] == 0: + mins[i] = empty + maxs[i] = empty + continue + mn = row[0] + mx = row[0] + for v in row[1:]: + if v < mn: + mn = v + if v > mx: + mx = v + mins[i] = mn + maxs[i] = mx + return mins, maxs + + +def _segment_summary(segment: np.ndarray, dtype: np.dtype, valid: np.ndarray | None = None): flags = np.uint8(0) + if valid is not None: + valid = np.asarray(valid, dtype=bool) + if not valid.all(): + if not valid.any(): + # Nothing to summarise. FLAG_ALL_NAN rides along so that a + # reader which knows only the two NaN flags still skips the + # segment instead of trusting the placeholder extrema. + zero = np.zeros((), dtype=dtype)[()] + return zero, zero, np.uint8(flags | FLAG_ALL_NULL | FLAG_ALL_NAN) + segment = segment[valid] if dtype.kind == "f": valid = ~np.isnan(segment) if not np.all(valid): @@ -1178,7 +1223,9 @@ def _segment_summary(segment: np.ndarray, dtype: np.dtype): return segment.min(), segment.max(), flags -def _compute_segment_summaries(values: np.ndarray, dtype: np.dtype, segment_len: int) -> np.ndarray: +def _compute_segment_summaries( + values: np.ndarray, dtype: np.dtype, segment_len: int, valid: np.ndarray | None = None +) -> np.ndarray: nsegments = math.ceil(values.shape[0] / segment_len) summary_dtype = _summary_dtype(dtype) summaries = np.empty(nsegments, dtype=summary_dtype) @@ -1187,7 +1234,7 @@ def _compute_segment_summaries(values: np.ndarray, dtype: np.dtype, segment_len: start = idx * segment_len stop = min(start + segment_len, values.shape[0]) segment = values[start:stop] - summaries[idx] = _segment_summary(segment, dtype) + summaries[idx] = _segment_summary(segment, dtype, None if valid is None else valid[start:stop]) return summaries @@ -1196,21 +1243,35 @@ def _fill_summaries_from_2d( summaries_arr: np.ndarray, offset: int, dtype: np.dtype, + valid_2d: np.ndarray | None = None, ) -> None: - """Fill summaries_arr[offset:offset+n] from data_2d (shape n×segment_len) with vectorized ops.""" + """Fill summaries_arr[offset:offset+n] from data_2d (shape n×segment_len) with vectorized ops. + + *valid_2d*, when given, has the same shape and excludes null rows from the + extrema, so a nullable column's summaries describe only the values a query + can actually match. Rows with no valid entry at all are flagged + ``FLAG_ALL_NULL``. + """ n = data_2d.shape[0] if n == 0: return + all_null = None if valid_2d is None else ~valid_2d.any(axis=1) if dtype.kind == "f": # All-NaN blocks make np.nanmin/nanmax emit "All-NaN slice encountered"; # their results are immediately overwritten with zero below, so silence # the (purely cosmetic) RuntimeWarning. + # NaN under a null row is the *fill*, not data, so the flags are taken + # over the valid rows only: FLAG_HAS_NAN must keep meaning "a real NaN + # value lives here", which is what makes the min()/max() shortcut safe + # to disable on exactly the columns that need it. + masked = data_2d if valid_2d is None else np.where(valid_2d, data_2d, np.nan) with np.errstate(all="ignore"), warnings.catch_warnings(): warnings.filterwarnings("ignore", r"All-NaN slice encountered", RuntimeWarning) - has_nan = np.any(np.isnan(data_2d), axis=1) - all_nan = np.all(np.isnan(data_2d), axis=1) - mins = np.nanmin(data_2d, axis=1) - maxs = np.nanmax(data_2d, axis=1) + is_nan = np.isnan(masked) + has_nan = np.any(is_nan if valid_2d is None else (is_nan & valid_2d), axis=1) + all_nan = np.all(is_nan, axis=1) + mins = np.nanmin(masked, axis=1) + maxs = np.nanmax(masked, axis=1) flags = np.where(has_nan, FLAG_HAS_NAN, np.uint8(0)).astype(np.uint8) flags = np.where(all_nan, np.uint8(FLAG_ALL_NAN | FLAG_HAS_NAN), flags) zero = dtype.type(0) @@ -1220,23 +1281,25 @@ def _fill_summaries_from_2d( if dtype.kind in "US": # String dtypes: numpy ufunc 'minimum'/'maximum' lack a loop for mx: - mx = v - mins[i] = mn - maxs[i] = mx - else: + mins, maxs = _rowwise_str_minmax(data_2d, dtype, valid_2d) + elif valid_2d is None: mins = data_2d.min(axis=1) maxs = data_2d.max(axis=1) + else: + # Substitute each row's *own* extremum for its null slots: it can + # never win the opposite reduction, so one masked pass gives the + # extrema over the valid rows. Using the row rather than the + # dtype's limits keeps this working for any orderable kind -- + # including datetime64, which has no np.iinfo. An all-null row + # degenerates to the row's own min/max and is overwritten below. + mins = np.where(valid_2d, data_2d, data_2d.max(axis=1, keepdims=True)).min(axis=1) + maxs = np.where(valid_2d, data_2d, data_2d.min(axis=1, keepdims=True)).max(axis=1) flags = np.zeros(n, dtype=np.uint8) + if all_null is not None and all_null.any(): + zero = np.zeros((), dtype=dtype)[()] + mins = np.where(all_null, zero, mins) + maxs = np.where(all_null, zero, maxs) + flags = np.where(all_null, np.uint8(FLAG_ALL_NULL | FLAG_ALL_NAN), flags).astype(np.uint8) summaries_arr["min"][offset : offset + n] = mins summaries_arr["max"][offset : offset + n] = maxs summaries_arr["flags"][offset : offset + n] = flags @@ -1429,12 +1492,14 @@ def _build_levels_descriptor( persistent: bool, cparams: dict | None = None, summary_levels: tuple[str, ...] | None = None, + validity: Callable[[np.ndarray, int, int], np.ndarray | None] | None = None, ) -> dict: levels = {} levels_to_build = summary_levels if summary_levels is not None else SEGMENT_LEVELS_BY_KIND[kind] + valid = None if validity is None else validity(values, 0, int(values.shape[0])) for level in levels_to_build: segment_len = _segment_len(array, level) - summaries = _compute_segment_summaries(values, dtype, segment_len) + summaries = _compute_segment_summaries(values, dtype, segment_len, valid) sidecar = _store_array_sidecar( array, token, kind, "summary", level, summaries, persistent, cparams=cparams ) @@ -1457,6 +1522,7 @@ def _build_levels_descriptor_ooc( cparams: dict | None = None, summary_levels: tuple[str, ...] | None = None, precomputed_summaries: dict[str, np.ndarray] | None = None, + validity: Callable[[np.ndarray, int, int], np.ndarray | None] | None = None, ) -> dict: size = int(array.shape[0]) summary_dtype = _summary_dtype(dtype) @@ -1472,11 +1538,21 @@ def _build_levels_descriptor_ooc( # column back just to recompute min/max. Only trusted when every requested # level is present with the exact expected segment count and dtype; otherwise # fall through to the decompression pass below. - use_precomputed = precomputed_summaries is not None and all( - level in precomputed_summaries - and precomputed_summaries[level].dtype == summary_dtype - and len(precomputed_summaries[level]) == nsegments_total[level] - for level in levels_to_build + # A validity provider always wins: the accumulator that produced the + # precomputed summaries folded min/max as rows were written, with no + # validity in hand, so trusting them would silently hand back extrema over + # the fill values. The caller keeps the two mutually exclusive + # (CTable._precomputed_summary_for declines for a column with nulls); the + # guard here makes that a property of the builder rather than of its caller. + use_precomputed = ( + validity is None + and precomputed_summaries is not None + and all( + level in precomputed_summaries + and precomputed_summaries[level].dtype == summary_dtype + and len(precomputed_summaries[level]) == nsegments_total[level] + for level in levels_to_build + ) ) if use_precomputed: for level in levels_to_build: @@ -1496,6 +1572,7 @@ def _build_levels_descriptor_ooc( chunk_stop = min(chunk_start + chunk_len, size) chunk_values = _slice_values_for_target(array, target, chunk_start, chunk_stop) chunk_size = chunk_stop - chunk_start + chunk_valid = None if validity is None else validity(chunk_values, chunk_start, chunk_stop) for level in levels_to_build: slen = segment_lens[level] summaries_arr = all_summaries[level] @@ -1504,10 +1581,17 @@ def _build_levels_descriptor_ooc( remainder = chunk_size % slen if n_complete > 0: data_2d = chunk_values[: n_complete * slen].reshape(n_complete, slen) - _fill_summaries_from_2d(data_2d, summaries_arr, offset, dtype) + valid_2d = ( + None + if chunk_valid is None + else chunk_valid[: n_complete * slen].reshape(n_complete, slen) + ) + _fill_summaries_from_2d(data_2d, summaries_arr, offset, dtype, valid_2d) if remainder > 0: summaries_arr[offset + n_complete] = _segment_summary( - chunk_values[n_complete * slen :], dtype + chunk_values[n_complete * slen :], + dtype, + None if chunk_valid is None else chunk_valid[n_complete * slen :], ) seg_offsets[level] = offset + n_complete + 1 else: @@ -1519,8 +1603,9 @@ def _build_levels_descriptor_ooc( for idx in range(nsegments_total[level]): start = idx * slen stop = min(start + slen, size) + seg_values = _slice_values_for_target(array, target, start, stop) all_summaries[level][idx] = _segment_summary( - _slice_values_for_target(array, target, start, stop), dtype + seg_values, dtype, None if validity is None else validity(seg_values, start, stop) ) levels = {} @@ -3990,6 +4075,7 @@ def _build_descriptor( full: dict | None, cparams: dict | None = None, opsi: dict | None = None, + null_aware: bool = False, ) -> dict: return { "name": name @@ -4013,6 +4099,11 @@ def _build_descriptor( "full": full, "opsi": opsi, "cparams": _plain_index_cparams(cparams), + # True when the segment summaries were built with a validity channel, + # so their extrema cover only rows that carry a value. Absent on every + # index built before null-aware summaries existed, which is why readers + # must spell this ``.get("null_aware", False)``. + "null_aware": bool(null_aware), } @@ -4042,6 +4133,7 @@ def create_index( opsi_max_cycles_arg = kwargs.pop("opsi_max_cycles", None) summary_levels = kwargs.pop("summary_levels", None) precomputed_summaries = kwargs.pop("precomputed_summaries", None) + validity = kwargs.pop("validity", None) if kwargs: unexpected = ", ".join(sorted(kwargs)) raise TypeError(f"unexpected keyword argument(s): {unexpected}") @@ -4074,6 +4166,7 @@ def create_index( cparams, summary_levels=summary_levels, precomputed_summaries=precomputed_summaries, + validity=validity, ) bucket = ( _build_bucket_descriptor_ooc(array, target, token, kind, dtype, optlevel, persistent, cparams) @@ -4115,11 +4208,21 @@ def create_index( full, cparams, opsi, + validity is not None, ) else: values = _values_for_target(array, target) levels = _build_levels_descriptor( - array, target, token, kind, dtype, values, persistent, cparams, summary_levels=summary_levels + array, + target, + token, + kind, + dtype, + values, + persistent, + cparams, + summary_levels=summary_levels, + validity=validity, ) bucket = ( _build_bucket_descriptor(array, token, kind, values, optlevel, persistent, cparams) @@ -4156,6 +4259,7 @@ def create_index( full, cparams, opsi, + validity is not None, ) store = _load_store(array) diff --git a/src/blosc2/schema.py b/src/blosc2/schema.py index bdb131ae0..a95152e6e 100644 --- a/src/blosc2/schema.py +++ b/src/blosc2/schema.py @@ -33,6 +33,127 @@ def _normalize_scalar_value(value): return value +# How a column represents its nulls. Defined here rather than in +# ``ctable_nulls``, which is where the null machinery otherwise lives, only +# because that module imports the spec classes from this one -- this is the +# lower layer. ``ctable_nulls`` re-exports these names. +NULL_NONE = "none" +NULL_MASK = "mask" +NULL_SENTINEL = "sentinel" +NULL_CODE = "code" +NULL_NATIVE = "native" + +#: The storages a spec may ask for explicitly. ``None`` means "let the null +#: policy decide when the schema is compiled". +_EXPLICIT_NULL_STORAGES = (NULL_MASK, NULL_SENTINEL) + + +class _NullableSpecMixin: + """Shared nullability plumbing for specs that can carry a null channel. + + ``supports_sentinel`` says whether this kind can represent nulls in band + at all. Complex is the one that cannot -- there is no value to steal from + the complex plane -- so it is mask-only. + + Every such spec declares the same three things: whether it is nullable, + which sentinel stands for its nulls, and where nullity is *stored* -- + in band as that sentinel, or in a sidecar validity array. Routing all + of them through one mixin is what lets ``null_storage`` reach nine specs + through a single kwarg and a single call each. + """ + + supports_sentinel = True + + #: True once ``__init__`` has physically widened a boolean column to + #: ``uint8`` to make room for the reserved ``255``. Only that flip may be + #: undone (``CTable._unflip_mask_bool_dtype``): a column whose *declared* + #: dtype is ``uint8`` is carrying real byte values, and turning it into + #: ``np.bool_`` would truncate every one of them to a flag. + bool_widened_to_uint8 = False + + def _init_nulls(self, *, nullable, null_value, null_storage) -> None: + if null_storage is not None and null_storage not in _EXPLICIT_NULL_STORAGES: + raise ValueError( + f"null_storage must be one of {_EXPLICIT_NULL_STORAGES} or None, got {null_storage!r}" + ) + if null_storage == NULL_MASK and null_value is not None: + raise ValueError( + "null_storage='mask' keeps nullity out of band, so it cannot be combined with an " + "explicit null_value (which is what in-band 'sentinel' storage means). " + "Drop one of the two." + ) + self.nullable = _builtin_bool(nullable or null_value is not None or null_storage is not None) + self.null_value = _normalize_scalar_value(null_value) + self.null_storage = null_storage + + @property + def uses_mask(self) -> _builtin_bool: + """True when nullity lives in a sidecar validity array.""" + return self.null_storage == NULL_MASK + + @property + def uses_sentinel(self) -> _builtin_bool: + """True when nullity is an in-band sentinel value. + + Note this is about *resolved* storage: a spec built as + ``nullable=True`` under the default policy has no sentinel yet, and + answers False until :meth:`CTable._resolve_nullable_specs` picks one. + """ + return self.null_value is not None + + def _null_metadata(self) -> dict[str, Any]: + """The nullability entries this spec contributes to its metadata dict. + + ``"sentinel"`` is deliberately never emitted: it is the original and + still the default storage, so a sentinel table serializes exactly as + it did before mask storage existed, and keeps opening in readers that + predate it. Only ``"mask"`` is recorded, and only that raises the + schema version. + """ + d: dict[str, Any] = {} + if self.nullable: + d["nullable"] = True + if self.null_value is not None: + d["null_value"] = self.null_value + if self.null_storage == NULL_MASK: + d["null_storage"] = NULL_MASK + return d + + +def fill_value_for(spec): + """The value written into a mask-backed column's null slots. + + Chosen to be *loud* where the dtype allows it -- ``NaN`` for floats, + ``int64.min`` for timestamps, which ``_maybe_decode_timestamp_values`` + already surfaces as ``NaT`` -- and the dtype's zero otherwise. + + This is **not** part of the format contract and is deliberately not + recorded in the schema: recording it would recreate the very sentinel + collisions that mask storage exists to avoid. Values under ``mask=False`` + are unobservable through the ``Column`` API, so the fill is an + implementation detail that may change. + """ + from blosc2.schema import timestamp as _timestamp # local: class defined below + + if isinstance(spec, _timestamp): + return int(np.iinfo(np.int64).min) + dtype = getattr(spec, "dtype", None) + if dtype is None: # utf8 and other dtype-less specs + return "" + dtype = np.dtype(dtype) + if dtype.kind == "f": + return float("nan") + if dtype.kind == "c": + return 0j + if dtype.kind == "U": + return "" + if dtype.kind == "S": + return b"" + if dtype.kind == "b": + return False + return dtype.type(0) + + # --------------------------------------------------------------------------- # Base spec class # --------------------------------------------------------------------------- @@ -81,23 +202,32 @@ def to_metadata_dict(self) -> dict[str, Any]: # and `_kind` as class attributes. -class _NumericSpec(SchemaSpec): - """Mixin for numeric specs that support constraints and null sentinels. +class _NumericSpec(_NullableSpecMixin, SchemaSpec): + """Mixin for numeric specs that support constraints and nulls. - ``nullable=True`` asks CTable to choose a null sentinel from the current - null policy when the schema is compiled. An explicit ``null_value`` takes - precedence. + ``nullable=True`` asks CTable to resolve the null representation from the + current null policy when the schema is compiled. An explicit + ``null_value`` (in-band sentinel) or ``null_storage`` takes precedence. """ _kind: str # set by each concrete subclass - def __init__(self, *, ge=None, gt=None, le=None, lt=None, nullable: bool = False, null_value=None): + def __init__( + self, + *, + ge=None, + gt=None, + le=None, + lt=None, + nullable: bool = False, + null_value=None, + null_storage: str | None = None, + ): self.ge = ge self.gt = gt self.le = le self.lt = lt - self.nullable = nullable or null_value is not None - self.null_value = _normalize_scalar_value(null_value) + self._init_nulls(nullable=nullable, null_value=null_value, null_storage=null_storage) def to_pydantic_kwargs(self) -> dict[str, Any]: # null_value is not a Pydantic constraint — exclude it from Pydantic kwargs. @@ -108,12 +238,7 @@ def to_pydantic_kwargs(self) -> dict[str, Any]: } def to_metadata_dict(self) -> dict[str, Any]: - d: dict[str, Any] = {"kind": self._kind, **self.to_pydantic_kwargs()} - if self.nullable: - d["nullable"] = True - if self.null_value is not None: - d["null_value"] = self.null_value - return d + return {"kind": self._kind, **self.to_pydantic_kwargs(), **self._null_metadata()} # ── Signed integers ────────────────────────────────────────────────────────── @@ -205,39 +330,54 @@ class float64(_NumericSpec): _kind = "float64" -class complex64(SchemaSpec): - """64-bit complex number column (two 32-bit floats).""" +class _ComplexSpec(_NullableSpecMixin, SchemaSpec): + """Shared body for the complex specs. + + Complex columns gain nullability here for the first time, and only with + ``null_storage="mask"``: there is no sensible in-band sentinel to steal + from the complex plane, which is precisely the argument for a side + channel. ``nullable=True`` alone therefore resolves to mask storage + regardless of the policy's default. + """ - dtype = np.dtype(np.complex64) python_type = complex + supports_sentinel = False + _kind: str - def __init__(self): - pass + def __init__(self, *, nullable: bool = False, null_value=None, null_storage: str | None = None): + if null_value is not None: + raise ValueError( + "complex columns cannot use an in-band null sentinel: no complex value is safe to " + "reserve. Use null_storage='mask' instead." + ) + if nullable and null_storage is None: + null_storage = NULL_MASK + self._init_nulls(nullable=nullable, null_value=None, null_storage=null_storage) + if not self.uses_mask and self.nullable: + raise ValueError("complex columns support null_storage='mask' only") def to_pydantic_kwargs(self) -> dict[str, Any]: return {} def to_metadata_dict(self) -> dict[str, Any]: - return {"kind": "complex64"} + return {"kind": self._kind, **self._null_metadata()} -class complex128(SchemaSpec): - """128-bit complex number column (two 64-bit floats).""" +class complex64(_ComplexSpec): + """64-bit complex number column (two 32-bit floats).""" - dtype = np.dtype(np.complex128) - python_type = complex + dtype = np.dtype(np.complex64) + _kind = "complex64" - def __init__(self): - pass - def to_pydantic_kwargs(self) -> dict[str, Any]: - return {} +class complex128(_ComplexSpec): + """128-bit complex number column (two 64-bit floats).""" - def to_metadata_dict(self) -> dict[str, Any]: - return {"kind": "complex128"} + dtype = np.dtype(np.complex128) + _kind = "complex128" -class timestamp(SchemaSpec): +class timestamp(_NullableSpecMixin, SchemaSpec): """Timestamp column stored as signed 64-bit epoch offsets. The physical storage dtype is ``int64``. ``unit`` follows Arrow/NumPy @@ -249,14 +389,19 @@ class timestamp(SchemaSpec): python_type = _builtin_object def __init__( - self, *, unit: str = "us", timezone: str | None = None, nullable: bool = False, null_value=None + self, + *, + unit: str = "us", + timezone: str | None = None, + nullable: bool = False, + null_value=None, + null_storage: str | None = None, ): if unit not in {"s", "ms", "us", "ns"}: raise ValueError("timestamp unit must be one of: 's', 'ms', 'us', 'ns'") self.unit = unit self.timezone = timezone - self.nullable = nullable or null_value is not None - self.null_value = _normalize_scalar_value(null_value) + self._init_nulls(nullable=nullable, null_value=null_value, null_storage=null_storage) def to_pydantic_kwargs(self) -> dict[str, Any]: return {} @@ -265,39 +410,42 @@ def to_metadata_dict(self) -> dict[str, Any]: d: dict[str, Any] = {"kind": "timestamp", "unit": self.unit} if self.timezone is not None: d["timezone"] = self.timezone - if self.nullable: - d["nullable"] = True - if self.null_value is not None: - d["null_value"] = self.null_value + d.update(self._null_metadata()) return d -class bool(SchemaSpec): +class bool(_NullableSpecMixin, SchemaSpec): """Boolean column. - Nullable bool columns use uint8 physical storage with values - ``0`` (false), ``1`` (true), and ``255`` (null). + Under sentinel storage a nullable bool column is physically ``uint8``, + with ``0`` (false), ``1`` (true) and ``255`` (null) -- the reserved + ``255`` is why raw reads and predicates need special handling. Under + mask storage it stays a real ``np.bool_`` column and nullity lives in + the sidecar, so none of that applies. """ dtype = np.dtype(np.bool_) python_type = _builtin_bool - def __init__(self, *, nullable: bool = False, null_value=None): - if null_value is not None and null_value != 255: + def __init__(self, *, nullable: bool = False, null_value=None, null_storage: str | None = None): + if null_value is not None and null_value != 255 and null_storage != NULL_MASK: raise ValueError("Nullable bool null_value must be 255") - self.nullable = nullable or null_value is not None - self.null_value = _normalize_scalar_value(null_value) - self.dtype = np.dtype(np.uint8) if self.nullable else np.dtype(np.bool_) + self._init_nulls(nullable=nullable, null_value=null_value, null_storage=null_storage) + # Resolve the physical dtype as far as it can be resolved here. This + # must happen in __init__ and not only in _resolve_nullable_specs, + # because *opening* a stored table rebuilds specs through + # ``spec_cls(**data)`` and never runs the resolver: a nullable bool + # persisted as uint8 has to come back as uint8 from its metadata + # alone. When storage is still unresolved (plain ``nullable=True``, + # policy decides later) the resolver corrects this both ways. + self.bool_widened_to_uint8 = _builtin_bool(self.nullable and not self.uses_mask) + self.dtype = np.dtype(np.uint8) if self.bool_widened_to_uint8 else np.dtype(np.bool_) def to_pydantic_kwargs(self) -> dict[str, Any]: return {} def to_metadata_dict(self) -> dict[str, Any]: - d: dict[str, Any] = {"kind": "bool"} - if self.nullable: - d["nullable"] = True - d["null_value"] = self.null_value - return d + return {"kind": "bool", **self._null_metadata()} # --------------------------------------------------------------------------- @@ -305,7 +453,7 @@ def to_metadata_dict(self) -> dict[str, Any]: # --------------------------------------------------------------------------- -class string(SchemaSpec): +class string(_NullableSpecMixin, SchemaSpec): """Fixed-width Unicode string column. Values longer than *max_length* are rejected at validation time (and @@ -327,19 +475,29 @@ class string(SchemaSpec): the current CTable null policy when the schema is compiled. null_value: Explicit null sentinel. Takes precedence over ``nullable=True``. + null_storage: + ``"sentinel"`` to reserve a value from the dtype's range for nulls, or + ``"mask"`` to keep nullity in a sidecar validity array so the whole + range stays usable. Defaults to the current null policy. """ python_type = str _DEFAULT_MAX_LENGTH = 32 def __init__( - self, *, min_length=None, max_length=None, pattern=None, nullable: bool = False, null_value=None + self, + *, + min_length=None, + max_length=None, + pattern=None, + nullable: bool = False, + null_value=None, + null_storage: str | None = None, ): self.min_length = min_length self.max_length = max_length if max_length is not None else self._DEFAULT_MAX_LENGTH self.pattern = pattern - self.nullable = nullable or null_value is not None - self.null_value = _normalize_scalar_value(null_value) + self._init_nulls(nullable=nullable, null_value=null_value, null_storage=null_storage) self.dtype = np.dtype(f"U{self.max_length}") def to_pydantic_kwargs(self) -> dict[str, Any]: @@ -353,15 +511,10 @@ def to_pydantic_kwargs(self) -> dict[str, Any]: return d def to_metadata_dict(self) -> dict[str, Any]: - d: dict[str, Any] = {"kind": "string", **self.to_pydantic_kwargs()} - if self.nullable: - d["nullable"] = True - if self.null_value is not None: - d["null_value"] = self.null_value - return d + return {"kind": "string", **self.to_pydantic_kwargs(), **self._null_metadata()} -class bytes(SchemaSpec): +class bytes(_NullableSpecMixin, SchemaSpec): """Fixed-width bytes column. Parameters @@ -376,16 +529,27 @@ class bytes(SchemaSpec): the current CTable null policy when the schema is compiled. null_value: Explicit null sentinel. Takes precedence over ``nullable=True``. + null_storage: + ``"sentinel"`` to reserve a value from the dtype's range for nulls, or + ``"mask"`` to keep nullity in a sidecar validity array so the whole + range stays usable. Defaults to the current null policy. """ python_type = _builtin_bytes _DEFAULT_MAX_LENGTH = 32 - def __init__(self, *, min_length=None, max_length=None, nullable: bool = False, null_value=None): + def __init__( + self, + *, + min_length=None, + max_length=None, + nullable: bool = False, + null_value=None, + null_storage: str | None = None, + ): self.min_length = min_length self.max_length = max_length if max_length is not None else self._DEFAULT_MAX_LENGTH - self.nullable = nullable or null_value is not None - self.null_value = _normalize_scalar_value(null_value) + self._init_nulls(nullable=nullable, null_value=null_value, null_storage=null_storage) self.dtype = np.dtype(f"S{self.max_length}") def to_pydantic_kwargs(self) -> dict[str, Any]: @@ -397,12 +561,7 @@ def to_pydantic_kwargs(self) -> dict[str, Any]: return d def to_metadata_dict(self) -> dict[str, Any]: - d: dict[str, Any] = {"kind": "bytes", **self.to_pydantic_kwargs()} - if self.nullable: - d["nullable"] = True - if self.null_value is not None: - d["null_value"] = self.null_value - return d + return {"kind": "bytes", **self.to_pydantic_kwargs(), **self._null_metadata()} # --------------------------------------------------------------------------- @@ -597,7 +756,7 @@ def to_metadata_dict(self) -> dict[str, Any]: return d -class UTF8Spec(SchemaSpec): +class UTF8Spec(_NullableSpecMixin, SchemaSpec): """Variable-length UTF-8 string column stored Arrow-style as offsets + bytes. Unlike :class:`string`, this spec does not use a fixed-width NumPy dtype: @@ -619,7 +778,13 @@ class UTF8Spec(SchemaSpec): python_type = str dtype = None - def __init__(self, *, nullable: _builtin_bool = False, null_value: str | None = None): + def __init__( + self, + *, + nullable: _builtin_bool = False, + null_value: str | None = None, + null_storage: str | None = None, + ): if null_value is not None and not isinstance(null_value, str): raise TypeError(f"utf8 null_value must be str, got {type(null_value).__name__!r}") if null_value == "\x00": @@ -633,19 +798,13 @@ def __init__(self, *, nullable: _builtin_bool = False, null_value: str | None = "match it against StringDType arrays, so nulls would go undetected. " "Use a longer sentinel (the default is '__BLOSC2_NULL__')." ) - self.nullable = nullable or null_value is not None - self.null_value = _normalize_scalar_value(null_value) + self._init_nulls(nullable=nullable, null_value=null_value, null_storage=null_storage) def to_pydantic_kwargs(self) -> dict[str, Any]: return {} def to_metadata_dict(self) -> dict[str, Any]: - d: dict[str, Any] = {"kind": "utf8"} - if self.nullable: - d["nullable"] = True - if self.null_value is not None: - d["null_value"] = self.null_value - return d + return {"kind": "utf8", **self._null_metadata()} def display_label(self) -> str: return "utf8" @@ -771,7 +930,7 @@ def to_metadata_dict(self) -> dict[str, Any]: # --------------------------------------------------------------------------- -class NDArraySpec(SchemaSpec): +class NDArraySpec(_NullableSpecMixin, SchemaSpec): """Fixed-shape N-D array column for CTable. Each row stores a NumPy-compatible array with shape ``item_shape`` and @@ -781,7 +940,15 @@ class NDArraySpec(SchemaSpec): python_type = _builtin_object - def __init__(self, item_shape, dtype=np.float64, *, nullable: bool = False, null_value=None): + def __init__( + self, + item_shape, + dtype=np.float64, + *, + nullable: bool = False, + null_value=None, + null_storage: str | None = None, + ): if isinstance(item_shape, int): item_shape = (item_shape,) item_shape = tuple(int(s) for s in item_shape) @@ -791,9 +958,14 @@ def __init__(self, item_shape, dtype=np.float64, *, nullable: bool = False, null raise ValueError("All NDArraySpec item_shape dimensions must be positive.") self.item_shape = item_shape self.dtype = np.dtype(dtype) - self.nullable = nullable or null_value is not None - if null_value is not None: - self.null_value = _normalize_scalar_value(null_value) + self._init_nulls(nullable=nullable, null_value=null_value, null_storage=null_storage) + if self.nullable and not self.uses_mask and self.dtype == np.dtype(np.bool_): + # Same reasoning as bool: opening a stored table rebuilds the spec + # without running the resolver, so the uint8 flip has to survive + # from metadata alone. Recorded, because a *declared* uint8 ndarray + # column holds real byte values and must never be unflipped. + self.bool_widened_to_uint8 = True + self.dtype = np.dtype(np.uint8) self.itemsize = self.dtype.itemsize self.kind = self.dtype.kind self.type = self.dtype.type @@ -809,19 +981,29 @@ def to_metadata_dict(self) -> dict[str, Any]: "item_shape": _builtin_list(self.item_shape), "dtype_str": self.dtype.str, } - if self.nullable: - d["nullable"] = True - if hasattr(self, "null_value"): - d["null_value"] = self.null_value + d.update(self._null_metadata()) return d def display_label(self) -> str: return f"ndarray{_builtin_list(self.item_shape)}[{self.dtype}]" -def ndarray(item_shape, dtype=np.float64, *, nullable: bool = False, null_value=None) -> NDArraySpec: +def ndarray( + item_shape, + dtype=np.float64, + *, + nullable: bool = False, + null_value=None, + null_storage: str | None = None, +) -> NDArraySpec: """Build a fixed-shape N-D array descriptor for CTable columns.""" - return NDArraySpec(item_shape=item_shape, dtype=dtype, nullable=nullable, null_value=null_value) + return NDArraySpec( + item_shape=item_shape, + dtype=dtype, + nullable=nullable, + null_value=null_value, + null_storage=null_storage, + ) def vlstring( @@ -850,7 +1032,9 @@ def vlstring( ) -def utf8(*, nullable: bool = False, null_value: str | None = None) -> UTF8Spec: +def utf8( + *, nullable: bool = False, null_value: str | None = None, null_storage: str | None = None +) -> UTF8Spec: """Build a variable-length UTF-8 string schema descriptor. Use this for high-cardinality or free-text string columns: values are @@ -880,6 +1064,11 @@ def utf8(*, nullable: bool = False, null_value: str | None = None) -> UTF8Spec: the current CTable null policy when the schema is compiled. null_value: Explicit null sentinel string. Takes precedence over ``nullable=True``. + null_storage: + ``"sentinel"`` to reserve a string value for nulls, or ``"mask"`` to + keep nullity in a sidecar validity array so that every string -- + including ``""`` and the default sentinel text -- stays an ordinary + value. Defaults to the current null policy. Examples -------- @@ -893,7 +1082,7 @@ def utf8(*, nullable: bool = False, null_value: str | None = None) -> UTF8Spec: from blosc2._utf8_array import string_dtype string_dtype() # fail early with a clear error on NumPy < 2.0 - return UTF8Spec(nullable=nullable, null_value=null_value) + return UTF8Spec(nullable=nullable, null_value=null_value, null_storage=null_storage) def object( diff --git a/src/blosc2/schema_compiler.py b/src/blosc2/schema_compiler.py index 2475474d7..fe782da0d 100644 --- a/src/blosc2/schema_compiler.py +++ b/src/blosc2/schema_compiler.py @@ -484,7 +484,17 @@ def schema_to_dict(schema: CompiledSchema) -> dict[str, Any]: entry["blocks"] = list(col.config.blocks) cols.append(entry) - schema_version = 2 if schema.metadata.get("nested") is not None else 1 + # Computed as an explicit feature max rather than a running counter, so + # each version says which feature demands it. The bump is *conditional on + # a mask column existing*: sentinel tables keep serializing as version 1/2 + # and stay readable by everything that came before mask storage. + uses_mask = any(getattr(col.spec, "null_storage", None) == "mask" for col in schema.columns) + if uses_mask: + schema_version = 3 + elif schema.metadata.get("nested") is not None: + schema_version = 2 + else: + schema_version = 1 result = { "version": schema_version, "columns": cols, @@ -507,8 +517,12 @@ def schema_from_dict(data: dict[str, Any]) -> CompiledSchema: If *data* uses an unknown schema version or an unknown column kind. """ version = data.get("version", 1) - if version not in (1, 2): - raise ValueError(f"Unsupported schema version {version!r}") + if version not in (1, 2, 3): + raise ValueError( + f"Unsupported schema version {version!r}. Version 3 is written only when a column uses " + f"validity-mask null storage; an older reader can be given a readable copy with " + f"table.convert_nulls(to='sentinel')." + ) columns: list[CompiledColumn] = [] for entry in data["columns"]: diff --git a/src/blosc2/schema_validation.py b/src/blosc2/schema_validation.py index faa7dd2a7..db8ff9390 100644 --- a/src/blosc2/schema_validation.py +++ b/src/blosc2/schema_validation.py @@ -14,13 +14,14 @@ from __future__ import annotations -import math from dataclasses import MISSING from typing import Any import numpy as np from pydantic import BaseModel, Field, ValidationError, create_model +from blosc2.ctable_nulls import is_null_value as _null_value_matches +from blosc2.ctable_nulls import sentinel_mask from blosc2.list_array import _coerce_struct_item, coerce_list_cell from blosc2.schema import ListSpec, NDArraySpec, StructSpec from blosc2.schema_compiler import CompiledSchema # noqa: TC001 @@ -71,16 +72,7 @@ def build_validator_model(schema: CompiledSchema) -> type[BaseModel]: def _is_null_value(val, null_value) -> bool: """Return True if *val* equals the null sentinel, handling NaN correctly.""" - import math - - if null_value is None: - return False - try: - if isinstance(null_value, (float, np.floating)) and math.isnan(null_value): - return isinstance(val, (float, np.floating)) and math.isnan(val) - except TypeError: - pass - return val == null_value + return _null_value_matches(val, null_value) def _mask_nulls(schema: CompiledSchema, row: dict[str, Any]) -> tuple[dict[str, Any], dict[str, Any]]: @@ -104,9 +96,7 @@ def _mask_nulls(schema: CompiledSchema, row: dict[str, Any]) -> tuple[dict[str, try: arr = np.asarray(val, dtype=col.spec.dtype) is_null = arr.shape == col.spec.item_shape and bool( - np.isnan(arr).all() - if isinstance(nv, (float, np.floating)) and math.isnan(nv) - else (arr == nv).all() + sentinel_mask(arr, nv, item_ndim=len(col.spec.item_shape))[0] ) except Exception: is_null = val is None diff --git a/src/blosc2/schema_vectorized.py b/src/blosc2/schema_vectorized.py index e08736eb8..550e48594 100644 --- a/src/blosc2/schema_vectorized.py +++ b/src/blosc2/schema_vectorized.py @@ -19,6 +19,7 @@ import numpy as np +from blosc2.ctable_nulls import is_na_marker, sentinel_mask from blosc2.list_array import _coerce_struct_item, coerce_list_cell from blosc2.schema import ListSpec, NDArraySpec, ObjectSpec, StructSpec from blosc2.schema_compiler import CompiledColumn, CompiledSchema # noqa: TC001 @@ -47,18 +48,22 @@ def _validate_string_lengths(col: CompiledColumn, arr: Any) -> None: def _null_mask_for_spec(arr: np.ndarray, spec) -> np.ndarray | None: - """Return a boolean mask True where values are the null sentinel, or None if no null_value.""" + """Return a boolean mask True where values are null, or ``None`` if none can be. + + A null cell has no value to constrain, so it must bypass the checks below + however the column spells it. Under sentinel storage that is the in-band + sentinel. Under mask storage validation runs *before* the batch is split + into values and validity, so what is still in the array is the caller's + ``None``/NA markers -- and only an object array can be carrying any. + """ + if getattr(spec, "uses_mask", False): + if arr.dtype.kind != "O" or arr.ndim != 1: + return None + return np.fromiter((is_na_marker(v) for v in arr), dtype=np.bool_, count=len(arr)) null_value = getattr(spec, "null_value", None) if null_value is None: return None - try: - import math - - if isinstance(null_value, float) and math.isnan(null_value): - return np.isnan(arr) - except TypeError: - pass - return arr == null_value + return sentinel_mask(arr, null_value) def validate_column_values(col: CompiledColumn, values: Any) -> None: # noqa: C901 @@ -89,12 +94,17 @@ def validate_column_values(col: CompiledColumn, values: Any) -> None: # noqa: C if isinstance(spec, ObjectSpec): return if isinstance(spec, NDArraySpec): - if getattr(spec, "null_value", None) is not None and not ( - isinstance(values, np.ndarray) and values.dtype != object - ): + uses_mask = getattr(spec, "uses_mask", False) + nullable = uses_mask or getattr(spec, "null_value", None) is not None + if nullable and not (isinstance(values, np.ndarray) and values.dtype != object): from blosc2.ctable import CTable for value in values: + # Under mask storage a null row is still a bare ``None`` here -- + # validation runs before the batch is split into values and + # validity -- and a null row has no item to shape-check. + if uses_mask and is_na_marker(value): + continue CTable._coerce_ndarray_value(col.name, spec, value) return arr = np.asarray(values, dtype=spec.dtype) diff --git a/src/blosc2/version.py b/src/blosc2/version.py index 77f4e2019..5014a7983 100644 --- a/src/blosc2/version.py +++ b/src/blosc2/version.py @@ -1,2 +1,2 @@ -__version__ = "4.10.2.dev0" +__version__ = "4.11.0.dev0" __array_api_version__ = "2024.12" diff --git a/tests/ctable/test_column.py b/tests/ctable/test_column.py index 8d880206e..fa9bdd1d6 100644 --- a/tests/ctable/test_column.py +++ b/tests/ctable/test_column.py @@ -528,7 +528,7 @@ class CRow: class MinMaxRow: i: int = blosc2.field(blosc2.int64()) # non-nullable int → fast path f: float = blosc2.field(blosc2.float64(null_value=float("nan"))) # NaN float → fast path - k: int = blosc2.field(blosc2.int64(null_value=INT_MIN)) # INT64_MIN sentinel → fallback + k: int = blosc2.field(blosc2.int64(null_value=INT_MIN)) # INT64_MIN sentinel → fast since null-aware s: str = blosc2.field(blosc2.string(max_length=8)) # non-nullable string → fast path @@ -560,10 +560,17 @@ def indexed_minmax(tmp_path_factory): return path, refs -@pytest.mark.parametrize(("col", "fast"), [("i", True), ("f", True), ("s", True), ("k", False)]) +@pytest.mark.parametrize(("col", "fast"), [("i", True), ("f", True), ("s", True), ("k", True)]) def test_minmax_matches_reference(indexed_minmax, col, fast): """min()/max() equal the live non-null reference, whether or not the - summary fast path is used.""" + summary fast path is used. + + ``k`` used to be the fallback case: its ``INT64_MIN`` sentinel *is* the + block minimum, so the summaries answered with the null. Null-aware + summaries (plans/mask-based-nulls.md, phase 10) take the extrema over the + valid rows only, so it now takes the shortcut like the rest — the payoff + that phase promised for sentinel columns, not only for mask ones. + """ path, refs = indexed_minmax t = blosc2.open(path, mode="r") try: diff --git a/tests/ctable/test_csv_interop.py b/tests/ctable/test_csv_interop.py index 8a904d807..d02097211 100644 --- a/tests/ctable/test_csv_interop.py +++ b/tests/ctable/test_csv_interop.py @@ -8,11 +8,13 @@ """Tests for CTable.to_csv() and CTable.from_csv().""" import csv +import dataclasses import os from dataclasses import dataclass import numpy as np import pytest +from utf8_compat import needs_utf8, utf8_spec import blosc2 from blosc2 import CTable @@ -538,5 +540,69 @@ def test_from_pandas_multi_dim_ndarray_roundtrip(): assert t2["label"][:].tolist() == t["label"][:].tolist() +@dataclass +class MaskTextRow: + text: str = blosc2.field(blosc2.string(max_length=16, nullable=True, null_storage="mask"), default="") + + +def test_csv_mask_text_roundtrip_empty_vs_null(tmp_csv): + """A mask-backed text column keeps "" apart from a null (and from whitespace).""" + values = ["", " ", None, "ok", "\\N", "\\E", "\\\\N", "N"] + t = CTable(MaskTextRow, new_data=[(v,) for v in values]) + t.to_csv(tmp_csv) + t2 = CTable.from_csv(tmp_csv, MaskTextRow) + + assert [None if n else v for v, n in zip(t2["text"][:], t2["text"].is_null(), strict=True)] == values + assert t2["text"].null_count() == 1 + + +# --------------------------------------------------------------------------- +# utf8 columns +# --------------------------------------------------------------------------- +# +# utf8 reports StringDType(), whose kind is "T" -- not the "U"/"S" a fixed-width +# text column reports, and not a fixed element dtype at all on the schema side. +# Both halves of the CSV path used to key off that dtype, so utf8 was the one +# text kind that neither escaped its empty cells nor could be read back. + + +def utf8_row(**kwargs): + """A one-utf8-column dataclass. Only ever called from a ``needs_utf8`` test.""" + return dataclasses.make_dataclass("Utf8Row", [("text", str, blosc2.field(utf8_spec(**kwargs)))]) + + +@needs_utf8 +def test_csv_utf8_column_roundtrips(tmp_csv): + """A plain utf8 column could not be read back at all -- from_csv raised.""" + row_cls = utf8_row() + values = ["x", "", " ", "unicode: \u00e9\u4e2d"] + t = CTable(row_cls, new_data=[(v,) for v in values]) + t.to_csv(tmp_csv) + t2 = CTable.from_csv(tmp_csv, row_cls) + assert list(t2["text"][:]) == values + + +@needs_utf8 +def test_csv_mask_utf8_keeps_empty_apart_from_null(tmp_csv): + """Same contract as the fixed-width text column above, for utf8.""" + row_cls = utf8_row(null_storage="mask") + values = ["", " ", None, "ok", "\\N", "\\E", "\\\\N", "N"] + t = CTable(row_cls, new_data=[(v,) for v in values]) + t.to_csv(tmp_csv) + t2 = CTable.from_csv(tmp_csv, row_cls) + + assert [None if n else v for v, n in zip(t2["text"][:], t2["text"].is_null(), strict=True)] == values + assert t2["text"].null_count() == 1 + + +def test_csv_rejects_a_kind_it_cannot_read(tmp_csv): + """A clear refusal, not an AttributeError from deep inside the conversion.""" + row_cls = dataclasses.make_dataclass("VlRow", [("v", str, blosc2.field(blosc2.vlstring()))]) + with open(tmp_csv, "w") as f: + f.write("v\nq\n") + with pytest.raises(ValueError, match="does not support"): + CTable.from_csv(tmp_csv, row_cls) + + if __name__ == "__main__": pytest.main(["-v", __file__]) diff --git a/tests/ctable/test_ctable_ndarray_columns.py b/tests/ctable/test_ctable_ndarray_columns.py index f3281ccff..df749d179 100644 --- a/tests/ctable/test_ctable_ndarray_columns.py +++ b/tests/ctable/test_ctable_ndarray_columns.py @@ -138,8 +138,12 @@ def test_stale_generated_col_read_stale_hatch(): @dataclass class NullableNDArrayRow: id: int = blosc2.field(blosc2.int32()) - embedding: object = blosc2.field(blosc2.ndarray((3,), dtype=blosc2.float32(), nullable=True)) - codes: object = blosc2.field(blosc2.ndarray((2,), dtype=blosc2.int16(), nullable=True)) + embedding: object = blosc2.field( + blosc2.ndarray((3,), dtype=blosc2.float32(), nullable=True, null_storage="sentinel") + ) + codes: object = blosc2.field( + blosc2.ndarray((2,), dtype=blosc2.int16(), nullable=True, null_storage="sentinel") + ) def test_nullable_ndarray_cols_write_and_reduce(): @@ -183,7 +187,9 @@ class RowWithExplicitNull: def test_nullable_bool_ndarray_uses_uint8_sentinel(): @dataclass class BoolRows: - flags: object = blosc2.field(blosc2.ndarray((2,), dtype=np.bool_, nullable=True)) + flags: object = blosc2.field( + blosc2.ndarray((2,), dtype=np.bool_, nullable=True, null_storage="sentinel") + ) t = blosc2.CTable(BoolRows, new_data=[(None,), ([True, False],)]) assert t.flags.dtype == np.dtype(np.uint8) diff --git a/tests/ctable/test_groupby.py b/tests/ctable/test_groupby.py index 9d1a76620..e0976fa69 100644 --- a/tests/ctable/test_groupby.py +++ b/tests/ctable/test_groupby.py @@ -19,7 +19,7 @@ class SalesRow: city: str = blosc2.field(blosc2.string(max_length=16)) category: int = blosc2.field(blosc2.int32()) - sales: float = blosc2.field(blosc2.float64(nullable=True), default=0.0) + sales: float = blosc2.field(blosc2.float64(nullable=True, null_storage="sentinel"), default=0.0) qty: int = blosc2.field(blosc2.int32(), default=0) @@ -479,7 +479,7 @@ def test_groupby_cython_int_key_null_aggs(): "IntKeyNullableFloatAggsRow", [ ("key", int, blosc2.field(blosc2.uint16())), - ("value", float, blosc2.field(blosc2.float64(nullable=True))), + ("value", float, blosc2.field(blosc2.float64(nullable=True, null_storage="sentinel"))), ], ) t = CTable(row_type, new_data=[(0, 1.5), (1, np.nan), (0, 2.5), (1, np.nan), (2, 10.0)]) @@ -531,7 +531,7 @@ def test_groupby_cython_float_key_nan_and_zero(): class TwoIntKeyFloatRow: key0: int = blosc2.field(blosc2.int16()) key1: int = blosc2.field(blosc2.uint16()) - value: float = blosc2.field(blosc2.float64(nullable=True), default=0.0) + value: float = blosc2.field(blosc2.float64(nullable=True, null_storage="sentinel"), default=0.0) def test_groupby_cython_two_integer_key_hash_aggs(): diff --git a/tests/ctable/test_nested_metadata_root.py b/tests/ctable/test_nested_metadata_root.py index 5b82794e2..261270968 100644 --- a/tests/ctable/test_nested_metadata_root.py +++ b/tests/ctable/test_nested_metadata_root.py @@ -19,7 +19,13 @@ def _table_with_empty_root_alias(): def test_schema_v2_nested_metadata_roundtrip(): - schema = pa.schema([pa.field("x.y", pa.float64())]) + """Nested metadata alone asks for version 2, not 3. + + The version is a feature max, so the field is declared non-nullable to + isolate it: an Arrow field is nullable by default, and a nullable column now + resolves to mask storage, which is what raises the version to 3. + """ + schema = pa.schema([pa.field("x.y", pa.float64(), nullable=False)]) batch = pa.record_batch([pa.array([1.0, 2.0])], schema=schema) t = blosc2.CTable.from_arrow(schema, [batch]) @@ -31,6 +37,18 @@ def test_schema_v2_nested_metadata_roundtrip(): assert restored.metadata["nested"]["logical_to_physical"]["x.y"] == "x.y" +def test_a_mask_column_raises_the_nested_schema_to_v3(): + """The version is the max of the features present, not a per-feature tag.""" + schema = pa.schema([pa.field("x.y", pa.float64())]) # nullable by default + batch = pa.record_batch([pa.array([1.0, 2.0])], schema=schema) + t = blosc2.CTable.from_arrow(schema, [batch]) + + d = schema_to_dict(t._schema) + assert d["version"] == 3 + assert "nested" in d["metadata"] + assert t["x.y"].null_storage == "mask" + + def test_empty_root_exports_empty_arrow_name(): t = _table_with_empty_root_alias() out = t.to_arrow() diff --git a/tests/ctable/test_null_aware_indexes.py b/tests/ctable/test_null_aware_indexes.py new file mode 100644 index 000000000..7fd96959d --- /dev/null +++ b/tests/ctable/test_null_aware_indexes.py @@ -0,0 +1,358 @@ +####################################################################### +# Copyright (c) 2019-present, Blosc Development Team +# All rights reserved. +# +# SPDX-License-Identifier: BSD-3-Clause +####################################################################### + +"""Null-aware column indexes (Phase 10 of plans/mask-based-nulls.md). + +An index summarises the column's *physical* array, so until now every nullable +column looked to it like a column of data: a sentinel entered the per-segment +extrema, and so did a mask column's fill. That is why ``min()``/``max()`` +declined the summary shortcut for any nullable column except the one case where +the null happens to be a NaN the summary builder already dropped. + +Phase 10 hands the builder the column's validity channel. The extrema then +cover only rows that carry a value, a segment with no such row is flagged +``FLAG_ALL_NULL``, and the descriptor records ``null_aware`` so that indexes +built before this keep the old bail instead of being trusted. The payoff lands +on *both* storages: a mask column and an ``INT64_MIN``-sentinel one are equally +unreadable to a summary that does not know about nulls. + +The second half is ``where()``: an OR over a nullable indexed column used to +fall back to a full scan, because the only null filtering available was global +and a global filter drops a row that is null in one branch but matches the +other. The segment path never needed that filter -- it *evaluates* the +predicate, which has been null-aware per leaf since Phase 1 -- so it now serves +OR directly. The ordered-range paths still bail, for the reason recorded in +§Expression layer of the plan: an index that answers by slicing the sorted +column never evaluates anything. +""" + +from __future__ import annotations + +import dataclasses + +import numpy as np +import pytest + +import blosc2 +from blosc2.indexing import ( + FLAG_ALL_NAN, + FLAG_ALL_NULL, + FLAG_HAS_NAN, + _compute_segment_summaries, +) + + +def one_col(values, spec, urlpath, capacity=None, name="a"): + Row = dataclasses.make_dataclass("R", [(name, spec.python_type, blosc2.field(spec))]) + t = blosc2.CTable(Row, expected_size=max(len(values), capacity or 0), urlpath=str(urlpath), mode="w") + t.extend([(v,) for v in values]) + return t + + +# --------------------------------------------------------------------------- +# The summaries themselves +# --------------------------------------------------------------------------- + + +def test_summaries_take_their_extrema_over_the_valid_rows_only(): + """The whole phase in one assertion: -1 is a null, not the minimum.""" + values = np.array([-1, 7, 3, -1, 9, 5], dtype=np.int64) + valid = np.array([False, True, True, False, True, True]) + naive = _compute_segment_summaries(values, values.dtype, 3) + aware = _compute_segment_summaries(values, values.dtype, 3, valid) + assert naive["min"].tolist() == [-1, -1] + assert aware["min"].tolist() == [3, 5] + assert aware["max"].tolist() == [7, 9] + + +def test_an_all_null_segment_is_flagged_rather_than_summarised(): + """FLAG_ALL_NAN rides along with FLAG_ALL_NULL so that a reader which knows + only the NaN flags still skips the segment instead of trusting its zeros.""" + values = np.array([1, 2, 8, 9], dtype=np.int64) + valid = np.array([True, True, False, False]) + summaries = _compute_segment_summaries(values, values.dtype, 2, valid) + assert not summaries["flags"][0] & FLAG_ALL_NULL + assert summaries["flags"][1] & FLAG_ALL_NULL + assert summaries["flags"][1] & FLAG_ALL_NAN + # The placeholder extrema must not look like data. + assert summaries["min"][1] == 0 + assert summaries["max"][1] == 0 + + +def test_has_nan_marks_a_real_nan_and_not_a_nan_fill(): + """This flag is what keeps min() honest on a mask float column. + + A mask column's fill *is* NaN, so a flag raised over all rows would mark + every segment holding a null. Taken over the valid rows it marks exactly + the segments where NaN is data -- the case decision 6 says must poison + min() to NaN, and so must not be answered from the summaries. + """ + values = np.array([1.0, np.nan, 5.0, np.nan], dtype=np.float64) + valid = np.array([True, False, True, True]) # index 1 is a null, index 3 a value + summaries = _compute_segment_summaries(values, values.dtype, 2, valid) + assert not summaries["flags"][0] & FLAG_HAS_NAN # the fill + assert summaries["flags"][1] & FLAG_HAS_NAN # the value + assert summaries["min"][0] == 1.0 + + +def test_string_segments_skip_their_nulls_too(): + values = np.array(["", "pear", "", "apple"], dtype="U8") + valid = np.array([False, True, False, True]) + summaries = _compute_segment_summaries(values, values.dtype, 4, valid) + assert summaries["min"][0] == "apple" + assert summaries["max"][0] == "pear" + + +# --------------------------------------------------------------------------- +# The descriptor claim +# --------------------------------------------------------------------------- + +STORAGES = [ + pytest.param(blosc2.int64(null_storage="mask"), False, id="mask"), + pytest.param(blosc2.int64(nullable=True, null_value=-1), True, id="sentinel"), +] + + +@pytest.mark.parametrize(("spec", "write_sentinel"), STORAGES) +def test_a_nullable_column_records_null_aware(spec, write_sentinel, tmp_path): + vals = [None if i % 7 == 0 else i for i in range(500)] + if write_sentinel: + vals = [-1 if v is None else v for v in vals] + t = one_col(vals, spec, tmp_path / "d.b2t") + t.create_index("a", kind="summary") + assert t._get_index_catalog()["a"]["null_aware"] is True + + +def test_a_null_free_mask_column_is_null_aware_without_a_sidecar(tmp_path): + """Nothing to exclude is not the same as nothing known: the summaries of a + column that has never held a null already agree with a null-skipping scan, + so it must not pay the bail.""" + t = one_col(list(range(500)), blosc2.int64(null_storage="mask"), tmp_path / "n.b2t") + assert t._null_mask("a") is None + t.create_index("a", kind="summary") + assert t._get_index_catalog()["a"]["null_aware"] is True + assert t["a"]._index_summary_minmax("min") == 0 + + +def test_an_index_built_before_phase_10_keeps_the_bail(tmp_path): + """The staleness rule for this phase. An older index carries no + ``null_aware`` key and its extrema cover the sentinel, so it must be read as + unusable rather than as False-meaning-anything-else.""" + vals = [-1 if i % 7 == 0 else i for i in range(500)] + t = one_col(vals, blosc2.int64(nullable=True, null_value=-1), tmp_path / "old.b2t") + t.create_index("a", kind="summary") + assert t["a"]._summary_minmax_source() is not None + + del t._get_index_catalog()["a"]["null_aware"] # simulate the older build + assert t["a"]._summary_minmax_source() is None + assert t["a"].min() == 1 # the scan still answers, and answers correctly + + t.rebuild_index("a") + assert t["a"]._summary_minmax_source() is not None + + +# --------------------------------------------------------------------------- +# min() / max() through the summaries +# --------------------------------------------------------------------------- + +MINMAX_KINDS = [ + pytest.param("int64", blosc2.int64(null_storage="mask"), lambda i: i * 3 % 977, id="int64"), + pytest.param( + "float64", blosc2.float64(null_storage="mask"), lambda i: (i * 7 % 991) / 3.0, id="float64" + ), + pytest.param("uint16", blosc2.uint16(null_storage="mask"), lambda i: i * 5 % 60_000, id="uint16"), + pytest.param("bool", blosc2.bool(null_storage="mask"), lambda i: bool(i % 3), id="bool"), + pytest.param( + "string", blosc2.string(max_length=8, null_storage="mask"), lambda i: f"s{i % 977:05d}", id="string" + ), +] + + +@pytest.mark.parametrize(("label", "spec", "value_of"), MINMAX_KINDS) +def test_the_shortcut_agrees_with_the_scan_for_every_kind(label, spec, value_of, tmp_path): + n = 20_000 + vals = [None if i % 11 == 0 else value_of(i) for i in range(n)] + live = [v for v in vals if v is not None] + t = one_col(vals, spec, tmp_path / f"{label}.b2t") + t.create_index("a", kind="summary") + + assert t["a"]._index_summary_minmax("min") is not NotImplemented + assert t["a"].min() == min(live) + assert t["a"].max() == max(live) + + +def test_a_full_range_int8_reports_its_true_extrema(tmp_path): + """The case no sentinel column can express, so nothing else could test it: + -128 and 127 are both data and there is no spare value to mean null.""" + n = 5_000 + vals = [None if i % 9 == 0 else (i % 256) - 128 for i in range(n)] + live = [v for v in vals if v is not None] + t = one_col(vals, blosc2.int8(null_storage="mask"), tmp_path / "i8.b2t") + t.create_index("a", kind="summary") + assert t["a"]._index_summary_minmax("min") == -128 + assert t["a"].min() == min(live) == -128 + assert t["a"].max() == max(live) == 127 + + +def test_the_sentinel_storage_gets_the_same_answer(tmp_path): + """Phase 10 is not a mask feature: an INT64_MIN sentinel is exactly as + invisible to a summary that does not consult a validity channel.""" + imin = np.iinfo(np.int64).min + n = 20_000 + vals = [imin if i % 11 == 0 else (i * 3 % 977) for i in range(n)] + live = [v for v in vals if v != imin] + t = one_col(vals, blosc2.int64(nullable=True, null_value=imin), tmp_path / "s.b2t") + t.create_index("a", kind="summary") + assert t["a"]._index_summary_minmax("min") == min(live) + assert t["a"].max() == max(live) + + +def test_nulls_in_the_straddling_block_are_skipped_by_hand(tmp_path): + """The one block the summaries cannot cover is rescanned, and that rescan + has to apply the same rule they did.""" + n = 100_003 # big enough for several whole blocks, and not a multiple of one + vals = list(range(10, n + 10)) + t = one_col(vals, blosc2.int64(null_storage="mask"), tmp_path / "tail.b2t") + t.create_index("a", kind="summary") + segment_len = t["a"]._summary_minmax_source()[-1] + tail_start = (n // segment_len) * segment_len + assert tail_start < n, "the fixture needs a partial trailing block" + + # Put both a new minimum and a null in the rescanned tail. + t["a"][tail_start] = 0 + t["a"][tail_start + 1] = None + t.rebuild_index("a") + assert t["a"]._index_summary_minmax("min") == 0 + assert t["a"].min() == 0 + assert t["a"].null_count() == 1 + + +def test_an_all_null_column_falls_back_to_the_scan(tmp_path): + """Every segment is FLAG_ALL_NULL, so there is nothing left to reduce and + the shortcut must decline rather than answer with the placeholder zeros.""" + t = one_col([None] * 2_000, blosc2.int64(null_storage="mask"), tmp_path / "allnull.b2t") + t.create_index("a", kind="summary") + assert t["a"]._index_summary_minmax("min") is NotImplemented + with pytest.raises(ValueError): + t["a"].min() + + +def test_a_null_written_after_the_build_takes_the_shortcut_away(tmp_path): + """A mask column with no sidecar claims null_aware, so the claim has to stop + holding the moment a null appears. It does, through the ordinary staleness + rule: the write invalidates the index.""" + t = one_col(list(range(5_000)), blosc2.int64(null_storage="mask"), tmp_path / "later.b2t") + t.create_index("a", kind="summary") + assert t["a"]._index_summary_minmax("min") == 0 + + t["a"][0] = None + assert t._get_index_catalog()["a"]["stale"] is True + assert t["a"]._index_summary_minmax("min") is NotImplemented + assert t["a"].min() == 1 + + t.rebuild_index("a") + assert t["a"]._index_summary_minmax("min") == 1 + + +# --------------------------------------------------------------------------- +# where(): OR over a nullable indexed column +# --------------------------------------------------------------------------- + + +def or_table(tmp_path, name, spec, sentinel=None): + """Two int columns whose large values are clustered, so a summary index has + something to prune and the planner prefers it to a scan. + + The row count is load-bearing and cannot be trimmed much: the planner's + cost model only prefers the index once the scan it would replace is big + enough, and measured here the switch happens between 200k and 400k rows. + Two million keeps a comfortable margin. It is not expensive despite the + size -- the values are deliberately repetitive, so the whole table plus + both indexes come to about 90 KB on disk. + """ + n = 2_000_000 + idx = np.arange(n) + a = np.where((idx >= 1000) & (idx < 1100), 5000 + idx, idx % 100).astype(np.int64) + b = np.where((idx >= 1_500_000) & (idx < 1_500_100), 7000 + idx, idx % 100).astype(np.int64) + nulls = (idx % 31) == 0 + + Row = dataclasses.make_dataclass( + "OrRow", [("a", int, blosc2.field(spec)), ("b", int, blosc2.field(blosc2.int64()))] + ) + t = blosc2.CTable(Row, expected_size=n, urlpath=str(tmp_path / name), mode="w") + if sentinel is None: + t.extend({"a": np.ma.MaskedArray(a, mask=nulls), "b": b}) + else: + col = a.copy() + col[nulls] = sentinel + t.extend({"a": col, "b": b}) + t.create_index("a", kind="summary") + t.create_index("b", kind="summary") + expected = int(np.count_nonzero(((a > 4000) & ~nulls) | (b > 6000))) + return t, expected + + +@pytest.mark.parametrize( + ("label", "spec", "sentinel"), + [ + ("mask", blosc2.int64(null_storage="mask"), None), + ("sentinel", blosc2.int64(nullable=True, null_value=-1), -1), + ], +) +def test_indexed_or_over_a_nullable_column_uses_its_index(label, spec, sentinel, tmp_path, monkeypatch): + """It used to bail to a full scan; the segment path evaluates the + (null-aware) predicate, so it is exact without any post-filter.""" + from blosc2.ctable_indexing import _CTableIndexingMixin + + t, expected = or_table(tmp_path, f"or-{label}.b2t", spec, sentinel) + + used = [] + original = _CTableIndexingMixin._try_index_where + + def spy(self, expr): + result = original(self, expr) + used.append(result is not None) + return result + + monkeypatch.setattr(_CTableIndexingMixin, "_try_index_where", spy) + + got = len(t.where("(a > 4000) | (b > 6000)")) + assert got == expected + assert used == [True], "the OR should now be answered from the index" + + +def test_a_row_null_in_one_branch_still_matches_the_other(tmp_path): + """What a global null post-filter would have got wrong, at small scale.""" + rng = np.random.default_rng(6) + n = 2000 + a = [None if i % 31 == 0 else int(rng.integers(0, 1000)) for i in range(n)] + b = [int(rng.integers(0, 1000)) for _ in range(n)] + Row = dataclasses.make_dataclass( + "R", + [ + ("a", int, blosc2.field(blosc2.int64(null_storage="mask"))), + ("b", int, blosc2.field(blosc2.int64())), + ], + ) + t = blosc2.CTable(Row, expected_size=n, urlpath=str(tmp_path / "small.b2t"), mode="w") + t.extend(list(zip(a, b, strict=True))) + t.create_index("a", kind="full") + t.create_index("b", kind="full") + + expected = sum(((av is not None and av > 800) or bv < 200) for av, bv in zip(a, b, strict=True)) + assert len(t.where("(a > 800) | (b < 200)")) == expected + + +def test_a_pipe_inside_a_string_literal_is_not_an_or(): + """The OR test decides whether a nullable column keeps its index, so it + parses rather than searching for a character that string data may contain.""" + from blosc2.ctable_indexing import _expression_has_or + + assert _expression_has_or("(a > 1) | (b < 2)") + assert _expression_has_or("a > 1 or b < 2") + assert not _expression_has_or("name == 'a|b'") + assert not _expression_has_or("(a > 1) & (b < 2)") diff --git a/tests/ctable/test_null_channel.py b/tests/ctable/test_null_channel.py new file mode 100644 index 000000000..aba2a04b2 --- /dev/null +++ b/tests/ctable/test_null_channel.py @@ -0,0 +1,283 @@ +####################################################################### +# Copyright (c) 2019-present, Blosc Development Team +# All rights reserved. +# +# SPDX-License-Identifier: BSD-3-Clause +####################################################################### + +"""Tests for the NullChannel abstraction (``blosc2.ctable_nulls``). + +Phase 0 of the mask-based-nulls plan: the channel unifies how a column's +nullity is read, without changing any observable behavior. These tests pin +the classification and the shared sentinel helpers so later phases (which add +a ``"mask"`` kind) have something to diff against. +""" + +from __future__ import annotations + +import dataclasses +from dataclasses import dataclass + +import numpy as np +import pytest +from utf8_compat import needs_utf8, utf8_spec + +import blosc2 +from blosc2 import CTable +from blosc2.ctable_nulls import ( + NULL_CODE, + NULL_MASK, + NULL_NATIVE, + NULL_NONE, + NULL_SENTINEL, + is_nan_sentinel, + is_null_value, + kind_of_spec, + sentinel_mask, +) + +# --------------------------------------------------------------------------- +# kind classification +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize( + ("spec", "expected"), + [ + (blosc2.int64(), NULL_NONE), + (blosc2.int64(null_value=-1), NULL_SENTINEL), + (blosc2.float64(null_value=float("nan")), NULL_SENTINEL), + (blosc2.string(max_length=8), NULL_NONE), + (blosc2.string(max_length=8, null_value=""), NULL_SENTINEL), + (blosc2.bytes(max_length=8, null_value=b""), NULL_SENTINEL), + (blosc2.timestamp(null_value=-1), NULL_SENTINEL), + # bool(nullable=True) has no sentinel until the schema is compiled: + # _resolve_nullable_specs is what picks 255. See + # test_kind_of_spec_resolves_on_compile. + (blosc2.bool(nullable=True), NULL_NONE), + (blosc2.bool(nullable=True, null_value=255), NULL_SENTINEL), + (blosc2.bool(), NULL_NONE), + # utf8 is a variable-length kind but stores nulls as a sentinel string. + pytest.param(utf8_spec(), NULL_NONE, marks=needs_utf8), + pytest.param(utf8_spec(null_value="__NULL__"), NULL_SENTINEL, marks=needs_utf8), + # Dictionary and native-None kinds report a channel either way: their + # storage can represent a null regardless of the nullable flag. + (blosc2.dictionary(), NULL_CODE), + (blosc2.dictionary(nullable=True), NULL_CODE), + (blosc2.vlstring(), NULL_NATIVE), + (blosc2.vlbytes(), NULL_NATIVE), + ], +) +def test_kind_of_spec(spec, expected): + assert kind_of_spec(spec) == expected + + +def test_kind_of_spec_none(): + assert kind_of_spec(None) == NULL_NONE + + +def test_kind_of_spec_resolves_on_compile(): + """``nullable=True`` only picks a channel once the table compiles it. + + Bare, it resolves to a mask -- the default since 4.11.0 -- and only an + explicit request keeps the sentinel. Either way the spec alone cannot say: + it is the resolver that decides. + """ + + @dataclass + class MaskRow: + flag: bool = blosc2.field(blosc2.bool(nullable=True)) + + t = CTable(MaskRow) + t.append({"flag": True}) + assert t["flag"].null_storage == NULL_MASK + assert t["flag"].null_value is None + + @dataclass + class SentinelRow: + flag: bool = blosc2.field(blosc2.bool(nullable=True, null_storage="sentinel")) + + t = CTable(SentinelRow) + t.append({"flag": True}) + assert t["flag"].null_storage == NULL_SENTINEL + assert t["flag"].null_value == 255 + + +def test_null_storage_property_matches_spec(): + @dataclass + class Row: + plain: int = blosc2.field(blosc2.int64()) + nulled: int = blosc2.field(blosc2.int64(null_value=-1)) + tag: str = blosc2.field(blosc2.dictionary()) + note: str = blosc2.field(blosc2.vlstring()) + + t = CTable(Row) + t.append({"plain": 1, "nulled": 2, "tag": "a", "note": "n"}) + assert t["plain"].null_storage == NULL_NONE + assert t["nulled"].null_storage == NULL_SENTINEL + assert t["tag"].null_storage == NULL_CODE + assert t["note"].null_storage == NULL_NATIVE + + +def test_channel_reads_through_to_live_schema(): + """A cached channel must not snapshot the sentinel. + + ``_resolve_nullable_specs`` assigns ``spec.null_value`` in place, so a + channel that copied it at construction time would report the wrong kind. + """ + + @dataclass + class Row: + v: int = blosc2.field(blosc2.int64()) + + t = CTable(Row) + t.append({"v": 1}) + col = t["v"] + channel = col._nulls + assert channel.kind == NULL_NONE + + t._schema.columns_by_name["v"].spec.null_value = -1 + assert channel.kind == NULL_SENTINEL + assert channel.sentinel == -1 + + +# --------------------------------------------------------------------------- +# sentinel_mask +# --------------------------------------------------------------------------- + + +def test_sentinel_mask_plain_value(): + arr = np.array([1, -1, 3, -1]) + np.testing.assert_array_equal(sentinel_mask(arr, -1), [False, True, False, True]) + + +def test_sentinel_mask_none_is_all_false(): + arr = np.array([1, 2, 3]) + np.testing.assert_array_equal(sentinel_mask(arr, None), [False, False, False]) + + +def test_sentinel_mask_none_does_not_coerce_ragged_input(): + """A list column reaches here with ragged rows; asarray would raise.""" + ragged = [[1, 2], [3], []] + np.testing.assert_array_equal(sentinel_mask(ragged, None), [False, False, False]) + + +def test_sentinel_mask_nan_sentinel(): + arr = np.array([1.0, np.nan, 3.0]) + np.testing.assert_array_equal(sentinel_mask(arr, float("nan")), [False, True, False]) + + +def test_sentinel_mask_nan_sentinel_narrow_float(): + """A float32 NaN sentinel must still be recognized as NaN, not compared.""" + arr = np.array([1.0, np.nan, 3.0], dtype=np.float32) + np.testing.assert_array_equal(sentinel_mask(arr, np.float32("nan")), [False, True, False]) + + +def test_sentinel_mask_datetime_uses_nat(): + """Timestamp values arrive already decoded to NaT, not as the raw sentinel.""" + arr = np.array(["2020-01-01", "NaT", "2021-01-01"], dtype="datetime64[s]") + np.testing.assert_array_equal(sentinel_mask(arr, np.iinfo(np.int64).min), [False, True, False]) + + +def test_sentinel_mask_ndarray_needs_every_element(): + """An ndarray row is null only when the whole item is the sentinel.""" + arr = np.array([[0, 0], [0, 5], [9, 9]]) + np.testing.assert_array_equal(sentinel_mask(arr, 0, item_ndim=1), [True, False, False]) + + +def test_sentinel_mask_ndarray_promotes_bare_row(): + arr = np.array([0, 0]) + np.testing.assert_array_equal(sentinel_mask(arr, 0, item_ndim=1), [True]) + + +# --------------------------------------------------------------------------- +# scalar helpers +# --------------------------------------------------------------------------- + + +def test_is_nan_sentinel(): + assert is_nan_sentinel(float("nan")) + assert is_nan_sentinel(np.float32("nan")) + assert is_nan_sentinel(np.float64("nan")) + assert not is_nan_sentinel(0.0) + assert not is_nan_sentinel(None) + assert not is_nan_sentinel("nan") + + +def test_is_null_value(): + assert is_null_value(-1, -1) + assert not is_null_value(0, -1) + assert is_null_value(float("nan"), float("nan")) + assert not is_null_value(1.0, float("nan")) + # No sentinel means no in-band null; native None cells are another kind. + assert not is_null_value(None, None) + + +# --------------------------------------------------------------------------- +# channel reads agree with the public Column API +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize( + ("spec", "annotation", "values", "null"), + [ + (blosc2.int64(null_value=-1), int, [1, 2, 3], -1), + (blosc2.float64(null_value=float("nan")), float, [1.0, 2.0, 3.0], float("nan")), + (blosc2.string(max_length=8, null_value=""), str, ["a", "b", "c"], ""), + pytest.param(utf8_spec(null_value="__NULL__"), str, ["a", "b", "c"], "__NULL__", marks=needs_utf8), + ], +) +def test_channel_agrees_with_column_api(spec, annotation, values, null): + Row = dataclasses.make_dataclass("Row", [("v", annotation, blosc2.field(spec))]) + + t = CTable(Row) + t.extend({"v": [*values, null]}) + col = t["v"] + + np.testing.assert_array_equal(col._nulls.null_mask(), col.is_null()) + assert col._nulls.null_count() == col.null_count() == 1 + assert col._nulls.is_nullable + assert col._nulls.sentinel == null or is_nan_sentinel(null) + + nonnull = np.concatenate(list(col._nulls.nonnull_chunks())) + assert len(nonnull) == len(values) + + +def test_null_pred_is_none_for_non_nullable(): + @dataclass + class Row: + v: int = blosc2.field(blosc2.int64()) + + t = CTable(Row) + t.extend({"v": [1, 2, 3]}) + assert t["v"]._nulls.null_pred() is None + assert t["v"]._nulls.valid_pred() is None + + +def test_null_pred_is_none_for_ndarray_column(): + """Per-item sentinel masks do not align 1:1 with row-level predicates.""" + + @dataclass + class Row: + v: object = blosc2.field(blosc2.ndarray((2,), dtype=blosc2.int64(), null_value=-1)) + + t = CTable(Row) + t.extend({"v": np.array([[1, 2], [-1, -1]])}) + col = t["v"] + assert col._nulls.null_pred() is None + # is_null() still works: it reduces per item. + np.testing.assert_array_equal(col.is_null(), [False, True]) + + +def test_null_pred_matches_is_null_for_sentinel_column(): + @dataclass + class Row: + v: int = blosc2.field(blosc2.int64(null_value=-1)) + + t = CTable(Row) + t.extend({"v": [1, -1, 3]}) + col = t["v"] + pred = np.asarray(col._nulls.null_pred().compute()[:]) + np.testing.assert_array_equal(pred[: t.nrows], col.is_null()) + valid = np.asarray(col._nulls.valid_pred().compute()[:]) + np.testing.assert_array_equal(valid[: t.nrows], col.notnull()) diff --git a/tests/ctable/test_null_expressions.py b/tests/ctable/test_null_expressions.py index a97ce22a7..29ab6b88c 100644 --- a/tests/ctable/test_null_expressions.py +++ b/tests/ctable/test_null_expressions.py @@ -39,7 +39,7 @@ class TsRow: @dataclass class FloatRow: id: int = blosc2.field(blosc2.int64()) - f: float = blosc2.field(blosc2.float64(nullable=True)) + f: float = blosc2.field(blosc2.float64(nullable=True, null_storage="sentinel")) # =========================================================================== diff --git a/tests/ctable/test_null_mask_api.py b/tests/ctable/test_null_mask_api.py new file mode 100644 index 000000000..836f6379b --- /dev/null +++ b/tests/ctable/test_null_mask_api.py @@ -0,0 +1,792 @@ +####################################################################### +# Copyright (c) 2019-present, Blosc Development Team +# All rights reserved. +# +# SPDX-License-Identifier: BSD-3-Clause +####################################################################### + +"""Reading and writing mask-storage nullable columns (Phase 4). + +Phase 3 gave the sidecar a place to live; this is where it becomes usable. +``None`` is now the canonical way to write a null into a fixed-width scalar +column -- which could not accept one at all before -- and the null API reads +validity from the sidecar rather than inferring it from the values. + +Two consequences are load-bearing and tested here: + +* **NaN is a value, not a null** (decision 6). A mask-backed float column + follows Arrow: only ``mask=False`` is missing. This is the whole point of + moving nullity to a side channel, and it is where mask and sentinel columns + deliberately diverge. +* **``fillna`` becomes correct when the fill collides with real data**, which + is impossible to get right under a sentinel. +""" + +from __future__ import annotations + +import dataclasses + +import numpy as np +import pytest +from utf8_compat import needs_utf8, utf8_spec + +import blosc2 + + +def annotation_for(spec): + """An annotation the schema compiler accepts for *spec*. + + ``validate_annotation_matches_spec`` keys off ``spec.python_type`` for + everything except the two kinds that take ``object``. + """ + if isinstance(spec, (blosc2.schema.NDArraySpec, blosc2.schema.timestamp)): + return object + return spec.python_type + + +def row_type(**cols): + """A dataclass from ``name=spec`` pairs, annotations derived from the specs.""" + return dataclasses.make_dataclass( + "MaskRow", [(n, annotation_for(spec), blosc2.field(spec)) for n, spec in cols.items()] + ) + + +def table(rows, capacity=32, **cols): + t = blosc2.CTable(row_type(**cols), expected_size=capacity) + if rows: + t.extend(rows) + return t + + +def simple(values, spec=None, capacity=32): + """A one-column table of *values*, nulls written as ``None``.""" + spec = blosc2.int64(null_storage="mask") if spec is None else spec + return table([(v,) for v in values], capacity=capacity, a=spec) + + +# --------------------------------------------------------------------------- +# None is accepted, and nothing else is mistaken for it +# --------------------------------------------------------------------------- + +WRITEABLE_SPECS = [ + ("int64", blosc2.int64(null_storage="mask"), 7), + ("int8", blosc2.int8(null_storage="mask"), -128), + ("uint8", blosc2.uint8(null_storage="mask"), 255), + ("float64", blosc2.float64(null_storage="mask"), 1.5), + ("complex128", blosc2.complex128(null_storage="mask"), 1 + 2j), + ("bool", blosc2.bool(null_storage="mask"), True), + ("string", blosc2.string(max_length=4, null_storage="mask"), "abcd"), + ("bytes", blosc2.bytes(max_length=4, null_storage="mask"), b"abcd"), + pytest.param("utf8", utf8_spec(null_storage="mask"), "hello", marks=needs_utf8), +] + + +@pytest.mark.parametrize(("label", "spec", "value"), WRITEABLE_SPECS) +def test_extend_accepts_none(label, spec, value): + t = simple([value, None, value], spec=spec) + assert t["a"].is_null().tolist() == [False, True, False] + assert t["a"].null_count() == 1 + + +@pytest.mark.parametrize(("label", "spec", "value"), WRITEABLE_SPECS) +def test_append_accepts_none(label, spec, value): + t = simple([value], spec=spec) + t.append((None,)) + assert t["a"].is_null().tolist() == [False, True] + + +@pytest.mark.parametrize(("label", "spec", "value"), WRITEABLE_SPECS) +def test_setitem_accepts_none(label, spec, value): + t = simple([value, value], spec=spec) + t["a"][1] = None + assert t["a"].is_null().tolist() == [False, True] + + +@pytest.mark.parametrize(("label", "spec", "value"), WRITEABLE_SPECS) +def test_assign_accepts_none(label, spec, value): + t = simple([value, value], spec=spec) + t["a"].assign([None, value]) + assert t["a"].is_null().tolist() == [True, False] + + +def test_nullable_bool_is_a_real_bool(): + """The case that motivated the whole design: no uint8, no reserved 255.""" + t = simple([True, None, False], spec=blosc2.bool(null_storage="mask")) + assert t["a"].dtype == np.dtype(np.bool_) + assert t["a"][:].tolist() == [True, False, False] + assert t["a"].is_null().tolist() == [False, True, False] + + +def test_int8_keeps_its_full_range_alongside_nulls(): + values = list(range(-128, 128)) + t = simple([*values, None], spec=blosc2.int8(null_storage="mask"), capacity=300) + assert t["a"].dtype == np.dtype(np.int8) + assert t["a"][:-1].tolist() == values + assert t["a"].is_null()[-1] + assert t["a"].null_count() == 1 + + +def test_string_keeps_its_declared_width(): + """No sentinel means no max_length widening to fit one.""" + t = simple(["abcd", None], spec=blosc2.string(max_length=4, null_storage="mask")) + assert t["a"].dtype == np.dtype("U4") + + +@needs_utf8 +def test_utf8_accepts_text_no_sentinel_could_survive(): + tricky = ["", "\x00", "__BLOSC2_NULL__", "🎉x"] + t = simple([*tricky, None], spec=blosc2.utf8(null_storage="mask")) + assert list(t["a"][:-1]) == tricky + assert t["a"].is_null().tolist() == [False] * 4 + [True] + + +def test_timestamp_accepts_none(): + spec = blosc2.timestamp(null_storage="mask") + t = simple(["2020-01-01", None, "2020-01-03"], spec=spec) + assert t["a"].is_null().tolist() == [False, True, False] + assert np.isnat(t["a"][:][1]) + + +def test_ndarray_column_accepts_none(): + spec = blosc2.ndarray((3,), dtype=blosc2.float32(), null_storage="mask") + item = np.ones(3, dtype=np.float32) + t = simple([item, None, item * 2], spec=spec) + assert t["a"].is_null().tolist() == [False, True, False] + assert t["a"][:][0].tolist() == [1.0, 1.0, 1.0] + + +# --------------------------------------------------------------------------- +# Decision 6: NaN is a value +# --------------------------------------------------------------------------- + + +def test_nan_is_a_value_not_a_null(): + t = simple([1.0, float("nan"), None], spec=blosc2.float64(null_storage="mask")) + assert t["a"].is_null().tolist() == [False, False, True] + assert t["a"].null_count() == 1 + assert np.isnan(t["a"][:][1]) + + +def test_sentinel_float_still_treats_nan_as_null(): + """The two storages diverge here on purpose, and both are documented.""" + t = simple([1.0, float("nan")], spec=blosc2.float64(null_value=float("nan"))) + assert t["a"].is_null().tolist() == [False, True] + + +def test_signed_zero_and_inf_are_values(): + values = [0.0, -0.0, float("inf"), float("-inf")] + t = simple([*values, None], spec=blosc2.float64(null_storage="mask")) + assert t["a"].is_null().tolist() == [False] * 4 + [True] + assert np.array_equal(t["a"][:-1], np.array(values), equal_nan=True) + + +def test_int64_min_is_a_value_in_a_timestamp_column(): + spec = blosc2.timestamp(null_storage="mask") + t = simple([np.datetime64(np.iinfo(np.int64).min + 1, "us"), None], spec=spec) + assert t["a"].is_null().tolist() == [False, True] + + +# --------------------------------------------------------------------------- +# Input forms that carry their own validity +# --------------------------------------------------------------------------- + + +def test_masked_array_input_supplies_validity_verbatim(): + spec = blosc2.float64(null_storage="mask") + data = np.ma.MaskedArray([1.0, 2.0, np.nan], mask=[False, True, False]) + t = blosc2.CTable(row_type(a=spec), expected_size=8) + t.extend({"a": data}) + # The NaN is *not* masked, so it stays a value; only index 1 is null. + assert t["a"].is_null().tolist() == [False, True, False] + + +def test_numpy_float_array_input_has_no_nulls(): + spec = blosc2.float64(null_storage="mask") + t = blosc2.CTable(row_type(a=spec), expected_size=8) + t.extend({"a": np.array([1.0, np.nan, 3.0])}) + assert t["a"].null_count() == 0 + assert t["a"].is_null().tolist() == [False, False, False] + + +def test_nat_reads_as_null_in_a_timestamp_column(): + # The NaT carries a unit: NumPy 2.5 deprecated the *generic* one, so a bare + # np.datetime64("NaT") warns in the caller. Taking the unit from the spec + # keeps this pinned to the column rather than to a literal. + spec = blosc2.timestamp(null_storage="mask") + t = simple([np.datetime64("2020-01-01"), np.datetime64("NaT", spec.unit)], spec=spec) + assert t["a"].is_null().tolist() == [False, True] + + +def test_a_nat_of_any_unit_reads_as_null(): + """Detection is np.isnat, so it does not care which unit the caller used.""" + spec = blosc2.timestamp(null_storage="mask") + for unit in ("s", "ms", "us", "ns"): + t = simple([np.datetime64("2020-01-01"), np.datetime64("NaT", unit)], spec=spec) + assert t["a"].is_null().tolist() == [False, True], unit + + +# --------------------------------------------------------------------------- +# Decision 9 through the public API +# --------------------------------------------------------------------------- + + +def test_a_null_free_batch_writes_no_sidecar(): + t = simple([1, 2, 3]) + assert t._null_mask("a") is None + assert t["a"].null_count() == 0 + assert t["a"].is_null().tolist() == [False, False, False] + + +def test_the_sidecar_appears_on_the_first_null(): + t = simple([1, 2, 3]) + assert t._null_mask("a") is None + t.append((None,)) + assert t._null_mask("a") is not None + + +def test_marking_a_row_valid_does_not_create_a_sidecar(): + t = simple([1, 2, 3]) + t["a"][0] = 5 + assert t._null_mask("a") is None + + +def test_info_reports_storage_and_whether_a_sidecar_exists(): + t = simple([1, 2, 3]) + items = dict(t["a"].info_items) + assert items["nullable"] is True + assert items["null_storage"] == "mask" + assert items["null_sidecar"] is False + t.append((None,)) + assert dict(t["a"].info_items)["null_sidecar"] is True + + +def test_non_nullable_column_reports_no_storage(): + t = table([(1,)], a=blosc2.int64()) + assert "null_storage" not in dict(t["a"].info_items) + + +# --------------------------------------------------------------------------- +# Overwrite semantics: a write replaces validity, it does not merge it +# --------------------------------------------------------------------------- + + +def test_assign_clears_previously_written_nulls(): + t = simple([1, None, 3]) + t["a"].assign([1, 2, 3]) + assert t["a"].null_count() == 0 + assert t["a"][:].tolist() == [1, 2, 3] + + +def test_setitem_clears_a_null(): + t = simple([1, None, 3]) + t["a"][1] = 2 + assert t["a"].null_count() == 0 + assert t["a"][:].tolist() == [1, 2, 3] + + +def test_setitem_slice_replaces_validity(): + t = simple([1, None, 3, None]) + t["a"][0:3] = [None, 2, 3] + assert t["a"].is_null().tolist() == [True, False, False, True] + + +def test_setitem_boolean_mask_replaces_validity(): + t = simple([1, None, 3, 4]) + t["a"][np.array([True, True, False, False])] = [None, 2] + assert t["a"].is_null().tolist() == [True, False, False, False] + + +def test_setitem_index_list_replaces_validity(): + t = simple([1, None, 3, 4]) + t["a"][[1, 3]] = [2, None] + assert t["a"].is_null().tolist() == [False, False, False, True] + + +# --------------------------------------------------------------------------- +# The null API +# --------------------------------------------------------------------------- + + +def test_notnull_is_the_complement_of_is_null(): + t = simple([1, None, 3]) + assert (t["a"].notnull() == ~t["a"].is_null()).all() + + +def test_fillna_is_correct_when_the_fill_collides_with_real_data(): + """Impossible under a sentinel: there, 7 *is* how a null would look.""" + t = simple([7, None, 3]) + assert t["a"].fillna(7).tolist() == [7, 7, 3] + # ...and the column itself is unchanged, still one null. + assert t["a"].null_count() == 1 + + +def test_to_numpy_masked_marks_only_the_nulls(): + t = simple([1.0, float("nan"), None], spec=blosc2.float64(null_storage="mask")) + out = t["a"].to_numpy(masked=True) + assert isinstance(out, np.ma.MaskedArray) + assert out.mask.tolist() == [False, False, True] + + +def test_to_numpy_masked_works_for_sentinel_columns_too(): + """One uniform way to ask for values-plus-validity, whatever the storage.""" + t = simple([1, -1, 3], spec=blosc2.int64(null_value=-1)) + out = t["a"].to_numpy(masked=True) + assert out.mask.tolist() == [False, True, False] + + +def test_to_numpy_without_masked_is_a_plain_array(): + t = simple([1, None, 3]) + assert isinstance(t["a"].to_numpy(), np.ndarray) + assert not isinstance(t["a"].to_numpy(), np.ma.MaskedArray) + + +def test_unique_and_value_counts_skip_nulls(): + t = simple([1, None, 3, 1]) + assert t["a"].unique().tolist() == [1, 3] + assert t["a"].value_counts() == {1: 2, 3: 1} + + +def test_dropna_uses_the_sidecar(): + t = table( + [(1, 10), (None, 20), (3, 30)], + a=blosc2.int64(null_storage="mask"), + b=blosc2.int64(), + ) + assert t.dropna()["b"][:].tolist() == [10, 30] + + +def test_reductions_skip_nulls(): + t = simple([2.0, None, 4.0], spec=blosc2.float64(null_storage="mask")) + assert t["a"].sum() == 6.0 + assert t["a"].mean() == 3.0 + + +def test_null_count_with_deletions(): + t = simple([None, 1, None, 3]) + t.delete(0) + assert t["a"].null_count() == 1 + assert t["a"].is_null().tolist() == [False, True, False] + + +# --------------------------------------------------------------------------- +# Views keep their nulls +# --------------------------------------------------------------------------- + + +def two_col_table(): + return table( + [(None if i in (1, 4) else i, 10 - i) for i in range(6)], + a=blosc2.int64(null_storage="mask"), + b=blosc2.int64(), + ) + + +def test_where_view_remaps_nulls(): + t = two_col_table() + assert t.where("b < 8")["a"].is_null().tolist() == [False, True, False] + + +def test_slice_view_remaps_nulls(): + assert two_col_table()[1:5]["a"].is_null().tolist() == [True, False, False, True] + + +def test_reversed_slice_view_remaps_nulls(): + assert two_col_table()[::-1]["a"].is_null().tolist() == [False, True, False, False, True, False] + + +@pytest.mark.parametrize("view", [True, False]) +def test_sort_by_keeps_nulls_with_their_rows(view): + sorted_t = two_col_table().sort_by("b", view=view) + assert sorted_t["a"].is_null().tolist() == [False, True, False, False, True, False] + assert sorted_t["b"][:].tolist() == [5, 6, 7, 8, 9, 10] + + +def test_sort_by_inplace_keeps_nulls_with_their_rows(): + t = two_col_table() + t.sort_by("b", inplace=True) + assert t["a"].is_null().tolist() == [False, True, False, False, True, False] + + +def test_sorting_a_view_keeps_nulls_with_their_rows(): + t = two_col_table() + sorted_view = t.where("b > 5").sort_by("b") + assert sorted_view["a"].is_null().tolist() == [True, False, False, True, False] + + +def test_take_keeps_nulls_with_their_rows(): + assert two_col_table().take([4, 1, 0])["a"].is_null().tolist() == [True, True, False] + + +def test_slice_copy_keeps_nulls_with_their_rows(): + assert two_col_table().slice(1, 5)["a"].is_null().tolist() == [True, False, False, True] + + +@pytest.mark.parametrize("compact", [True, False]) +def test_copy_keeps_nulls(compact): + assert two_col_table().copy(compact=compact)["a"].null_count() == 2 + + +def test_compact_keeps_nulls_with_their_rows(): + t = two_col_table() + t.delete(0) + t.compact() + assert t["a"].is_null().tolist() == [True, False, False, True, False] + + +def test_a_view_of_a_null_free_column_stays_sidecar_free(): + t = simple([1, 2, 3, 4]) + assert t.take([0, 2])._null_mask("a") is None + + +def test_gathering_only_valid_rows_needs_no_sidecar(): + """Decision 9 again: the copy has no nulls, so it gets no bytes.""" + t = two_col_table() + assert t.take([0, 2, 3])._null_mask("a") is None + + +# --------------------------------------------------------------------------- +# Persistence of nulls written through the API +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize("suffix", [".b2d", ".b2z"]) +def test_nulls_survive_save_and_reopen(tmp_path, suffix): + path = str(tmp_path / f"t{suffix}") + two_col_table().save(path) + reopened = blosc2.CTable.open(path) + try: + assert reopened["a"].is_null().tolist() == [False, True, False, False, True, False] + assert reopened["a"].null_count() == 2 + finally: + reopened.close() + + +def test_nulls_survive_a_cframe_round_trip(): + rebuilt = blosc2.ctable_from_cframe(two_col_table().to_cframe()) + assert rebuilt["a"].is_null().tolist() == [False, True, False, False, True, False] + + +def test_appending_to_a_reopened_table_keeps_earlier_nulls(tmp_path): + path = str(tmp_path / "grow.b2d") + two_col_table().save(path) + reopened = blosc2.CTable.open(path, mode="a") + try: + reopened.append((None, 99)) + assert reopened["a"].is_null().tolist() == [ + False, True, False, False, True, False, True, + ] # fmt: skip + finally: + reopened.close() + + +# --------------------------------------------------------------------------- +# Sentinel columns are untouched +# --------------------------------------------------------------------------- + + +def test_sentinel_columns_keep_their_behaviour(): + t = simple([1, -1, 3], spec=blosc2.int64(null_value=-1)) + assert t["a"].null_storage == "sentinel" + assert t["a"].is_null().tolist() == [False, True, False] + assert t._null_mask("a") is None + + +def test_sentinel_bool_is_still_uint8_backed(): + t = simple([True, 255, False], spec=blosc2.bool(nullable=True, null_value=255)) + assert t["a"].dtype == np.dtype(np.uint8) + assert t["a"].is_null().tolist() == [False, True, False] + + +def test_channel_is_not_cached_on_its_column(): + """A cached channel would close a Column-CTable reference cycle.""" + col = simple([1, 2])["a"] + assert col._nulls is not col._nulls + assert col._nulls.kind == "mask" + + +def test_a_table_is_freed_without_a_gc_pass(): + """The regression the no-caching rule above exists to prevent.""" + import gc + import weakref + + def build(): + return weakref.ref(simple([1, None, 3])) + + gc.disable() + try: + ref = build() + assert ref() is None + finally: + gc.enable() + + +# --------------------------------------------------------------------------- +# Extending from another CTable +# --------------------------------------------------------------------------- +# +# A CTable source is the one input shape whose nulls do not travel with its +# values: under mask storage nullity lives in a sidecar, so copying the raw +# column alone turns every null into its fill -- a plausible-looking 0 or "", +# not an error. The sentinel and list-of-rows forms below are the controls +# that say what the answer has to be. + + +def test_extend_from_a_table_carries_the_nulls_over(): + src = simple([1, None, 3]) + dst = simple([]) + dst.extend(src) + assert dst["a"][:].tolist() == [1, 0, 3] + assert dst["a"].is_null().tolist() == [False, True, False] + assert dst["a"].null_count() == 1 + + +def test_extend_from_a_table_agrees_with_the_other_input_shapes(): + """The same rows, spelled three ways, must land identically.""" + from_table = simple([]) + from_table.extend(simple([1, None, 3])) + + from_rows = simple([]) + from_rows.extend([(1,), (None,), (3,)]) + + sentinel = table([], a=blosc2.int64(nullable=True, null_value=-9)) + sentinel.extend(table([(1,), (-9,), (3,)], a=blosc2.int64(nullable=True, null_value=-9))) + + assert from_table["a"].is_null().tolist() == from_rows["a"].is_null().tolist() + assert from_table["a"].is_null().tolist() == sentinel["a"].is_null().tolist() + + +@needs_utf8 +def test_extend_from_a_table_carries_utf8_nulls(): + """The fill is "" here, which is also a legal value -- so it has to be the sidecar.""" + src = table([("x",), (None,), ("",)], s=utf8_spec(null_storage="mask")) + dst = table([], s=utf8_spec(null_storage="mask")) + dst.extend(src) + assert list(dst["s"][:]) == ["x", "", ""] + assert dst["s"].is_null().tolist() == [False, True, False] + + +def test_extend_from_a_table_with_deleted_rows_copies_the_live_ones(): + """Live rows scatter through the physical extent, so a leading slice is the wrong rows.""" + src = simple([10, 11, 12, 13]) + src.delete(0) + dst = simple([]) + dst.extend(src) + assert dst["a"][:].tolist() == [11, 12, 13] + assert dst.nrows == 3 + + +def test_extend_from_a_table_with_holes_keeps_values_and_nulls_aligned(): + src = simple([10, None, 12, None, 14]) + src.delete(0) + dst = simple([]) + dst.extend(src) + assert dst["a"].is_null().tolist() == src["a"].is_null().tolist() + assert dst["a"][:].tolist() == src["a"][:].tolist() + + +def test_extend_from_a_sorted_view_copies_it_in_sorted_order(): + src = simple([3, None, 1, 2]) + dst = simple([]) + dst.extend(src.sort_by("a", view=True)) + # Nulls sort last, in both directions, and the copy has to agree. + assert dst["a"][:].tolist() == [1, 2, 3, 0] + assert dst["a"].is_null().tolist() == [False, False, False, True] + + +def test_extend_from_a_null_free_table_writes_no_sidecar(): + """Decision 9: an absent sidecar is the common state and must stay absent.""" + dst = simple([]) + dst.extend(simple([1, 2, 3])) + assert dst["a"].is_null().tolist() == [False, False, False] + assert dst._null_mask("a") is None + + +# --------------------------------------------------------------------------- +# add_column +# --------------------------------------------------------------------------- +# +# A mask column is the only kind that can say "this row has no value" without +# reserving one, so it is the only way to add a column to a populated table +# and be honest about the rows that predate it. A sentinel column has to +# spell that with its sentinel, and says so. + + +def test_add_column_values_accept_none(): + t = simple([1, 2, 3], spec=blosc2.int64()) + t.add_column("b", blosc2.int64(nullable=True), values=[10, None, 30]) + assert t["b"][:].tolist() == [10, 0, 30] + assert t["b"].is_null().tolist() == [False, True, False] + assert t["b"].null_count() == 1 + + +def test_add_column_default_none_backfills_nulls(): + """The rows that predate the column have no value, and now can say so.""" + t = simple([1, 2, 3], spec=blosc2.int64()) + t.add_column("b", blosc2.field(blosc2.int64(nullable=True), default=None)) + assert t["b"].is_null().tolist() == [True, True, True] + + +def test_add_column_default_none_still_applies_to_later_rows(): + t = simple([1, 2], spec=blosc2.int64()) + t.add_column("b", blosc2.field(blosc2.int64(nullable=True), default=None)) + t.append((3, 9)) + t.extend([(4, None)]) + assert t["b"][:].tolist() == [0, 0, 9, 0] + assert t["b"].is_null().tolist() == [True, True, False, True] + + +def test_add_column_without_a_null_writes_no_sidecar(): + """Decision 9 again: the lazy sidecar must stay lazy here too.""" + t = simple([1, 2, 3], spec=blosc2.int64()) + t.add_column("b", blosc2.int64(nullable=True), values=[10, 20, 30]) + assert t["b"].is_null().tolist() == [False, False, False] + assert t._null_mask("b") is None + + +def test_add_column_scatters_nulls_past_deleted_rows(): + """values= is one entry per *live* row, so validity has to scatter with them.""" + t = simple([0, 1, 2, 3], spec=blosc2.int64()) + t.delete(1) + t.add_column("b", blosc2.int64(nullable=True), values=[10, None, 30]) + assert t["b"].is_null().tolist() == [False, True, False] + t.compact() + assert t["b"].is_null().tolist() == [False, True, False] + assert t["b"][:].tolist() == [10, 0, 30] + + +@needs_utf8 +def test_add_column_utf8_values_accept_none(): + """The fill is "" here, which a genuine row may also hold.""" + t = simple([1, 2, 3], spec=blosc2.int64()) + t.add_column("s", utf8_spec(null_storage="mask"), values=["p", None, ""]) + assert list(t["s"][:]) == ["p", "", ""] + assert t["s"].is_null().tolist() == [False, True, False] + + +def test_add_column_ndarray_values_accept_none(): + t = simple([1, 2], spec=blosc2.int64()) + t.add_column( + "v", + blosc2.ndarray((3,), dtype=blosc2.int64(), nullable=True), + values=[np.array([1, 2, 3]), None], + ) + assert t["v"][:].tolist() == [[1, 2, 3], [0, 0, 0]] + assert t["v"].is_null().tolist() == [False, True] + + +def test_add_column_nulls_survive_a_reopen(tmp_path): + t = simple([1, 2, 3], spec=blosc2.int64()) + urlpath = str(tmp_path / "added.b2t") + t.save(urlpath) + live = blosc2.CTable.open(urlpath, mode="a") + try: + live.add_column("b", blosc2.int64(nullable=True), values=[10, None, 30]) + finally: + live.close() + reopened = blosc2.CTable.open(urlpath) + try: + assert reopened["b"].null_storage == "mask" + assert reopened["b"].is_null().tolist() == [False, True, False] + finally: + reopened.close() + + +@pytest.mark.parametrize( + ("spec", "match"), + [ + (blosc2.int64(), "nullable"), + (blosc2.int64(nullable=True, null_value=-1), r"null_value \(-1\)"), + ], +) +def test_add_column_says_why_a_null_will_not_fit(spec, match): + """Not NumPy's "int() argument must be ...", which names neither the column nor the way out.""" + t = simple([1, 2, 3], spec=blosc2.int64()) + with pytest.raises(TypeError, match=match): + t.add_column("b", spec, values=[10, None, 30]) + + +# A timestamp column stores int64 in the spec's unit, so add_column has to +# convert through that unit exactly as extend does. It did not, which put a +# datetime64[s] value in a microsecond column and read it back as 1970. + + +@pytest.mark.parametrize("nullable", [False, True]) +def test_add_column_timestamps_keep_their_unit(nullable): + when = np.datetime64("2020-01-01T00:00:00", "s") + t = simple([1, 2], spec=blosc2.int64()) + t.add_column("ts", blosc2.timestamp(nullable=nullable), values=[when, when]) + assert t["ts"][:].tolist() == [when.astype("datetime64[us]").item()] * 2 + + +def test_add_column_timestamp_default_keeps_its_unit(): + when = np.datetime64("2020-01-01T00:00:00", "s") + t = simple([1, 2], spec=blosc2.int64()) + t.add_column("ts", blosc2.field(blosc2.timestamp(), default=when)) + assert t["ts"][:].tolist() == [when.astype("datetime64[us]").item()] * 2 + + +def test_add_column_timestamp_null_reads_as_nat(): + when = np.datetime64("2020-01-01T00:00:00", "s") + t = simple([1, 2], spec=blosc2.int64()) + t.add_column("ts", blosc2.timestamp(nullable=True), values=[when, None]) + assert t["ts"].is_null().tolist() == [False, True] + assert np.isnat(t["ts"][:][1]) + + +# --------------------------------------------------------------------------- +# isin +# --------------------------------------------------------------------------- +# +# Membership is asked of the row's *value*, and a null row has none. Testing +# the raw values instead matched whatever stands in for a null -- the fill +# here, a sentinel elsewhere -- neither of which the column ever means +# literally. Cross-storage agreement is pinned in +# test_null_storage_equivalence.py; these are the mask-only corners. + + +def test_isin_does_not_match_the_fill(): + t = simple([1, None, 0]) + # Rows 1 and 2 both hold a physical 0; only row 2 holds it as a value. + assert t["a"][:].tolist() == [1, 0, 0] + assert t["a"].isin([0]).tolist() == [False, False, True] + + +def test_isin_none_selects_the_nulls(): + t = simple([1, None, 3, None]) + assert t["a"].isin([None]).tolist() == t["a"].is_null().tolist() + + +def test_isin_none_can_be_combined_with_values(): + t = simple([1, None, 3]) + assert t["a"].isin([3, None]).tolist() == [False, True, True] + + +def test_isin_pandas_na_spells_the_same_request(): + pd = pytest.importorskip("pandas") + t = simple([1, None, 3]) + assert t["a"].isin([pd.NA]).tolist() == [False, True, False] + + +@needs_utf8 +def test_isin_does_not_match_the_empty_string_fill(): + """The utf8 fill is "", which a genuine row may hold -- so it has to be the sidecar.""" + t = table([("x",), (None,), ("",)], s=utf8_spec(null_storage="mask")) + assert list(t["s"][:]) == ["x", "", ""] + assert t["s"].isin([""]).tolist() == [False, False, True] + + +def test_isin_on_a_view_uses_the_views_rows(): + t = simple([1, None, 3, None, 5]) + assert t.sort_by("a", view=True)["a"].isin([None]).tolist() == [False, False, False, True, True] + assert t.take([1, 2])["a"].isin([None]).tolist() == [True, False] + + +def test_isin_keeps_nan_a_value(): + """Decision 6: only mask=False is missing, so a NaN row is not a null row.""" + t = simple([1.0, float("nan"), None], spec=blosc2.float64(null_storage="mask")) + assert t["a"].is_null().tolist() == [False, False, True] + assert t["a"].isin([None]).tolist() == [False, False, True] + + +def test_isin_on_an_empty_column(): + assert simple([])["a"].isin([1, None]).tolist() == [] diff --git a/tests/ctable/test_null_mask_arrow.py b/tests/ctable/test_null_mask_arrow.py new file mode 100644 index 000000000..8bd22b7cc --- /dev/null +++ b/tests/ctable/test_null_mask_arrow.py @@ -0,0 +1,450 @@ +####################################################################### +# Copyright (c) 2019-present, Blosc Development Team +# All rights reserved. +# +# SPDX-License-Identifier: BSD-3-Clause +####################################################################### + +"""Lossless Arrow/Parquet interop for mask-storage columns (Phase 6). + +This is what the whole design is for. A sentinel steals a value from the +dtype's range, so importing Arrow data has always been lossy for any column +whose data happens to use that value — silently lossy, since nothing raises. +``test_sentinel_storage_is_lossy_where_mask_storage_is_not`` measures exactly +that, and is the reason the rest of this file exists. + +Phase 6 ships **opt-in**: ``null_storage="mask"`` on ``from_arrow`` / +``from_parquet``, or a ``NullPolicy``. The default stays ``"sentinel"`` until +Phase 9, so every existing caller is unaffected. + +On comparing round-trips: ``pyarrow.Array.equals`` uses IEEE semantics, so two +*identical* arrays containing NaN compare unequal, and the plan's literal +``to_arrow(from_arrow(x)).equals(x)`` contract is unachievable for float data +by construction. :func:`assert_same_logical` compares what is actually +observable — the validity bitmap, and the values under valid rows — treating +NaN as equal to NaN and keeping signed zeros distinct. +""" + +from __future__ import annotations + +import numpy as np +import pytest +from utf8_compat import needs_utf8, utf8_spec + +import blosc2 + +pa = pytest.importorskip("pyarrow") +pq = pytest.importorskip("pyarrow.parquet") + + +# --------------------------------------------------------------------------- +# Comparison helper +# --------------------------------------------------------------------------- + + +def _flat(column): + """A plain Array, whether the input was one already or a ChunkedArray.""" + return column.combine_chunks() if hasattr(column, "combine_chunks") else column + + +def assert_same_logical(got, want): + """Assert two Arrow columns hold the same observable data. + + Values sitting under ``valid=False`` are deliberately *not* compared: they + are the fill, which is explicitly not part of the format contract. + """ + got, want = _flat(got), _flat(want) + assert len(got) == len(want), "length" + assert got.null_count == want.null_count, "null_count" + + got_valid = got.is_valid().to_numpy(zero_copy_only=False) + want_valid = want.is_valid().to_numpy(zero_copy_only=False) + assert np.array_equal(got_valid, want_valid), "validity bitmap" + + for i in range(len(got)): + if not want_valid[i]: + continue + g, w = got[i], want[i] + if pa.types.is_timestamp(g.type): + g, w = g.cast(pa.int64()), w.cast(pa.int64()) + g, w = g.as_py(), w.as_py() + if isinstance(g, float) and isinstance(w, float): + assert g == w or (np.isnan(g) and np.isnan(w)), f"row {i}: {g!r} != {w!r}" + if g == 0.0: + assert np.copysign(1, g) == np.copysign(1, w), f"row {i}: signed zero" + else: + assert g == w, f"row {i}: {g!r} != {w!r}" + + +def round_trip(arrow_array, **kwargs): + """Import one Arrow column into a mask-backed CTable and export it again.""" + table = pa.table({"v": arrow_array}) + ct = blosc2.CTable.from_arrow(table, null_storage="mask", **kwargs) + return ct, ct.to_arrow().column("v") + + +# --------------------------------------------------------------------------- +# The round-trip contract: none of these survive under a sentinel +# --------------------------------------------------------------------------- + +ROUND_TRIP_CASES = [ + ("bool", pa.array([True, None, False, True], type=pa.bool_())), + ("int8_full_range", pa.array([*range(-128, 128), None], type=pa.int8())), + ("uint8_full_range", pa.array([*range(256), None], type=pa.uint8())), + ( + "float64_specials", + pa.array( + [float("nan"), None, 0.0, -0.0, float("inf"), float("-inf")], + type=pa.float64(), + ), + ), + ( + "utf8_free_text", + pa.array(["", "\x00", "__BLOSC2_NULL__", None, "\U0001f389x"], type=pa.string()), + ), + ( + "timestamp_int64_min", + pa.array( + [ + np.datetime64("2020-01-01", "us"), + None, + np.datetime64(np.iinfo(np.int64).min + 1, "us"), + ], + type=pa.timestamp("us"), + ), + ), +] + + +@pytest.mark.parametrize(("label", "arrow_array"), ROUND_TRIP_CASES) +def test_arrow_round_trip_is_lossless(label, arrow_array): + _, exported = round_trip(arrow_array) + assert_same_logical(exported, arrow_array) + + +@pytest.mark.parametrize(("label", "arrow_array"), ROUND_TRIP_CASES) +def test_parquet_round_trip_is_lossless(label, arrow_array, tmp_path): + src = str(tmp_path / "in.parquet") + out = str(tmp_path / "out.parquet") + original = pa.table({"v": arrow_array}) + pq.write_table(original, src) + + ct = blosc2.CTable.from_parquet(src, null_storage="mask") + ct.to_parquet(out) + assert_same_logical(pq.read_table(out).column("v"), original.column("v")) + + +@pytest.mark.parametrize("max_length", [4]) +def test_fixed_width_string_fully_occupying_its_width(max_length): + arrow_array = pa.array(["abcd", None, "", "wxyz"], type=pa.string()) + table = pa.table({"v": arrow_array}) + ct = blosc2.CTable.from_arrow(table, null_storage="mask", string_max_length=max_length) + assert ct["v"].dtype == np.dtype(f"U{max_length}") # no widening for a sentinel + assert_same_logical(ct.to_arrow().column("v"), arrow_array) + + +def test_fixed_width_bytes_fully_occupying_its_width(): + arrow_array = pa.array([b"abcd", None, b"", b"wxyz"], type=pa.binary()) + table = pa.table({"v": arrow_array}) + ct = blosc2.CTable.from_arrow(table, null_storage="mask", string_max_length=4) + assert ct["v"].dtype == np.dtype("S4") + exported = ct.to_arrow().column("v") + assert exported.to_pylist() == [b"abcd", None, b"", b"wxyz"] + + +def test_ndarray_column_round_trip(): + arrow_array = pa.array([[1, 2], None, [3, 4]], type=pa.list_(pa.int64(), 2)) + ct, exported = round_trip(arrow_array) + assert ct["v"].null_storage == "mask" + assert exported.to_pylist() == [[1, 2], None, [3, 4]] + + +# --------------------------------------------------------------------------- +# The measurement that justifies the design +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize( + ("label", "arrow_array", "lossy_result"), + [ + # -128 is the sentinel int8 picks, so real -128 data reads back as null. + ("int8_min", pa.array([-128, None, 127], type=pa.int8()), [None, None, 127]), + # "__BLOSC2_NULL__" is literally the utf8 sentinel. Only on NumPy >= 2: + # without StringDType an Arrow string column imports as vlstring, whose + # nulls are native None, so there is no sentinel to collide with and + # nothing is lost. + pytest.param( + "utf8_sentinel_literal", + pa.array(["", "__BLOSC2_NULL__", None], type=pa.string()), + ["", None, None], + marks=needs_utf8, + ), + ], +) +def test_sentinel_storage_is_lossy_where_mask_storage_is_not(label, arrow_array, lossy_result): + """Silent corruption, not an error — which is what makes it worth fixing.""" + table = pa.table({"v": arrow_array}) + + sentinel = blosc2.CTable.from_arrow(table, null_storage="sentinel") + assert sentinel.to_arrow().column("v").to_pylist() == lossy_result + + masked = blosc2.CTable.from_arrow(table, null_storage="mask") + assert masked.to_arrow().column("v").to_pylist() == arrow_array.to_pylist() + + +def test_nullable_bool_imports_without_the_255_reservation(): + table = pa.table({"v": pa.array([True, None, False], type=pa.bool_())}) + ct = blosc2.CTable.from_arrow(table, null_storage="mask") + assert ct["v"].dtype == np.dtype(np.bool_) + assert ct["v"][:].tolist() == [True, False, False] + assert ct["v"].is_null().tolist() == [False, True, False] + + +# --------------------------------------------------------------------------- +# Choosing the storage +# --------------------------------------------------------------------------- + + +def test_default_is_a_mask(): + """Lossless round-trip is why the sidecar exists, so imports get one. + + Which is the point of the flip: an Arrow column carries a validity bitmap, + and reading it into a sidecar preserves it exactly, where a sentinel has to + steal a value from the range to stand in for it. + """ + table = pa.table({"v": pa.array([1, None, 3], type=pa.int64())}) + assert blosc2.CTable.from_arrow(table)["v"].null_storage == "mask" + + +def test_sentinel_storage_is_one_parameter_away(): + table = pa.table({"v": pa.array([1, None, 3], type=pa.int64())}) + ct = blosc2.CTable.from_arrow(table, null_storage="sentinel") + assert ct["v"].null_storage == "sentinel" + assert ct["v"].null_value == np.iinfo(np.int64).min + + +def test_explicit_parameter_selects_mask(): + table = pa.table({"v": pa.array([1, None, 3], type=pa.int64())}) + assert blosc2.CTable.from_arrow(table, null_storage="mask")["v"].null_storage == "mask" + + +def test_null_policy_selects_mask(): + table = pa.table({"v": pa.array([1, None, 3], type=pa.int64())}) + with blosc2.null_policy(blosc2.NullPolicy(null_storage="mask")): + assert blosc2.CTable.from_arrow(table)["v"].null_storage == "mask" + + +def test_explicit_parameter_overrides_the_policy(): + table = pa.table({"v": pa.array([1, None, 3], type=pa.int64())}) + with blosc2.null_policy(blosc2.NullPolicy(null_storage="mask")): + ct = blosc2.CTable.from_arrow(table, null_storage="sentinel") + assert ct["v"].null_storage == "sentinel" + + +def test_column_null_values_still_forces_sentinel_per_column(): + table = pa.table( + { + "a": pa.array([1, None, 3], type=pa.int64()), + "b": pa.array([1, None, 3], type=pa.int64()), + } + ) + with blosc2.null_policy(blosc2.NullPolicy(column_null_values={"a": -7})): + ct = blosc2.CTable.from_arrow(table, null_storage="mask") + assert ct["a"].null_storage == "sentinel" + assert ct["a"].null_value == -7 + assert ct["b"].null_storage == "mask" + + +def test_non_nullable_columns_get_no_null_channel(): + """Arrow fields are nullable by default, so this needs an explicit schema.""" + schema = pa.schema([pa.field("v", pa.int64(), nullable=False)]) + table = pa.table({"v": pa.array([1, 2, 3], type=pa.int64())}, schema=schema) + ct = blosc2.CTable.from_arrow(table, null_storage="mask") + assert ct["v"].null_storage == "none" + + +def test_a_nullable_column_with_no_nulls_writes_no_sidecar(): + """Decision 9 survives the import path.""" + field = pa.field("v", pa.int64(), nullable=True) + table = pa.table({"v": pa.array([1, 2, 3], type=pa.int64())}, schema=pa.schema([field])) + ct = blosc2.CTable.from_arrow(table, null_storage="mask") + assert ct["v"].null_storage == "mask" + assert ct._null_mask("v") is None + assert ct["v"].null_count() == 0 + + +def test_import_error_now_points_at_mask_storage(): + """A type with no available sentinel still fails under sentinel storage... + + ...but the message now names the way out, which is the whole point: mask + storage needs no sentinel, so every type can carry nulls. + """ + table = pa.table({"v": pa.array([1.0, None], type=pa.float64())}) + with blosc2.null_policy(blosc2.NullPolicy(float_value=None)): + with pytest.raises(TypeError, match="null_storage='mask'"): + blosc2.CTable.from_arrow(table, null_storage="sentinel") + + +def test_mask_storage_imports_what_no_sentinel_could(): + """The same input the previous test rejects, accepted.""" + table = pa.table({"v": pa.array([1.0, None], type=pa.float64())}) + with blosc2.null_policy(blosc2.NullPolicy(float_value=None)): + ct = blosc2.CTable.from_arrow(table, null_storage="mask") + assert ct["v"].is_null().tolist() == [False, True] + + +def test_auto_null_sentinels_false_is_irrelevant_under_mask(): + """Mask storage picks no sentinel, so disabling the picker changes nothing.""" + table = pa.table({"v": pa.array([1, None, 3], type=pa.int64())}) + ct = blosc2.CTable.from_arrow(table, null_storage="mask", auto_null_sentinels=False) + assert ct["v"].is_null().tolist() == [False, True, False] + + +# --------------------------------------------------------------------------- +# The utf8 validity bitmap: bit order, pinned to literal bytes +# --------------------------------------------------------------------------- + + +def test_utf8_validity_bitmap_is_lsb_first(): + """Arrow validity bitmaps are LSB-first, and a round-trip cannot prove it. + + Import would unpack a wrongly-packed bitmap the same wrong way, so this + asserts the literal buffer bytes for a known pattern instead. Rows + ``[valid, null, valid, valid, valid, null, valid, valid]`` must pack to + ``0b1101_1101`` = 0xDD, not to the MSB-first ``0b1011_1011`` = 0xBB. + """ + values = ["a", None, "c", "d", "e", None, "g", "h"] + table = pa.table({"v": pa.array(values, type=pa.string())}) + ct = blosc2.CTable.from_arrow(table, null_storage="mask") + exported = ct.to_arrow().column("v").combine_chunks() + + validity = exported.buffers()[0] + assert validity is not None, "a column with nulls must carry a validity buffer" + assert validity.to_pybytes()[0] == 0xDD + + +def test_utf8_dense_buffer_export_matches_the_generic_path(): + """The dense fast path builds Arrow buffers directly; it must not diverge.""" + values = ["a", None, "c", None, "e"] + table = pa.table({"v": pa.array(values, type=pa.string())}) + ct = blosc2.CTable.from_arrow(table, null_storage="mask") + + dense = ct.to_arrow().column("v") + # Deleting and re-compacting leaves a table the fast path declines (its + # guard is _last_pos == _n_rows over a dense root table), so this exercises + # the per-row path over the same data. + generic = ct.where("True" if False else ct["v"].notnull() | ct["v"].is_null()).to_arrow().column("v") + assert dense.to_pylist() == generic.to_pylist() == values + + +def test_utf8_export_of_a_filtered_view_keeps_its_nulls(): + values = ["a", None, "c", None, "e"] + table = pa.table({"v": pa.array(values, type=pa.string()), "k": pa.array([0, 1, 1, 1, 0])}) + ct = blosc2.CTable.from_arrow(table, null_storage="mask") + view = ct.where("k == 1") + assert view.to_arrow().column("v").to_pylist() == [None, "c", None] + + +# --------------------------------------------------------------------------- +# Export of every V1 kind, from a natively-built table +# --------------------------------------------------------------------------- + + +def build(spec, values, capacity=32): + import dataclasses + + ann = ( + object + if isinstance(spec, (blosc2.schema.NDArraySpec, blosc2.schema.timestamp)) + else spec.python_type + ) + Row = dataclasses.make_dataclass("R", [("v", ann, blosc2.field(spec))]) + t = blosc2.CTable(Row, expected_size=capacity) + t.extend([(v,) for v in values]) + return t + + +EXPORT_CASES = [ + ("bool", blosc2.bool(null_storage="mask"), [True, None, False], [True, None, False]), + ("int8", blosc2.int8(null_storage="mask"), [-128, None, 127], [-128, None, 127]), + ("uint8", blosc2.uint8(null_storage="mask"), [0, None, 255], [0, None, 255]), + ( + "string", + blosc2.string(max_length=4, null_storage="mask"), + ["abcd", None, ""], + ["abcd", None, ""], + ), + ( + "bytes", + blosc2.bytes(max_length=4, null_storage="mask"), + [b"abcd", None, b""], + [b"abcd", None, b""], + ), + pytest.param( + "utf8", + utf8_spec(null_storage="mask"), + ["", "\x00", None], + ["", "\x00", None], + marks=needs_utf8, + ), +] + + +@pytest.mark.parametrize(("label", "spec", "values", "expected"), EXPORT_CASES) +def test_export_carries_the_sidecar_into_arrow(label, spec, values, expected): + exported = build(spec, values).to_arrow().column("v") + assert exported.to_pylist() == expected + assert exported.null_count == 1 + + +def test_export_of_a_null_free_mask_column_has_no_nulls(): + exported = build(blosc2.int64(null_storage="mask"), [1, 2, 3]).to_arrow().column("v") + assert exported.null_count == 0 + assert exported.to_pylist() == [1, 2, 3] + + +def test_export_of_a_table_with_deletions_remaps_nulls(): + t = build(blosc2.int64(null_storage="mask"), [1, None, 3, None, 5]) + t.delete(0) + assert t.to_arrow().column("v").to_pylist() == [None, 3, None, 5] + + +def test_ndarray_export_uses_row_level_validity(): + """A sentinel ndarray column needs *every* element to match; a mask does not.""" + spec = blosc2.ndarray((2,), dtype=blosc2.int64(), null_storage="mask") + t = build(spec, [np.array([0, 0]), None, np.array([1, 2])]) + exported = t.to_arrow().column("v") + # Row 0 is all-zero — the sentinel rule's definition of null — but is a + # real value here, and only row 1 is null. + assert exported.to_pylist() == [[0, 0], None, [1, 2]] + + +# --------------------------------------------------------------------------- +# Persistence of imported nulls +# --------------------------------------------------------------------------- + + +def test_imported_nulls_survive_a_persistent_import(tmp_path): + path = str(tmp_path / "imported.b2d") + table = pa.table({"v": pa.array([1, None, 3], type=pa.int64())}) + ct = blosc2.CTable.from_arrow(table, null_storage="mask", urlpath=path, mode="w") + ct.close() + + reopened = blosc2.CTable.open(path) + try: + assert reopened["v"].null_storage == "mask" + assert reopened["v"].is_null().tolist() == [False, True, False] + assert reopened.to_arrow().column("v").to_pylist() == [1, None, 3] + finally: + reopened.close() + + +def test_import_spanning_several_batches_keeps_its_nulls(): + """The sidecar is created mid-import, on the first batch that has a null.""" + n = 300 + values = [None if i % 50 == 49 else i for i in range(n)] + table = pa.table({"v": pa.array(values, type=pa.int64())}) + batches = table.to_batches(max_chunksize=64) + ct = blosc2.CTable.from_arrow(table.schema, iter(batches), null_storage="mask") + assert ct["v"].null_count() == n // 50 + assert ct.to_arrow().column("v").to_pylist() == values diff --git a/tests/ctable/test_null_mask_expressions.py b/tests/ctable/test_null_mask_expressions.py new file mode 100644 index 000000000..256d2a24a --- /dev/null +++ b/tests/ctable/test_null_mask_expressions.py @@ -0,0 +1,349 @@ +####################################################################### +# Copyright (c) 2019-present, Blosc Development Team +# All rights reserved. +# +# SPDX-License-Identifier: BSD-3-Clause +####################################################################### + +"""Expressions and reductions over mask-storage columns (Phase 5). + +The expression layer was already storage-agnostic — it consumes an opaque +boolean null predicate — so most of this is a differential check that mask and +sentinel columns agree wherever they are supposed to, and diverge only where +decision 6 says they must (NaN is a value under a mask, a null under a +sentinel). + +The reductions needed real work: ``argmin``/``argmax`` and the ndarray +reduction path both keyed off ``null_value is not None``, so under mask storage +they silently reduced over the *fill*. For an int column the fill is ``0``, +which is a plausible-looking minimum — the kind of wrong answer nobody notices. + +Two things this file pins as **deliberately not done**, because measurement +showed the plan's premises for them were wrong: + +* the summary-index ``min``/``max`` shortcut stays disabled for mask columns; +* ndarray columns still get no null predicate in the lazy layer. + +Both are explained where they are asserted. +""" + +from __future__ import annotations + +import dataclasses + +import numpy as np +import pytest + +import blosc2 + + +def annotation_for(spec): + if isinstance(spec, (blosc2.schema.NDArraySpec, blosc2.schema.timestamp)): + return object + return spec.python_type + + +def one_col(spec, values, capacity=32): + Row = dataclasses.make_dataclass("R", [("v", annotation_for(spec), blosc2.field(spec))]) + t = blosc2.CTable(Row, expected_size=capacity) + t.extend([(v,) for v in values]) + return t + + +def mask_col(values, factory=blosc2.int64, **kw): + return one_col(factory(null_storage="mask", **kw), values) + + +# --------------------------------------------------------------------------- +# Reductions skip nulls rather than reducing over the fill +# --------------------------------------------------------------------------- + + +def test_min_ignores_the_zero_fill(): + """The int fill is 0 — a plausible-looking minimum for positive data.""" + assert mask_col([5, None, 1, 9])["v"].min() == 1 + + +def test_max_ignores_the_zero_fill(): + """...and a plausible-looking maximum for negative data.""" + assert mask_col([-5, None, -1])["v"].max() == -1 + + +def test_argmin_points_at_a_real_row(): + assert mask_col([5, None, 1, 9])["v"].argmin() == 2 + + +def test_argmax_points_at_a_real_row(): + assert mask_col([-5, None, -1])["v"].argmax() == 2 + + +def test_argmin_would_have_picked_the_fill(): + """Regression: the fill sits at row 1 and is smaller than every real value.""" + col = mask_col([5, None, 1, 9])["v"] + assert col[:][1] == 0 # the fill really is there in the values + assert col.argmin() == 2 # ...and is not what argmin reports + + +@pytest.mark.parametrize("op", ["argmin", "argmax"]) +def test_arg_reductions_raise_on_an_all_null_column(op): + col = mask_col([None, None])["v"] + with pytest.raises(ValueError, match="all values are null"): + getattr(col, op)() + + +def test_sum_and_mean_skip_nulls(): + col = mask_col([2.0, None, 4.0], factory=blosc2.float64)["v"] + assert col.sum() == 6.0 + assert col.mean() == 3.0 + + +def test_std_skips_nulls(): + col = mask_col([1.0, None, 3.0], factory=blosc2.float64)["v"] + assert col.std() == pytest.approx(1.0) + + +def test_reductions_ignore_capacity_padding(): + """Padding is fill-valued too, and must not reach a reduction either.""" + col = one_col(blosc2.int64(null_storage="mask"), [5, None, 1], capacity=64)["v"] + assert col.min() == 1 + assert col.max() == 5 + assert col.sum() == 6 + + +# --------------------------------------------------------------------------- +# ndarray columns: the gain is in the NumPy reduction paths +# --------------------------------------------------------------------------- + + +def ndarray_col(items, dtype=blosc2.int64): + spec = blosc2.ndarray((3,), dtype=dtype(), null_storage="mask") + return one_col(spec, items)["v"] + + +def test_ndarray_min_skips_null_rows(): + col = ndarray_col([np.array([1, 2, 3]), None, np.array([7, 8, 9])]) + assert col.min() == 1 # not 0, the fill item + + +def test_ndarray_max_skips_null_rows(): + col = ndarray_col([np.array([-1, -2, -3]), None, np.array([-7, -8, -9])]) + assert col.max() == -1 # not 0, the fill item + + +def test_ndarray_sum_skips_null_rows(): + col = ndarray_col([np.array([1, 2, 3]), None, np.array([7, 8, 9])]) + assert col.sum() == 30 + + +def test_ndarray_mean_divides_by_live_non_null_elements(): + col = ndarray_col([np.array([1, 2, 3]), None, np.array([7, 8, 9])]) + assert col.mean() == pytest.approx(30 / 6) + + +def test_ndarray_reduction_where_composes_with_nulls(): + spec = blosc2.ndarray((2,), dtype=blosc2.int64(), null_storage="mask") + Row = dataclasses.make_dataclass( + "R", + [ + ("e", object, blosc2.field(spec)), + ("k", int, blosc2.field(blosc2.int64())), + ], + ) + t = blosc2.CTable(Row, expected_size=16) + t.extend([(np.array([1, 1]), 0), (None, 1), (np.array([5, 5]), 1)]) + assert t["e"].sum(where="k == 1") == 10 + + +# --------------------------------------------------------------------------- +# Propagation through arithmetic and comparisons +# --------------------------------------------------------------------------- + + +def test_arithmetic_marks_null_rows_nan(): + expr = mask_col([1, None, 3])["v"] * 2 + values = np.asarray(expr[:3]) + assert values[0] == 2.0 + assert np.isnan(values[1]) + assert values[2] == 6.0 + + +def test_arithmetic_result_reduces_without_nan_poisoning(): + assert (mask_col([1, None, 3])["v"] * 2).sum() == 8.0 + + +def test_comparison_gives_null_rows_false(): + """SQL WHERE semantics: a null satisfies no comparison.""" + col = mask_col([1, None, 3])["v"] + assert (col > 0)[:3].tolist() == [True, False, True] + + +def test_where_over_a_mask_column_drops_nulls(): + t = mask_col([1, None, 3]) + assert t.where(t["v"] > 0)["v"][:].tolist() == [1, 3] + + +def test_not_equal_does_not_leak_nulls(): + """IEEE says nan != x is True; SQL says a null satisfies nothing.""" + col = mask_col([1, None, 3])["v"] + assert (col != 1)[:3].tolist() == [False, False, True] + + +def test_two_mask_columns_combine_their_nulls(): + Row = dataclasses.make_dataclass( + "R", + [ + ("a", int, blosc2.field(blosc2.int64(null_storage="mask"))), + ("b", int, blosc2.field(blosc2.int64(null_storage="mask"))), + ], + ) + t = blosc2.CTable(Row, expected_size=16) + t.extend([(1, 10), (None, 20), (3, None), (4, 40)]) + values = np.asarray((t["a"] + t["b"])[:4]) + assert values[0] == 11 + assert np.isnan(values[1]) + assert np.isnan(values[2]) + assert values[3] == 44 + + +def test_a_null_free_mask_column_costs_no_operand(): + """No sidecar means no predicate, so arithmetic stays a plain LazyExpr.""" + col = mask_col([1, 2, 3])["v"] + assert col._raw_null_pred() is None + assert isinstance(col * 2, blosc2.LazyExpr) + + +# --------------------------------------------------------------------------- +# Differential: mask and sentinel agree, except where decision 6 says not to +# --------------------------------------------------------------------------- + +AGREEING_CASES = [ + ("min", lambda c: c.min()), + ("max", lambda c: c.max()), + ("sum", lambda c: c.sum()), + ("mean", lambda c: c.mean()), + ("argmin", lambda c: c.argmin()), + ("argmax", lambda c: c.argmax()), + ("null_count", lambda c: c.null_count()), + ("is_null", lambda c: c.is_null().tolist()), + ("unique", lambda c: c.unique().tolist()), + ("gt", lambda c: (c > 2)[:4].tolist()), +] + + +@pytest.mark.parametrize(("label", "op"), AGREEING_CASES) +def test_mask_and_sentinel_agree_on_integer_data(label, op): + """Same logical data, two storages, one answer.""" + values = [5, None, 1, 9] + masked = mask_col(values)["v"] + sentinel = one_col(blosc2.int64(null_value=-999), [-999 if v is None else v for v in values])["v"] + assert op(masked) == op(sentinel) + + +def test_mask_and_sentinel_diverge_on_nan_by_design(): + """decision 6: NaN is a value under a mask, the null itself under a sentinel.""" + values = [1.0, float("nan"), 3.0] + masked = one_col(blosc2.float64(null_storage="mask"), values)["v"] + sentinel = one_col(blosc2.float64(null_value=float("nan")), values)["v"] + + assert masked.null_count() == 0 + assert sentinel.null_count() == 1 + # The NaN is data for the mask column, so it poisons the reduction the way + # NumPy does; for the sentinel column it is the null and is skipped. + assert np.isnan(masked.sum()) + assert sentinel.sum() == 4.0 + + +# --------------------------------------------------------------------------- +# Nullable bool needs no predicate rewrite under mask storage +# --------------------------------------------------------------------------- + + +def test_mask_bool_is_not_treated_as_a_sentinel_bool(): + col = mask_col([True, None, False], factory=blosc2.bool)["v"] + assert col._is_nullable_bool is False + + +def test_sentinel_bool_is_still_treated_as_one(): + col = one_col(blosc2.bool(nullable=True, null_value=255), [True, 255, False])["v"] + assert col._is_nullable_bool is True + + +def test_mask_bool_filters_directly(): + t = mask_col([True, None, False, True], factory=blosc2.bool) + assert t.where(t["v"])["v"][:].tolist() == [True, True] + + +# --------------------------------------------------------------------------- +# Deliberately not done — with the measurement that says why +# --------------------------------------------------------------------------- + + +def test_a_genuine_nan_still_keeps_a_mask_float_column_off_the_shortcut(tmp_path): + """Phase 5 disabled the whole column here; Phase 10 narrows that to the NaN. + + Null-aware summaries let a mask float column qualify as a *source* — its + nulls are excluded from the extrema like anything else. What cannot be + shortcut is a genuine NaN, which decision 6 makes a value: the scan poisons + to NaN while the summaries would report a real extremum. ``FLAG_HAS_NAN`` + is raised over the valid rows only, so it now marks exactly that case, and + ``_index_summary_minmax`` declines on it while the source stays usable. + """ + spec = blosc2.float64(null_storage="mask") + Row = dataclasses.make_dataclass("R", [("v", float, blosc2.field(spec))]) + path = str(tmp_path / "s.b2d") + t = blosc2.CTable(Row, expected_size=5, urlpath=path, mode="w") + t.extend([(1.0,), (float("nan"),), (5.0,), (None,), (3.0,)]) + t.close() + + reopened = blosc2.CTable.open(path, mode="a") + try: + assert reopened["v"]._summary_minmax_source() is not None + assert reopened["v"]._index_summary_minmax("min") is NotImplemented + # The scanned answer, which the shortcut would have contradicted. + assert np.isnan(reopened["v"].min()) + finally: + reopened.close() + + +def test_a_nan_free_mask_float_column_now_takes_the_shortcut(tmp_path): + """The other half: with no NaN in the data there is nothing to disagree on.""" + spec = blosc2.float64(null_storage="mask") + Row = dataclasses.make_dataclass("R", [("v", float, blosc2.field(spec))]) + path = str(tmp_path / "nn.b2d") + t = blosc2.CTable(Row, expected_size=5, urlpath=path, mode="w") + t.extend([(1.0,), (2.0,), (5.0,), (None,), (3.0,)]) + t.close() + + reopened = blosc2.CTable.open(path, mode="a") + try: + assert reopened["v"]._index_summary_minmax("min") == 1.0 + assert reopened["v"].min() == 1.0 + assert reopened["v"].max() == 5.0 + finally: + reopened.close() + + +def test_sentinel_nan_float_keeps_its_summary_shortcut(tmp_path): + """The contrast: there NaN *is* the null, so dropping it is exactly right.""" + spec = blosc2.float64(null_value=float("nan")) + Row = dataclasses.make_dataclass("R", [("v", float, blosc2.field(spec))]) + path = str(tmp_path / "n.b2d") + t = blosc2.CTable(Row, expected_size=4, urlpath=path, mode="w") + t.extend([(1.0,), (float("nan"),), (5.0,), (3.0,)]) + t.close() + + reopened = blosc2.CTable.open(path, mode="a") + try: + assert reopened["v"]._summary_minmax_source() is not None + assert reopened["v"].min() == 1.0 + finally: + reopened.close() + + +def test_ndarray_columns_still_get_no_lazy_null_predicate(): + """Not an oversight: see NullChannel._mask_pred for the two blockers.""" + col = ndarray_col([np.array([1, 2, 3]), None]) + assert col._raw_null_pred() is None + assert col._nulls.valid_pred() is None + # ...and the reduction paths, which are row-level NumPy, cover it instead. + assert col.min() == 1 diff --git a/tests/ctable/test_null_mask_sort_groupby.py b/tests/ctable/test_null_mask_sort_groupby.py new file mode 100644 index 000000000..88a135be1 --- /dev/null +++ b/tests/ctable/test_null_mask_sort_groupby.py @@ -0,0 +1,758 @@ +####################################################################### +# Copyright (c) 2019-present, Blosc Development Team +# All rights reserved. +# +# SPDX-License-Identifier: BSD-3-Clause +####################################################################### + +"""Sort, group_by and query paths over mask-storage columns (Phase 7). + +Every path here had the same shape of bug, and it is worth stating once: a +mask column's null rows hold the column's *fill*, and a fill is an ordinary +value to anything that only looks at the values. So + +* ``sort_by`` sorted nulls by their fill — first for an ascending int column + (``0``), first for a string one (``""``) — where the nulls-last contract and + every sentinel column put them last; +* ``group_by`` merged the nulls into the genuine ``0`` / ``""`` group, and + reduced over the fill instead of skipping it; +* ``where("a < 10")`` matched every null, because ``0 < 10``; +* a ``FULL`` index on a float column returned every null for ``f > 0.5``, + because the NaN fill sorts into the ordered range the index hands back. + +The tests are written as a differential oracle against sentinel storage: the +same logical data both ways, asserted to give the same answer. That is the +strongest form available, because the sentinel path has been correct since +Phase 1 and is independently tested. Where the two are *supposed* to differ, +the divergence is asserted directly instead. +""" + +from __future__ import annotations + +import dataclasses + +import numpy as np +import pytest +from utf8_compat import needs_utf8, utf8_spec + +import blosc2 + + +def annotation_for(spec): + if isinstance(spec, (blosc2.schema.NDArraySpec, blosc2.schema.timestamp)): + return object + return spec.python_type + + +def table(rows, capacity=64, urlpath=None, **cols): + Row = dataclasses.make_dataclass( + "SortRow", [(n, annotation_for(s), blosc2.field(s)) for n, s in cols.items()] + ) + kwargs = {"urlpath": str(urlpath), "mode": "w"} if urlpath is not None else {} + t = blosc2.CTable(Row, expected_size=max(capacity, len(rows)), **kwargs) + if rows: + t.extend(rows) + return t + + +def one_col(values, spec, capacity=64, urlpath=None): + return table([(v,) for v in values], capacity=capacity, urlpath=urlpath, a=spec) + + +#: The V1 kinds a sort key can be, each with a mask spec, a sentinel spec, and +#: the value the sentinel path has to write where the mask path writes ``None``. +KEY_KINDS = [ + ("int64", blosc2.int64, {}, -(2**62)), + ("float64", blosc2.float64, {}, np.nan), + ("string", blosc2.string, {"max_length": 4}, "\x7f\x7f"), + pytest.param("utf8", blosc2.utf8, {}, "__BLOSC2_NULL__", marks=needs_utf8), +] + + +def pair(values, factory, kw, sentinel): + """The same logical *values* as a mask table and a sentinel table.""" + mask = one_col(values, factory(null_storage="mask", **kw)) + raw = [sentinel if v is None else v for v in values] + sent = one_col(raw, factory(nullable=True, null_value=sentinel, **kw)) + return mask, sent + + +def as_list(col): + """A column's live values with nulls spelled ``None``.""" + null = col.is_null() + return [None if null[i] else _scalar(col[i]) for i in range(len(col))] + + +def _scalar(value): + return value.item() if hasattr(value, "item") else value + + +# --------------------------------------------------------------------------- +# sort_by: nulls last, in both directions, for every kind +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize(("label", "factory", "kw", "sentinel"), KEY_KINDS) +@pytest.mark.parametrize("ascending", [True, False]) +def test_sort_by_puts_mask_nulls_last(label, factory, kw, sentinel, ascending): + """The contract sentinel columns already keep: nulls last, either direction. + + Before this phase the null-indicator lexsort key was only built for a + sentinel, so a mask column sorted its nulls by the fill — ``0`` and ``""`` + both sort *first* ascending, which silently reversed the contract. + """ + values = ["e", None, "a", "z", None, "b"] if label in ("string", "utf8") else [5, None, 1, 9, None, 2] + mask, sent = pair(values, factory, kw, sentinel) + + got = as_list(mask.sort_by("a", ascending=ascending)["a"]) + n_nulls = sum(v is None for v in values) + assert got[-n_nulls:] == [None] * n_nulls + assert got[:-n_nulls] == sorted((v for v in values if v is not None), reverse=not ascending) + # And the same answer the sentinel path gives, null spelling aside. + assert got == as_list(sent.sort_by("a", ascending=ascending)["a"]) + + +def test_sort_by_null_free_mask_column_needs_no_indicator_key(): + """No sidecar, no nulls, no extra key — decision 9 all the way through.""" + t = one_col([3, 1, 2], blosc2.int64(null_storage="mask")) + assert t._null_mask("a") is None + assert as_list(t.sort_by("a")["a"]) == [1, 2, 3] + + +def test_sort_by_multi_key_orders_nulls_last_per_key(): + t = table( + [(1, "b"), (1, None), (None, "a"), (1, "a"), (None, None)], + a=blosc2.int64(null_storage="mask"), + s=blosc2.string(max_length=2, null_storage="mask"), + ) + st = t.sort_by(["a", "s"]) + assert as_list(st["a"]) == [1, 1, 1, None, None] + assert as_list(st["s"]) == ["a", "b", None, "a", None] + + +def test_sort_by_inplace_keeps_values_with_their_nulls(): + t = one_col([5, None, 1], blosc2.int64(null_storage="mask")) + t.sort_by("a", inplace=True) + assert as_list(t["a"]) == [1, 5, None] + + +def test_sort_by_view_orders_nulls_last(): + t = one_col([5, None, 1], blosc2.int64(null_storage="mask")) + assert as_list(t.sort_by("a", view=True)["a"]) == [1, 5, None] + + +@pytest.mark.parametrize( + ("spec", "values", "expected"), + [ + (blosc2.int8(null_storage="mask"), [3, None, -128, 7], [7, 3, -128, None]), + (blosc2.uint8(null_storage="mask"), [3, None, 255, 0], [255, 3, 0, None]), + (blosc2.bool(null_storage="mask"), [True, None, False], [True, False, None]), + (blosc2.bytes(max_length=2, null_storage="mask"), [b"b", None, b""], [b"b", b"", None]), + ], +) +def test_descending_sort_over_a_full_range_mask_column(spec, values, expected): + """Two storage-independent sort bugs that only mask storage can reach. + + The descending value key is built by negating the values, and that broke on + the two dtypes a nullable column could not previously *be*: a ``bool`` + column has no unary minus (a nullable bool used to be physically ``uint8``), + and a narrow signed dtype wraps on its own minimum -- ``-(-128) == -128`` in + int8 -- so that row sorted as if it were the largest. A sentinel had to + reserve int8's ``-128``, so no nullable int8 column could hold it. + """ + assert as_list(one_col(values, spec).sort_by("a", ascending=False)["a"]) == expected + + +def test_descending_sort_of_a_plain_bool_column(): + """The same fix, stated for the case that was broken with no nulls at all.""" + t = one_col([True, False, True], blosc2.bool()) + assert as_list(t.sort_by("a", ascending=False)["a"]) == [True, True, False] + + +# --------------------------------------------------------------------------- +# The FULL-index sort path reads the sidecar, not the whole column +# --------------------------------------------------------------------------- + + +def indexed_pair(values, factory, kw, sentinel, tmp_path): + mask = one_col(values, factory(null_storage="mask", **kw), urlpath=tmp_path / "m.b2t") + raw = [sentinel if v is None else v for v in values] + sent = one_col(raw, factory(nullable=True, null_value=sentinel, **kw), urlpath=tmp_path / "s.b2t") + for t in (mask, sent): + t.create_index("a", kind="full") + return mask, sent + + +@pytest.mark.parametrize(("label", "factory", "kw", "sentinel"), KEY_KINDS) +@pytest.mark.parametrize("ascending", [True, False]) +def test_full_index_sort_matches_the_lexsort(label, factory, kw, sentinel, ascending, tmp_path): + """A FULL index sorts by stored value, so nulls have to be repartitioned. + + Which is not new — the sentinel path already did it — but a mask column had + no branch, so its nulls came back wherever the fill sorted. + """ + if label in ("string", "utf8"): + pool = ["alfa", "beta", "", "zeta", "gam"] # within string's max_length=4 + else: + pool = [5, 1, 9, 2, 7] + values = [None if i % 7 == 0 else pool[i % len(pool)] for i in range(200)] + mask, sent = indexed_pair(values, factory, kw, sentinel, tmp_path) + + got = as_list(mask.sort_by("a", ascending=ascending)["a"]) + assert got == as_list(sent.sort_by("a", ascending=ascending)["a"]) + n_nulls = sum(v is None for v in values) + assert got[-n_nulls:] == [None] * n_nulls + + +def whole_column_reads(t, name, monkeypatch, body): + """Run *body*, returning which arrays *t* read whole (``arr[:]``).""" + seen = [] + real_getitem = blosc2.NDArray.__getitem__ + values_arr = t._cols[name] + sidecar = t._null_mask(name) + + def counting(self, key): + if isinstance(key, slice) and key == slice(None): + if self is values_arr: + seen.append("values") + elif sidecar is not None and self is sidecar: + seen.append("sidecar") + return real_getitem(self, key) + + monkeypatch.setattr(blosc2.NDArray, "__getitem__", counting) + body() + monkeypatch.undo() + return seen + + +def test_full_index_sort_reads_the_sidecar_instead_of_the_values(tmp_path, monkeypatch): + """The highest value-per-line change in the phase, measured as such. + + Locating the null rows in a FULL index's permutation means reading a whole + column: the sentinel path reads the *values* to compare against the + sentinel, 8 bytes a row for ``int64`` and 64 for a ``U16`` string. A mask + column reads its sidecar instead — one byte a row, and bool NDArrays + compress to almost nothing. + """ + values = [None if i % 5 == 0 else i for i in range(200)] + mask = one_col(values, blosc2.int64(null_storage="mask"), urlpath=tmp_path / "m.b2t") + sent = one_col( + [-1 if v is None else v for v in values], + blosc2.int64(nullable=True, null_value=-1), + urlpath=tmp_path / "s.b2t", + ) + for t in (mask, sent): + t.create_index("a", kind="full") + + assert whole_column_reads(mask, "a", monkeypatch, lambda: mask.sort_by("a")) == ["sidecar"] + assert whole_column_reads(sent, "a", monkeypatch, lambda: sent.sort_by("a")) == ["values"] + + +def test_sorted_slice_falls_back_when_the_column_has_nulls(tmp_path): + """The window read cannot locate a mask column's null block. + + It finds the null block by bisecting the *sorted values* sidecar for the + null's stored value; under mask storage that value is the fill, which + genuine rows share, and the validity sidecar is indexed by physical + position rather than sorted position so the window cannot consult it. So + it declines, and the full sorted view — which is mask-aware — answers. + """ + values = [None if i % 9 == 0 else (i % 17) for i in range(300)] + t = one_col(values, blosc2.int64(null_storage="mask"), urlpath=tmp_path / "m.b2t") + t.create_index("a", kind="full") + + assert t._sorted_slice_positions("a", True, slice(0, 5)) is None + for key in (slice(0, 5), slice(-5, None), slice(280, 300)): + for ascending in (True, False): + window = as_list(t.sorted_slice("a", key, ascending=ascending)["a"]) + full = as_list(t.sort_by("a", ascending=ascending, view=True)[key]["a"]) + assert window == full + + +def test_sorted_slice_keeps_its_window_read_when_there_are_no_nulls(tmp_path): + """A mask column with no sidecar has no null block to locate.""" + t = one_col(list(range(300)), blosc2.int64(null_storage="mask"), urlpath=tmp_path / "m.b2t") + t.create_index("a", kind="full") + assert t._sorted_slice_positions("a", True, slice(0, 5)) is not None + + +# --------------------------------------------------------------------------- +# utf8 rank index: the fill must not factorize as an ordinary value +# --------------------------------------------------------------------------- + + +@needs_utf8 +def test_utf8_rank_index_separates_nulls_from_genuine_empty_strings(tmp_path): + """The landmine this phase was warned about, and it is a real one. + + A utf8 rank index factorizes the column; under mask storage the ``""`` fill + is just another vocabulary entry, and it factorizes to **rank 0** — the + smallest — so nulls both sorted first and answered ``a == ''`` alongside + the rows that really are empty. + """ + values = ["b", None, "", "a", None, ""] + t = one_col(values, blosc2.utf8(null_storage="mask"), urlpath=tmp_path / "m.b2t") + t.create_index("a", kind="full") + + assert len(t.where("a == ''")) == 2 # the two real empty strings, not the nulls + assert as_list(t.where("a == ''")["a"]) == ["", ""] + assert as_list(t.sort_by("a")["a"]) == ["", "", "a", "b", None, None] + + +@needs_utf8 +def test_utf8_rank_arrays_stamps_nulls_with_the_null_rank(): + """Directly, since this is where the recoding happens.""" + from blosc2.ctable_indexing import _utf8_rank_arrays + + t = one_col(["b", None, "", "a"], blosc2.utf8(null_storage="mask")) + col = t._cols["a"] + valid = t._null_mask("a") + ranks, meta, vocab = _utf8_rank_arrays(col, 4, None, valid=valid) + assert meta["null_aware"] is True + assert list(vocab) == ["", "a", "b"] + assert int(ranks[1]) == meta["null_rank"] == 3 + assert int(ranks[2]) == 0 # the genuine "" keeps rank 0 + + +@needs_utf8 +def test_a_pre_mask_utf8_rank_index_is_treated_as_stale(tmp_path): + """An index built before nulls got their own rank cannot be trusted. + + Neither of the O(1) staleness signals can catch it — the column's row count + and byte size are exactly what they were — so the absence of ``null_aware`` + from the meta is what marks it, and only for a column that has a sidecar. + """ + t = one_col(["b", None, ""], blosc2.utf8(null_storage="mask"), urlpath=tmp_path / "m.b2t") + t.create_index("a", kind="full") + meta = t._get_index_catalog()["a"]["full"]["utf8_rank"] + assert t._utf8_rank_index_stale("a", meta) is False + + del meta["null_aware"] + assert t._utf8_rank_index_stale("a", meta) is True + + +@needs_utf8 +def test_a_sentinel_utf8_rank_index_stays_fresh_without_the_flag(tmp_path): + """The staleness rule must not fire for sentinel columns. + + Their nulls have always carried ``null_rank``, so an index with no + ``null_aware`` flag is merely an older one, not a wrong one — invalidating + it would rebuild every stored utf8 index in the wild for nothing. + """ + t = one_col( + ["b", "__BLOSC2_NULL__", ""], + blosc2.utf8(nullable=True), + urlpath=tmp_path / "s.b2t", + ) + t.create_index("a", kind="full") + meta = dict(t._get_index_catalog()["a"]["full"]["utf8_rank"]) + meta.pop("null_aware", None) + assert t._utf8_rank_index_stale("a", meta) is False + + +# --------------------------------------------------------------------------- +# where(): the string form, indexed and not +# --------------------------------------------------------------------------- + +WHERE_CASES = [ + "a < 500", + "a > 500", + "a != 7", + "a == 3", + "~(a > 500)", +] + + +def numeric_pair(tmp_path, indexed): + rng = np.random.default_rng(5) + values = [None if i % 31 == 0 else int(rng.integers(0, 1000)) for i in range(2000)] + mask = one_col(values, blosc2.int64(null_storage="mask"), urlpath=tmp_path / f"m{indexed}.b2t") + sent = one_col( + [-1 if v is None else v for v in values], + blosc2.int64(nullable=True, null_value=-1), + urlpath=tmp_path / f"s{indexed}.b2t", + ) + if indexed: + mask.create_index("a", kind="full") + sent.create_index("a", kind="full") + return mask, sent + + +@pytest.mark.parametrize("query", WHERE_CASES) +@pytest.mark.parametrize("indexed", [False, True]) +def test_string_predicates_reject_mask_nulls(query, indexed, tmp_path): + """``a < 10`` matched every null before this: the int fill is ``0``. + + The operator form was already correct (``_null_aware_compare`` collapses + null to False at the leaf); it was only the string form that compared the + stored fill, exactly as it used to compare the stored sentinel before + Phase 1. + """ + mask, sent = numeric_pair(tmp_path, indexed) + got = mask.where(query) + assert got["a"].null_count() == 0 + assert len(got) == len(sent.where(query)) + + +@pytest.mark.parametrize("indexed", [False, True]) +def test_or_over_a_mask_column_keeps_the_other_branch(indexed, tmp_path): + """The per-leaf guard is what makes OR right; a global filter would not be. + + A row null in ``a`` but matching ``b`` must survive ``(a > x) | (b < y)``. + """ + rng = np.random.default_rng(6) + n = 2000 + a = [None if i % 31 == 0 else int(rng.integers(0, 1000)) for i in range(n)] + b = [int(rng.integers(0, 1000)) for _ in range(n)] + t = table( + list(zip(a, b, strict=True)), + capacity=n, + urlpath=tmp_path / f"or{indexed}.b2t", + a=blosc2.int64(null_storage="mask"), + b=blosc2.int64(), + ) + if indexed: + t.create_index("a", kind="full") + + expected = sum(((av is not None and av > 800) or bv < 200) for av, bv in zip(a, b, strict=True)) + assert len(t.where("(a > 800) | (b < 200)")) == expected + + +def test_a_nan_fill_does_not_come_back_from_an_ordered_index(tmp_path): + """The index failure this phase found, and the one masks make worse. + + An ordered index answers ``f > 0.5`` by taking a *range of the sorted + column* — it never evaluates the predicate — and NaN sorts to the end of + that range. A mask float column's fill is NaN, so every null row was + returned. Making the expression null-aware cannot fix this; the + positions have to be filtered, which is what a mask column now joins + ``nullable_indexed`` to get. + """ + rng = np.random.default_rng(9) + values = [None if i % 23 == 0 else float(rng.random()) for i in range(2000)] + t = one_col(values, blosc2.float64(null_storage="mask"), urlpath=tmp_path / "f.b2t") + unindexed = len(t.where("a > 0.5")) + t.create_index("a", kind="full") + indexed = t.where("a > 0.5") + assert indexed["a"].null_count() == 0 + assert len(indexed) == unindexed + + +def test_null_free_mask_column_stays_out_of_the_null_bookkeeping(tmp_path): + """No sidecar means no guard operand and no post-filter to pay for.""" + t = one_col(list(range(2000)), blosc2.int64(null_storage="mask"), urlpath=tmp_path / "n.b2t") + t.create_index("a", kind="full") + rewritten, operands = t._rewrite_null_predicates("a < 500", {"a": t._cols["a"]}) + assert rewritten == "a < 500" + assert list(operands) == ["a"] + assert len(t.where("a < 500")) == 500 + + +def test_a_guard_is_only_emitted_where_the_fill_could_match(): + """Keeping needless guards out is what keeps a query on its index. + + The fill takes the sentinel's place in the can-match test, and it is the + right stand-in: it is what a null row actually holds. ``0`` cannot satisfy + ``a > 10``, so that leaf is left alone. + """ + t = one_col([1, None, 20], blosc2.int64(null_storage="mask")) + operands = {"a": t._cols["a"]} + assert t._rewrite_null_predicates("a > 10", operands)[0] == "a > 10" + guarded, extended = t._rewrite_null_predicates("a < 10", operands) + assert guarded != "a < 10" + assert any(name.startswith("__nv") for name in extended) + + +# --------------------------------------------------------------------------- +# utf8 comparisons and the span driver +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize( + "query", + ["a < 'b'", "a > 'b'", "a != 'a'", "a == ''", "startswith(a, 'a')"], +) +@needs_utf8 +def test_utf8_predicates_agree_between_storages(query): + values = ["e", None, "a", "z", "", "b"] + mask, sent = pair(values, blosc2.utf8, {}, "__BLOSC2_NULL__") + assert as_list(mask.where(query)["a"]) == as_list(sent.where(query)["a"]) + + +@needs_utf8 +def test_utf8_span_driver_excludes_mask_nulls(): + """The span driver materializes nulls to ``""`` and re-applies nullity. + + It found them by comparing against the sentinel, which a mask column does + not have — so a boolean result over the ``""`` fill came back True for + nulls whenever the fill satisfied the expression. + """ + t = one_col(["ax", None, "bx"], blosc2.utf8(null_storage="mask")) + assert as_list(t.where("startswith(a, '')")["a"]) == ["ax", "bx"] + + +@needs_utf8 +def test_utf8_column_vs_column_comparison_excludes_nulls(): + t = table( + [("a", "a"), (None, "a"), ("b", None), ("c", "c")], + a=blosc2.utf8(null_storage="mask"), + b=blosc2.utf8(null_storage="mask"), + ) + assert as_list(t.where(t["a"] == t["b"])["a"]) == ["a", "c"] + + +# --------------------------------------------------------------------------- +# group_by: as a key and as a value +# --------------------------------------------------------------------------- + +AGGS = { + "cnt": ("v", "count"), + "sm": ("v", "sum"), + "mn": ("v", "min"), + "mx": ("v", "max"), + "av": ("v", "mean"), +} + + +def grouped(t, keys, dropna): + """``{key tuple -> {agg -> value}}``, nulls spelled ``None`` throughout.""" + g = t.group_by(keys, dropna=dropna, sort=True).agg(**AGGS) + # Each column is read once, not once per row: as_list is O(rows), so + # calling it inside the loop below made this O(rows^2 * columns) and cost + # more than every other test in the file put together. + cols = {name: as_list(g[name]) for name in (*keys, *AGGS)} + out = {} + for i in range(len(g)): + key = tuple(cols[name][i] for name in keys) + row = {} + for name in AGGS: + value = cols[name][i] + if isinstance(value, float) and np.isnan(value): + value = None # a non-nullable float output spells "missing" NaN + row[name] = value + out[key] = row + return out + + +def groupby_pair(): + rng = np.random.default_rng(11) + n = 400 + k = [None if i % 13 == 0 else int(rng.integers(0, 4)) for i in range(n)] + s = [None if i % 17 == 0 else ["aa", "bb", "cc"][int(rng.integers(0, 3))] for i in range(n)] + v = [None if i % 7 == 0 else float(rng.integers(0, 100)) for i in range(n)] + mask = table( + list(zip(k, s, v, strict=True)), + capacity=n, + k=blosc2.int64(null_storage="mask"), + s=blosc2.string(max_length=2, null_storage="mask"), + v=blosc2.float64(null_storage="mask"), + ) + sent = table( + [ + (-1 if a is None else a, "ZZ" if b is None else b, np.nan if c is None else c) + for a, b, c in zip(k, s, v, strict=True) + ], + capacity=n, + k=blosc2.int64(nullable=True, null_value=-1), + s=blosc2.string(max_length=2, nullable=True, null_value="ZZ"), + v=blosc2.float64(nullable=True, null_value=np.nan), + ) + return mask, sent + + +@pytest.mark.parametrize("keys", [["k"], ["s"], ["k", "s"]]) +@pytest.mark.parametrize("dropna", [True, False]) +def test_group_by_agrees_with_sentinel_storage(keys, dropna, tmp_path): + """The differential oracle for group_by, keys and values together. + + A mask key column used to merge its nulls into the group of whatever its + fill was — ``0`` for the int key, ``""`` for the string one — and a mask + value column reduced over the fill, turning any group containing a null + into ``NaN`` for ``sum``/``mean``. + """ + mask, sent = groupby_pair() + got = grouped(mask, keys, dropna) + expected = { + tuple(None if part in (-1, "ZZ") else part for part in key): row + for key, row in grouped(sent, keys, dropna).items() + } + assert got == expected + + +def test_group_by_a_mask_key_keeps_nulls_out_of_the_fill_group(): + """Stated on its own, because it is the failure that is easiest to miss. + + ``0`` is a perfectly ordinary key, so a null row landing in its group + inflates a real answer rather than producing an obviously wrong one. + """ + t = table( + [(0, 1.0), (None, 2.0), (0, 4.0)], + k=blosc2.int64(null_storage="mask"), + v=blosc2.float64(null_storage="mask"), + ) + g = t.group_by(["k"], dropna=True).agg(total=("v", "sum")) + assert as_list(g["k"]) == [0] + assert as_list(g["total"]) == [5.0] + + +def test_group_by_a_mask_key_writes_its_null_group_as_a_null(): + """With ``dropna=False`` the null group's key is a *real* null. + + A sentinel column can only offer its sentinel here, which is why + ``group_by(dropna=False)`` over one returns a group keyed ``-1``. Mask + storage can say what it means. + """ + t = table( + [(1, 1.0), (None, 2.0), (None, 4.0)], + k=blosc2.int64(null_storage="mask"), + v=blosc2.float64(null_storage="mask"), + ) + g = t.group_by(["k"], dropna=False, sort=True).agg(total=("v", "sum")) + assert as_list(g["k"]) == [None, 1] + assert as_list(g["total"]) == [6.0, 1.0] + + +def test_group_by_a_mask_value_skips_nulls_not_fills(): + t = table( + [("x", 3), ("x", None), ("y", None)], + s=blosc2.string(max_length=1), + v=blosc2.int64(null_storage="mask"), + ) + g = t.group_by(["s"], sort=True).agg(n=("v", "count"), lo=("v", "min")) + assert as_list(g["n"]) == [1, 0] + # min over a group with no non-null value is a null, not the 0 fill. + assert as_list(g["lo"]) == [3, None] + + +def test_group_by_a_mask_float_value_treats_nan_as_a_value(): + """Decision 6, in the one place it is observable in an aggregate. + + A sentinel float column's NaN *is* its null, so ``sum`` skips it. A mask + column's NaN is data, so it propagates — which is Arrow's answer and + NumPy's. + """ + t = table( + [("x", 1.0), ("x", float("nan")), ("x", None)], + s=blosc2.string(max_length=1), + v=blosc2.float64(null_storage="mask"), + ) + g = t.group_by(["s"]).agg(total=("v", "sum"), n=("v", "count")) + assert np.isnan(g["total"][0]) + assert as_list(g["n"]) == [2] # the NaN counts; the null does not + + +def test_group_by_a_mask_float_key_groups_nan_with_the_nulls(): + """Keys keep NaN-as-missing, so ``dropna`` stays predictable. + + This is the one place a float mask column does *not* follow decision 6, and + deliberately: the rule is about values, and a key that sometimes forms its + own NaN group and sometimes does not would make ``dropna`` unusable. It is + also what the sentinel path does. + """ + t = table( + [(1.0, 1), (float("nan"), 2), (None, 4)], + k=blosc2.float64(null_storage="mask"), + v=blosc2.int64(), + ) + assert len(t.group_by(["k"], dropna=True).agg(total=("v", "sum"))) == 1 + g = t.group_by(["k"], dropna=False, sort=True).agg(total=("v", "sum")) + assert as_list(g["total"]) == [6, 1] + + +@needs_utf8 +def test_group_by_utf8_mask_key_separates_nulls_from_empty_strings(): + t = table( + [("", 1), (None, 2), ("", 4)], + s=blosc2.utf8(null_storage="mask"), + v=blosc2.int64(), + ) + g = t.group_by(["s"], dropna=False, sort=True).agg(total=("v", "sum")) + assert as_list(g["s"]) == [None, ""] + assert as_list(g["total"]) == [2, 5] + + +@pytest.mark.parametrize( + ("spec", "values"), + [ + (blosc2.int8(null_storage="mask"), [3, None, -128, 3]), + (blosc2.uint8(null_storage="mask"), [255, None, 0, 255]), + (blosc2.bool(null_storage="mask"), [True, None, False, True]), + (blosc2.bytes(max_length=2, null_storage="mask"), [b"aa", None, b"", b"aa"]), + pytest.param(utf8_spec(null_storage="mask"), ["aa", None, "", "aa"], marks=needs_utf8), + ], +) +def test_group_by_a_mask_key_of_every_v1_kind(spec, values): + """Each kind's fill is a value some real row could hold, so each needs the recode.""" + t = table([(v, i) for i, v in enumerate(values)], k=spec, v=blosc2.int64()) + distinct = {v for v in values if v is not None} + dropped = t.group_by(["k"], dropna=True, sort=True).agg(total=("v", "sum")) + assert len(dropped) == len(distinct) + assert None not in as_list(dropped["k"]) + kept = t.group_by(["k"], dropna=False, sort=True).agg(total=("v", "sum")) + assert as_list(kept["k"]).count(None) == 1 + assert sum(as_list(kept["total"])) == sum(range(len(values))) + + +def test_group_by_min_over_a_bool_value_column(): + """A storage-independent bug the mask path is what routed into. + + ``min``/``max`` over a group seed an accumulator with the dtype's opposite + identity, and ``bool`` had none -- ``np.full(n, None, dtype=bool)`` is + ``False``, so a min accumulator could never rise above it and every all-True + group reduced to ``False``. Reachable with a plain non-nullable bool column + on any generic-path aggregation; a nullable one used to be ``uint8``, whose + identities are fine, which is why mask storage surfaced it. + """ + t = table( + [("a", True), ("a", True), ("b", False), ("b", True)], + k=blosc2.string(max_length=1), + v=blosc2.bool(), + ) + g = t.group_by(["k"], sort=True).agg(lo=("v", "min"), hi=("v", "max")) + assert as_list(g["lo"]) == [True, False] + assert as_list(g["hi"]) == [True, True] + + +def fast_path_taken(t, keys, **aggs): + """Whether ``group_by(keys).agg(**aggs)`` is served by a fast path.""" + gb = t.group_by(keys) + specs = gb._normalize_aggs((), aggs) + return gb._mask_null_columns(specs), gb._try_fast_paths(specs, False) is not None + + +def test_null_free_mask_columns_keep_every_groupby_fast_path(): + """Decision 9 again: the deoptimization is scoped to columns with nulls.""" + t = table( + [(1, 2), (1, 3), (2, 4)], + k=blosc2.int64(null_storage="mask"), + v=blosc2.int64(null_storage="mask"), + ) + assert fast_path_taken(t, ["k"], total=("v", "sum")) == ([], True) + + +def test_a_mask_value_column_keeps_the_dense_single_key_path(): + """It is plain NumPy and already asks ``_null_mask``; it just needs the sidecar. + + Only the Cython kernels have to give up, because a kernel is handed a + ``skip_nan`` flag rather than a validity array. Keeping this path matters: + deferring a mask value column all the way to the generic hash-and-merge path + cost ~4.5x on a 2M-row sum. + """ + t = table( + [(1, 2), (1, 3), (2, 4)], + k=blosc2.int64(), + v=blosc2.int64(null_storage="mask"), + ) + t["v"][0] = None + assert fast_path_taken(t, ["k"], total=("v", "sum")) == (["v"], True) + + +def test_a_mask_key_column_leaves_the_fast_paths(): + """Its recoded chunk is a ``_CodedKeyChunk``, not an array of dense ints.""" + t = table( + [(1, 2), (1, 3), (2, 4)], + k=blosc2.int64(null_storage="mask"), + v=blosc2.int64(), + ) + t["k"][0] = None + assert fast_path_taken(t, ["k"], total=("v", "sum")) == (["k"], False) diff --git a/tests/ctable/test_null_migration.py b/tests/ctable/test_null_migration.py new file mode 100644 index 000000000..cef2612af --- /dev/null +++ b/tests/ctable/test_null_migration.py @@ -0,0 +1,558 @@ +####################################################################### +# Copyright (c) 2019-present, Blosc Development Team +# All rights reserved. +# +# SPDX-License-Identifier: BSD-3-Clause +####################################################################### + +"""``convert_nulls``: moving a column's nulls between channels (Phase 8). + +Two guarantees frame everything here. **Nothing auto-migrates** -- opening, +copying and saving a table all preserve each column's ``null_storage``, so this +is the only thing that changes it. And a conversion either happens or does not: +every reason one can fail is decided before a byte is written, so a refusal +leaves the table exactly as it was rather than half converted. + +The crash-safety ordering is asserted directly, because it is the whole argument +for why an in-place conversion is allowed at all: each intermediate state on disk +has to read correctly under *some* schema, and the schema is only ever the last +thing to move. +""" + +from __future__ import annotations + +import dataclasses + +import numpy as np +import pytest +from utf8_compat import needs_utf8 + +import blosc2 + + +def annotation_for(spec): + if isinstance(spec, (blosc2.schema.NDArraySpec, blosc2.schema.timestamp)): + return object + return spec.python_type + + +def table(rows, capacity=64, urlpath=None, **cols): + Row = dataclasses.make_dataclass( + "MigrateRow", [(n, annotation_for(s), blosc2.field(s)) for n, s in cols.items()] + ) + kwargs = {"urlpath": str(urlpath), "mode": "w"} if urlpath is not None else {} + t = blosc2.CTable(Row, expected_size=max(capacity, len(rows)), **kwargs) + if rows: + t.extend(rows) + return t + + +def one_col(values, spec, capacity=64, urlpath=None): + return table([(v,) for v in values], capacity=capacity, urlpath=urlpath, a=spec) + + +def as_list(col): + """A column's live values with nulls spelled ``None``.""" + null = col.is_null() + return [None if null[i] else _scalar(col[i]) for i in range(len(col))] + + +def _scalar(value): + return value.item() if hasattr(value, "item") else value + + +#: Every V1 kind, with a value it can hold and the sentinel its policy picks. +V1_KINDS = [ + ("int64", blosc2.int64, {}, [5, None, 1]), + ("int8", blosc2.int8, {}, [5, None, 1]), + ("uint8", blosc2.uint8, {}, [5, None, 1]), + ("float64", blosc2.float64, {}, [1.5, None, 2.5]), + ("bool", blosc2.bool, {}, [True, None, False]), + ("string", blosc2.string, {"max_length": 4}, ["ab", None, "cd"]), + ("bytes", blosc2.bytes, {"max_length": 4}, [b"ab", None, b"cd"]), + pytest.param("utf8", blosc2.utf8, {}, ["ab", None, "cdefgh"], marks=needs_utf8), + ( + "timestamp", + blosc2.timestamp, + {}, + [np.datetime64("2020-01-01"), None, np.datetime64("2021-06-01")], + ), +] + + +# --------------------------------------------------------------------------- +# Round trip, both directions, every kind +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize(("label", "factory", "kw", "values"), V1_KINDS) +def test_mask_to_sentinel_to_mask_preserves_the_data(label, factory, kw, values): + """Both directions, back to back, for each kind that has two channels. + + The sentinel leg is the lossy one by construction -- it has to steal a value + from the range -- so what this pins is that the *data present here* survives, + not that any data would. ``test_to_sentinel_refuses_*`` covers the rest. + """ + mask = one_col(values, factory(null_storage="mask", **kw)) + assert mask["a"].null_storage == "mask" + + sent = mask.convert_nulls("a", to="sentinel") + assert sent["a"].null_storage == "sentinel" + assert sent["a"].null_value is not None + assert as_list(sent["a"]) == as_list(mask["a"]) + + back = sent.convert_nulls("a", to="mask") + assert back["a"].null_storage == "mask" + assert as_list(back["a"]) == as_list(mask["a"]) + assert back["a"].dtype == mask["a"].dtype or label in ("string", "bytes") + + +def test_conversion_leaves_the_source_untouched(): + t = one_col([5, None, 1], blosc2.int64(null_storage="mask")) + converted = t.convert_nulls("a", to="sentinel") + assert t["a"].null_storage == "mask" + assert t["a"].null_value is None + assert converted["a"].null_storage == "sentinel" + + +def test_implicit_sweep_converts_every_convertible_column(): + t = table( + [(5, "ab", 1), (None, None, 2)], + k=blosc2.int64(null_storage="mask"), + s=blosc2.string(max_length=2, null_storage="mask"), + plain=blosc2.int64(), + ) + sent = t.convert_nulls(to="sentinel") + assert sent["k"].null_storage == "sentinel" + assert sent["s"].null_storage == "sentinel" + assert sent["plain"].null_storage == "none" # not nullable: nothing to convert + + +def test_implicit_sweep_skips_what_it_cannot_convert(): + """A dictionary column has one representation, so a sweep passes over it.""" + t = table( + [("x", 5), (None, None)], + d=blosc2.dictionary(), + k=blosc2.int64(null_storage="mask"), + ) + sent = t.convert_nulls(to="sentinel") + assert sent["d"].null_storage == "code" + assert sent["k"].null_storage == "sentinel" + + +def test_converting_is_idempotent(): + t = one_col([5, None, 1], blosc2.int64(null_storage="mask")) + once = t.convert_nulls("a", to="mask") + twice = once.convert_nulls("a", to="mask") + assert twice["a"].null_storage == "mask" + assert as_list(twice["a"]) == [5, None, 1] + + +# --------------------------------------------------------------------------- +# A null-free column converts as a pure schema update +# --------------------------------------------------------------------------- + + +def test_null_free_sentinel_column_writes_no_sidecar(): + """Decision 9 reaching migration: an absent sidecar already says all-valid.""" + t = one_col([1, 2, 3], blosc2.int64(nullable=True, null_value=-1)) + converted = t.convert_nulls("a", to="mask") + assert converted["a"].null_storage == "mask" + assert converted._null_mask("a") is None + assert converted["a"].is_null().tolist() == [False, False, False] + assert as_list(converted["a"]) == [1, 2, 3] + + +def test_null_free_mask_column_needs_no_value_rewrite(): + t = one_col([1, 2, 3], blosc2.int64(null_storage="mask")) + assert t._null_mask("a") is None + converted = t.convert_nulls("a", to="sentinel") + assert converted["a"].null_storage == "sentinel" + assert as_list(converted["a"]) == [1, 2, 3] + + +def test_converting_drops_the_sidecar_on_the_way_to_sentinel(): + t = one_col([5, None, 1], blosc2.int64(null_storage="mask")) + converted = t.convert_nulls("a", to="sentinel") + assert converted._null_mask("a") is None + assert converted["a"].is_null().tolist() == [False, True, False] + + +# --------------------------------------------------------------------------- +# to="sentinel" refuses what it cannot represent +# --------------------------------------------------------------------------- + + +def test_to_sentinel_refuses_a_full_range_int8(): + """The case that motivated the design, refused rather than silently lossy. + + ``int8`` using all 256 values has nothing left to reserve, which is exactly + why mask storage exists; converting back would relabel a real ``-128``. + """ + t = one_col([*range(-128, 128), None], blosc2.int8(null_storage="mask"), capacity=300) + with pytest.raises(ValueError, match="already contains -128"): + t.convert_nulls("a", to="sentinel") + + +@needs_utf8 +def test_to_sentinel_refuses_utf8_holding_the_sentinel_string(): + t = one_col(["__BLOSC2_NULL__", None], blosc2.utf8(null_storage="mask")) + with pytest.raises(ValueError, match="already contains"): + t.convert_nulls("a", to="sentinel") + + +@needs_utf8 +def test_to_sentinel_accepts_a_different_sentinel_instead(): + """The refusal names the offending value, and a free one is accepted.""" + t = one_col(["__BLOSC2_NULL__", None], blosc2.utf8(null_storage="mask")) + converted = t.convert_nulls("a", to="sentinel", null_value="\x00\x01") + assert converted["a"].null_value == "\x00\x01" + assert as_list(converted["a"]) == ["__BLOSC2_NULL__", None] + + +def test_to_sentinel_refuses_complex(): + """No value can be spared from the complex plane, so it is mask or nothing.""" + t = one_col([1 + 2j, None], blosc2.complex128(null_storage="mask")) + with pytest.raises(ValueError, match="no value can be reserved"): + t.convert_nulls("a", to="sentinel") + + +def test_a_nan_already_present_does_not_block_the_nan_sentinel(): + """Folding NaN into "null" is the documented semantic change, not data loss. + + A sentinel float column has always spelled its nulls ``NaN``; converting a + mask column that contains a real NaN therefore *changes what that row means*, + which decision 6 covers, rather than losing a value the way an integer + sentinel collision would. + """ + t = one_col([1.0, float("nan"), None], blosc2.float64(null_storage="mask")) + converted = t.convert_nulls("a", to="sentinel") + assert converted["a"].is_null().tolist() == [False, True, True] + + +def test_the_refusal_leaves_the_column_alone(): + t = one_col([*range(-128, 128), None], blosc2.int8(null_storage="mask"), capacity=300) + with pytest.raises(ValueError): + t.convert_nulls("a", to="sentinel") + assert t["a"].null_storage == "mask" + assert t["a"].null_value is None + assert t["a"][:-1].tolist() == list(range(-128, 128)) + + +# --------------------------------------------------------------------------- +# Naming a column that cannot convert is an error +# --------------------------------------------------------------------------- + + +def test_naming_a_dictionary_column_raises(): + t = one_col(["x", None], blosc2.dictionary()) + with pytest.raises(ValueError, match="only representation its kind has"): + t.convert_nulls("a", to="mask") + + +def test_naming_a_non_nullable_column_raises(): + t = one_col([1, 2], blosc2.int64()) + with pytest.raises(ValueError, match="not nullable"): + t.convert_nulls("a", to="mask") + + +def test_naming_an_unknown_column_raises(): + t = one_col([1, 2], blosc2.int64()) + with pytest.raises(KeyError, match="not found"): + t.convert_nulls("nope") + + +def test_bad_to_value_raises(): + t = one_col([1, 2], blosc2.int64()) + with pytest.raises(ValueError, match="must be 'mask' or 'sentinel'"): + t.convert_nulls(to="bitmap") + + +def test_null_value_with_to_mask_raises(): + t = one_col([1, 2], blosc2.int64()) + with pytest.raises(ValueError, match="only applies to to='sentinel'"): + t.convert_nulls(to="mask", null_value=7) + + +def test_null_value_across_several_columns_raises(): + """One value cannot be right for several kinds, so it has to name a column.""" + t = table( + [(5, "ab"), (None, None)], + k=blosc2.int64(null_storage="mask"), + s=blosc2.string(max_length=2, null_storage="mask"), + ) + with pytest.raises(ValueError, match="applies to a single column"): + t.convert_nulls(to="sentinel", null_value=-1) + + +# --------------------------------------------------------------------------- +# ndarray columns +# --------------------------------------------------------------------------- + + +def ndarray_table(dtype, item_shape, rows, **spec_kw): + spec = blosc2.ndarray(dtype=dtype, item_shape=item_shape, **spec_kw) + Row = dataclasses.make_dataclass("NdRow", [("v", object, blosc2.field(spec))]) + t = blosc2.CTable(Row, expected_size=16) + t.extend([(r,) for r in rows]) + return t + + +def test_ndarray_column_converts_both_ways(): + t = ndarray_table(np.int32, (3,), [[1, 2, 3], None, [4, 5, 6]], null_storage="mask") + sent = t.convert_nulls("v", to="sentinel") + assert sent["v"].is_null().tolist() == [False, True, False] + # The old rule: a row is null only when *every* element holds the sentinel. + assert sent["v"][:][1].tolist() == [np.iinfo(np.int32).min] * 3 + back = sent.convert_nulls("v", to="mask") + assert back["v"].is_null().tolist() == [False, True, False] + assert back["v"][:][0].tolist() == [1, 2, 3] + + +def test_bool_ndarray_column_changes_dtype_both_ways(): + t = ndarray_table(np.bool_, (2,), [[True, False], None], null_storage="mask") + assert t["v"].dtype == np.dtype(np.bool_) + sent = t.convert_nulls("v", to="sentinel") + assert sent["v"].dtype == np.dtype(np.uint8) + assert sent["v"][:].tolist() == [[1, 0], [255, 255]] + back = sent.convert_nulls("v", to="mask") + assert back["v"].dtype == np.dtype(np.bool_) + assert back["v"].is_null().tolist() == [False, True] + assert back["v"][:][0].tolist() == [True, False] + + +def test_a_declared_uint8_ndarray_column_is_not_mistaken_for_a_widened_bool(): + """The unflip must undo only the flip it made. + + ``bool`` columns are physically ``uint8`` under sentinel storage, and mask + storage undoes that -- but a column whose *declared* dtype is ``uint8`` holds + real byte values, and turning it into ``np.bool_`` would truncate every one + of them to a flag. Keyed off a recorded flag rather than off the dtype. + """ + spec = blosc2.ndarray(dtype=np.uint8, item_shape=(2,), nullable=True, null_storage="mask") + Row = dataclasses.make_dataclass("U8Row", [("v", object, blosc2.field(spec))]) + t = blosc2.CTable(Row, expected_size=8) + t.extend([([7, 200],), (None,)]) + assert t["v"].dtype == np.dtype(np.uint8) + assert t["v"][:][0].tolist() == [7, 200] + + +# --------------------------------------------------------------------------- +# Persistence, in place, and the crash ordering +# --------------------------------------------------------------------------- + + +def test_inplace_conversion_survives_a_reopen(tmp_path): + t = one_col([5, -1, 1], blosc2.int64(nullable=True, null_value=-1), urlpath=tmp_path / "t.b2t") + assert t.convert_nulls("a", to="mask", inplace=True) is t + assert t["a"].null_storage == "mask" + + reopened = blosc2.open(str(tmp_path / "t.b2t")) + assert reopened["a"].null_storage == "mask" + assert reopened._null_mask("a") is not None + assert as_list(reopened["a"]) == [5, None, 1] + + +def test_inplace_conversion_back_to_sentinel_survives_a_reopen(tmp_path): + t = one_col([5, None, 1], blosc2.int64(null_storage="mask"), urlpath=tmp_path / "t.b2t") + t.convert_nulls("a", to="sentinel", inplace=True) + reopened = blosc2.open(str(tmp_path / "t.b2t")) + assert reopened["a"].null_storage == "sentinel" + assert reopened._null_mask("a") is None + assert as_list(reopened["a"]) == [5, None, 1] + + +def test_inplace_on_a_persistent_table_refuses_a_dtype_change(tmp_path): + """No ordering of "replace the array" and "update the schema" is crash-safe.""" + t = one_col([1, 255, 0], blosc2.bool(nullable=True, null_value=255), urlpath=tmp_path / "t.b2t") + with pytest.raises(ValueError, match="changes its physical dtype"): + t.convert_nulls("a", to="mask", inplace=True) + assert t["a"].null_storage == "sentinel" + assert t["a"].dtype == np.dtype(np.uint8) + + +def test_the_same_column_converts_fine_out_of_place(tmp_path): + t = one_col([1, 255, 0], blosc2.bool(nullable=True, null_value=255), urlpath=tmp_path / "t.b2t") + converted = t.convert_nulls("a", to="mask") + assert converted["a"].dtype == np.dtype(np.bool_) + assert as_list(converted["a"]) == [True, None, False] + landed = converted.copy(urlpath=str(tmp_path / "out.b2d")) + assert landed["a"].null_storage == "mask" + assert as_list(landed["a"]) == [True, None, False] + + +def test_inplace_on_an_in_memory_table_may_change_dtype(): + """There is no crash window in memory, so the restriction does not apply.""" + t = one_col([1, 255, 0], blosc2.bool(nullable=True, null_value=255)) + t.convert_nulls("a", to="mask", inplace=True) + assert t["a"].dtype == np.dtype(np.bool_) + assert as_list(t["a"]) == [True, None, False] + + +def test_inplace_refuses_a_view(): + t = one_col([5, None, 1], blosc2.int64(null_storage="mask")) + with pytest.raises(ValueError, match="view"): + t.where("a > 0").convert_nulls("a", to="sentinel", inplace=True) + + +def test_inplace_refuses_a_read_only_table(tmp_path): + one_col([5, -1, 1], blosc2.int64(nullable=True, null_value=-1), urlpath=tmp_path / "t.b2t") + ro = blosc2.open(str(tmp_path / "t.b2t"), mode="r") + with pytest.raises(ValueError, match="read-only"): + ro.convert_nulls("a", to="mask", inplace=True) + + +def test_an_orphan_sidecar_still_reads_as_sentinel(tmp_path): + """The crash-safety argument for ``to="mask"``, in the state it protects. + + Step 1 writes the complete sidecar; step 2 flips the schema. A crash in + between leaves a ``.notnull`` key beside a schema that still says + ``sentinel`` -- and a sentinel column never opens a sidecar, so the table + reads exactly as it did before the conversion started. + """ + t = one_col([5, -1, 1], blosc2.int64(nullable=True, null_value=-1), urlpath=tmp_path / "t.b2t") + # Simulate an interruption after step 1 by writing the sidecar by hand. + mask = t._ensure_null_mask("a") + mask[:] = True + mask[1] = False + del t + + reopened = blosc2.open(str(tmp_path / "t.b2t"), mode="a") + assert reopened["a"].null_storage == "sentinel" + assert reopened._storage.has_null_mask("a") # the orphan is on disk + assert as_list(reopened["a"]) == [5, None, 1] # and is not consulted + # Finishing the conversion is then just the schema update. + reopened.convert_nulls("a", to="mask", inplace=True) + assert as_list(reopened["a"]) == [5, None, 1] + + +def test_a_mask_column_whose_nulls_still_hold_the_sentinel_reads_correctly(tmp_path): + """The other side of the same window: schema flipped, values not normalized. + + Step 3 rewrites the null slots to the fill and is purely cosmetic -- what + sits under ``valid=False`` is unobservable through the Column API, and the + fill is explicitly not part of the format contract. + """ + t = one_col([5, None, 1], blosc2.int64(null_storage="mask"), urlpath=tmp_path / "t.b2t") + t._cols["a"][1] = -12345 # as if step 3 had never run + assert t["a"].is_null().tolist() == [False, True, False] + assert t["a"].null_count() == 1 + assert t["a"].fillna(0).tolist() == [5, 0, 1] + assert t["a"].min() == 1 + + +# --------------------------------------------------------------------------- +# Nothing auto-migrates +# --------------------------------------------------------------------------- + + +def test_copy_preserves_each_columns_null_storage(): + t = table( + [(5, "ab"), (None, "__BLOSC2_NULL__")], + k=blosc2.int64(null_storage="mask"), + s=blosc2.string(max_length=2, nullable=True, null_value="__BLOSC2_NULL__"), + ) + c = t.copy() + assert c["k"].null_storage == "mask" + assert c["s"].null_storage == "sentinel" + + +def test_saving_preserves_null_storage_under_a_mask_default_policy(tmp_path): + """A policy governs *creation*; it must never rewrite what already exists.""" + t = one_col([5, -1, 1], blosc2.int64(nullable=True, null_value=-1)) + with blosc2.null_policy(blosc2.NullPolicy(null_storage="mask")): + t.save(str(tmp_path / "t.b2d")) + reopened = blosc2.open(str(tmp_path / "t.b2d")) + assert reopened["a"].null_storage == "sentinel" + assert reopened["a"].null_value == -1 + + +def test_opening_an_old_table_changes_nothing(tmp_path): + t = one_col([1, 255, 0], blosc2.bool(nullable=True, null_value=255), urlpath=tmp_path / "t.b2t") + del t + with blosc2.null_policy(blosc2.NullPolicy(null_storage="mask")): + reopened = blosc2.open(str(tmp_path / "t.b2t")) + assert reopened["a"].null_storage == "sentinel" + assert reopened["a"].dtype == np.dtype(np.uint8) + assert reopened["a"].null_value == 255 + + +# --------------------------------------------------------------------------- +# Detection +# --------------------------------------------------------------------------- + + +def test_null_storage_is_reported_per_column(): + t = table( + [(5, "ab", 1, "x")], + k=blosc2.int64(null_storage="mask"), + s=blosc2.string(max_length=2, nullable=True, null_value="__BLOSC2_NULL__"), + plain=blosc2.int64(), + d=blosc2.dictionary(), + ) + assert t["k"].null_storage == "mask" + assert t["s"].null_storage == "sentinel" + assert t["plain"].null_storage == "none" + assert t["d"].null_storage == "code" + + +def test_info_tags_each_column_with_where_its_nulls_live(): + """What you read to decide whether a table needs converting.""" + t = table( + [(5, "ab", 1)], + k=blosc2.int64(null_storage="mask"), + s=blosc2.string(max_length=2, nullable=True, null_value="__BLOSC2_NULL__"), + plain=blosc2.int64(), + ) + summary = dict(t.info_items)["columns"] + assert "nullable[mask]" in str(summary["k"]) + assert "nullable[sentinel]" in str(summary["s"]) + assert "nullable" not in str(summary["plain"]) + + +def test_column_info_reports_whether_a_sidecar_exists_yet(): + """The visible difference between "nullable" and "has ever held a null".""" + t = one_col([1, 2], blosc2.int64(null_storage="mask")) + assert dict(t["a"].info_items)["null_sidecar"] is False + t["a"][0] = None + assert dict(t["a"].info_items)["null_sidecar"] is True + + +# --------------------------------------------------------------------------- +# The converted table behaves like a natively created one +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize("to", ["mask", "sentinel"]) +def test_a_converted_column_still_sorts_groups_and_queries(to): + """A converted column is indistinguishable from a natively created one.""" + if to == "mask": + t = one_col([5, -1, 1, 9, -1, 2], blosc2.int64(nullable=True, null_value=-1)) + else: + t = one_col([5, None, 1, 9, None, 2], blosc2.int64(null_storage="mask")) + converted = t.convert_nulls("a", to=to) + + assert as_list(converted.sort_by("a")["a"]) == [1, 2, 5, 9, None, None] + assert converted["a"].null_count() == 2 + assert converted["a"].min() == 1 + assert len(converted.where("a < 3")) == 2 + grouped = converted.group_by(["a"], dropna=True, sort=True).agg(n=("a", "count")) + assert len(grouped) == 4 + + +def test_a_converted_column_round_trips_through_arrow(): + """The point of converting to mask in the first place.""" + pa = pytest.importorskip("pyarrow") + t = one_col([True, 255, False], blosc2.bool(nullable=True, null_value=255)) + converted = t.convert_nulls("a", to="mask") + arrow = converted.to_arrow() + assert arrow.column("a").type == pa.bool_() + assert arrow.column("a").to_pylist() == [True, None, False] + + +def test_extending_a_converted_column_accepts_none(): + t = one_col([5, -1], blosc2.int64(nullable=True, null_value=-1)) + converted = t.convert_nulls("a", to="mask") + converted.append((None,)) + converted.extend([(7,), (None,)]) + assert as_list(converted["a"]) == [5, None, None, 7, None] diff --git a/tests/ctable/test_null_persistence.py b/tests/ctable/test_null_persistence.py new file mode 100644 index 000000000..d6715e6bd --- /dev/null +++ b/tests/ctable/test_null_persistence.py @@ -0,0 +1,463 @@ +####################################################################### +# Copyright (c) 2019-present, Blosc Development Team +# All rights reserved. +# +# SPDX-License-Identifier: BSD-3-Clause +####################################################################### + +"""The ``.notnull`` validity sidecar, through every persistence path (Phase 3). + +A mask-storage nullable column keeps its nullity in a bool NDArray beside the +values. Nothing reads or writes one through the public API yet -- ``extend``, +``is_null`` and friends arrive in Phase 4 -- so these tests build the sidecar +by hand via ``_ensure_null_mask`` and check that every path that moves, grows, +shrinks or serializes a column carries it along unchanged. + +The load-bearing invariants here: + +* **An absent sidecar means all-valid.** A nullable column that has never + been given a null has no ``.notnull`` key at all, and must not acquire one + just by being saved, copied or reopened. +* **Chunk pinning.** The sidecar shares its column's row grid, so mask and + values never need re-aligning on a paired read. +""" + +from __future__ import annotations + +import dataclasses +import os + +import numpy as np +import pytest +from utf8_compat import needs_utf8 + +import blosc2 +from blosc2.ctable_storage import _NOTNULL_SUFFIX, FileTableStorage + +# --------------------------------------------------------------------------- +# Fixtures / helpers +# --------------------------------------------------------------------------- + +NULL_POSITIONS = (3, 7, 41) + + +def make_row_type(): + return dataclasses.make_dataclass( + "MaskRow", + [ + ("a", int, blosc2.field(blosc2.int64(null_storage="mask"))), + ("b", float, blosc2.field(blosc2.float64())), + ("c", str, blosc2.field(blosc2.string(max_length=8, null_storage="mask"))), + ], + ) + + +def make_table(n_rows=50, capacity=100, *, with_nulls=True, urlpath=None): + """A table with two mask columns; only ``a`` is given nulls.""" + kwargs = {"expected_size": capacity} + if urlpath is not None: + kwargs |= {"urlpath": urlpath, "mode": "w"} + t = blosc2.CTable(make_row_type(), **kwargs) + t.extend([(i, float(i), f"s{i}") for i in range(n_rows)]) + if with_nulls: + mask = t._ensure_null_mask("a") + for pos in NULL_POSITIONS: + if pos < n_rows: + mask[pos] = False + return t + + +def null_positions(table, name="a", n=None): + """Physical positions the sidecar marks null, or ``None`` if there is none.""" + mask = table._null_mask(name) + if mask is None: + return None + n = table._n_rows if n is None else n + return np.flatnonzero(~mask[:n]).tolist() + + +def sidecar_key_exists(table, name): + storage = table._storage + return storage.has_null_mask(name) + + +# --------------------------------------------------------------------------- +# Decision 9: an absent sidecar means all-valid +# --------------------------------------------------------------------------- + + +def test_a_fresh_mask_column_has_no_sidecar(): + t = make_table(with_nulls=False) + assert t._null_mask_names == ["a", "c"] + assert t._null_mask("a") is None + assert t._null_mask("c") is None + assert t._existing_null_masks() == {} + + +def test_non_nullable_columns_are_not_listed(): + t = make_table(with_nulls=False) + assert "b" not in t._null_mask_names + assert t._null_mask("b") is None + + +def test_sentinel_columns_are_not_listed(): + Row = dataclasses.make_dataclass("SentinelRow", [("v", int, blosc2.field(blosc2.int64(null_value=-1)))]) + t = blosc2.CTable(Row, expected_size=8) + assert t._null_mask_names == [] + + +def test_materializing_the_sidecar_starts_all_valid(): + t = make_table(with_nulls=False) + mask = t._ensure_null_mask("a") + assert mask.shape == (len(t._valid_rows),) + assert mask.dtype == np.dtype(np.bool_) + # An explicit all-True fill, not zeros: a fresh sidecar must say exactly + # what its absence said. + assert bool(np.asarray(mask[:]).all()) + + +def test_ensure_null_mask_is_idempotent(): + t = make_table(with_nulls=False) + first = t._ensure_null_mask("a") + first[5] = False + second = t._ensure_null_mask("a") + assert second is first + assert null_positions(t) == [5] + + +def test_a_null_free_column_writes_no_sidecar_to_disk(tmp_path): + path = str(tmp_path / "clean.b2d") + make_table(with_nulls=False).save(path) + reopened = blosc2.CTable.open(path) + try: + assert not sidecar_key_exists(reopened, "a") + assert reopened._null_mask("a") is None + finally: + reopened.close() + + +def test_only_the_column_given_nulls_gets_a_sidecar(tmp_path): + path = str(tmp_path / "one.b2d") + make_table().save(path) + reopened = blosc2.CTable.open(path) + try: + assert sidecar_key_exists(reopened, "a") + assert not sidecar_key_exists(reopened, "c") + finally: + reopened.close() + + +# --------------------------------------------------------------------------- +# Chunk pinning +# --------------------------------------------------------------------------- + + +def test_sidecar_shares_its_column_row_grid(): + t = make_table() + col = t._cols["a"] + mask = t._null_mask("a") + assert mask.chunks[0] == col.chunks[0] + assert mask.blocks[0] == col.blocks[0] + + +def test_sidecar_grid_survives_a_chunk_override(tmp_path): + """``copy(chunks=…)`` reblocks the column; the sidecar must follow it.""" + t = make_table() + copied = t.copy(urlpath=str(tmp_path / "ovr.b2d"), chunks=16, blocks=8) + try: + assert copied._cols["a"].chunks[0] == 16 + assert copied._null_mask("a").chunks[0] == 16 + assert null_positions(copied) == list(NULL_POSITIONS) + finally: + copied.close() + + +@needs_utf8 +def test_utf8_sidecar_falls_back_to_the_table_grid(): + """utf8 offsets carry ``n + 1`` entries, so they are not the grid to pin to.""" + Row = dataclasses.make_dataclass("Utf8Row", [("u", str, blosc2.field(blosc2.utf8(null_storage="mask")))]) + t = blosc2.CTable(Row, expected_size=64) + t.extend([(f"v{i}",) for i in range(64)]) + mask = t._ensure_null_mask("u") + assert mask.chunks[0] == t._valid_rows.chunks[0] + assert mask.blocks[0] == t._valid_rows.blocks[0] + + +# --------------------------------------------------------------------------- +# Persistence round-trips +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize("suffix", [".b2d", ".b2z"]) +def test_sidecar_survives_save_and_open(tmp_path, suffix): + path = str(tmp_path / f"t{suffix}") + make_table().save(path) + reopened = blosc2.CTable.open(path) + try: + assert null_positions(reopened) == list(NULL_POSITIONS) + finally: + reopened.close() + + +def test_sidecar_survives_mmap_open(tmp_path): + path = str(tmp_path / "m.b2z") + make_table().save(path) + reopened = blosc2.CTable.open(path, mmap_mode="r") + try: + assert null_positions(reopened) == list(NULL_POSITIONS) + finally: + reopened.close() + + +def test_sidecar_survives_ctable_load(tmp_path): + path = str(tmp_path / "l.b2d") + make_table().save(path) + loaded = blosc2.CTable.load(path) + assert null_positions(loaded) == list(NULL_POSITIONS) + # load() rebuilds every array in RAM, so the pinning must be re-established + # rather than inherited. + assert loaded._null_mask("a").chunks[0] == loaded._cols["a"].chunks[0] + + +def test_sidecar_survives_cframe_round_trip(): + original = make_table() + rebuilt = blosc2.ctable_from_cframe(original.to_cframe()) + assert null_positions(rebuilt) == list(NULL_POSITIONS) + + +def test_cframe_of_a_null_free_table_carries_no_sidecar(): + rebuilt = blosc2.ctable_from_cframe(make_table(with_nulls=False).to_cframe()) + assert rebuilt._null_mask("a") is None + + +def test_sidecar_survives_inline_treestore(tmp_path): + path = str(tmp_path / "tree.b2d") + with blosc2.TreeStore(path, mode="w") as store: + store["/tbl"] = make_table() + with blosc2.TreeStore(path, mode="r") as store: + assert null_positions(store["/tbl"]) == list(NULL_POSITIONS) + + +def test_sidecar_survives_physical_pack_and_unpack(tmp_path): + """``to_b2z``/``to_b2d`` zip the leaves as-is; the sidecar is one of them.""" + src_path = str(tmp_path / "src.b2d") + make_table().save(src_path) + + packed = str(tmp_path / "packed.b2z") + src = blosc2.CTable.open(src_path, mode="r") + try: + src.to_b2z(packed) + finally: + src.close() + + zipped = blosc2.CTable.open(packed, mode="r") + try: + assert null_positions(zipped) == list(NULL_POSITIONS) + unpacked_path = str(tmp_path / "unpacked.b2d") + zipped.to_b2d(unpacked_path) + finally: + zipped.close() + + unpacked = blosc2.CTable.open(unpacked_path) + try: + assert null_positions(unpacked) == list(NULL_POSITIONS) + finally: + unpacked.close() + + +@pytest.mark.parametrize("compact", [True, False]) +def test_sidecar_survives_in_memory_copy(compact): + copied = make_table().copy(compact=compact) + assert null_positions(copied) == list(NULL_POSITIONS) + + +def test_copy_of_a_null_free_table_stays_sidecar_free(): + assert make_table(with_nulls=False).copy()._null_mask("a") is None + + +def test_view_materialization_remaps_the_sidecar(): + t = make_table() + materialized = t.where("b >= 5").copy() + # Physical rows 0-4 are filtered out, so physical nulls 7 and 41 land at + # logical 2 and 36; physical 3 is not in the view at all. + assert null_positions(materialized) == [2, 36] + + +def test_a_view_shares_its_base_sidecar_cache(): + t = make_table() + view = t.where("b >= 5") + assert view._null_masks is t._null_masks + assert view._null_mask("a") is t._null_mask("a") + + +# --------------------------------------------------------------------------- +# Capacity management +# --------------------------------------------------------------------------- + + +def test_grow_extends_the_sidecar_as_all_valid(): + t = make_table(n_rows=100, capacity=100) + before = len(t._valid_rows) + t.extend([(i, float(i), f"s{i}") for i in range(100, 150)]) + mask = t._null_mask("a") + assert mask.shape[0] == len(t._valid_rows) > before + # resize() zero-fills, i.e. *invalid*; the new tail must be corrected. + assert bool(np.asarray(mask[before:]).all()) + assert null_positions(t) == list(NULL_POSITIONS) + + +def test_repeated_grow_cycles_keep_the_sidecar_aligned(): + t = make_table(n_rows=20, capacity=20) + for start in range(20, 200, 20): + t.extend([(i, float(i), f"s{i}") for i in range(start, start + 20)]) + mask = t._null_mask("a") + assert mask.shape[0] == len(t._valid_rows) + assert null_positions(t) == [p for p in NULL_POSITIONS if p < 20] + assert int((~np.asarray(mask[: t._n_rows])).sum()) == 2 + + +def test_trim_capacity_shrinks_the_sidecar(): + t = make_table(n_rows=50, capacity=100) + t.trim_capacity() + mask = t._null_mask("a") + assert mask.shape == (50,) + assert null_positions(t) == list(NULL_POSITIONS) + + +def test_capacity_paths_find_an_unopened_sidecar(tmp_path): + """A reopened table has opened no sidecar; grow and trim must still find it. + + Iterating only the *materialized* sidecars would silently leave this one + at its old length, out of step with its column. + """ + path = str(tmp_path / "trim.b2d") + make_table(n_rows=50, capacity=100).save(path) + reopened = blosc2.CTable.open(path, mode="a") + try: + assert reopened._null_masks.materialized() == {} + reopened.extend([(i, float(i), f"s{i}") for i in range(50, 90)]) + assert reopened._null_mask("a").shape[0] == len(reopened._valid_rows) > 50 + reopened.trim_capacity() + assert reopened._null_mask("a").shape == (90,) + assert null_positions(reopened) == list(NULL_POSITIONS) + finally: + reopened.close() + + +def test_compact_gathers_the_sidecar_with_its_column(): + t = make_table() + t.delete(0) + t.compact() + # Every surviving row shifts down one, so the nulls do too. + assert null_positions(t) == [p - 1 for p in NULL_POSITIONS] + + +def test_compact_leaves_the_freed_tail_valid(): + t = make_table() + t.delete(0) + t.compact() + mask = np.asarray(t._null_mask("a")[:]) + assert bool(mask[t._n_rows :].all()) + + +# --------------------------------------------------------------------------- +# Column-level mutation +# --------------------------------------------------------------------------- + + +def test_drop_column_removes_the_sidecar_key(tmp_path): + path = str(tmp_path / "drop.b2d") + make_table().save(path) + table = blosc2.CTable.open(path, mode="a") + try: + key = table._storage._col_key("a") + _NOTNULL_SUFFIX + assert key in table._storage._open_store() + table.drop_column("a") + assert key not in table._storage._open_store() + assert table._null_mask_names == ["c"] + finally: + table.close() + + +@pytest.mark.parametrize("persistent", [True, False]) +def test_rename_column_carries_the_sidecar(tmp_path, persistent): + if persistent: + path = str(tmp_path / "ren.b2d") + make_table().save(path) + table = blosc2.CTable.open(path, mode="a") + else: + table = make_table() + try: + table.rename_column("a", "renamed") + assert null_positions(table, "renamed") == list(NULL_POSITIONS) + assert table._null_mask("a") is None + finally: + if persistent: + table.close() + + +def test_renamed_sidecar_survives_reopen(tmp_path): + path = str(tmp_path / "ren2.b2d") + make_table().save(path) + table = blosc2.CTable.open(path, mode="a") + try: + table.rename_column("a", "renamed") + finally: + table.close() + reopened = blosc2.CTable.open(path) + try: + assert null_positions(reopened, "renamed") == list(NULL_POSITIONS) + finally: + reopened.close() + + +# --------------------------------------------------------------------------- +# Storage-backend surface +# --------------------------------------------------------------------------- + + +def test_storage_reports_absence_before_creation(tmp_path): + path = str(tmp_path / "backend.b2d") + table = make_table(with_nulls=False, urlpath=path) + try: + assert table._storage.has_null_mask("a") is False + table._ensure_null_mask("a") + assert table._storage.has_null_mask("a") is True + finally: + table.close() + + +def test_delete_null_mask_is_a_no_op_when_absent(tmp_path): + path = str(tmp_path / "nodel.b2d") + table = make_table(with_nulls=False, urlpath=path) + try: + table._storage.delete_null_mask("a") # must not raise + assert table._storage.has_null_mask("a") is False + finally: + table.close() + + +def test_in_memory_storage_never_reports_a_stored_sidecar(): + """In-memory sidecars are held by the CTable, mirroring ``open_column``.""" + t = make_table() + assert t._storage.has_null_mask("a") is False + assert t._null_mask("a") is not None + + +def test_sidecar_key_is_beside_the_column_key(tmp_path): + path = str(tmp_path / "layout.b2d") + make_table().save(path) + storage = FileTableStorage(path, "r") + try: + keys = set(storage._open_store()) + assert "/_cols/a" in keys + assert "/_cols/a" + _NOTNULL_SUFFIX in keys + assert "/_cols/c" + _NOTNULL_SUFFIX not in keys + finally: + storage.close() + + +def test_sidecar_leaf_lands_on_disk(tmp_path): + path = str(tmp_path / "leaf.b2d") + make_table().save(path) + assert os.path.exists(os.path.join(path, "_cols", "a.notnull.b2nd")) diff --git a/tests/ctable/test_null_predicate_rewrite.py b/tests/ctable/test_null_predicate_rewrite.py new file mode 100644 index 000000000..20761ba7d --- /dev/null +++ b/tests/ctable/test_null_predicate_rewrite.py @@ -0,0 +1,284 @@ +####################################################################### +# Copyright (c) 2019-present, Blosc Development Team +# All rights reserved. +# +# SPDX-License-Identifier: BSD-3-Clause +####################################################################### + +"""String predicates over nullable columns follow SQL ``WHERE`` semantics. + +A null operand satisfies no comparison. The operator form has always done +this (``Column._null_aware_compare``); the string form gets there through +``CTable._rewrite_null_predicates``, which conjoins a validity guard onto each +comparison leaf that reads a nullable column. + +Per *leaf*, not once over the whole expression: a global ``result & valid_a`` +would drop a row that is null in ``a`` but matches the other branch of +``(a > 10) | (b == 0)``, which SQL says qualifies. +""" + +from __future__ import annotations + +import dataclasses + +import numpy as np +import pytest + +import blosc2 +from blosc2 import CTable +from blosc2.ctable_nulls import rewrite_null_predicates, sentinel_guard_expr + +# --------------------------------------------------------------------------- +# The pure string -> string transform +# --------------------------------------------------------------------------- + +# name -> (guard text, sentinel value). 999 satisfies "> 10", so every leaf +# below genuinely needs its guard. +GUARDS = {"a": ("(a != 999)", 999), "trip.lon": ("(trip.lon != 999)", 999)} + + +@pytest.mark.parametrize( + ("expr", "expected"), + [ + ("a > 10", "(a > 10) & (a != 999)"), + # Per leaf, so the OR branch that does not read `a` stays reachable. + ("(a > 10) | (b == 0)", "(a > 10) & (a != 999) | (b == 0)"), + ("a + b > 10", "(a + b > 10) & (a != 999)"), + # Comparing against the sentinel itself must match nothing. + ("a == 999", "(a == 999) & (a != 999)"), + # Nested leaves keep their dotted name; the nested rewrite runs later. + ("trip.lon > 10", "(trip.lon > 10) & (trip.lon != 999)"), + # and/or are normalized to &/| so one precedence rule governs the result. + ("a > 10 and b < 3", "(a > 10) & (a != 999) & (b < 3)"), + ("a > 10 or b < 3", "(a > 10) & (a != 999) | (b < 3)"), + ], +) +def test_rewrite_shapes(expr, expected): + assert rewrite_null_predicates(expr, GUARDS) == expected + + +def test_rewrite_guards_outside_a_negation(): + """``not (a > 10)`` must be False for a null ``a``, not True. + + Guarding inside -- ``~((a > 10) & valid)`` -- would yield True. + """ + out = rewrite_null_predicates("~(a > 10)", GUARDS) + assert out == "~(a > 10) & (a != 999)" + out = rewrite_null_predicates("not (a > 10)", GUARDS) + assert out == "(not a > 10) & (a != 999)" + + +def test_rewrite_skips_leaves_the_sentinel_cannot_satisfy(): + """A guard that cannot change the answer is not emitted. + + Beyond tidiness: an unguarded single-predicate expression is the shape the + index planner recognizes, so a needless guard would push a query that is + correct today off its index onto a full scan. + """ + guards = {"a": ("(a != -1)", -1)} + assert rewrite_null_predicates("a > 10", guards) is None + assert rewrite_null_predicates("a < 10", guards) == "(a < 10) & (a != -1)" + # ...but never inside a negation: a sentinel that fails `a > 10` passes + # `not (a > 10)`. + assert rewrite_null_predicates("~(a > 10)", guards) == "~(a > 10) & (a != -1)" + + +def test_rewrite_nan_sentinel_only_guards_not_equal(): + """Every comparison with NaN is False -- except ``!=``, which is True.""" + guards = {"a": ("(a == a)", float("nan"))} + assert rewrite_null_predicates("a > 10", guards) is None + assert rewrite_null_predicates("a == 10", guards) is None + assert rewrite_null_predicates("a != 10", guards) == "(a != 10) & (a == a)" + + +def test_rewrite_returns_none_when_nothing_applies(): + assert rewrite_null_predicates("b > 0", GUARDS) is None + assert rewrite_null_predicates("a > 10", {}) is None + assert rewrite_null_predicates("a > > 10", GUARDS) is None # unparseable + + +@pytest.mark.parametrize( + ("null_value", "expected"), + [ + (-1, "(x != -1)"), + (255, "(x != 255)"), + (0.5, "(x != 0.5)"), + ("", "(x != '')"), + (b"", "(x != b'')"), + (np.int64(-1), "(x != -1)"), + ], +) +def test_sentinel_guard_expr(null_value, expected): + assert sentinel_guard_expr("x", null_value) == expected + + +def test_sentinel_guard_expr_nan_compares_to_itself(): + """NaN is the only value unequal to itself, so this needs no isnan().""" + assert sentinel_guard_expr("x", float("nan")) == "(x == x)" + + +# --------------------------------------------------------------------------- +# End-to-end SQL semantics +# --------------------------------------------------------------------------- + + +def _table(spec, annotation, values, sentinel): + Row = dataclasses.make_dataclass( + "Row", + [("a", annotation, blosc2.field(spec)), ("b", int, blosc2.field(blosc2.int64()))], + ) + t = CTable(Row) + t.extend({"a": values, "b": np.arange(len(values))}) + return t + + +# Each case: sentinel, the four `a` values (index 2 is the null), and a probe +# value. The int-999 case is the one that is wrong before this change: the +# sentinel satisfies `a > 10`, so the null row leaked into the result. +CASES = [ + ("int-sentinel-below", blosc2.int64(null_value=-1), int, [1, 20, -1, 30], -1), + ("int-sentinel-above", blosc2.int64(null_value=999), int, [1, 20, 999, 30], 999), + ("nan", blosc2.float64(null_value=float("nan")), float, [1.0, 20.0, np.nan, 30.0], np.nan), + ("string", blosc2.string(max_length=8, null_value=""), str, ["aa", "zz", "", "mm"], ""), +] + + +@pytest.mark.parametrize(("label", "spec", "annotation", "values", "sentinel"), CASES) +def test_string_predicate_matches_sql(label, spec, annotation, values, sentinel): + t = _table(spec, annotation, values, sentinel) + arr = np.array(values) + valid = np.isnan(arr) == False if label == "nan" else arr != sentinel # noqa: E712 + b = np.arange(len(values)) + threshold = "'mm'" if label == "string" else "10" + cmp = arr > (np.array("mm") if label == "string" else 10) + + got = sorted(t.where(f"a > {threshold}")["b"][:].tolist()) + assert got == sorted(b[cmp & valid].tolist()) + + # OR: the null row must still qualify through the other branch. + got = sorted(t.where(f"(a > {threshold}) | (b == 2)")["b"][:].tolist()) + assert got == sorted(b[(cmp & valid) | (b == 2)].tolist()) + + # Negation: a null operand makes `not (...)` False, not True. + got = sorted(t.where(f"~(a > {threshold})")["b"][:].tolist()) + assert got == sorted(b[~cmp & valid].tolist()) + + +@pytest.mark.parametrize(("label", "spec", "annotation", "values", "sentinel"), CASES) +def test_string_form_agrees_with_operator_form(label, spec, annotation, values, sentinel): + """The two ways of spelling a predicate must return the same rows.""" + t = _table(spec, annotation, values, sentinel) + threshold = "mm" if label == "string" else 10 + quoted = f"'{threshold}'" if label == "string" else threshold + + assert sorted(t.where(f"a > {quoted}")["b"][:].tolist()) == sorted( + t.where(t["a"] > threshold)["b"][:].tolist() + ) + assert sorted(t.where(f"(a > {quoted}) | (b == 2)")["b"][:].tolist()) == sorted( + t.where((t["a"] > threshold) | (t["b"] == 2))["b"][:].tolist() + ) + + +def test_negation_over_and_corner_is_conservative(): + """The one known corner where the two query forms disagree -- pinned. + + ``~((a > 10) & (b == 999))`` with ``a`` null and the second term False: + SQL collapses ``NULL AND FALSE`` to ``FALSE``, so the negation is True and + the row qualifies. The string form guards at the negation point + (``... & (a != -1)``) and drops it. That is by design -- pushing validity + to the negation point is exact for every simpler form, and errs only by + *dropping*: a null row is never wrongly returned (see + plans/mask-based-nulls.md, "Negation caveat, resolved"). + + The operator form returns the SQL answer here, but not by reasoning in + three values: ``_null_aware_compare`` collapses null to False at the leaf, + so ``~`` sees plain booleans -- which is also why it leaks nulls on a bare + negation (the xfail below). If either assertion starts failing, the two + forms moved; make sure they moved *toward* SQL, together. + """ + t = _table(blosc2.int64(null_value=-1), int, [1, 20, -1, 30], -1) + + got = sorted(t.where("~((a > 10) & (b == 999))")["b"][:].tolist()) + assert got == [0, 1, 3] # null row dropped; SQL would keep it + + got = sorted(t.where(~((t["a"] > 10) & (t["b"] == 999)))["b"][:].tolist()) + assert got == [0, 1, 2, 3] # SQL-exact + + +@pytest.mark.xfail( + strict=True, + reason="operator-form negation is not null-aware: `a > 10` collapses null to " + "False at the leaf, so ~ turns the null row into a match. The string form " + "gets this right via the negation-point guard. Fixing it needs the " + "comparison result to carry its null predicate through ~.", +) +def test_operator_form_negation_drops_nulls(): + t = _table(blosc2.int64(null_value=-1), int, [1, 20, -1, 30], -1) + assert sorted(t.where(~(t["a"] > 10))["b"][:].tolist()) == [0] + + +def test_comparing_against_the_sentinel_matches_nothing(): + """The sentinel is not a value: SQL has no row whose `a` equals NULL.""" + t = _table(blosc2.int64(null_value=-1), int, [1, 20, -1, 30], -1) + assert t.where("a == -1").nrows == 0 + assert sorted(t.where("a != -1")["b"][:].tolist()) == [0, 1, 3] + + +def test_non_nullable_column_is_untouched(): + t = _table(blosc2.int64(null_value=-1), int, [1, 20, -1, 30], -1) + assert sorted(t.where("b > 1")["b"][:].tolist()) == [2, 3] + + +def test_dropna_then_predicate_is_consistent(): + t = _table(blosc2.int64(null_value=999), int, [1, 20, 999, 30], 999) + assert sorted(t.where("a > 10")["b"][:].tolist()) == sorted(t.dropna().where("a > 10")["b"][:].tolist()) + + +# --------------------------------------------------------------------------- +# Index and scan must agree +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize("sentinel", [-1.0, 999.0, float("nan")]) +@pytest.mark.parametrize( + "expr", ["a > 90", "a != 50", "(a > 90) | (b < 10)", "(a > 90) & (b < 5000)", "~(a > 90)"] +) +def test_indexed_matches_unindexed(sentinel, expr): + """An ordered index answers by sorted range and never evaluates the + predicate -- a NaN sentinel sorts last, so it lands inside every ``>`` + range. Making the expression null-aware cannot fix that, so the index path + keeps its own null exclusion; this pins the two paths together. + """ + Row = dataclasses.make_dataclass( + "Row", + [ + ("a", float, blosc2.field(blosc2.float64(null_value=sentinel))), + ("b", int, blosc2.field(blosc2.int64())), + ], + ) + n = 20_000 + rng = np.random.default_rng(0) + a = rng.integers(0, 100, n).astype(np.float64) + a[::13] = sentinel + payload = {"a": a, "b": np.arange(n)} + + plain = CTable(Row) + plain.extend(payload) + indexed = CTable(Row) + indexed.extend(payload) + indexed.create_index("a", kind=blosc2.IndexKind.FULL) + + expected = np.sort(plain.where(expr)["b"][:]) + assert np.array_equal(np.sort(indexed.where(expr)["b"][:]), expected) + + # ...and both agree with SQL. + valid = ~np.isnan(a) if np.isnan(sentinel) else a != sentinel + b = np.arange(n) + sql = { + "a > 90": (a > 90) & valid, + "a != 50": (a != 50) & valid, + "(a > 90) | (b < 10)": ((a > 90) & valid) | (b < 10), + "(a > 90) & (b < 5000)": (a > 90) & valid & (b < 5000), + "~(a > 90)": ~(a > 90) & valid, + }[expr] + assert np.array_equal(expected, np.sort(b[sql])) diff --git a/tests/ctable/test_null_storage_equivalence.py b/tests/ctable/test_null_storage_equivalence.py new file mode 100644 index 000000000..9f78c56b0 --- /dev/null +++ b/tests/ctable/test_null_storage_equivalence.py @@ -0,0 +1,702 @@ +####################################################################### +# Copyright (c) 2019-present, Blosc Development Team +# All rights reserved. +# +# SPDX-License-Identifier: BSD-3-Clause +####################################################################### + +"""The differential oracle: sentinel and mask storage must answer alike. + +CTable can keep a column's nulls in two places -- an in-band **sentinel**, or a +sidecar **validity mask** -- and a second implementation of anything is a +standing invitation to drift. This suite builds the *same logical data* twice, +once each way, and asserts that every public API gives the same answer. That +turns "mask is a second implementation" from a permanent liability into a +checked invariant. + +Comparison is always **logical**, never physical: a column is read as its +values with ``None`` substituted wherever ``is_null()`` is True (:func:`logical` +below). What sits underneath a null is the fill for one storage and the +sentinel for the other, and neither is part of the format contract -- asserting +on it would pin down something the design explicitly says may change. + +Three differences are **deliberate** and asserted as differences rather than +papered over: + +* **NaN is a value under mask storage** and null under a sentinel one + (decision 6). This is the entire point of a side channel, so the float cases + here carry no NaN, and :func:`test_nan_is_a_value_only_under_mask_storage` + pins the divergence on its own. +* **A nullable bool is physically ``uint8``** under a sentinel, to leave room + for the reserved ``255``, and plain ``np.bool_`` under a mask. The values + still compare equal (``True == 1``), which is what the oracle checks. +* **complex is mask-only**: no complex value is safe to reserve, so there is no + sentinel column to compare against. + +Everything else agrees, and there are no known unintentional divergences left: +the ``isin`` and ``to_pandas`` leaks this suite was first written to pin are +both fixed, and their tests are now plain assertions. The one remaining +inexactness is not a divergence but a limit of the destination format -- +pandas has no float dtype separating NaN from missing, so a mask float column +cannot round-trip through it (:func:`test_a_mask_float_cannot_round_trip_through_pandas`). +""" + +from __future__ import annotations + +import dataclasses + +import numpy as np +import pytest +from utf8_compat import HAVE_UTF8, needs_utf8, utf8_spec + +import blosc2 +from blosc2.ctable_nulls import is_na_marker + +T0 = np.datetime64("2021-03-04T05:06:07", "s") + + +def secs(n: int) -> np.timedelta64: + """A timedelta with an explicit unit; a bare int is deprecated in NumPy.""" + return np.timedelta64(n, "s") + + +#: ``kind -> (spec factory, logical values, sentinel literal)``. +#: +#: Every V1 kind, and the values are chosen to be awkward on purpose: a full +#: range of ``int8``/``uint8``, a genuine ``""`` beside a null in each text +#: kind, and a ``0``/``0.0`` that a fill could be confused with. No float NaN +#: -- that is decision 6's business and is tested separately. +KINDS: dict = { + "int64": (lambda **k: blosc2.int64(**k), [5, None, -3, 0, 7], -9), + "int8": (lambda **k: blosc2.int8(**k), [1, None, -5, 0, 127], -128), + "uint8": (lambda **k: blosc2.uint8(**k), [1, None, 200, 0, 7], 255), + "float64": (lambda **k: blosc2.float64(**k), [1.5, None, -2.5, 0.0, 7.25], float("nan")), + "float32": (lambda **k: blosc2.float32(**k), [1.5, None, -2.5, 0.0, 7.25], float("nan")), + "bool": (lambda **k: blosc2.bool(**k), [True, None, False, True, False], 255), + "timestamp": ( + lambda **k: blosc2.timestamp(**k), + [T0, None, T0 + secs(60), T0 + secs(5), T0 + secs(1)], + int(np.iinfo(np.int64).min), + ), + "string": (lambda **k: blosc2.string(max_length=8, **k), ["a", None, "", "zz", "m"], "ZZZZZZZZ"), + "bytes": ( + lambda **k: blosc2.bytes(max_length=8, **k), + [b"a", None, b"", b"zz", b"m"], + b"ZZZZZZZZ", + ), +} +if HAVE_UTF8: + KINDS["utf8"] = (utf8_spec, ["a", None, "", "zz", "m"], "__BLOSC2_NULL__") + +ALL_KINDS = sorted(KINDS) +#: Kinds whose values support ordered arithmetic reductions. +NUMERIC_KINDS = [k for k in ALL_KINDS if k.startswith(("int", "uint", "float"))] +#: The row that is null in every table this module builds. +NULL_ROW = 1 + + +def annotation_for(spec): + if isinstance(spec, (blosc2.schema.NDArraySpec, blosc2.schema.timestamp)): + return object + return spec.python_type + + +def build(kind: str, storage: str, *, capacity: int = 64): + """The same logical rows, stored *storage*'s way. + + Column ``a`` carries the nulls; ``g`` is a plain non-nullable key so the + group-by and multi-key sort cases have something to work with. + """ + factory, values, sentinel = KINDS[kind] + if storage == "mask": + spec = factory(nullable=True, null_storage="mask") + cells = values + else: + spec = factory(nullable=True, null_value=sentinel) + # A sentinel column has no way to spell "null" other than its sentinel. + cells = [sentinel if v is None else v for v in values] + row_cls = dataclasses.make_dataclass( + "EquivRow", + [ + ("a", annotation_for(spec), blosc2.field(spec)), + ("g", str, blosc2.field(blosc2.string(max_length=4))), + ], + ) + t = blosc2.CTable(row_cls, expected_size=capacity) + t.extend([(v, f"g{i % 2}") for i, v in enumerate(cells)]) + return t + + +def both(kind: str, **kwargs): + """The mask-backed and sentinel-backed tables for *kind*.""" + return build(kind, "mask", **kwargs), build(kind, "sentinel", **kwargs) + + +def logical(col) -> list: + """A column as its observable contents: values, with ``None`` where null. + + This is the whole comparison discipline of this suite. Reading ``col[:]`` + alone would compare the fill against the sentinel and fail everywhere for + reasons that are not bugs. + """ + nulls = col.is_null() + out = [] + for value, is_null in zip(col[:], nulls, strict=True): + if is_null: + out.append(None) + else: + out.append(value.item() if hasattr(value, "item") else value) + return out + + +def logical_rows(t, cols=("a", "g")) -> list[tuple]: + """Every live row of *t*, each column read through :func:`logical`.""" + columns = [logical(t[c]) for c in cols] + return [tuple(col[i] for col in columns) for i in range(t.nrows)] + + +def _null_last(value): + """Sort key putting ``None`` last and ordering the rest by value. + + By value, not by ``repr``: a nullable bool is ``np.bool_`` under a mask and + ``uint8`` under a sentinel, so ``False``/``0`` sort together here and + compare equal afterwards, while their reprs would not. + """ + return (value is None, 0 if value is None else value) + + +def assert_same(mask_result, sentinel_result, what: str) -> None: + """Assert the two storages agree, reporting which API disagreed.""" + assert mask_result == sentinel_result, ( + f"{what}: mask storage gave {mask_result!r}, sentinel storage gave {sentinel_result!r}" + ) + + +# --------------------------------------------------------------------------- +# The null API +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize("kind", ALL_KINDS) +def test_reads_agree(kind): + """The baseline: the same rows, and the same rows are null.""" + m, s = both(kind) + assert_same(logical(m["a"]), logical(s["a"]), "column contents") + assert_same(logical(m["a"])[NULL_ROW], None, "the null row") + + +@pytest.mark.parametrize("kind", ALL_KINDS) +def test_null_api_agrees(kind): + m, s = both(kind) + assert_same(m["a"].is_null().tolist(), s["a"].is_null().tolist(), "is_null") + assert_same(m["a"].notnull().tolist(), s["a"].notnull().tolist(), "notnull") + assert_same(m["a"].null_count(), s["a"].null_count(), "null_count") + + +@pytest.mark.parametrize("kind", ALL_KINDS) +def test_to_numpy_masked_agrees(kind): + """The masked view is the one read that must work for *both* storages.""" + m, s = both(kind) + got, want = m["a"].to_numpy(masked=True), s["a"].to_numpy(masked=True) + assert_same(got.mask.tolist(), want.mask.tolist(), "to_numpy(masked=True).mask") + assert_same(got.compressed().tolist(), want.compressed().tolist(), "to_numpy(masked=True) values") + + +@pytest.mark.parametrize("kind", ALL_KINDS) +def test_dropna_agrees(kind): + m, s = both(kind) + assert_same(logical_rows(m.dropna()), logical_rows(s.dropna()), "dropna") + assert m.dropna().nrows == len(KINDS[kind][1]) - 1 + + +@pytest.mark.parametrize("kind", NUMERIC_KINDS) +def test_fillna_agrees(kind): + m, s = both(kind) + assert_same(m["a"].fillna(42).tolist(), s["a"].fillna(42).tolist(), "fillna") + + +@pytest.mark.parametrize("kind", ALL_KINDS) +def test_unique_and_value_counts_agree(kind): + """Both exclude the null, and neither leaks what sits under it.""" + m, s = both(kind) + assert_same(sorted(m["a"].unique().tolist()), sorted(s["a"].unique().tolist()), "unique") + assert_same( + sorted(m["a"].value_counts().items()), + sorted(s["a"].value_counts().items()), + "value_counts", + ) + + +# --------------------------------------------------------------------------- +# Ordering +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize("kind", ALL_KINDS) +@pytest.mark.parametrize("ascending", [True, False]) +def test_sort_by_agrees(kind, ascending): + """Nulls sort last in both directions, whichever storage they live in.""" + m, s = both(kind) + got = logical(m.sort_by("a", ascending=ascending)["a"]) + want = logical(s.sort_by("a", ascending=ascending)["a"]) + assert_same(got, want, f"sort_by(ascending={ascending})") + assert got[-1] is None, "nulls sort last" + + +@pytest.mark.parametrize("kind", ALL_KINDS) +def test_multi_key_sort_agrees(kind): + m, s = both(kind) + assert_same(logical_rows(m.sort_by(["g", "a"])), logical_rows(s.sort_by(["g", "a"])), "sort_by 2 keys") + + +@pytest.mark.parametrize("kind", ALL_KINDS) +def test_sorted_view_agrees(kind): + m, s = both(kind) + assert_same( + logical(m.sort_by("a", view=True)["a"]), + logical(s.sort_by("a", view=True)["a"]), + "sort_by(view=True)", + ) + + +# --------------------------------------------------------------------------- +# Reductions +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize("kind", NUMERIC_KINDS) +@pytest.mark.parametrize("op", ["sum", "mean", "min", "max", "std"]) +def test_reductions_agree(kind, op): + """Every reduction skips the null rather than folding in what stands for it.""" + m, s = both(kind) + got, want = getattr(m["a"], op)(), getattr(s["a"], op)() + assert got == pytest.approx(want), f"{op}: {got!r} != {want!r}" + + +@pytest.mark.parametrize("kind", NUMERIC_KINDS + ["timestamp"]) +@pytest.mark.parametrize("op", ["argmin", "argmax"]) +def test_arg_reductions_agree(kind, op): + m, s = both(kind) + got, want = int(getattr(m["a"], op)()), int(getattr(s["a"], op)()) + assert_same(got, want, op) + assert got != NULL_ROW, "a null row can never be the extremum" + + +@pytest.mark.parametrize("kind", ["timestamp", "string", "bytes"]) +@pytest.mark.parametrize("op", ["min", "max"]) +def test_ordered_non_numeric_reductions_agree(kind, op): + m, s = both(kind) + assert_same(getattr(m["a"], op)(), getattr(s["a"], op)(), op) + + +@pytest.mark.parametrize("kind", [*NUMERIC_KINDS, "bool"]) +def test_cov_agrees(kind): + """cov() drops a null row listwise, and must learn which rows those are. + + It asked the values, which for a mask column means asking a fill that + looks like ordinary data, so the null was averaged in. The reference is a + table holding the same rows with the null one simply absent. + """ + m, s = both(kind) + factory, values, _sentinel = KINDS[kind] + kept = [(v, i) for i, v in enumerate(values) if v is not None] + row_cls = dataclasses.make_dataclass( + "CovRef", + [ + ("a", annotation_for(factory()), blosc2.field(factory())), + ("n", int, blosc2.field(blosc2.int64())), + ], + ) + reference = blosc2.CTable(row_cls, expected_size=16) + reference.extend(kept) + + # Column "g" is text, which cov() rejects, so pair "a" with a numeric one. + got = np.asarray(m.select(["a"]).cov()) + want = np.asarray(s.select(["a"]).cov()) + np.testing.assert_allclose(got, want, err_msg="mask and sentinel disagree") + np.testing.assert_allclose( + got, np.asarray(reference.select(["a"]).cov()), err_msg="neither matches a null-free table" + ) + + +def test_cov_drops_a_null_row_from_every_column(): + """Listwise: a null in one column removes the row from the others too.""" + row_cls = dataclasses.make_dataclass( + "CovRow", + [ + ("a", int, blosc2.field(blosc2.int64(nullable=True, null_storage="mask"))), + ("b", int, blosc2.field(blosc2.int64())), + ], + ) + t = blosc2.CTable(row_cls, expected_size=16) + t.extend([(1, 2), (None, 7), (3, 6), (5, 1)]) + + plain_cls = dataclasses.make_dataclass( + "PlainCovRow", + [("a", int, blosc2.field(blosc2.int64())), ("b", int, blosc2.field(blosc2.int64()))], + ) + reference = blosc2.CTable(plain_cls, expected_size=16) + reference.extend([(1, 2), (3, 6), (5, 1)]) + + np.testing.assert_allclose(np.asarray(t.cov()), np.asarray(reference.cov())) + + +def test_cov_keeps_a_mask_floats_nan_as_data(): + """Decision 6 again: only mask=False is dropped, so a NaN poisons the result.""" + row_cls = dataclasses.make_dataclass( + "NanCovRow", + [ + ("a", float, blosc2.field(blosc2.float64(nullable=True, null_storage="mask"))), + ("b", float, blosc2.field(blosc2.float64())), + ], + ) + t = blosc2.CTable(row_cls, expected_size=16) + t.extend([(1.0, 2.0), (float("nan"), 7.0), (3.0, 6.0)]) + assert t["a"].null_count() == 0 + assert np.isnan(np.asarray(t.cov())[0, 0]) + + +# --------------------------------------------------------------------------- +# Grouping +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize("kind", ALL_KINDS) +@pytest.mark.parametrize("dropna", [True, False]) +def test_group_by_key_agrees(kind, dropna): + """As a *key*: the null forms its own group, keyed None, under both storages. + + A sentinel column stores that group's key as its sentinel and a mask column + as its fill, so this only holds when the key column is read logically -- + which is the point. It is also the one place a sentinel column has to + round-trip its own sentinel back into a null. + """ + m, s = both(kind) + got = sorted(logical(m.group_by("a", dropna=dropna).count("g")["a"]), key=_null_last) + want = sorted(logical(s.group_by("a", dropna=dropna).count("g")["a"]), key=_null_last) + assert_same(got, want, f"group_by key (dropna={dropna})") + assert (None in got) is not dropna, "dropna decides whether the null group exists" + + +@pytest.mark.parametrize("kind", NUMERIC_KINDS) +@pytest.mark.parametrize("op", ["sum", "min", "max", "count"]) +def test_group_by_value_agrees(kind, op): + """As a *value*: the null must not be aggregated as if it were its fill.""" + m, s = both(kind) + got = getattr(m.group_by("g"), op)("a") + want = getattr(s.group_by("g"), op)("a") + assert_same(logical_rows(got, got.col_names), logical_rows(want, want.col_names), f"group_by {op}") + + +# --------------------------------------------------------------------------- +# Queries +# --------------------------------------------------------------------------- + +#: Expression shapes the two storages must answer identically. Each is written +#: so the *stored* null value would satisfy it under at least one storage if +#: nullity leaked -- which is what makes them worth asserting. +NUMERIC_EXPRESSIONS = [ + "a > 0", + "a < 3", + "a == 0", + "a != 0", + "(a > 0) & (g == 'g0')", + "(a > 0) | (g == 'g1')", + "~(a > 0)", + "(a < 3) & (a > -100)", +] + + +@pytest.mark.parametrize("kind", ["int64", "int8", "uint8", "float64"]) +@pytest.mark.parametrize("expression", NUMERIC_EXPRESSIONS) +def test_where_agrees(kind, expression): + m, s = both(kind) + assert_same(logical_rows(m.where(expression)), logical_rows(s.where(expression)), f"where({expression})") + + +def build_big(storage: str, *, indexed: bool, n: int = 4000): + """A table big enough for the planner to actually reach for an index. + + Every seventh row is null, and the values straddle zero so the expressions + below select a real range rather than everything or nothing. + """ + values = [None if i % 7 == 0 else (i % 101) - 50 for i in range(n)] + sentinel = -9999 + if storage == "mask": + spec = blosc2.int64(nullable=True, null_storage="mask") + cells = values + else: + spec = blosc2.int64(nullable=True, null_value=sentinel) + cells = [sentinel if v is None else v for v in values] + row_cls = dataclasses.make_dataclass( + "BigRow", + [("a", int, blosc2.field(spec)), ("g", str, blosc2.field(blosc2.string(max_length=4)))], + ) + t = blosc2.CTable(row_cls, expected_size=n + 16, create_summary_index=False) + t.extend([(v, f"g{i % 2}") for i, v in enumerate(cells)]) + if indexed: + t.create_index("a", kind="summary") + assert "a" in t._get_index_catalog(), "the index this test is about was not built" + else: + assert "a" not in t._get_index_catalog(), "this table was meant to have no index" + return t + + +@pytest.mark.parametrize("expression", NUMERIC_EXPRESSIONS) +def test_indexed_where_agrees(expression): + """The same answers with an index in play. + + An ordered index answers by slicing the sorted column rather than by + evaluating the predicate, so this is the path where a leaked null shows up + as a *different* result from the identical unindexed query -- the failure + the plan's Addendum 2 found leaking every null through a float index. + """ + m = build_big("mask", indexed=True) + s = build_big("sentinel", indexed=True) + assert_same( + logical_rows(m.where(expression)), logical_rows(s.where(expression)), f"indexed {expression}" + ) + + +@pytest.mark.parametrize("storage", ["mask", "sentinel"]) +@pytest.mark.parametrize("expression", NUMERIC_EXPRESSIONS) +def test_an_index_does_not_change_the_answer(storage, expression): + """Each storage must agree with *itself*, indexed versus not. + + Cross-storage agreement alone would not catch an index that leaks nulls + into both storages the same way, so this is the other half of the oracle: + the scan is the reference, and the index has to match it. + """ + indexed = build_big(storage, indexed=True) + scanned = build_big(storage, indexed=False) + assert_same( + logical_rows(indexed.where(expression)), + logical_rows(scanned.where(expression)), + f"{storage}: indexed vs scanned {expression}", + ) + + +@needs_utf8 +@pytest.mark.parametrize("expression", ["a == 'a'", "a != 'a'", "(a == 'a') | (g == 'g1')"]) +def test_text_where_agrees(expression): + m, s = both("utf8") + assert_same(logical_rows(m.where(expression)), logical_rows(s.where(expression)), f"where({expression})") + + +@pytest.mark.parametrize("kind", ALL_KINDS) +def test_a_null_never_satisfies_a_predicate(kind): + """SQL WHERE semantics, which is what both storages promise (decision 8).""" + for t in both(kind): + for row in logical_rows(t.where("a == a")): + assert row[0] is not None + + +# --------------------------------------------------------------------------- +# Export +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize("kind", ALL_KINDS) +def test_to_arrow_agrees(kind): + """Arrow has a validity bitmap of its own, so this is the exact comparison.""" + pytest.importorskip("pyarrow") + m, s = both(kind) + got, want = m.to_arrow()["a"], s.to_arrow()["a"] + assert_same(got.null_count, want.null_count, "to_arrow null_count") + assert_same(got.to_pylist(), want.to_pylist(), "to_arrow values") + assert got.to_pylist()[NULL_ROW] is None + + +# --------------------------------------------------------------------------- +# Deliberate divergences +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize("kind", ["float64", "float32"]) +def test_nan_is_a_value_only_under_mask_storage(kind): + """Decision 6, and the reason the float rows above carry no NaN. + + A sentinel float column spells its null ``NaN``, so it cannot also hold one + as data; a mask column follows Arrow and keeps NaN an ordinary value. This + is the divergence the side channel exists to create, so it is asserted + rather than tolerated. + """ + factory = KINDS[kind][0] + nan = float("nan") + + mask_t = build_one(factory(nullable=True, null_storage="mask"), [1.0, nan, 3.0]) + sent_t = build_one(factory(nullable=True, null_value=nan), [1.0, nan, 3.0]) + + assert mask_t["a"].is_null().tolist() == [False, False, False], "NaN is a value under a mask" + assert sent_t["a"].is_null().tolist() == [False, True, False], "NaN is the null under a sentinel" + assert mask_t["a"].null_count() == 0 + assert sent_t["a"].null_count() == 1 + # And it follows through to the reductions, which is where a user meets it. + assert np.isnan(mask_t["a"].sum()) + assert sent_t["a"].sum() == 4.0 + + +def build_one(spec, values): + """A one-column table of *values*, with no null coercion of any kind.""" + row_cls = dataclasses.make_dataclass("NanRow", [("a", annotation_for(spec), blosc2.field(spec))]) + t = blosc2.CTable(row_cls, expected_size=16) + t.extend([(v,) for v in values]) + return t + + +def test_nullable_bool_is_uint8_only_under_a_sentinel(): + """The reserved 255 needs room; a mask column has np.bool_ and needs none.""" + m, s = both("bool") + assert m["a"].dtype == np.dtype(np.bool_) + assert s["a"].dtype == np.dtype(np.uint8) + # The physical difference is invisible to the logical read, which is why + # every other bool case in this suite compares equal. + assert_same(logical(m["a"]), logical(s["a"]), "bool contents") + + +def test_complex_has_no_sentinel_to_compare_against(): + """complex is mask-only: no complex value is safe to reserve.""" + with pytest.raises((TypeError, ValueError)): + blosc2.complex128(nullable=True, null_value=0j) + assert blosc2.complex128(nullable=True).null_storage == "mask" + + +@pytest.mark.parametrize("kind", ALL_KINDS) +def test_isin_agrees(kind): + """A null matches nothing -- not even the value that stands in for it.""" + m, s = both(kind) + # Probe with each storage's own stand-in for a null. Neither should match, + # because neither stand-in is the row's value: the fill is not part of the + # format contract, and the sentinel is reserved. + for probe in (m["a"][:][NULL_ROW], s["a"][:][NULL_ROW]): + got, want = m["a"].isin([probe]).tolist(), s["a"].isin([probe]).tolist() + assert_same(got, want, f"isin([{probe!r}])") + if is_na_marker(probe): + # A timestamp's fill decodes to NaT, which *is* a way of spelling + # "missing", so probing with it is asking for the nulls. + assert got[NULL_ROW], f"{probe!r} is a null marker, so it selects nulls" + else: + assert not got[NULL_ROW], "a null row is not a member of anything" + + +@pytest.mark.parametrize("kind", ALL_KINDS) +def test_isin_none_selects_the_nulls(kind): + """``None`` is how you ask for them, and it agrees with is_null exactly.""" + for t in both(kind): + assert_same(t["a"].isin([None]).tolist(), t["a"].is_null().tolist(), "isin([None]) vs is_null") + + +@pytest.mark.parametrize("kind", ALL_KINDS) +def test_isin_none_beside_a_real_value(kind): + """Asking for a value *and* the nulls gets both, and nothing else.""" + m, s = both(kind) + rows = logical(m["a"]) + present = rows[0] + want = [v is None or v == present for v in rows] + assert m["a"].isin([present, None]).tolist() == want + assert s["a"].isin([present, None]).tolist() == want + + +@pytest.mark.parametrize("kind", ALL_KINDS) +def test_to_pandas_agrees(kind): + """A null reaches pandas as NA, not as whatever stands in for it.""" + pytest.importorskip("pandas") + m, s = both(kind) + got, want = m.to_pandas()["a"], s.to_pandas()["a"] + assert_same(got.isna().tolist(), want.isna().tolist(), "to_pandas isna") + assert_same(got.isna().tolist(), m["a"].is_null().tolist(), "to_pandas isna vs is_null") + assert got.isna()[NULL_ROW], "the null row should read as NA" + + +@pytest.mark.parametrize("kind", ALL_KINDS) +def test_to_pandas_keeps_the_values_it_does_have(kind): + """Expressing the nulls must not disturb the rows that are not null.""" + pytest.importorskip("pandas") + for t in both(kind): + series = t.to_pandas()["a"] + for i, value in enumerate(logical(t["a"])): + if value is None: + continue + got = series[i] + got = got.item() if hasattr(got, "item") else got + assert got == value, f"row {i}: {got!r} != {value!r}" + + +if __name__ == "__main__": + pytest.main(["-v", __file__]) + + +# --------------------------------------------------------------------------- +# The pandas round trip +# --------------------------------------------------------------------------- + + +def row_cls_for(kind: str, storage: str): + """A one-column dataclass matching what :func:`build` produces.""" + factory, _values, sentinel = KINDS[kind] + spec = ( + factory(nullable=True, null_storage="mask") + if storage == "mask" + else factory(nullable=True, null_value=sentinel) + ) + return dataclasses.make_dataclass("RoundTripRow", [("a", annotation_for(spec), blosc2.field(spec))]) + + +#: Float is excluded: pandas spells missing as NaN in every float dtype it +#: has, and a mask float column keeps NaN a value (decision 6), so the two +#: cannot survive the trip. :func:`test_a_mask_float_cannot_round_trip_through_pandas` +#: pins that limitation rather than leaving it implicit. +ROUND_TRIP_KINDS = [k for k in ALL_KINDS if not k.startswith("float")] + + +@pytest.mark.parametrize("kind", ROUND_TRIP_KINDS) +@pytest.mark.parametrize("storage", ["mask", "sentinel"]) +def test_pandas_round_trip_keeps_the_nulls(kind, storage): + """to_pandas must emit something from_pandas can read back unchanged.""" + pytest.importorskip("pandas") + t = build(kind, storage) + back = blosc2.CTable.from_pandas(t.to_pandas()[["a"]], row_cls_for(kind, storage)) + assert_same(logical(back["a"]), logical(t["a"]), f"{storage} round trip") + assert logical(back["a"])[NULL_ROW] is None + + +@pytest.mark.parametrize("storage", ["mask", "sentinel"]) +def test_pandas_round_trip_of_a_null_free_column(storage): + """The common case must not be disturbed by any of the above.""" + pytest.importorskip("pandas") + spec = ( + blosc2.int64(nullable=True, null_storage="mask") + if storage == "mask" + else blosc2.int64(nullable=True, null_value=-9) + ) + row_cls = dataclasses.make_dataclass("PlainRow", [("a", int, blosc2.field(spec))]) + t = blosc2.CTable(row_cls, expected_size=16) + t.extend([(1,), (2,), (3,)]) + df = t.to_pandas() + # No null, so nothing widens: the dtype is what it has always been here. + assert df["a"].dtype == np.dtype(np.int64) + back = blosc2.CTable.from_pandas(df, row_cls) + assert logical(back["a"]) == [1, 2, 3] + + +@pytest.mark.parametrize("kind", ["float32", "float64"]) +def test_a_mask_float_cannot_round_trip_through_pandas(kind): + """The one stated limitation, pinned so it stays a known quantity. + + pandas has no float dtype that distinguishes NaN from missing -- even + ``Float64`` folds a NaN into ``NA`` -- so a mask float column's null comes + back as a NaN *value*. A sentinel float column is lossless precisely + because NaN is what it means by null in the first place. + """ + pytest.importorskip("pandas") + mask_back = blosc2.CTable.from_pandas(build(kind, "mask").to_pandas()[["a"]], row_cls_for(kind, "mask")) + assert mask_back["a"].null_count() == 0, "the null came back as a NaN value" + assert np.isnan(mask_back["a"][:][NULL_ROW]) + + sentinel_back = blosc2.CTable.from_pandas( + build(kind, "sentinel").to_pandas()[["a"]], row_cls_for(kind, "sentinel") + ) + assert sentinel_back["a"].null_count() == 1 + # to_arrow is the export that keeps the distinction for both storages. + pytest.importorskip("pyarrow") + assert build(kind, "mask").to_arrow()["a"].null_count == 1 diff --git a/tests/ctable/test_null_storage_schema.py b/tests/ctable/test_null_storage_schema.py new file mode 100644 index 000000000..10bf1287d --- /dev/null +++ b/tests/ctable/test_null_storage_schema.py @@ -0,0 +1,490 @@ +####################################################################### +# Copyright (c) 2019-present, Blosc Development Team +# All rights reserved. +# +# SPDX-License-Identifier: BSD-3-Clause +####################################################################### + +"""Schema plumbing for validity-mask null storage (Phase 2). + +Specs can now say *where* their nulls live -- in band as a sentinel, or in a +sidecar validity array -- and the schema records it. Nothing reads or writes +a sidecar yet; this is the declaration layer. + +The default is still ``"sentinel"``, so every existing table and every schema +written by this release is byte-identical to before. +""" + +from __future__ import annotations + +import dataclasses + +import numpy as np +import pytest +from utf8_compat import HAVE_UTF8, needs_utf8, utf8_spec + +import blosc2 +from blosc2 import CTable +from blosc2.schema import fill_value_for +from blosc2.schema_compiler import schema_from_dict, schema_to_dict + +# --------------------------------------------------------------------------- +# Spec-level declaration +# --------------------------------------------------------------------------- + +MASKABLE_SPECS = [ + ("int8", blosc2.int8, {}), + ("int64", blosc2.int64, {}), + ("uint8", blosc2.uint8, {}), + ("float64", blosc2.float64, {}), + ("bool", blosc2.bool, {}), + ("timestamp", blosc2.timestamp, {}), + ("string", blosc2.string, {"max_length": 4}), + ("bytes", blosc2.bytes, {"max_length": 4}), + pytest.param("utf8", blosc2.utf8, {}, marks=needs_utf8), +] + + +@pytest.mark.parametrize(("label", "factory", "kwargs"), MASKABLE_SPECS) +def test_spec_accepts_null_storage(label, factory, kwargs): + spec = factory(null_storage="mask", **kwargs) + assert spec.null_storage == "mask" + assert spec.uses_mask + assert spec.nullable + assert spec.null_value is None + + +@pytest.mark.parametrize(("label", "factory", "kwargs"), MASKABLE_SPECS) +def test_spec_defaults_to_unresolved_storage(label, factory, kwargs): + """``nullable=True`` alone defers the choice to the policy.""" + assert factory(nullable=True, **kwargs).null_storage is None + assert factory(**kwargs).null_storage is None + + +def test_mask_and_sentinel_together_is_rejected(): + with pytest.raises(ValueError, match="cannot be combined with an explicit null_value"): + blosc2.int64(null_storage="mask", null_value=-1) + + +def test_unknown_null_storage_is_rejected(): + with pytest.raises(ValueError, match="null_storage must be one of"): + blosc2.int64(null_storage="bitmap") + + +def test_ndarray_spec_accepts_null_storage(): + spec = blosc2.ndarray((2,), dtype=blosc2.int64(), null_storage="mask") + assert spec.uses_mask + assert spec.null_value is None + + +# --------------------------------------------------------------------------- +# Nullable bool loses the 255 reservation under mask storage +# --------------------------------------------------------------------------- + + +def test_sentinel_bool_is_still_uint8(): + spec = blosc2.bool(nullable=True, null_value=255) + assert spec.dtype == np.dtype(np.uint8) + + +def test_mask_bool_stays_bool(): + """The point of the design: no reserved 255, no uint8 leak.""" + assert blosc2.bool(null_storage="mask").dtype == np.dtype(np.bool_) + + +def test_mask_bool_accepts_any_null_value_rejection(): + with pytest.raises(ValueError, match="Nullable bool null_value must be 255"): + blosc2.bool(nullable=True, null_value=7) + + +def test_stored_uint8_bool_reopens_as_uint8_without_the_resolver(): + """Opening a table rebuilds specs through ``spec_cls(**data)`` and never + runs ``_resolve_nullable_specs``, so the uint8 flip has to survive from + metadata alone. + """ + spec = blosc2.bool(nullable=True, null_value=255) + rebuilt = schema_from_dict({"version": 1, "columns": [{"name": "flag", **spec.to_metadata_dict()}]}) + assert rebuilt.columns[0].spec.dtype == np.dtype(np.uint8) + assert rebuilt.columns[0].dtype == np.dtype(np.uint8) + + +# --------------------------------------------------------------------------- +# Complex gains nullability, mask-only +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize("factory", [blosc2.complex64, blosc2.complex128]) +def test_complex_is_nullable_via_mask(factory): + spec = factory(nullable=True) + assert spec.uses_mask, "complex has no representable sentinel, so nullable implies mask" + + +@pytest.mark.parametrize("factory", [blosc2.complex64, blosc2.complex128]) +def test_complex_rejects_a_sentinel(factory): + with pytest.raises(ValueError, match="cannot use an in-band null sentinel"): + factory(null_value=0j) + + +@pytest.mark.parametrize("factory", [blosc2.complex64, blosc2.complex128]) +def test_complex_non_nullable_is_unchanged(factory): + spec = factory() + assert not spec.nullable + assert spec.to_metadata_dict() == {"kind": spec._kind} + + +# --------------------------------------------------------------------------- +# Serialization and version gating +# --------------------------------------------------------------------------- + + +def _schema(**cols): + fields = [(name, ann, blosc2.field(spec)) for name, (ann, spec) in cols.items()] + return CTable(dataclasses.make_dataclass("Row", fields))._schema + + +def test_sentinel_schema_is_unchanged_on_disk(): + """A sentinel table must serialize exactly as it did before mask storage. + + ``"sentinel"`` is never emitted, so old readers keep working. + """ + d = schema_to_dict(_schema(v=(int, blosc2.int64(null_value=-1)))) + assert d["version"] == 1 + assert d["columns"][0] == {"name": "v", "kind": "int64", "nullable": True, "null_value": -1} + assert "null_storage" not in d["columns"][0] + + +def test_mask_column_records_storage_and_bumps_version(): + d = schema_to_dict(_schema(v=(int, blosc2.int64(null_storage="mask")))) + assert d["version"] == 3 + assert d["columns"][0]["null_storage"] == "mask" + assert "null_value" not in d["columns"][0] + + +def test_version_bump_is_conditional_on_a_mask_column(): + """Only mask-using tables become unreadable by older readers.""" + mixed = _schema( + s=(int, blosc2.int64(null_value=-1)), + plain=(int, blosc2.int64()), + ) + assert schema_to_dict(mixed)["version"] == 1 + + +def test_schema_round_trip_preserves_storage(): + d = schema_to_dict( + _schema( + m=(int, blosc2.int64(null_storage="mask")), + s=(int, blosc2.int64(null_value=-1)), + ) + ) + back = schema_from_dict(d) + assert back.columns_by_name["m"].spec.uses_mask + assert back.columns_by_name["s"].spec.null_value == -1 + assert not back.columns_by_name["s"].spec.uses_mask + assert schema_to_dict(back) == d + + +def test_schema_from_dict_accepts_version_3(): + schema_from_dict({"version": 3, "columns": [{"name": "v", "kind": "int64"}]}) + + +def test_unsupported_version_names_the_version_and_the_way_out(): + with pytest.raises(ValueError, match="Unsupported schema version 9") as exc: + schema_from_dict({"version": 9, "columns": []}) + assert "convert_nulls" in str(exc.value), "the message should say how to get a readable copy" + + +# --------------------------------------------------------------------------- +# NullPolicy +# --------------------------------------------------------------------------- + + +def test_policy_defaults_to_a_mask(): + """A newly created nullable column keeps its nulls out of band. + + ``null_storage`` stays ``None`` -- *unspecified* -- rather than being + resolved at construction, and that distinction is load-bearing: it is what + lets a type-wide sentinel field imply sentinel storage without + contradicting anything the caller wrote. + """ + assert blosc2.NullPolicy().null_storage is None + assert blosc2.NullPolicy().resolve_null_storage() == "mask" + assert blosc2.NullPolicy(null_storage="sentinel").resolve_null_storage() == "sentinel" + + +def test_policy_rejects_an_unknown_storage(): + with pytest.raises(ValueError, match="null_storage must be 'mask', 'sentinel' or None"): + blosc2.NullPolicy(null_storage="bitmap") + + +def test_policy_mask_with_a_type_wide_sentinel_field_is_a_contradiction(): + with pytest.raises(ValueError, match="contradicts the type-wide sentinel field"): + blosc2.NullPolicy(null_storage="mask", string_value="") + + +def test_policy_mask_still_allows_per_column_sentinels(): + """``column_null_values`` forces sentinel storage per column, which is + a refinement of the default rather than a contradiction of it.""" + policy = blosc2.NullPolicy(null_storage="mask", column_null_values={"v": -1}) + assert policy.null_storage == "mask" + + +def test_untouched_nan_float_value_does_not_read_as_set(): + """float_value defaults to NaN, which never equals itself.""" + assert not blosc2.NullPolicy()._sentinel_field_is_set("float_value") + assert blosc2.NullPolicy(float_value=-1.0)._sentinel_field_is_set("float_value") + + +# --------------------------------------------------------------------------- +# Resolution order +# --------------------------------------------------------------------------- + + +def _resolved(spec, annotation=int, name="v", **policy_kw): + Row = dataclasses.make_dataclass("Row", [(name, annotation, blosc2.field(spec))]) + if policy_kw: + with blosc2.null_policy(blosc2.NullPolicy(**policy_kw)): + return CTable(Row)._schema.columns_by_name[name] + return CTable(Row)._schema.columns_by_name[name] + + +def test_explicit_storage_beats_the_policy(): + col = _resolved(blosc2.int64(null_storage="mask"), null_storage="sentinel") + assert col.spec.uses_mask + + +def test_explicit_null_value_implies_sentinel_under_a_mask_policy(): + col = _resolved(blosc2.int64(null_value=-1), null_storage="mask") + assert col.spec.null_value == -1 + assert not col.spec.uses_mask + + +def test_column_null_values_implies_sentinel_under_a_mask_policy(): + col = _resolved(blosc2.int64(nullable=True), null_storage="mask", column_null_values={"v": -1}) + assert col.spec.null_value == -1 + assert not col.spec.uses_mask + + +@pytest.mark.parametrize( + ("field", "value", "spec", "annotation"), + [ + ("string_value", "", blosc2.string(max_length=4, nullable=True), str), + ("bytes_value", b"", blosc2.bytes(max_length=4, nullable=True), bytes), + ("float_value", -1.0, blosc2.float64(nullable=True), float), + ("timestamp_value", -1, blosc2.timestamp(nullable=True), object), + ("signed_int_strategy", "max", blosc2.int64(nullable=True), int), + ("unsigned_int_strategy", "min", blosc2.uint64(nullable=True), int), + ], +) +def test_type_wide_sentinel_field_implies_sentinel_for_its_kinds(field, value, spec, annotation): + """Existing ``NullPolicy(float_value=...)`` code must keep working once the + default flips to mask, so setting a sentinel field opts those types in. + """ + col = _resolved(spec, annotation, **{field: value}) + assert not col.spec.uses_mask + assert col.spec.null_value is not None + + +def test_bool_value_cannot_imply_sentinel_storage(): + """``bool_value``'s default is the only value it may hold, so "was it set?" + has no answer for it: ``255`` is the sole legal sentinel for a nullable + bool. ``NullPolicy(bool_value=255)`` therefore states the default and + changes nothing -- a bool column that wants sentinel storage has to say so, + with ``null_storage`` or ``column_null_values``. + """ + col = _resolved(blosc2.bool(nullable=True), bool, bool_value=255) + assert col.spec.uses_mask + col = _resolved(blosc2.bool(nullable=True), bool, null_storage="sentinel") + assert not col.spec.uses_mask + assert col.spec.null_value == 255 + + +def test_policy_mask_applies_to_plain_nullable(): + col = _resolved(blosc2.int64(nullable=True), null_storage="mask") + assert col.spec.uses_mask + assert col.spec.null_value is None + + +def test_mask_skips_string_max_length_widening(): + """The sentinel path widens ``U4`` to fit ``__BLOSC2_NULL__``; mask does not.""" + col = _resolved(blosc2.string(max_length=4, nullable=True), str, null_storage="mask") + assert col.dtype == np.dtype("U4") + + sentinel_col = _resolved(blosc2.string(max_length=4, nullable=True), str, null_storage="sentinel") + assert sentinel_col.dtype == np.dtype("U15") + + +def test_mask_skips_the_bool_uint8_flip(): + col = _resolved(blosc2.bool(nullable=True), bool, null_storage="mask") + assert col.dtype == np.dtype(np.bool_) + assert col.spec.dtype == np.dtype(np.bool_) + + +def test_mask_skips_the_ndarray_bool_uint8_flip(): + col = _resolved(blosc2.ndarray((2,), dtype=blosc2.bool(), nullable=True), object, null_storage="mask") + assert col.spec.dtype == np.dtype(np.bool_) + assert col.spec.itemsize == 1 + + +def test_sentinel_storage_is_unchanged_when_asked_for(): + """The uint8 widening and the reserved 255 stay exactly as they were. + + Every stored table keeps them, and so does any column that asks -- which is + what makes the default flip a change to *new* columns only. + """ + col = _resolved(blosc2.bool(nullable=True), bool, null_storage="sentinel") + assert col.dtype == np.dtype(np.uint8) + assert col.spec.null_value == 255 + + +# --------------------------------------------------------------------------- +# fill_value_for +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize( + ("spec", "expected"), + [ + (blosc2.int32(), 0), + (blosc2.uint8(), 0), + (blosc2.float32(), None), # NaN, checked separately + (blosc2.complex128(), 0j), + (blosc2.bool(), False), + (blosc2.string(max_length=4), ""), + (blosc2.bytes(max_length=4), b""), + pytest.param(utf8_spec(), "", marks=needs_utf8), + ], +) +def test_fill_value_for(spec, expected): + got = fill_value_for(spec) + if expected is None: + assert np.isnan(got) + else: + assert got == expected + + +def test_fill_value_for_timestamp_decodes_to_nat(): + """int64.min is NaT's bit pattern, so a null timestamp reads back as NaT.""" + fill = fill_value_for(blosc2.timestamp()) + assert fill == np.iinfo(np.int64).min + assert np.isnat(np.array([fill], dtype="datetime64[us]")[0]) + + +def test_fill_value_is_not_recorded_in_the_schema(): + """Recording it would recreate sentinel collisions at the metadata layer.""" + d = schema_to_dict(_schema(v=(float, blosc2.float64(null_storage="mask")))) + assert "fill_value" not in d["columns"][0] + assert "null_value" not in d["columns"][0] + + +# --------------------------------------------------------------------------- +# The declaration survives every persistence route +# --------------------------------------------------------------------------- + +_MASK_ROW_FIELDS = [ + ("m", int, blosc2.field(blosc2.int64(null_storage="mask"))), + ("f", bool, blosc2.field(blosc2.bool(null_storage="mask"))), + ("s", str, blosc2.field(blosc2.string(max_length=4, null_storage="mask"))), + ("c", complex, blosc2.field(blosc2.complex128(nullable=True))), + ("plain", int, blosc2.field(blosc2.int64())), +] +_MASK_PAYLOAD = { + "m": [1, 2], + "f": [True, False], + "s": ["ab", "cd"], + "c": [1 + 2j, 3 + 4j], + "plain": [7, 8], +} +_EXPECTED_STORAGE = {"m": "mask", "f": "mask", "s": "mask", "c": "mask", "plain": None} + + +def _mask_table(): + t = CTable(dataclasses.make_dataclass("MaskRow", _MASK_ROW_FIELDS)) + t.extend(_MASK_PAYLOAD) + return t + + +def _storage_of(table): + return {c.name: c.spec.null_storage for c in table._schema.columns} + + +@pytest.mark.parametrize("ext", [".b2d", ".b2z"]) +def test_storage_survives_save_and_open(tmp_path, ext): + path = tmp_path / f"masked{ext}" + _mask_table().save(str(path)) + reopened = blosc2.open(str(path)) + assert _storage_of(reopened) == _EXPECTED_STORAGE + assert reopened["s"][:].tolist() == ["ab", "cd"] + assert reopened["c"][:].tolist() == [1 + 2j, 3 + 4j] + # A mask column keeps its natural dtype across the round trip. + assert reopened._schema.columns_by_name["f"].dtype == np.dtype(np.bool_) + assert reopened._schema.columns_by_name["s"].dtype == np.dtype("U4") + + +def test_storage_survives_to_cframe(): + restored = blosc2.ctable_from_cframe(_mask_table().to_cframe()) + assert _storage_of(restored) == _EXPECTED_STORAGE + + +def test_storage_survives_copy(): + """Nothing auto-migrates: copy() preserves each column's storage.""" + assert _storage_of(_mask_table().copy()) == _EXPECTED_STORAGE + + +# --------------------------------------------------------------------------- +# The default flip (4.11.0) +# --------------------------------------------------------------------------- + + +def test_a_bare_nullable_column_gets_a_mask(): + """What the flip changes, stated once for each kind that has a choice.""" + for spec, annotation in [ + (blosc2.int8(nullable=True), int), + (blosc2.float64(nullable=True), float), + (blosc2.bool(nullable=True), bool), + (blosc2.string(max_length=4, nullable=True), str), + (blosc2.bytes(max_length=4, nullable=True), bytes), + *([(blosc2.utf8(nullable=True), str)] if HAVE_UTF8 else []), + (blosc2.timestamp(nullable=True), object), + ]: + col = _resolved(spec, annotation) + assert col.spec.uses_mask, spec + assert col.spec.null_value is None, spec + + +def test_a_stored_table_keeps_the_storage_it_was_written_with(tmp_path): + """The flip governs *creation* only; opening never re-resolves anything. + + Which is the whole reason it is safe: every table already on disk carries + its own answer, sentinel included, and reading one does not consult a policy. + """ + Row = dataclasses.make_dataclass( + "OldRow", + [("flag", bool, blosc2.field(blosc2.bool(nullable=True, null_storage="sentinel")))], + ) + t = blosc2.CTable(Row, urlpath=str(tmp_path / "t.b2t"), mode="w", expected_size=4) + t.extend([(1,), (255,), (0,)]) + del t + + reopened = blosc2.open(str(tmp_path / "t.b2t")) + assert reopened["flag"].null_storage == "sentinel" + assert reopened["flag"].dtype == np.dtype(np.uint8) + assert reopened["flag"].null_value == 255 + assert reopened["flag"].is_null().tolist() == [False, True, False] + + +def test_a_mask_default_table_records_schema_version_3(tmp_path): + """Only a table that *uses* a mask needs a reader that understands one.""" + from blosc2.schema_compiler import schema_to_dict + + Row = dataclasses.make_dataclass("V3Row", [("v", int, blosc2.field(blosc2.int64(nullable=True)))]) + t = blosc2.CTable(Row, expected_size=4) + assert schema_to_dict(t._schema)["version"] == 3 + + Plain = dataclasses.make_dataclass("V1Row", [("v", int, blosc2.field(blosc2.int64()))]) + assert schema_to_dict(blosc2.CTable(Plain, expected_size=4)._schema)["version"] == 1 + + +def test_the_flip_is_one_kwarg_from_the_old_behaviour(): + col = _resolved(blosc2.int64(nullable=True), int, null_storage="sentinel") + assert not col.spec.uses_mask + assert col.spec.null_value == np.iinfo(np.int64).min diff --git a/tests/ctable/test_nullable.py b/tests/ctable/test_nullable.py index 779fe34c3..3f328f756 100644 --- a/tests/ctable/test_nullable.py +++ b/tests/ctable/test_nullable.py @@ -107,14 +107,21 @@ def test_null_value_string(): def test_nullable_true_uses_default_null_policy(): + """The type-wide sentinels a policy picks, once sentinel storage is asked for. + + A bare ``nullable=True`` resolves to a mask since 4.11.0, so this pins the + *sentinel* half of the resolution -- which is still what + ``null_storage="sentinel"`` and any type-wide policy field select. + """ + @dataclass class Row: - i: int = blosc2.field(blosc2.int32(nullable=True)) - u: int = blosc2.field(blosc2.uint32(nullable=True)) - f: float = blosc2.field(blosc2.float64(nullable=True)) - flag: bool = blosc2.field(blosc2.bool(nullable=True)) - s: str = blosc2.field(blosc2.string(max_length=4, nullable=True)) - b: bytes = blosc2.field(blosc2.bytes(max_length=4, nullable=True)) + i: int = blosc2.field(blosc2.int32(nullable=True, null_storage="sentinel")) + u: int = blosc2.field(blosc2.uint32(nullable=True, null_storage="sentinel")) + f: float = blosc2.field(blosc2.float64(nullable=True, null_storage="sentinel")) + flag: bool = blosc2.field(blosc2.bool(nullable=True, null_storage="sentinel")) + s: str = blosc2.field(blosc2.string(max_length=4, nullable=True, null_storage="sentinel")) + b: bytes = blosc2.field(blosc2.bytes(max_length=4, nullable=True, null_storage="sentinel")) t = CTable(Row) assert t["i"].null_value == np.iinfo(np.int32).min diff --git a/tests/ctable/test_parquet_interop.py b/tests/ctable/test_parquet_interop.py index 908becedb..cff49f058 100644 --- a/tests/ctable/test_parquet_interop.py +++ b/tests/ctable/test_parquet_interop.py @@ -430,8 +430,7 @@ def test_utf8_arrow_roundtrip_no_singleton_list(self): out = t.to_arrow() if HAVE_STRING_DTYPE: assert t["txt"].is_utf8 - nv = t["txt"].null_value - assert list(t["txt"][:]) == ["short", long_str, nv, "end"] + assert t["txt"].is_null().tolist() == [False, False, True, False] # Export back to Arrow → still a scalar string column, not list assert pa.types.is_large_string(out.schema.field("txt").type) else: @@ -467,8 +466,7 @@ def test_utf8_parquet_roundtrip(self, tmp_path): assert t["txt"].is_varlen_scalar if HAVE_STRING_DTYPE: assert t["txt"].is_utf8 - nv = t["txt"].null_value - assert list(t["txt"][:]) == ["short", long_str, nv, "end"] + assert t["txt"].is_null().tolist() == [False, False, True, False] else: assert not t["txt"].is_utf8 # vlstring fallback, native-None nulls assert list(t["txt"][:]) == ["short", long_str, None, "end"] @@ -507,12 +505,25 @@ class NullableListRow: assert t2["vals"][2] == [3] def test_scalar_null_no_sentinel_raises(self, tmp_path): - """Importing Parquet scalar nulls without a null_value sentinel fails.""" + """Under *sentinel* storage, nulls with no sentinel available still fail. + + The default is mask storage, which needs no sentinel and so has no such + failure mode -- see :meth:`test_scalar_null_defaults_to_a_mask`. + """ at = pa.table({"score": pa.array([1.0, None, 3.0], type=pa.float64())}) path = tmp_path / "nulls.parquet" pq.write_table(at, path) with pytest.raises(TypeError, match="null_value sentinel"): - CTable.from_parquet(path, auto_null_sentinels=False) + CTable.from_parquet(path, auto_null_sentinels=False, null_storage="sentinel") + + def test_scalar_null_defaults_to_a_mask(self, tmp_path): + """No sentinel to choose, and none needed: nullity goes in the sidecar.""" + at = pa.table({"score": pa.array([1.0, None, 3.0], type=pa.float64())}) + path = tmp_path / "nulls.parquet" + pq.write_table(at, path) + t = CTable.from_parquet(path, auto_null_sentinels=False) + assert t["score"].null_storage == "mask" + assert t["score"].is_null().tolist() == [False, True, False] def test_scalar_null_exported_as_parquet_null(self, tmp_path): """Sentinel values become Parquet nulls on export.""" @@ -546,7 +557,10 @@ def test_auto_nullable_scalars_roundtrip(self, tmp_path): assert t["s"].null_count() == 1 assert t["b"].null_count() == 1 assert t["flag"].null_count() == 1 - assert t["flag"][:].tolist() == [1, 255, 0] + # A mask-backed nullable bool is a real np.bool_ column: no reserved 255, + # and the null slot holds the (unobservable) False fill. + assert t["flag"].dtype == np.dtype(np.bool_) + assert t["flag"].is_null().tolist() == [False, True, False] out = tmp_path / "nullable_scalars_out.parquet" t.to_parquet(out) rt = pq.read_table(out) @@ -653,7 +667,7 @@ def test_nullable_bool_filter_semantics(self, tmp_path): at = pa.table({"flag": pa.array([True, None, False], type=pa.bool_())}) path = tmp_path / "nullable_bool.parquet" pq.write_table(at, path) - t = CTable.from_parquet(path) + t = CTable.from_parquet(path, null_storage="sentinel") assert t.where(t.flag).flag[:].tolist() == [1] assert t.where(~t.flag).flag[:].tolist() == [0] assert t.where(t.flag == True).flag[:].tolist() == [1] # noqa: E712 @@ -661,6 +675,18 @@ def test_nullable_bool_filter_semantics(self, tmp_path): assert t.where(t.flag != True).flag[:].tolist() == [0] # noqa: E712 assert t.where(t.flag != False).flag[:].tolist() == [1] # noqa: E712 + def test_nullable_bool_filter_semantics_under_a_mask(self, tmp_path): + """The same semantics with no reserved 255 and no rewrite to get there.""" + at = pa.table({"flag": pa.array([True, None, False], type=pa.bool_())}) + path = tmp_path / "nullable_bool_mask.parquet" + pq.write_table(at, path) + t = CTable.from_parquet(path) + assert t["flag"].dtype == np.dtype(np.bool_) + assert t.where(t.flag).flag[:].tolist() == [True] + assert t.where(~t.flag).flag[:].tolist() == [False] + assert t.where(t.flag == True).flag[:].tolist() == [True] # noqa: E712 + assert t.where(t.flag == False).flag[:].tolist() == [False] # noqa: E712 + # --------------------------------------------------------------------------- # Error handling diff --git a/tests/ctable/test_utf8.py b/tests/ctable/test_utf8.py index 13bdd3a07..e0d7bfa29 100644 --- a/tests/ctable/test_utf8.py +++ b/tests/ctable/test_utf8.py @@ -32,7 +32,7 @@ class Row: @dataclass class NullableRow: - name: str = blosc2.field(blosc2.utf8(nullable=True)) + name: str = blosc2.field(blosc2.utf8(nullable=True, null_storage="sentinel")) x: int = blosc2.field(blosc2.int64()) @@ -712,8 +712,8 @@ def test_ctable_utf8_comparison_excludes_null_rows(): def test_ctable_utf8_column_vs_column_comparison(): @dataclass class TwoCols: - a: str = blosc2.field(blosc2.utf8(nullable=True)) - b: str = blosc2.field(blosc2.utf8(nullable=True)) + a: str = blosc2.field(blosc2.utf8(nullable=True, null_storage="sentinel")) + b: str = blosc2.field(blosc2.utf8(nullable=True, null_storage="sentinel")) t = CTable(TwoCols, new_data={"a": ["x", "y", None, "z"], "b": ["x", "z", "q", None]}) eq = t[t.a == t.b] @@ -1296,15 +1296,31 @@ def test_utf8_from_arrow_large_string_ingest(): def test_utf8_from_arrow_nulls_use_sentinel(): + """Importing under an explicitly sentinel policy still reserves a string. + + The *default* is mask storage now, which is what makes a free-text utf8 + column round-trip at all (any string is a legal value, so no sentinel is + safe); this pins that asking for the old behaviour still gets it. + """ pa = pytest.importorskip("pyarrow") at = pa.table({"name": pa.array(["a", None, "c"], type=pa.string())}) - t = CTable.from_arrow(at.schema, at.to_batches()) + t = CTable.from_arrow(at.schema, at.to_batches(), null_storage="sentinel") nv = t["name"].null_value assert nv is not None assert list(t["name"][:]) == ["a", nv, "c"] assert t["name"].null_count() == 1 +def test_utf8_from_arrow_nulls_default_to_a_mask(): + pa = pytest.importorskip("pyarrow") + at = pa.table({"name": pa.array(["a", None, "c"], type=pa.string())}) + t = CTable.from_arrow(at.schema, at.to_batches()) + assert t["name"].null_storage == "mask" + assert t["name"].null_value is None + assert t["name"].is_null().tolist() == [False, True, False] + assert t["name"].null_count() == 1 + + def test_utf8_from_arrow_fixed_width_max_len(): pa = pytest.importorskip("pyarrow") at = pa.table({"name": pa.array(["hi", "there"], type=pa.string())}) @@ -1759,7 +1775,9 @@ def test_ctable_utf8_index_reopen_nulls_last(tmp_path): from dataclasses import make_dataclass path = str(tmp_path / "utf8_index.b2t") - row_cls = make_dataclass("Row", [("name", str, blosc2.field(blosc2.utf8(nullable=True)))]) + row_cls = make_dataclass( + "Row", [("name", str, blosc2.field(blosc2.utf8(nullable=True, null_storage="sentinel")))] + ) values = ["pear", "apple", None, "banana"] t = blosc2.CTable(row_cls, urlpath=path, mode="w") t.extend({"name": values}, validate=False) @@ -1817,7 +1835,7 @@ def test_ctable_utf8_index_answers_scalar_predicates(nullable, tmp_path): values = ["pear", "apple", "café", "banana", "apple", "pear"] if nullable: values = [*values, None] - spec = blosc2.utf8(nullable=True) if nullable else blosc2.utf8() + spec = blosc2.utf8(nullable=True, null_storage="sentinel") if nullable else blosc2.utf8() row_cls = make_dataclass("Row", [("c", str, blosc2.field(spec))]) masks = {} @@ -1902,7 +1920,9 @@ def test_ctable_utf8_index_ne_on_all_null_column(tmp_path): """ from dataclasses import make_dataclass - row_cls = make_dataclass("Row", [("c", str, blosc2.field(blosc2.utf8(nullable=True)))]) + row_cls = make_dataclass( + "Row", [("c", str, blosc2.field(blosc2.utf8(nullable=True, null_storage="sentinel")))] + ) t = blosc2.CTable(row_cls, urlpath=str(tmp_path / "t.b2t"), mode="w") t.extend({"c": [None] * 6}, validate=False) t._flush_varlen_columns() diff --git a/tests/ctable/utf8_compat.py b/tests/ctable/utf8_compat.py new file mode 100644 index 000000000..41afc0698 --- /dev/null +++ b/tests/ctable/utf8_compat.py @@ -0,0 +1,43 @@ +####################################################################### +# Copyright (c) 2019-present, Blosc Development Team +# All rights reserved. +# +# SPDX-License-Identifier: BSD-3-Clause +####################################################################### + +"""Letting utf8 cases sit in suites that must still run on NumPy 1.x. + +``blosc2.utf8()`` raises on NumPy < 2.0, which has no ``StringDType``. That is +fine inside a test body -- :data:`needs_utf8` skips it -- but a parametrize list +is built at *collection* time, so the call raises before any ``skipif`` can act +and takes the whole module down with it. :func:`utf8_spec` returns ``None`` +there instead, and the mark keeps the placeholder from ever being dereferenced. + +Suites that are entirely about utf8 do not need this: they skip at module level +(see ``test_utf8.py``). This is for the null-storage suites, where utf8 is one +kind among many and everything else must still be exercised. +""" + +from __future__ import annotations + +import numpy as np +import pytest + +import blosc2 + +#: Whether this NumPy can back a utf8 column at all. +HAVE_UTF8 = hasattr(np.dtypes, "StringDType") + +#: Skip a test that builds a utf8 column in its body. +needs_utf8 = pytest.mark.skipif( + not HAVE_UTF8, reason="utf8 columns require NumPy >= 2.0 (numpy.dtypes.StringDType)" +) + + +def utf8_spec(**kwargs): + """``blosc2.utf8(**kwargs)``, or ``None`` where NumPy cannot build one. + + Pair the ``None`` with ``marks=needs_utf8`` on the parametrize entry, so the + placeholder is collected but never used. + """ + return blosc2.utf8(**kwargs) if HAVE_UTF8 else None