Skip to content

Reconcile the data reader parameters - #37

Merged
glopesdev merged 3 commits into
mainfrom
refactor-data-api
Aug 22, 2026
Merged

Reconcile the data reader parameters#37
glopesdev merged 3 commits into
mainfrom
refactor-data-api

Conversation

@glopesdev

@glopesdev glopesdev commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Two parameter changes on the data reader, both from one mechanism: a flag that only tests truthiness sitting next to a plausible value someone might put there instead. Neither changes behaviour for a call that passes the defaults.

The message type collision

harp.data used message_type for two different kinds of thing. On the writer it takes a value, to_buffer(reg, values, message_type=MessageType.Event). On the reader it was a boolean asking for a column. Because the flag only tested truthiness, the confusion was accepted rather than reported:

parse_to_dataframe(reg, buf, message_type=MessageType.Read)
# -> columns ['message_type', 'value'], 2 rows, no filter applied, no error

A caller who has written the writer call and expects to select reads gets a plausible DataFrame instead. The reader now takes keep_type, which is what harp-python called it, so a migrating script keeps the spelling as well as the behaviour, and message_type belongs to the writer alone. Worth noting harp-python had both words too and kept them apart deliberately, keep_type for the request and message_type for the value, so this restores a distinction we removed rather than inventing one.

The inserted column is still named message_type, matching the snake_case that payload columns already use. Only columns derived from mask members differ, since those are enum spellings.

The time index

The reader had timestamp: bool beside epoch: datetime | None, so the epoch had a truthy slot to fall into:

parse_to_dataframe(reg, buf, epoch=REFERENCE_EPOCH)      # DatetimeIndex, as meant
parse_to_dataframe(reg, buf, timestamp=REFERENCE_EPOCH)  # Index [1.0, 2.0], datetime dropped

timestamp becomes time_index, and the two are separated by level rather than merged. Whether to index by time is a per-read choice, while the epoch that anchors the index describes how the recording was made, so open_dataset and the DatasetReader constructor take the epoch for the whole dataset:

reader = open_dataset("session.harp", epoch=REFERENCE_EPOCH)
reader.read("AnalogData")                    # DatetimeIndex
reader.read("AnalogData", time_index=False)  # RangeIndex

DatasetReader.read therefore has no epoch of its own, so no read can index on a different clock from its siblings and frames from several registers concatenate on one index type. parse_to_dataframe and read keep an epoch parameter, having no dataset to take it from, and read gains the absolute case it previously had no parameter for at all.

Naming it time_index rather than keeping timestamp also says what the parameter decides, since the data carries timestamps either way and only the index changes.

The reason a flag exists here rather than inference is worth recording, since harp-python needs none. It reads the timestamp bit from the first frame, and an unlogged register has no frames, so inference has nothing to read and must default. The default matters: DatasetReader.read returns an empty DataFrame with matching columns for a register that was never logged, and it carries an empty Time index only because the flag says so. Give the empty case a RangeIndex and it silently drops out of a concatenation across sessions rather than contributing zero rows. Keeping the parameter keeps that decision explicit.

The last three Union annotations in harp-data also become pipe unions, so the package spells every union the same way.

@glopesdev
glopesdev requested a review from bruno-f-cruz August 21, 2026 22:40
Base automatically changed from fix-device-dispatch to main August 22, 2026 19:43
read, parse_to_dataframe and DatasetReader.read now take keep_type,
leaving message_type to to_buffer, to_file and format_bulk, where it
carries a MessageType. Passing message_type to a reader now raises
instead of accepting any truthy value. The inserted column keeps its
name.
read, parse_to_dataframe and DatasetReader.read take time_index in place
of timestamp and epoch. True indexes by Harp time in float seconds, a
datetime such as REFERENCE_EPOCH gives an absolute DatetimeIndex, and
False gives a RangeIndex. Anything else raises TypeError, so a value
that happens to be truthy no longer passes for a request to index by
time.

read gains the absolute case, which it had no parameter for before.
time_index narrows to a bool, and the epoch it used to carry becomes its
own parameter. open_dataset and the DatasetReader constructor take the
epoch for the whole dataset, so every register reads on one clock and
DatasetReader.read no longer sets one at all. parse_to_dataframe and
read keep an epoch parameter, having no dataset to take it from.
_resolve_epoch goes with the union, and with it the TypeError it raised
for a value that was neither a bool nor a datetime, since epoch is now
the parameter that takes one.

The three remaining Union annotations in harp-data become pipe unions,
so the package now spells every union the same way.
@glopesdev
glopesdev merged commit c3d9ab4 into main Aug 22, 2026
13 checks passed
@glopesdev
glopesdev deleted the refactor-data-api branch August 22, 2026 19:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants