Reconcile the data reader parameters - #37
Merged
Merged
Conversation
bruno-f-cruz
approved these changes
Aug 21, 2026
read, parse_to_dataframe and DatasetReader.read now take keep_type, leaving message_type to to_buffer, to_file and format_bulk, where it carries a MessageType. Passing message_type to a reader now raises instead of accepting any truthy value. The inserted column keeps its name.
read, parse_to_dataframe and DatasetReader.read take time_index in place of timestamp and epoch. True indexes by Harp time in float seconds, a datetime such as REFERENCE_EPOCH gives an absolute DatetimeIndex, and False gives a RangeIndex. Anything else raises TypeError, so a value that happens to be truthy no longer passes for a request to index by time. read gains the absolute case, which it had no parameter for before.
time_index narrows to a bool, and the epoch it used to carry becomes its own parameter. open_dataset and the DatasetReader constructor take the epoch for the whole dataset, so every register reads on one clock and DatasetReader.read no longer sets one at all. parse_to_dataframe and read keep an epoch parameter, having no dataset to take it from. _resolve_epoch goes with the union, and with it the TypeError it raised for a value that was neither a bool nor a datetime, since epoch is now the parameter that takes one. The three remaining Union annotations in harp-data become pipe unions, so the package now spells every union the same way.
glopesdev
force-pushed
the
refactor-data-api
branch
from
August 22, 2026 19:43
cabd486 to
faae84d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two parameter changes on the data reader, both from one mechanism: a flag that only tests truthiness sitting next to a plausible value someone might put there instead. Neither changes behaviour for a call that passes the defaults.
The message type collision
harp.datausedmessage_typefor two different kinds of thing. On the writer it takes a value,to_buffer(reg, values, message_type=MessageType.Event). On the reader it was a boolean asking for a column. Because the flag only tested truthiness, the confusion was accepted rather than reported:A caller who has written the writer call and expects to select reads gets a plausible DataFrame instead. The reader now takes
keep_type, which is whatharp-pythoncalled it, so a migrating script keeps the spelling as well as the behaviour, andmessage_typebelongs to the writer alone. Worth notingharp-pythonhad both words too and kept them apart deliberately,keep_typefor the request andmessage_typefor the value, so this restores a distinction we removed rather than inventing one.The inserted column is still named
message_type, matching the snake_case that payload columns already use. Only columns derived from mask members differ, since those are enum spellings.The time index
The reader had
timestamp: boolbesideepoch: datetime | None, so the epoch had a truthy slot to fall into:timestampbecomestime_index, and the two are separated by level rather than merged. Whether to index by time is a per-read choice, while the epoch that anchors the index describes how the recording was made, soopen_datasetand theDatasetReaderconstructor take the epoch for the whole dataset:DatasetReader.readtherefore has no epoch of its own, so no read can index on a different clock from its siblings and frames from several registers concatenate on one index type.parse_to_dataframeandreadkeep anepochparameter, having no dataset to take it from, andreadgains the absolute case it previously had no parameter for at all.Naming it
time_indexrather than keepingtimestampalso says what the parameter decides, since the data carries timestamps either way and only the index changes.The reason a flag exists here rather than inference is worth recording, since
harp-pythonneeds none. It reads the timestamp bit from the first frame, and an unlogged register has no frames, so inference has nothing to read and must default. The default matters:DatasetReader.readreturns an empty DataFrame with matching columns for a register that was never logged, and it carries an emptyTimeindex only because the flag says so. Give the empty case aRangeIndexand it silently drops out of a concatenation across sessions rather than contributing zero rows. Keeping the parameter keeps that decision explicit.The last three
Unionannotations inharp-dataalso become pipe unions, so the package spells every union the same way.