From fee81da94c7981265ce0dba70d5009fcf20fe211 Mon Sep 17 00:00:00 2001 From: bruno-f-cruz <7049351+bruno-f-cruz@users.noreply.github.com> Date: Thu, 13 Aug 2026 08:37:42 -0700 Subject: [PATCH 1/2] Add harp-python migration guide --- docs/articles/migrating_from_harp_python.md | 188 ++++++++++++++++++++ mkdocs.yml | 2 + 2 files changed, 190 insertions(+) create mode 100644 docs/articles/migrating_from_harp_python.md diff --git a/docs/articles/migrating_from_harp_python.md b/docs/articles/migrating_from_harp_python.md new file mode 100644 index 0000000..fdd015c --- /dev/null +++ b/docs/articles/migrating_from_harp_python.md @@ -0,0 +1,188 @@ +# Migrating from harp-python + +`harp-data` is the successor to `harp-python` for reading Harp binary data files into +pandas DataFrames. The core concepts are unchanged — device schemas, register maps, +binary files — but the API has been reorganized to separate data reading from device +communication. + +This guide covers the three workflows most users relied on in `harp-python`. + +--- + +## Swap the package + +Replace the old dependency: + +```sh title="Before" +pip install harp-python +``` + +```sh title="After" +pip install harp-data +``` + +If you want the full toolkit — serial transport, device client, and data reading — install +the umbrella package instead: + +```sh +pip install harp +``` + +--- + +## Loading a device schema at runtime + +In `harp-python`, `harp.create_reader()` accepted a dataset folder and handled +schema loading internally. `create_dataset_reader` is the direct replacement — it +finds the `device.yml` inside the folder automatically: + +```python title="Before" +import harp + +reader = harp.create_reader("session.harp") +``` + +```python title="After" +from harp.data import create_dataset_reader + +reader = create_dataset_reader("session.harp") +``` + +If the schema lives outside the data folder, pass it explicitly: + +```python +reader = create_dataset_reader("session.harp", schema="/path/to/device.yml") +``` + +### Accessing the device module + +The reader holds a reference to the compiled device module at `reader.device_module`. +Use it to look up register classes by name — no need to keep a separate variable: + +```python +reader = create_dataset_reader("session.harp") + +# access any register class through the reader +df = reader.read(reader.device_module.AnalogData) +``` + +!!! note + The old `harp.read_schema()` had no direct equivalent you needed to call separately. + `create_dataset_reader` handles schema loading in one step, matching the convenience + of the original API. + +--- + +## Reading a single register + +The old API gave you attribute access on the reader — `reader.AnalogData.read()`. The +new API inverts this: you call `reader.read()` and pass the register class or its +address as the argument. + +```python title="Before" +# by attribute name +df = reader.AnalogData.read() + +# by name string +df = reader.registers["AnalogData"].read() + +# by address +df = reader.registers[44].read() +``` + +```python title="After" +# by register class (accessed through the reader) +df = reader.read(reader.device_module.AnalogData) + +# by address +df = reader.read(44) +``` + +### Absolute timestamps + +The `epoch` parameter moves from the reader constructor into the `read()` call: + +```python title="Before" +reader = harp.create_reader("session.harp", epoch=harp.REFERENCE_EPOCH) +df = reader.AnalogData.read() +``` + +```python title="After" +from harp.data import REFERENCE_EPOCH + +df = reader.read(reader.device_module.AnalogData, epoch=REFERENCE_EPOCH) +``` + +### Reading the whole session at once + +`read_all()` returns a dictionary of DataFrames keyed by register name. Registers +with no corresponding `.bin` file are skipped: + +```python +everything: dict[str, pd.DataFrame] = reader.read_all() +``` + +### Parameter reference + +| harp-python | harp-data | Notes | +|---|---|---| +| `keep_type=True` | `message_type=True` | Renamed | +| `epoch=REFERENCE_EPOCH` | `epoch=REFERENCE_EPOCH` | Same | +| `epoch=None` | `epoch=None` (default) | Float seconds; same | +| — | `decode_enums=True` | New: enum fields as `pd.Categorical` | +| — | `demux_bit_masks=False` | New: expand bitmask flags into one column per flag | + +--- + +## Schemaless read + +If you have a raw `.bin` file and no schema — or you just want to inspect the data +quickly — the `read()` function works the same as before. Only the import path and +one parameter name change: + +```python title="Before" +import harp + +df = harp.read("Behavior_44.bin") +df = harp.read("Behavior_44.bin", keep_type=True) +``` + +```python title="After" +from harp.data import read + +df = read("Behavior_44.bin") +df = read("Behavior_44.bin", message_type=True) +``` + +Both functions infer the payload type and element count automatically from the first +frame — no register metadata needed. + +--- + +## Going further: static device packages + +Loading a YAML at runtime is convenient, but for production workflows — or when you +want IDE autocompletion and type safety — Harp device packages are pre-compiled Python +modules that give you the same interface without any schema parsing at startup. + +A static device package installs its register map as a proper Python module. You +import it, pass it directly to `DatasetReader`, and the rest of the API is identical: + +```python +pip install harp-device-behavior +``` + +```python +from harp.device.behavior import device as behavior +from harp.data import DatasetReader + +reader = DatasetReader(behavior, "session.harp") + +# everything works the same +df = reader.read(behavior.AnalogData) +everything = reader.read_all() +``` + +The static module is faster to start up and ships with stubs for autocompletion. See +[Generating Registers from a Schema](../examples/create_device_module/create_device_module.md) +for how device modules are structured. diff --git a/mkdocs.yml b/mkdocs.yml index 357ca93..089afb1 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -79,6 +79,8 @@ nav: - Subscribing to Events: examples/subscribing_to_events/subscribing_to_events.md - Reading a Whole Dataset Folder: examples/read_dataset/read_dataset.md - Reading Data into a DataFrame: examples/read_data_to_dataframe/read_data_to_dataframe.md + - Articles: + - Migrating from harp-python: articles/migrating_from_harp_python.md - API: - Protocol: api/protocol.md - Serial: api/serial.md From c555285d70b3da66de4658b7abe4f9ba515d6e2f Mon Sep 17 00:00:00 2001 From: glopesdev Date: Sat, 22 Aug 2026 01:39:33 +0100 Subject: [PATCH 2/2] Correct the migration guide against the API open_dataset replaces create_dataset_reader throughout, read takes a register name as well as a class or address, and the epoch example and the parameter table match the current reader: epoch is set once on open_dataset, keep_type keeps its harp-python spelling, and time_index is new, since harp-python inferred the index from the first frame. read_all is gone, so whole-session loading becomes a comprehension over contents, which gains a section of its own. A new section warns that harp-data returns a bitmask register as a single integer column unless demux_bit_masks is set, so a script selecting a flag by name needs it. The static package example imports from the harp.device namespace and names no distribution. Em-dashes, second person and two mid-sentence enumerations are rephrased, and the horizontal rules are removed. --- docs/articles/migrating_from_harp_python.md | 142 +++++++++++--------- 1 file changed, 82 insertions(+), 60 deletions(-) diff --git a/docs/articles/migrating_from_harp_python.md b/docs/articles/migrating_from_harp_python.md index fdd015c..a468f30 100644 --- a/docs/articles/migrating_from_harp_python.md +++ b/docs/articles/migrating_from_harp_python.md @@ -1,14 +1,12 @@ # Migrating from harp-python `harp-data` is the successor to `harp-python` for reading Harp binary data files into -pandas DataFrames. The core concepts are unchanged — device schemas, register maps, -binary files — but the API has been reorganized to separate data reading from device +pandas DataFrames. The core concepts of device schemas, register maps and binary files +are unchanged, but the API has been reorganized to separate data reading from device communication. This guide covers the three workflows most users relied on in `harp-python`. ---- - ## Swap the package Replace the old dependency: @@ -21,20 +19,18 @@ pip install harp-python pip install harp-data ``` -If you want the full toolkit — serial transport, device client, and data reading — install -the umbrella package instead: +For the full toolkit of serial transport, device client and data reading, install the +umbrella package instead: ```sh pip install harp ``` ---- - ## Loading a device schema at runtime In `harp-python`, `harp.create_reader()` accepted a dataset folder and handled -schema loading internally. `create_dataset_reader` is the direct replacement — it -finds the `device.yml` inside the folder automatically: +schema loading internally. `open_dataset` is the direct replacement, and it finds the +`device.yml` inside the folder automatically: ```python title="Before" import harp @@ -43,41 +39,44 @@ reader = harp.create_reader("session.harp") ``` ```python title="After" -from harp.data import create_dataset_reader +from harp.data import open_dataset -reader = create_dataset_reader("session.harp") +reader = open_dataset("session.harp") ``` If the schema lives outside the data folder, pass it explicitly: ```python -reader = create_dataset_reader("session.harp", schema="/path/to/device.yml") +reader = open_dataset("session.harp", schema="/path/to/device.yml") ``` -### Accessing the device module +### Finding out what a session holds -The reader holds a reference to the compiled device module at `reader.device_module`. -Use it to look up register classes by name — no need to keep a separate variable: +`reader.contents` maps the name of every register with data in the folder to its address, +in address order, which is the quickest way to see what was recorded: ```python -reader = create_dataset_reader("session.harp") +reader = open_dataset("session.harp") +print(reader.contents) # {'WhoAmI': 0, 'DigitalInputState': 32, ...} +``` -# access any register class through the reader +The reader also holds the compiled device module at `reader.device_module`, so a register +class can be reached without keeping a separate variable: + +```python df = reader.read(reader.device_module.AnalogData) ``` !!! note - The old `harp.read_schema()` had no direct equivalent you needed to call separately. - `create_dataset_reader` handles schema loading in one step, matching the convenience + The old `harp.read_schema()` had no direct equivalent that needed calling separately. + `open_dataset` handles schema loading in one step, matching the convenience of the original API. ---- - ## Reading a single register -The old API gave you attribute access on the reader — `reader.AnalogData.read()`. The -new API inverts this: you call `reader.read()` and pass the register class or its -address as the argument. +The old API allowed reads through attribute access on the reader. The new +API inverts this, so `reader.read()` takes the register class, its name or +its address as the argument. ```python title="Before" # by attribute name @@ -94,13 +93,22 @@ df = reader.registers[44].read() # by register class (accessed through the reader) df = reader.read(reader.device_module.AnalogData) +# by name, the direct analogue of the old string lookup +df = reader.read("AnalogData") + # by address df = reader.read(44) ``` +Names resolve against the device address space rather than the module namespace, so a +common register such as `reader.read("WhoAmI")` works even though a device module does +not name it. + ### Absolute timestamps -The `epoch` parameter moves from the reader constructor into the `read()` call: +The `epoch` parameter keeps its name and moves from `create_reader` to `open_dataset`. +`harp-python` also allowed it per read. `harp-data` sets it once for the dataset, so +every register is read on the same clock. ```python title="Before" reader = harp.create_reader("session.harp", epoch=harp.REFERENCE_EPOCH) @@ -110,35 +118,53 @@ df = reader.AnalogData.read() ```python title="After" from harp.data import REFERENCE_EPOCH -df = reader.read(reader.device_module.AnalogData, epoch=REFERENCE_EPOCH) +reader = open_dataset("session.harp", epoch=REFERENCE_EPOCH) +df = reader.read(reader.device_module.AnalogData) ``` ### Reading the whole session at once -`read_all()` returns a dictionary of DataFrames keyed by register name. Registers -with no corresponding `.bin` file are skipped: +There is no `read_all()`. Whole-session loading is a comprehension over `contents`, +which keeps the choice of what to load with the caller: ```python -everything: dict[str, pd.DataFrame] = reader.read_all() +everything = {name: reader.read(name) for name in reader.contents} ``` +### Bitmask registers lose their per-flag columns by default + +This is the change most likely to break working code. `harp-python` always expanded a +bitmask register into one boolean column per flag, so a script could select a flag by +name. `harp-data` returns a single integer column instead, and expands the flags only +when asked: + +```python title="Before" +led = reader.DigitalOutputSet.read()["GP15"] +``` + +```python title="After" +led = reader.read("DigitalOutputSet", demux_bit_masks=True)["GP15"] +``` + +Group masks need no such flag. `harp-python` mapped each value to its member name, and +`harp-data` decodes them by default, as a `pd.Categorical` rather than plain strings, so +a comparison against a string still reads naturally. + ### Parameter reference | harp-python | harp-data | Notes | |---|---|---| -| `keep_type=True` | `message_type=True` | Renamed | -| `epoch=REFERENCE_EPOCH` | `epoch=REFERENCE_EPOCH` | Same | -| `epoch=None` | `epoch=None` (default) | Float seconds; same | -| — | `decode_enums=True` | New: enum fields as `pd.Categorical` | -| — | `demux_bit_masks=False` | New: expand bitmask flags into one column per flag | - ---- +| `keep_type=True` | `keep_type=True` | Same, but the column is named `message_type` rather than `MessageType` | +| `epoch=REFERENCE_EPOCH` | `epoch=REFERENCE_EPOCH` | Same, but set on `open_dataset` rather than per read | +| `epoch=None` | `epoch=None` (default) | Float seconds, same | +| inferred from the first frame | `time_index=False` | Needed for data carrying no timestamp | +| always on | `demux_bit_masks=True` | Needed to keep per-flag columns | +| always on | `decode_enums=True` (default) | Group mask values, now `pd.Categorical` | ## Schemaless read -If you have a raw `.bin` file and no schema — or you just want to inspect the data -quickly — the `read()` function works the same as before. Only the import path and -one parameter name change: +For a raw `.bin` file with no schema, or a quick look at the data, the `read()` +function works the same as before. Only the import path changes: ```python title="Before" import harp @@ -151,38 +177,34 @@ df = harp.read("Behavior_44.bin", keep_type=True) from harp.data import read df = read("Behavior_44.bin") -df = read("Behavior_44.bin", message_type=True) +df = read("Behavior_44.bin", keep_type=True) ``` -Both functions infer the payload type and element count automatically from the first -frame — no register metadata needed. - ---- +Both functions infer the payload type and element count from the frame, so no register +metadata is needed. The new one assumes timestamped data, which is what a device sends, +and takes `time_index=False` for the rare buffer that is not. It also takes `epoch` per +call, since a single file has no dataset to set one on. ## Going further: static device packages -Loading a YAML at runtime is convenient, but for production workflows — or when you -want IDE autocompletion and type safety — Harp device packages are pre-compiled Python -modules that give you the same interface without any schema parsing at startup. - -A static device package installs its register map as a proper Python module. You -import it, pass it directly to `DatasetReader`, and the rest of the API is identical: +Loading a YAML at runtime is convenient, but for production workflows, or where IDE +autocompletion and type checking matter, a generated device package gives the same +interface without parsing a schema at startup. -```python -pip install harp-device-behavior -``` +Such a package is an ordinary Python module under the `harp.device` namespace. Import +it, pass it to `open_dataset`, and the rest of the API is identical: ```python -from harp.device.behavior import device as behavior -from harp.data import DatasetReader +from harp.device import behavior +from harp.data import open_dataset -reader = DatasetReader(behavior, "session.harp") +reader = open_dataset("session.harp", behavior) -# everything works the same +# a register class now resolves statically df = reader.read(behavior.AnalogData) -everything = reader.read_all() ``` -The static module is faster to start up and ships with stubs for autocompletion. See +A generated module starts up faster and resolves under a type checker, which a module +built from a schema at runtime cannot. See [Generating Registers from a Schema](../examples/create_device_module/create_device_module.md) for how device modules are structured.