Add MIAPPE metadata with an editor and legacy migration - #116
Merged
Merged
Conversation
…alidation Define the canonical MIAPPE-aligned biological metadata structure for Scan and validate it at the create and metadata-write boundaries (metadata.py, core.py). The block is optional; when present, known fields are type-checked.
…CLIs Add fsdb_migrate_metadata (one-time conversion of legacy Metadata.object to the MIAPPE tree, idempotent) and fsdb_clean_images_metadata (removes the now-redundant biological/hardware entries duplicated in images.json). Drop the legacy images.json fallback read in _load_scan_metadata.
get_scan_info now reads the canonical biologicalMaterial/study structure for species, plant and environment instead of the legacy Metadata.object block. Update scan API tests accordingly.
Add a Dash web UI to view, edit and bulk-fill MIAPPE-aligned biological metadata of local FSDB scans, with per-field specs, flatten/unflatten helpers and write-back with schema validation. Wire up the metadata_app CLI entry point.
Drop support for loose scan directories; a database root must now be a proper ROMI DB. Surface a clearer NotAnFSDBError message from FSDB.connect.
…r UI Auto-load the DB from --db-path, flag scans still on the legacy object schema, and add a one-click 'Migrate now' action in the editor. Add scan_needs_migration/migratable_scans/migrate_scans helpers in db_ops.
diskcache backs Dash background callbacks (used by the metadata editor for streaming migration progress). Ignore the .dashcache runtime directory.
Wrap list_scans in try/finally so the database connection is released even on error, matching other call sites.
Cache connected FSDB instances per database path (connect is expensive) with thread-safe access, close_db to switch databases, and atexit cleanup. Add migrate_scans_progress to report per-scan progress to a caller.
…gration modal Restructure the layout (FSDB location + About on top, scan selection beside a single field-edit form), merging the separate per-scan and bulk-edit flows into one form driven by the edit-accordion mode. Move migration behind a modal with a streaming progress bar via a DiskCache background callback, auto-closing on completion. Close cached DBs when switching databases.
Add Bootstrap icons to card headers, accordion and MIAPPE section titles, move field tooltips to a question-mark icon on each row, and add app.css to size the field labels consistently.
ArthurLuciani2
requested changes
Sep 16, 2026
- Remove duplicated `_load_fileset_metadata` implementation and relocate it after scan helper. - Simplify `_load_scan_metadata` to directly return `_load_metadata`. - Update `file_ops` to use `_load_scan_metadata` and `_fileset_metadata_path` for images metadata.
Stop reading and writing metadata.json via raw path manipulation in the editor helpers. Go through the Scan API (get_metadata/set_metadata) so the editor transparently handles both flat and nested (timelapse) scans. - replace load_db with all_scan_metadata (get_scans + get_metadata) - read/write_scan_metadata now take a Scan; write uses _scan_metadata_path and set_metadata, keeping the .bak backup and MIAPPE validation - apply_bulk, scan_needs_migration, migratable_scans use the Scan API - drop the _SCAN_METADATA_REL constant and the dead migrate_scans wrapper - remove now-unused read_scan_metadata and the json import - fix a NameError in apply_edit by importing _connect in app.py - update tests for the new API and drop the migration-wrapper test
…imelapse scans In _load_scan_at, populate scan.metadata before loading filesets so _scan_path can resolve nested (timelapse) scan paths from the timelapse id. Also import Fileset locally for the legacy images.json metadata merge. Rewrite test_dual_read_scans to build a member scan through the timelapse API (create_timelapse/create_scan/set_metadata) instead of hand-crafting the on-disk layout.
Remove a stray leading space before the >>> prompt so the doctest runs.
ArthurLuciani2
approved these changes
Sep 23, 2026
- New requirement for Open3D. - Updated `docker/Dockerfile` to install `libusb-1.0-0` alongside existing Open3D system dependencies (`libegl1`, `libgl1`, `libgomp1`) in both the runtime and build stages.
- Update `metadata.py` to call `_fileset_metadata_json_path` instead of `_fileset_metadata_path` when loading fileset metadata.
- In `file_ops.py`, extract possible `timelapse_id` from the scan path and pass it to `Scan`. - Extend `Scan.__init__` in `core.py` to accept `timelapse_id=None`, auto‑create the timelapse if needed, and store it in metadata. - Update doctest examples in `core.py` to showcase initializing a `Scan` with a `timelapse_id`. - Add safe fallback when the path does not contain a timelapse component.
… tests - Add `test_fsdb_reload_timelapse_scan` in `src/commons/tests/test_fsdb.py` to verify `FSDB.reload()` correctly loads a scan nested inside a timelapse container without raising an exception. - Add `test_scan_timelapse_id_parameter` to ensure `Scan(timelapse_id=…)` creates the timelapse, registers it in the DB, and stores the correct metadata. - Add `test_scan_timelapse_id_none` to confirm that creating a `Scan` without a `timelapse_id` results in empty timelapse metadata.
- Update `src/commons/plantdb/commons/fsdb/file_ops.py`: - Refactor directory listing for readability. - Store scans in the result dict by `scan.id` instead of directory name. - Adjust error reporting to use actual child directory names. - Remove the now‑unused `scan_id` argument from `_load_scan_at` and update its signature. - Resolve scan paths with `Path(...).resolve()` and extract `timelapse_id`/`scan_id` from the path relative to the DB root. - Add unit tests in `src/commons/tests/test_file_ops.py`: - `test_load_scan_at_success` verifies that a scan can be loaded from a valid explicit path and that its `id` matches. - `test_load_scan_at_invalid_path` ensures that loading from a non‑scan path returns `None`.
- Replace `self.set_metadata("timelapse", {"id": timelapse_id})` with `self.metadata["timelapse"] = {"id": timelapse_id}` in `src/commons/plantdb/commons/fsdb/core.py`.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds MIAPPE-aligned biological metadata support to PlantDB: a write-time schema/validator in the commons layer, CLI migration for legacy scans, and a web UI in the metadata editor to view, edit, and migrate scan metadata.
Changes
fsdb_migrate_metadata, now with backup /--no-backup) andimages.jsoncleanup commands.developers/miappe_metadata.md(schema mapping & design) and wired it into the mkdocs nav.What's new since the last update
fsdb_migrate_metadatawrites a.bakbackup by default and gained a--no-backupoption.get_metadata/set_metadata) instead of raw path manipulation, so it transparently handles both flat and nested (timelapse) scans.load_databaseto return 7 values.Notes
metadata_miappe_plan.md→miappe_metadata.md.