Add timelapse container support across fsdb, server API, client and CLI - #115
Closed
jlegrand62 wants to merge 153 commits into
Closed
jlegrand62 wants to merge 153 commits into
jlegrand62 wants to merge 153 commits into
Conversation
- Import `NotAnFSDBError` in `core.py` - Import `_is_fsdb` from `plantdb.commons.fsdb.validation` in `core.py` - Add a `_is_fsdb(basedir)` check in `FSDB.__init__` and raise `NotAnFSDBError` if the directory is not a valid FSDB
- Add return type hint `-> bool` to `_is_fsdb` in `validation.py` - Expand `_is_fsdb` docstring with detailed description and usage examples - Verify the provided path is a directory before proceeding - Ensure the presence of the `MARKER_FILE_NAME` file - Introduce scan‑directory validation using new helper `_is_scan_dataset` - Log warnings for empty databases and for any bad scan directories found - Add new function `_is_scan_dataset` to validate FSDB datasets: - Checks for required `files.json` and valid JSON structure - Optionally validates filesets if `validate_json_fileset` is true - Confirms presence of required `metadata` subdirectory - Update `_is_safe_to_delete` signature to `-> bool` and improve its docstring.
- Import `_fileset_path` and `_scan_json_file` from `path_helpers` in `validation.py` - Extend `_is_fsdb` signature to accept `validate_json_fileset` and pass it to scan checks - Rename parameters to `scan_path` in `_is_scan_dataset` and update all internal path usages - Add optional `validate_json_fileset` flag documentation to both `_is_fsdb` and `_is_scan_dataset` - Implement `_is_valid_fileset` to verify fileset directories and required files listed in `files.json` - Update `_is_scan_dataset` to invoke `_is_valid_fileset` when validation is enabled - Adjust calls to `_is_scan_dataset` in `_is_fsdb` to include the new flag - Refactor variable names and path handling for clarity across the validation module
- Import `_is_scan_dataset` in `file_ops.py` (pre‑load for validation utilities). - Remove `required_fs` handling in `get_scans`; now only checks that the scan directory exists before loading filesets. - Comment out the user prompt (`yes_no_choice`) and deletion loop for bad scans, preventing interactive prompts in non‑TTY environments. - Keep existing logic for loading filesets and updating scans unchanged.
…mentation
- Update `pyproject.toml` script entry: replace `fsdb_check` with `fsdb_healthcheck`.
- Modify `validation.py` to suggest the new `fsdb_healthcheck` CLI instead of a TODO comment.
- Add new CLI module `src/commons/plantdb/commons/cli/fsdb_healthcheck.py`:
- Replace `argparse` with `click` for argument parsing.
- Introduce `--log-level`, `--fix`, `--fix-missing`, and `--fix-extra` options.
- Configure logger via `get_logger('fsdb_healthcheck', ...)`.
- Implement missing‑reference fixing logic with progress bar and backup handling.
- Stub `--fix-extra` with `NotImplementedError`.
- Remove old `fsdb_check.py` implementation.
- Log an error when the provided path is not a directory. - Log an error when the required marker file `MARKER_FILE_NAME` is missing. - Update empty‑FSDB warning to use the path string directly. - Store bad scan directories as strings and simplify the bad‑scan log output. - Add explicit error logging for missing `metadata` subdirectory. - Add error logs for missing `files.json`, JSON parse failures, and missing `filesets` entry. - Log an error when a fileset directory defined in `files.json` is absent. - Introduce `_fileset_files_exists` helper to verify all required files exist and log the count of missing files. - Update `_is_valid_fileset` to use the new helper for detailed missing‑file reporting.
…ional files‑json updates - Updated `file_ops.py` to use `Path` from `pathlib` and added `typing` imports for clearer type hints. - Replaced direct `shutil.rmtree` calls with `send2trash` for safer, reversible deletions of scans, filesets, and metadata directories. - Introduced `backup_file` usage before overwriting `files.json` when `updates_files_json` is enabled. - Extended `_load_scan` signature to `def _load_scan(db: 'FSDB', scan_id: str, updates_files_json: bool = False) -> 'Scan | None'` and added detailed docstring. - Modified `_load_scans` to return a `dict[str, 'Scan']`, accept `updates_files_json` flag, and improved handling of bad scans (no interactive prompts). - Added validation imports (`_is_valid_fileset`, `_is_scan_dataset`) and guarded type‑checking imports with `TYPE_CHECKING` to avoid circular dependencies. - Updated helper functions (`_load_scan_filesets`, `_load_fileset`, `_load_fileset_files`) to return a `(result, needs_update)` tuple, propagating the update flag. - Adjusted internal calls to reflect new return signatures and update logic. - Replaced legacy `yes_no_choice` prompt handling with commented‑out code, eliminating interactive deletion in non‑TTY environments. - Updated function signatures for `_load_dummy_fileset`, `_load_file`, `_load_measures`, `_load_scan_measures`, `_delete_file`, `_delete_fileset`, `_delete_scan`, `_make_fileset`, `_make_scan`, and `_store_scan` with explicit type hints and return annotations.
- Reformat `yes_no_choice` signature to use explicit spacing (`default: bool = True`). - Introduce `yes_no_abort_choice` in `utils.py` to allow aborting a yes/no prompt and return `None` when aborted. - Add `backup_filename` function to create a timestamped backup path for a given file. - Add `backup_file` function that copies the original file to the backup path generated by `backup_filename`.
…nnect - Drop the `required_filesets` attribute, its `__init__` parameter, and related docstring sections in `core.py` - Initialize scans without setting `self.required_filesets`; default validation now relies on the presence of a `metadata` fileset - Add an `_is_fsdb(self.basedir)` check in `FSDB.connect` to raise `NotAnFSDBError` when the directory is not a valid FSDB - Clean up import comments and unused code related to required filesets.
- Updated `plantdb/src/commons/pyproject.toml` to include ``send2trash`` in the `dependencies` list, enabling safe recycle‑bin deletions.
- Grouped CLI options with `click_option_group` into “Fix” and “Logging” sections and reordered parameters (`fsdb_path`, `fix`, `fix_missing`, `fix_extra`, `log_level`) - Added explicit FSDB marker validation using `MARKER_FILE_NAME` and raise `NotAnFSDBError` when missing - Replaced direct directory check with `Path.is_dir()` and added early error handling for non‑directory paths - Collected scan directories while ignoring hidden folders and added empty‑FSDB warning - Implemented `fix_missing_scans_reference` helper: - Validates each scan with `_is_scan_dataset` - Updates `files.json` via `_load_scan(..., updates_files_json=True)` - Tracks bad scans, prompts user with `yes_no_abort_choice`, and moves them to trash using `send2trash` - Removed old backup and progress‑bar logic; introduced new interactive deletion flow with clear warnings - Updated imports: added `Logger`, `OptionGroup`, `optgroup`, `send2trash`, and `yes_no_abort_choice`; removed unused `datetime`, `json`, `shutil`, and `tqdm` - Adjusted logger initialization comment and eliminated unnecessary `db.connect()`/`db.disconnect()` calls - Updated documentation strings to reflect new behavior and parameters in `fsdb_healthcheck.py`
- Extend `_is_fsdb` signature with `extra_dirs:list[str]=['configs']` and update docstring. - Skip verification of directories listed in `extra_dirs` during scan dataset checks. - Add `extra_dirs` parameter to `FSDB.__init__` (default `['configs']`) and store it as an instance attribute. - Pass `self.extra_dirs` to `_is_fsdb` in `FSDB.connect` to respect extra directory handling.
- Updated `src/commons/pyproject.toml` to include `click_option_group` in the `dependencies` list.
- No functional code changes; the move improves file organization and readability.
Improve FSDB validation and health‑check CLI with extra roots and safe deletions
…REST API registration - Switch imports from `plantdb.client` to `plantdb.commons` in client modules (`rest_api.py`, `plantdb_client.py`) and tests - Relocate `api_endpoints` to `plantdb.commons`, add missing `home` endpoint and update all usage examples - Rewrite server CLI (`fsdb_rest_api.py`) to import endpoints directly and replace verbose `api.add_resource` calls with a concise mapping helper - Reformat function signatures, logging calls, and type hints for consistency - Update docstrings and examples to reflect the new import path and endpoint structure
…pers - Introduce comprehensive path constants (`HOME`, `HEALTH`, `REFRESH`, `REGISTER`, `LOGIN`, `LOGOUT`, `TOKEN_REFRESH`, `TOKEN_VALIDATION`, `CREATE_API_TOKEN`, `SCANS`, `SCANS_INFO`, `SCAN`, `SCAN_MD`, `SCAN_FILESETS`, `FILESET`, `FILESET_MD`, `FILESET_FILES`, `FILE`, `FILE_MD`, `IMAGE`, `POINTCLOUD`, `MESH`, `SKELETON`, `ARCHIVE`, `FILE_PATH`) in **`src/commons/plantdb/commons/api_endpoints.py`**. - Replace all hard‑coded endpoint strings with the new constants and use `.format(...)` for dynamic segments. - Update endpoint functions: - **`pointcloud`** with size, coords, and type validation. - **`mesh`** with size and coords handling. - **`skeleton`** returning the skeleton path. - **`file_path`** endpoint to use the `FILE_PATH` constant. - Adjust docstrings and examples to reflect the new constant‑based paths and updated signatures.
- Replace all JSON error payloads from `{'message': ...}` to `{'error': ...}` across asset, image, pointcloud, mesh, and zip handling endpoints.
- Add `resource_file` helper to retrieve a `File` object with unified error handling and consistent `error` key.
- Refactor `PointCloud`, `Mesh`, `CurveSkeleton`, and `AnglesAndInternodes` resources to use `resource_file`, removing redundant `fileset_id`/`file_id` parameters and related sanitizers.
- Extend PointCloud endpoint to support a `type` query parameter (`default` or `gt`) and simplify request signature.
- Remove `PointCloudGroundTruth` resource as it is now accessible from `PointCloud`.
- Update logging and success responses to align with the new error key convention.
…ource registration - Replace bulk import of `plantdb.commons.api_endpoints` with explicit constant imports (e.g., `ARCHIVE`, `FILE`, `SCAN_MD`, `SCAN_FILESETS`, etc.) - Reorder typing imports for clarity (`Optional` and `Union` on separate lines) - Simplify `_register_resources` by removing the custom `_add` helper, using a flat `RESOURCE_MAP` with endpoint constants formatted via ``.format`` and registering each with `api.add_resource(..., resource_class_args=(db, logger))` - Update function signatures in `_configure_api`, `_setup_test_database`, and `rest_api` for consistent indentation and type hint style - Adjust import ordering and formatting throughout the file for readability
- Define constructor `def __init__(self, db, logger=None)` with detailed docstring - Store `self.db: FSDB = db` and `self.logger: logging.Logger` (fallback to `get_logger`) - Enables dependency injection of database instance and logger for the home endpoint.
- Replace all JSON error payloads from `{'message': ...}` to `{'error': ...}` across authentication endpoints (register, login, logout, token validation, refresh, and API token creation).
- Consolidate response construction using a single `response` variable with proper status codes and return it at the end of each method.
- Add success response for user registration (`User <username> successfully created`).
- Refactor logout to always return a response variable and use the `error` key for failure cases.
- Simplify token validation flow: early return on missing token and unified error handling.
- Update login to build the successful response in the `else` block and use `error` for failure cases.
- Adjust refresh token and API token creation endpoints to follow the new error-key convention and to return responses consistently.
- Remove unused imports (`requests`, `jsonify`, `make_response`).
- Change health‑check error response key from `message` to `error` and move exception handling before the success `else` block.
- Update `Refresh` resource:
- Return `{'error': ...}` on failure instead of `{'message': ...}`.
- Add explicit `else` block for successful reload response.
- Update full‑database reload endpoint:
- Use `{'error': ...}` for exception case.
- Add `else` block for successful reload message.
- Apply consistent try/except/else pattern across the modified endpoints.
- Move `resource_file`, `is_within_directory`, and `is_directory_in_archive` to new **`plantdb/server/api/utils.py`** - Clean up unused imports in `assets.py` (e.g., `os`, `pathlib`, `BytesIO`, `ZipFile`, `numpy`, `requests`) - Import `resource_file` from utils in `assets.py` - Delete the original helper implementations from `assets.py` to avoid duplication - Keep existing functionality unchanged while improving module organization.
- Replace all `{'message': ...}` responses with `{'error': ...}` in **scan**, **fileset**, **file**, and **assets** endpoints
- Update related docstrings and success response structures to reflect the new error key
- Adjust metadata, metadata‑update, and file‑upload error handling accordingly
- Minor refactor: improve logger initialization formatting and simplify `wants_base64` flag parsing in assets API.
…umentation
- Rename `FILE_PATH` constant to use `/files/{scan_id}/{file_path}`
- Minor comment formatting tweak for lower‑case JSON‑style boolean handling
- Import `ARCHIVE`, `FILE`, `HEALTH`, `IMAGE`, `LOGIN`, `SCAN`, and `SCANS` from `plantdb.commons.api_endpoints` - Replace hard‑coded URL strings with the corresponding formatted constants in all request calls - Adjust request constructions for health check, login, scans retrieval, scan metadata, file serving, image thumbnails, and archive download - Add a temporary print of the retrieved file name (debug aid)
- Updated endpoint hierarchy comment block to reflect new `/auth` and `/assets` routes - Prefixed authentication paths (`/register`, `/login`, `/logout`, token routes) with `/auth` and adjusted corresponding constants (`REGISTER`, `LOGIN`, `LOGOUT`, `TOKEN_REFRESH`, `TOKEN_VALIDATION`, `CREATE_API_TOKEN`) - Moved all static asset routes under `/assets` and updated related constants (`IMAGE`, `POINTCLOUD`, `MESH`, `SEQUENCE`, `SKELETON`, `ARCHIVE`, `FILE_PATH`) - Revised URL building functions to use the new constants, including updated example docstrings for `register`, `login`, `logout`, `token_refresh`, `token_validation`, `create_api_token`, `scans_info`, `scan`, `scan_metadata`, `scan_filesets_list`, `fileset`, `fileset_metadata`, `fileset_files_list`, `file_metadata`, `image`, `sequence`, `pointcloud`, `mesh`, `skeleton`, `archive` - Modified `file_path` helper signature to accept only `file_path` (removed `scan_id` argument) and formatted URL with the new `FILE_PATH` pattern - Adjusted all related import paths and comments to match the new namespace structure.
A request to the bare server root (http://host:port/ or the reverse-proxy
root http://host:port/{prefix}/) previously returned 404, because all
resources are mounted under the /api/v1 prefix. Register a lightweight
route at "/" that issues a 302 redirect to the home endpoint, keeping the
deployment (reverse-proxy) prefix in the generated Location so clients
stay under the proxy path.
- fsdb_rest_api.py: add _register_root_redirect() helper that maps the
root path to home(prefix=deploy_prefix) and wire it into rest_api() so
both the fsdb_rest_api CLI and the WSGI entrypoint (wsgi.py) expose the
redirect, honoring the --api-prefix / API_PREFIX value.
- test_rest_api_server.py: add RootRedirectTests covering the no-prefix
and /plantdb-prefix cases (asserting the 302 and exact Location header)
plus a follow-redirect check that lands on the Home payload.
handle missing ScanPath kwargs with fallback values and improve error handling in get_scan_info()
- Remove leading spaces before `>>>` prompts to clean up doctest formatting in `urls.py`
- Update `refresh` signature to `def refresh(scan_id: str | None = None, **kwargs) -> str` and adjust its docstring to "str or None, optional" - Insert a blank line after the `urllib` import for consistency - Remove the stray blank line preceding the `refresh` definition - Reformat parameter indentation in `fileset_metadata`, `file_metadata`, `image`, `pointcloud`, and `mesh` for clearer readability - Minor docstring wording updates (e.g., `key : str or None, optional`)
RE: Fix `/api/v1` prefix consistency and add a root resource to the REST API
Define the abstract timelapse contract before any backend implements it: a TimeLapse groups temporal member scans plus shared metadata. Adds the TimeLapse base class, the Series alias, abstract DB timelapse methods (create/get/list/delete), and Scan.get_timelapse/get_series navigation.
Exclude the .junie plan/draft directory and the large real_plant dataset downloaded via test_database.py (150MB of image fixtures) from version control.
Store a timelapse as a top-level directory holding a timelapse.json marker
and its scans nested under <db>/{tl_id}/{scan_id}/. Standalone scans stay
at the root, and new plantdb still opens old flat databases.
Adds TimeLapse and Series FSDB implementations (create/get/list/delete,
member scan management, metadata, dict-style access), timelapse path and
marker helpers, ID validation, dual-read scan loading (_load_scan_at plus
nested discovery in _load_scans/_load_scan), nested scan directory creation,
_timelapse delete with a non-recursive guard, chronological scan sorting by
timelapse.scheduled with index tie-break, and immutable timelapse.id in
metadata updates. Scan/timelapse creation reject cross-namespace collisions.
…hcheck The healthcheck previously treated each root directory as a scan. It now descends into timelapse containers (marked by timelapse.json) and validates their child scans, and reports the correct total scan count.
Adds /timelapses, /timelapses/{id} and /timelapses/{id}/scans resources with
URL builders in api_endpoints, plus timelapse_id and sort filtering on the
existing scan list and table endpoints.
Adds create/get/list/delete_timelapse methods to PlantDBClient and
timelapse_id/sort filter parameters on scan listing. Filesystem sync now
detects timelapse membership from scan metadata and preserves the nested
<db>/{tl_id}/{scan_id}/ layout across local, HTTP and SFTP transfers.
Tests the abstract and FSDB TimeLapse lifecycle (CRUD, sorting, immutable timelapse.id, collision guards), timelapse path/id validation and nested scan loading, the timelapse REST endpoints, the PlantDBClient SDK methods, and nested-layout preservation during local sync.
- Extend `create_timelapse`, `get_timelapse`, `list_timelapses`, and `delete_timelapse` docstrings with concrete doctest-style examples.
…se docstrings - Delete the `Series` alias for `TimeLapse` in `db.py`. - Remove the `Scan.get_series` method and its documentation. - Refine the `TimeLapse` class docstring to reference only a timelapse object. - Update `Scan.get_timelapse` docstring to reflect the parent timelapse instance.
…fixtures Rewrite the romi import test to source data from the shared dummy dataset instead of the hand-committed testscan fixtures, then retire those fixtures and ignore the rest of testdata so large datasets are not tracked.
…pse_id Collapse the timelapse-specific validator into a single tightened _is_valid_id used by scans, filesets, files and timelapses: 128-char cap, no dots, no leading separator, precompiled regex.
Add dedicated TimeLapseNotFound/Exists errors, persist owner and scans index in timelapse markers, register/unregister member scans on create/delete, refresh last_modified, and expose owner/scans via to_dict and item access.
Switch timelapse handlers from scan errors to the dedicated timelapse errors so missing/existing timelapses return the right status, and assert owner and scans in API responses.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds end-to-end support for timelapses — named containers holding a set of member scans with a shared owner and chronological index — spanning the database layer, the REST API, the Python client, and the CLI.
A timelapse is a directory in the FSDB root identified by a
timelapse.jsonmarker. Each member scan carries ametadata.timelapse.id(andindex) reference. The marker is maintained append/remove-only (never rebuilt on load) and records the container's owner, the ordered list of member scans, and alast_modifiedtimestamp.Highlights
create_timelapse/get_timelapse/delete_timelapse+TimeLapsecontainer class; dedicatedTimeLapseNotFoundError/TimeLapseExistsError; member scans are registered/unregistered in the marker on create/delete; owner is set on first registration and kept on conflict;last_modifiedrefreshed on each mutation./api/v1/timelapsesREST endpoints (list, CRUD, member scans) with auth/rate-limit integration; API handlers map the dedicated timelapse errors to the correct HTTP statuses._is_valid_id(128-char cap, no dots, no leading separator, precompiled regex), removing the timelapse-specific validator.client,commons,server).Tests
Backend (
test_fsdb.py,test_file_ops.py,test_validation.py), REST API (test_timelapse_api.py), and client sync tests were added/updated.The romi import test now sources from the shared dummy dataset instead of hand-committed fixtures (those legacy fixtures are removed and
testdata/is ignored exceptromidb).Breaking / notes
_is_valid_idnow rejects dots and caps length at 128 chars (previously allowed dots and up to 255).