cloudscope-data

Sample datasets for AcqStore and CloudScope.

AcqStore discovers available samples from the catalog on the main branch:

https://raw.githubusercontent.com/mapmanager/cloudscope-data/main/catalog.json

Large sample ZIP files are stored as GitHub Release assets rather than committed to this repository. CloudScope uses the AcqStore sample-data API and does not maintain its own dataset registry.

Repository layout

cloudscope-data/
├── README.md
├── catalog.json
└── scripts/
    ├── build_sample_archive.py
    └── validate_catalog.py

Generated ZIP files are written to ignored dist/.

Catalog format

catalog.json is an ordered JSON list of objects. List order is the client display order.

A folder sample:

{
  "id": "diameter-sample-data",
  "label": "Diameter Sample Data",
  "description": "TIF kymograph sample data for diameter analysis demos.",
  "url": "https://github.com/mapmanager/cloudscope-data/releases/download/v0.1.0/diameter-sample-data.zip",
  "sha256": "c84048c404497585cfa2813b00cf8993d838237f4c796d609abe6024b5ebbc24"
}

A single-file sample adds primary_file:

{
  "id": "kymograph-flow",
  "label": "Kymograph Flow (single file)",
  "description": "One OIR line-scan kymograph for velocity and heart-rate demos.",
  "url": "https://github.com/mapmanager/cloudscope-data/releases/download/v0.1.0/kymograph-flow.zip",
  "sha256": "0000000000000000000000000000000000000000000000000000000000000000",
  "primary_file": "20251014_A98_0002.oir"
}

Key reference

Consumers must treat unlisted keys as an error, so a typo cannot silently disable a feature. scripts/validate_catalog.py rejects unknown keys.

Key	Type	Required	Meaning
id	string	yes	Permanent sample identifier. Lowercase letters, numbers, and single hyphens (^[a-z0-9]+(?:-[a-z0-9]+)*$). Also the required name of the ZIP's single top-level directory, the ZIP file name (<id>.zip), and the key callers pass to the AcqStore sample-data API. Never reuse or rename an id; publish a new one instead.
label	string	yes	User-facing display text. CloudScope renders one Load <label> menu item per entry, in list order. Safe to edit after publication.
description	string	yes	One-sentence user-facing summary, suitable for UI help text. Safe to edit after publication.
url	string	yes	Full HTTPS URL of the immutable GitHub Release asset holding the ZIP. Must begin with https://github.com/mapmanager/cloudscope-data/releases/download/. Point at a release tag, never at a branch, so the bytes behind the URL never change.
sha256	string	yes	Lowercase 64-character hex SHA-256 digest of the ZIP at url. Clients verify the download against this digest and refuse the sample on mismatch. It is also the sample's cache identity: AcqStore extracts into <id>-<first 12 hex chars>, so changing the digest makes clients re-download rather than reuse stale data.
primary_file	string	no	Presence marks the entry as a single-file sample. Path, relative to the top-level <id>/ directory, of the one recording that AcqImage(...) should open. Uses / separators, and must not be absolute, end in /, or contain . or .. segments. For TIFF samples this is the .tif file (not a wrapper folder). For real directory-backed stores such as *.ome.zarr, it may name that directory. Absent means the entry is a folder sample loaded with AcqImageList.

How consumers read an entry

• Every entry, folder or single-file, extracts to a folder that AcqImageList can load. primary_file is purely additive; it never changes how an existing entry behaves.
• acqstore.sample_data.ensure_sample(id) returns the extracted <id>/ folder for any entry.
• acqstore.sample_data.ensure_sample_file(id) returns <extracted>/<primary_file> and raises for entries without primary_file.
• Clients written before primary_file existed ignore the key and keep treating the entry as a folder sample.

ZIP layout

Every ZIP must contain exactly one top-level directory whose name equals the catalog id. This rule is the same for folder and single-file samples.

A folder sample:

diameter-sample-data.zip
└── diameter-sample-data/
    ├── condition-1/
    ├── condition-2/
    └── ...

A single-file sample (raw microscopy; Olympus .txt kept for TIFF):

kymograph-diameter.zip
└── kymograph-diameter/
    ├── 220110n_0005.tif   <- primary_file
    └── 220110n_0005.txt   <- Olympus header (physical units)

A single-file sample may also ship AcqStore sidecars when built without --no-acqstore-sidecars (default packs every file under the source directory):

kymograph-flow.zip
└── kymograph-flow/
    ├── 20251014_A98_0002.oir
    ├── 20251014_A98_0002.oir.json
    └── 20251014_A98_0002.oir.radon_velocity.csv

The top-level directory must be directly loadable by AcqImageList as a folder.

"Single file" means one loadable recording. The archive may also contain that recording's companions: an Olympus .txt header next to a TIFF, and optionally AcqStore <source>.json / analysis CSV files. Wrapper folders such as *.tif.frames are a data-layout convention only; they are not an AcqStore import extension. Point primary_file at the .tif (or .oir, …) that AcqImage opens.

Build a sample archive

From the repository root:

python scripts/build_sample_archive.py \
    data-samples/diameter-sample-data \
    --label "Diameter Sample Data" \
    --description "TIF kymograph sample data for diameter analysis demos." \
    --release-tag v0.1.0

The source directory name is used as the default id. Use --id only when a different stable ID is required.

The command:

1. Creates dist/<id>.zip with the required top-level directory.
2. Validates the ZIP layout, including that primary_file is present when given.
3. Computes its SHA-256 digest.
4. Constructs the GitHub Release asset URL.
5. Adds the item to catalog.json.

Build a single-file archive

Pass a single file as the source. Its name becomes the default id and the primary_file:

python scripts/build_sample_archive.py \
    data-samples/velocity-sample-data/"7d Control"/20251014/20251014_A98_0002.oir \
    --id kymograph-flow \
    --label "Kymograph Flow (single file)" \
    --description "One OIR line-scan kymograph for velocity and heart-rate demos." \
    --release-tag v0.1.0

To pack from a directory that holds one recording plus neighbors (for example a *.tif.frames wrapper folder), name the primary path and choose a sidecar policy:

• Default: every file under the source directory is packed (AcqStore .json and analysis CSV included when present).
• --no-acqstore-sidecars: pack only the primary file plus a sibling Olympus <stem>.txt when present. Omit AcqStore .json and analysis CSV.

python scripts/build_sample_archive.py \
    data-samples/diameter-sample-data/Control/220110n_0005.tif.frames \
    --id kymograph-diameter \
    --primary-file 220110n_0005.tif \
    --no-acqstore-sidecars \
    --label "Kymograph Diameter (single file)" \
    --description "One TIF line-scan kymograph for diameter and sum-intensity demos." \
    --release-tag v0.1.0

--primary-file is relative to the top-level <id>/ directory, which is the same as being relative to the source directory. The build fails if that path is not in the resulting ZIP. Omit --primary-file for a directory source and the entry stays a folder sample.

Preview without writing files:

python scripts/build_sample_archive.py \
    data-samples/diameter-sample-data \
    --label "Diameter Sample Data" \
    --description "TIF kymograph sample data for diameter analysis demos." \
    --release-tag v0.1.0 \
    --dry-run

To intentionally update an existing catalog item, add --replace:

python scripts/build_sample_archive.py \
    data-samples/diameter-sample-data \
    --label "Diameter Sample Data" \
    --description "Updated TIF kymograph sample data for diameter analysis demos." \
    --release-tag v0.1.0 \
    --replace

Validate the catalog

python scripts/validate_catalog.py

This checks required fields, ID and digest formats, release URL prefix, duplicate IDs, unknown keys, and primary_file path safety. Run it before every push.

Publish a sample

1. Prepare the local source directory or file.
2. Run scripts/build_sample_archive.py.
3. Inspect the generated ZIP and the catalog.json change.
4. Run scripts/validate_catalog.py.
5. Create or open the matching GitHub release.
6. Upload dist/<id>.zip as a release asset.
7. Open the URL recorded in catalog.json and verify that it downloads.
8. Commit and push catalog.json and any documentation changes.

Upload and verify the ZIP before pushing the catalog update so clients never see an entry whose release asset is unavailable.

Managed sample data

Downloaded samples are AcqStore-managed example data. AcqStore may replace a local sample when its catalog checksum changes. Copy modified sample data elsewhere before treating it as permanent work.