Any data used for analyses in the KBase Lakehouse needs to remain accessible so that future users can rerun the same analyses with the same input datasets.
Small datasets (less than 500GB)
Raw reference data downloaded from external data sources should be stored in the refdata tenant using the following directory structure:
cdm-lake/tenant-general-warehouse/refdata/datasets/<dataset_name>/raw_data/<version>/<files>
For example, the last couple of UniProt releases ("2025-03", "2026-01", "2026-02") are stored as follows:
[refdata path]/datasets/uniprot/2025-03/<files>
/2026-01/<files>
/2026-02/<files>
GTDB:
[refdata path]/datasets/gtdb/220.0/<files>
/226.0/<files>
/232.0/<files>
For datasets without a formal release number or cadence, the import date (date accessed) can be used instead of release number:
[refdata path]/tcdb/2026_05_01/<files>
/2026_06_01/<files>
Large datasets (500GB+)
For large datasets, storing the entire data dump is likely to be time-consuming to perform and may well result in a lot of redundant data, particularly if only a small proportion of the dataset is updated. In these cases, it would be preferable to store only what has been updated in the release.
For example:
[refdata path]/ncbi/genbank_271.0/GCA_000123456.1/<files>
/GCA_000123678.2/<files>
[refdata path]/ncbi/genbank_272.0/GCA_000654321.1/<files>
[refdata path]/ncbi/genbank_273.0/GCA_000123456.1/<files> # data updated in 271 and 273
File metadata
All files in the raw data directory should have file metadata stored in frictionless format; files should be represented as Data Resources, and cohesive groupings of files can be stored as a Data Package. These metadata files will be used to create an index of the raw data files available in the Lakehouse.
Metadata files are in JSON format and are stored in a subdirectory of the raw_data directory, handily named metadata. They can contain any of the frictionless data types -- resource, package, etc.
See #167 for the proposed file indexing script that can be used to maintain a db table of available raw data by parsing the files in the metadata directory.
TODO
Any data used for analyses in the KBase Lakehouse needs to remain accessible so that future users can rerun the same analyses with the same input datasets.
Small datasets (less than 500GB)
Raw reference data downloaded from external data sources should be stored in the refdata tenant using the following directory structure:
For example, the last couple of UniProt releases ("2025-03", "2026-01", "2026-02") are stored as follows:
GTDB:
For datasets without a formal release number or cadence, the import date (date accessed) can be used instead of release number:
Large datasets (500GB+)
For large datasets, storing the entire data dump is likely to be time-consuming to perform and may well result in a lot of redundant data, particularly if only a small proportion of the dataset is updated. In these cases, it would be preferable to store only what has been updated in the release.
For example:
File metadata
All files in the raw data directory should have file metadata stored in frictionless format; files should be represented as Data Resources, and cohesive groupings of files can be stored as a Data Package. These metadata files will be used to create an index of the raw data files available in the Lakehouse.
Metadata files are in JSON format and are stored in a subdirectory of the
raw_datadirectory, handily namedmetadata. They can contain any of the frictionless data types -- resource, package, etc.See #167 for the proposed file indexing script that can be used to maintain a db table of available raw data by parsing the files in the metadata directory.
TODO