Skip to content

Metadata restructure - #391

Open
varunghat wants to merge 3 commits into
phoible:masterfrom
varunghat:refactor/metadata-restructure
Open

Metadata restructure#391
varunghat wants to merge 3 commits into
phoible:masterfrom
varunghat:refactor/metadata-restructure

Conversation

@varunghat

Copy link
Copy Markdown
Contributor

Narrows the two wide InventoryID mapping files down to just what the aggregation pipeline actually uses, adds InventoryID-Filenames.csv and moves phoible-references.bib into mappings/ so all the metadata lives together. Based on the PHOIBLE re-design doc "metadata restructure" section

Changelog:

InventoryID-Bibtex.csv: 5 cols (InventoryID, BibtexKey, Source, Filename, URI) -> 2 cols (InventoryID, BibtexKey). Dropped 23 exact-duplicate rows. 3840 -> 3817 rows.
InventoryID-LanguageCodes.csv: 5 cols (InventoryID, ISO6393, Glottocode, LanguageName, Source) -> 3 cols (InventoryID, Glottocode, ISO6393).
InventoryID-Filenames.csv (new): 3 cols (InventoryID, Filename, URI) take from the wide Bibtex file. 3031 rows for 3020 InventoryIDs.
phoible-references.bib: moved from data/ to mappings/

Fixed the links to phoible-references.bib in 7 source READMEs that referenced it (EA, ER, PH, RA, SAPHON, SPA, UPSID).

Added mappings/README.md

Tested with creating fresh clone and generating phoible.csv. Preserves parity with current committed phoible.csv

Notes:

Current changes remove source and language from the rows, which reduce human readability but add better metadata structure.

@drammock

Copy link
Copy Markdown
Member

@varunghat re: the removed mapping of InventoryID:LanguageName:Source (from InventoryID-LanguageCodes.csv): where does the pipeline get it from if not that CSV file?

@bambooforest probably one for you to review?

@varunghat

Copy link
Copy Markdown
Contributor Author

@varunghat re: the removed mapping of InventoryID:LanguageName:Source (from InventoryID-LanguageCodes.csv): where does the pipeline get it from if not that CSV file?

@drammock "Source" is set in validate_data <- function(dataframe,source_id,...) in aggregation-helper-functions.R
Function called in aggregate-raw-data.R manually for each source

eg:
saphon_data <- validate_data(saphon_data, "saphon", debug=debug)

LanguageName is taken from each raw data file directly

eg:
saphon_data$LanguageName <- keep_first(saphon_data$LanguageName, split.on=" (")

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants