Problem
The dependency table is indexed by filename, and the actual files inside the zip archives on the backend are stored under those same names. Renaming a file between versions is treated as "old file removed + new file added", requiring a full re-upload even though the content is identical.
Proposed solution: archive_path column
Add a column to the dependency table that records the original path of the file inside the archive, used only when it differs from the current index name.
Schema change
In audb/core/define.py, add archive_path to DEPENDENCY_TABLE:
DEPENDENCY_TABLE = {
"archive": "string[pyarrow]",
"archive_path": "string[pyarrow]", # original filename in archive (if renamed)
"bit_depth": "int32[pyarrow]",
"channels": "int32[pyarrow]",
"checksum": "string[pyarrow]",
"duration": "float64[pyarrow]",
"format": "string[pyarrow]",
"removed": "int32[pyarrow]",
"sampling_rate": "int32[pyarrow]",
"type": "int32[pyarrow]",
"version": "string[pyarrow]",
}Publish logic
In _find_media() in audb/core/publish.py: when a file is "new" (not in deps) but its checksum matches a file that's about to be removed, treat it as a rename:
- Copy the old file's dep entry (archive, version, checksum, etc.) to the new filename
- Set
archive_path to the old filename
- Skip upload entirely
Rename detection could be done:
- Automatically via checksum matching (convenient, but could produce false positives if two different files happen to have the same content)
- Explicitly via a
renamed={"old_name": "new_name"} parameter on publish() (safer, more transparent)
Load logic
In _get_media_from_backend() in audb/core/load.py: after extracting an archive, if archive_path is set for a file, rename from archive_path to the actual index name.
Example
# Version 1.0.0: file "audio/old_name.wav" uploaded in archive abc123.zip
# Version 2.0.0: user renames to "audio/new_name.wav"
# Dependency table v2.0.0:
# file archive archive_path version checksum
# audio/new_name.wav abc123 audio/old_name.wav 1.0.0 md5...
During load(), audb downloads abc123.zip (from version 1.0.0), extracts audio/old_name.wav, then renames it to audio/new_name.wav. No re-upload needed.
Things to consider
- Multi-file archives: If two files in the same archive are both renamed, the mapping still works since each file has its own
archive_path.
- Chained renames: If a file is renamed across multiple versions (A -> B in v2, B -> C in v3),
archive_path should always point to the name inside the archive (the original name from when it was uploaded), not the intermediate name. So C's entry would have archive_path=A and version=1.0.0.
- Backward compatibility: Older audb versions won't know about
archive_path. The column can be made optional (empty string = no rename), which is backward compatible for reading. A dependency table format version bump may still be needed.
Alternative: renamed parameter without schema change
A simpler variant that avoids the schema change: add a renamed dict parameter to publish(). For each renamed file, the publish logic would download the old archive, re-pack it with the new filename, and upload. This still requires a round-trip download + upload but avoids the user having to have the media files locally. It's simpler but doesn't fully avoid the upload cost.
Problem
The dependency table is indexed by filename, and the actual files inside the zip archives on the backend are stored under those same names. Renaming a file between versions is treated as "old file removed + new file added", requiring a full re-upload even though the content is identical.
Proposed solution:
archive_pathcolumnAdd a column to the dependency table that records the original path of the file inside the archive, used only when it differs from the current index name.
Schema change
In
audb/core/define.py, addarchive_pathtoDEPENDENCY_TABLE:Publish logic
In
_find_media()inaudb/core/publish.py: when a file is "new" (not in deps) but its checksum matches a file that's about to be removed, treat it as a rename:archive_pathto the old filenameRename detection could be done:
renamed={"old_name": "new_name"}parameter onpublish()(safer, more transparent)Load logic
In
_get_media_from_backend()inaudb/core/load.py: after extracting an archive, ifarchive_pathis set for a file, rename fromarchive_pathto the actual index name.Example
During
load(), audb downloadsabc123.zip(from version 1.0.0), extractsaudio/old_name.wav, then renames it toaudio/new_name.wav. No re-upload needed.Things to consider
archive_path.archive_pathshould always point to the name inside the archive (the original name from when it was uploaded), not the intermediate name. So C's entry would havearchive_path=Aandversion=1.0.0.archive_path. The column can be made optional (empty string = no rename), which is backward compatible for reading. A dependency table format version bump may still be needed.Alternative:
renamedparameter without schema changeA simpler variant that avoids the schema change: add a
renameddict parameter topublish(). For each renamed file, the publish logic would download the old archive, re-pack it with the new filename, and upload. This still requires a round-trip download + upload but avoids the user having to have the media files locally. It's simpler but doesn't fully avoid the upload cost.