WIP Read tracks from a mounted library instead of downloading them - #821
Draft
lachlan-00 wants to merge 1 commit into
Draft
lachlan-00 wants to merge 1 commit into
lachlan-00 wants to merge 1 commit into
Conversation
Every provider reports the path it holds a track at, so an install whose library is mounted into the container can read the file directly: no HTTP round trip, no second copy of the bytes, and no load on the media server. Provider-agnostic - it lives in the dispatcher, and all six backends populate Path/FilePath. Off by default. LOCAL_FILE_ACCESS enables it, LOCAL_FILE_ROOTS allowlists the directories a track may come from, and LOCAL_FILE_PATH_MAP rewrites the server's prefix onto this container's mount point. Two properties the implementation is built around: * The pipeline DELETES whatever download_track returns (the finally in tasks/analysis/album.py), so the library file is never returned. What the caller gets is a link inside TEMP_DIR - a symlink, or a hardlink where symlinks are not permitted - and removing it never touches the target. * The path comes from the media server and is therefore untrusted. The REAL location is resolved first, so a symlink planted in the library cannot escape, and anything outside LOCAL_FILE_ROOTS is refused. With no root configured nothing is read at all. Every failure returns None and the track is downloaded as before, so local access is an optimisation that cannot become a new way for analysis to fail.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
WIP My server has too many files to download and look at files individually so I am mounting the local files in the container using a ro volume
Keeping at draft while i test on my server.
Every provider reports the path it holds a track at, so an install whose library is mounted into the container can read the file directly: no HTTP round trip, no second copy of the bytes, and no load on the media server.
Provider-agnostic - it lives in the dispatcher, and all six backends populate Path/FilePath.
Off by default. LOCAL_FILE_ACCESS enables it, LOCAL_FILE_ROOTS allowlists the directories a track may come from, and LOCAL_FILE_PATH_MAP rewrites the server's prefix onto this container's mount point.
Two properties the implementation is built around:
Every failure returns None and the track is downloaded as before, so local access is an optimisation that cannot become a new way for analysis to fail.
AS-IS
TO-BE
Test
Other useful information
Checklist
Type of change:
Tested on architecture:
Tested on media server:
Updated:
Other:
Related ISSUE: Closes #xx