feat(fs): let a file crawl leave symbolic links alone - #199
Merged
Merged
Conversation
FileSystemClient walks a directory with java.io.File, which follows a symbolic link and offers no way not to. A link is therefore crawled as though it were the directory or the file it points at, and the target is indexed a second time -- under the crawling configuration that reached the link, not the one that covers the target. The second copy carries the permissions of the link. Where a share is readable by more people than the tree it links into, anyone who can create a link on that share can have documents from the other tree re-indexed under their own roles and read them back through search. The same file also ends up in the index twice, which distorts the hit count and the ranking. Until now the only way to avoid it was to name the link in the excluded paths of every configuration. A followSymlink init parameter is added instead. It defaults to true, so nothing changes for an existing crawl; set client.followSymlink=false in the configuration parameters of a file crawling configuration and a link is refused with 403 and left out of the child urls of the directory it sits in, with one INFO line naming what was skipped.
marevol
force-pushed
the
fix/filesystem-symlink
branch
from
September 6, 2026 03:01
4c0d347 to
6aeba4a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
FileSystemClient walks a directory with java.io.File, which follows a symbolic
link and offers no way not to. A link is therefore crawled as though it were the
directory or the file it points at, and the target is indexed a second time --
under the crawling configuration that reached the link, not the one that covers
the target.
The second copy carries the permissions of the link. Where a share is readable
by more people than the tree it links into, anyone who can create a link on that
share can have documents from the other tree re-indexed under their own roles
and read them back through search. The same file also ends up in the index
twice, which distorts the hit count and the ranking.
Until now the only way to avoid it was to name the link in the excluded paths of
every configuration. A followSymlink init parameter is added instead. It
defaults to true, so nothing changes for an existing crawl; set
client.followSymlink=false in the configuration parameters of a file crawling
configuration and a link is refused with 403 and left out of the child urls of
the directory it sits in, with one INFO line naming what was skipped.