Skip to content

feat(fs): let a file crawl leave symbolic links alone - #199

Merged
marevol merged 1 commit into
masterfrom
fix/filesystem-symlink
Sep 6, 2026
Merged

marevol merged 1 commit into
masterfrom
fix/filesystem-symlink

Conversation

@marevol

@marevol marevol commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

FileSystemClient walks a directory with java.io.File, which follows a symbolic
link and offers no way not to. A link is therefore crawled as though it were the
directory or the file it points at, and the target is indexed a second time --
under the crawling configuration that reached the link, not the one that covers
the target.

The second copy carries the permissions of the link. Where a share is readable
by more people than the tree it links into, anyone who can create a link on that
share can have documents from the other tree re-indexed under their own roles
and read them back through search. The same file also ends up in the index
twice, which distorts the hit count and the ranking.

Until now the only way to avoid it was to name the link in the excluded paths of
every configuration. A followSymlink init parameter is added instead. It
defaults to true, so nothing changes for an existing crawl; set
client.followSymlink=false in the configuration parameters of a file crawling
configuration and a link is refused with 403 and left out of the child urls of
the directory it sits in, with one INFO line naming what was skipped.

FileSystemClient walks a directory with java.io.File, which follows a symbolic
link and offers no way not to. A link is therefore crawled as though it were the
directory or the file it points at, and the target is indexed a second time --
under the crawling configuration that reached the link, not the one that covers
the target.

The second copy carries the permissions of the link. Where a share is readable
by more people than the tree it links into, anyone who can create a link on that
share can have documents from the other tree re-indexed under their own roles
and read them back through search. The same file also ends up in the index
twice, which distorts the hit count and the ranking.

Until now the only way to avoid it was to name the link in the excluded paths of
every configuration. A followSymlink init parameter is added instead. It
defaults to true, so nothing changes for an existing crawl; set
client.followSymlink=false in the configuration parameters of a file crawling
configuration and a link is refused with 403 and left out of the child urls of
the directory it sits in, with one INFO line naming what was skipped.
@marevol
marevol force-pushed the fix/filesystem-symlink branch from 4c0d347 to 6aeba4a Compare September 6, 2026 03:01
@marevol marevol added this to the 15.9.0 milestone Sep 6, 2026
@marevol marevol self-assigned this Sep 6, 2026
@marevol
marevol merged commit 734a71b into master Sep 6, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant