fix(deps): restore zstd-jni so .zst and .tar.zst files are extracted again - #211
Merged
Merged
Conversation
…again zstd-jni used to reach Fess through the OpenSearch jar. codelibs/fess#3439 dropped that jar, and zstd-jni went with it. Nothing failed loudly: Tika's CompressorParser registers application/zstd only when com.github.luben.zstd.ZstdInputStream can be loaded (CompressorParser.java:105), and commons-compress declares zstd-jni optional. Since then a crawled .zst or .tar.zst file is indexed without its text and without a warning, while .gz and .tar.gz keep working. The same missing class breaks fess-ds-box: box-java-sdk decodes zstd-encoded responses in com.box.sdk.ZstdInterceptor with ZstdInputStream, and the plugin does not bundle zstd-jni, so such a response ends in NoClassDefFoundError. Declare zstd-jni 1.5.6-1 at runtime scope in fess-crawler, the version 15.8.0 shipped. Every zstd-jni member that commons-compress 1.28.0 and box-java-sdk 5.15.0 reference exists in it. fess-parent does not manage zstd-jni, so the version is declared inline. The jar is 6.48 MiB because it carries native libraries for 17 platforms. TikaExtractorTest gains a .zst, a .tar.zst and a .gz fixture, each with a token that appears nowhere else. The two zstd tests fail without the dependency; the gzip test passes either way.
marevol
added a commit
to codelibs/docker-fess
that referenced
this pull request
Sep 16, 2026
codelibs/fess-crawler#211 put zstd-jni back into the distribution so that .zst and .tar.zst files are extracted again. The jar is 6.5 MB because it bundles its native library for 17 OS and architecture combinations, while a Linux container only ever loads linux/<os.arch>, the resource zstd-jni looks up. Every snapshot image, full and slim, now deletes the other libraries from the jar right after installing Fess from the zip, deb or rpm, which leaves 390 KB on arm64 and 466 KB on amd64. The architecture name comes from the image's own java, and the build fails if the jar has no library under it rather than producing an image that cannot extract those files. zip is installed for the step and removed again, with the unzip it pulls in on Amazon Linux 2023. On Noble and Amazon Linux 2023 the rewritten jar is given back to fess:fess, the owner the packages set. The README's Image Variants section mentions it.
marevol
added a commit
to codelibs/docker-fess
that referenced
this pull request
Sep 16, 2026
) codelibs/fess-crawler#211 put zstd-jni back into the distribution so that .zst and .tar.zst files are extracted again. The jar is 6.5 MB because it bundles its native library for 17 OS and architecture combinations, while a Linux container only ever loads linux/<os.arch>, the resource zstd-jni looks up. Every snapshot image, full and slim, now deletes the other libraries from the jar right after installing Fess from the zip, deb or rpm, which leaves 390 KB on arm64 and 466 KB on amd64. The architecture name comes from the image's own java, and the build fails if the jar has no library under it rather than producing an image that cannot extract those files. zip is installed for the step and removed again, with the unzip it pulls in on Amazon Linux 2023. On Noble and Amazon Linux 2023 the rewritten jar is given back to fess:fess, the owner the packages set. The README's Image Variants section mentions it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found while verifying 15.9.0. A
.zstor.tar.zstfile that 15.8.0 indexes with its text is indexed by 15.9.0 with no text at all. There is no error or warning, and the crawl reports success..gzand.tar.gzfiles are not affected.Cause
zstd-jniwas never declared anywhere in the Fess build. It arrived transitively with theorg.opensearch:opensearchjar. codelibs/fess#3439 dropped that jar, andzstd-jniwent with it.Losing it does not fail loudly because both libraries that use it treat it as optional:
org.apache.tika.parser.pkg.CompressorParseraddsapplication/zstdto its supported types only ifClass.forName("com.github.luben.zstd.ZstdInputStream")succeeds. It swallows theClassNotFoundException(CompressorParser.java:105in tika-parser-pkg-module 3.3.2).zstd-jnias an optional dependency.Without the class, Tika stops treating the file as compressed. The document still reaches
tikaExtractor: Fess'sdefaultRulehands it toFessStandardTransformer, which falls back totikaExtractorforapplication/zstd. But nothing is decompressed, so the extracted content is empty.The same class is missing for fess-ds-box. box-java-sdk 5.15.0 decodes zstd-encoded responses in
com.box.sdk.ZstdInterceptorwithnew ZstdInputStream(InputStream). The plugin shades box-java-sdk but notzstd-jni, and expects Fess to provide it. A zstd-encoded response therefore ends inNoClassDefFoundError: com/github/luben/zstd/ZstdInputStream.Change
fess-crawler/pom.xmlnow declarescom.github.luben:zstd-jni:1.5.6-1atruntimescope, next totika-parser-pkg-module, with a comment explaining why a dependency nothing calls is needed. Runtime scope carries it intofess-crawler-lastaand from there into the Fess war; Fess's exclusions onfess-crawler-lastado not cover it.This puts back what #3439 removed by accident; it is not a new capability.
zstd-jnimember it references (Zstdlevel and parameter bounds,ZstdInputStreamread, mark and reset, and theZstdOutputStreamsetters) exists in 1.5.6-1. box-java-sdk 5.15.0 only needsZstdInputStream(InputStream). fess-parent does not managezstd-jni, so the version is declared inline, the same waycommons-csvalready is. Moving to 1.5.7-x can be a separate bump..zstdecompression work without a system library. It is the same jar 15.8.0 distributed.No configuration or API changes.
Tests
TikaExtractorTestgains three tests, each with a tiny fixture whose token appears nowhere else:test_getTika_zst:extractor/zst/test.txt.zst(48 bytes)test_getTika_tarzst:extractor/zst/test.tar.zst(106 bytes, a tar with one text member)test_getTika_gz:extractor/gz/test.txt.gz(49 bytes), the gzip counterpart that has to pass with or without this changeVerification
mvn -pl fess-crawler test -Dtest=TikaExtractorTestwithout thepom.xmlchange: 50 tests, 2 failures (test_getTika_zst,test_getTika_tarzst).test_getTika_gzand the existingtest_getTika_targzpass.mvn dependency:tree -pl fess-crawler,fess-crawler-lasta:com.github.luben:zstd-jni:jar:1.5.6-1:runtimeappears in both modules. Before the change it appears in neither.mvn -pl fess-crawler test: 2,060 tests, 1 failure, unrelated to this change.Hc5HttpClientTest.test_doHead_accessTimeoutTargetfailed while the machine was under load from other builds and passes when its class is run alone (32 tests, 0 failures).