Skip to content

fix(deps): restore zstd-jni so .zst and .tar.zst files are extracted again - #211

Merged
marevol merged 1 commit into
mainfrom
fix/restore-zstd-jni
Sep 15, 2026
Merged

marevol merged 1 commit into
mainfrom
fix/restore-zstd-jni

Conversation

@marevol

@marevol marevol commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

Found while verifying 15.9.0. A .zst or .tar.zst file that 15.8.0 indexes with its text is indexed by 15.9.0 with no text at all. There is no error or warning, and the crawl reports success. .gz and .tar.gz files are not affected.

Cause

zstd-jni was never declared anywhere in the Fess build. It arrived transitively with the org.opensearch:opensearch jar. codelibs/fess#3439 dropped that jar, and zstd-jni went with it.

Losing it does not fail loudly because both libraries that use it treat it as optional:

  • Tika's org.apache.tika.parser.pkg.CompressorParser adds application/zstd to its supported types only if Class.forName("com.github.luben.zstd.ZstdInputStream") succeeds. It swallows the ClassNotFoundException (CompressorParser.java:105 in tika-parser-pkg-module 3.3.2).
  • commons-compress declares zstd-jni as an optional dependency.

Without the class, Tika stops treating the file as compressed. The document still reaches tikaExtractor: Fess's defaultRule hands it to FessStandardTransformer, which falls back to tikaExtractor for application/zstd. But nothing is decompressed, so the extracted content is empty.

The same class is missing for fess-ds-box. box-java-sdk 5.15.0 decodes zstd-encoded responses in com.box.sdk.ZstdInterceptor with new ZstdInputStream(InputStream). The plugin shades box-java-sdk but not zstd-jni, and expects Fess to provide it. A zstd-encoded response therefore ends in NoClassDefFoundError: com/github/luben/zstd/ZstdInputStream.

Change

fess-crawler/pom.xml now declares com.github.luben:zstd-jni:1.5.6-1 at runtime scope, next to tika-parser-pkg-module, with a comment explaining why a dependency nothing calls is needed. Runtime scope carries it into fess-crawler-lasta and from there into the Fess war; Fess's exclusions on fess-crawler-lasta do not cover it.

This puts back what #3439 removed by accident; it is not a new capability.

  • Version. 1.5.6-1 is the jar 15.8.0 shipped. commons-compress 1.28.0 is built against 1.5.7-4, but every zstd-jni member it references (Zstd level and parameter bounds, ZstdInputStream read, mark and reset, and the ZstdOutputStream setters) exists in 1.5.6-1. box-java-sdk 5.15.0 only needs ZstdInputStream(InputStream). fess-parent does not manage zstd-jni, so the version is declared inline, the same way commons-csv already is. Moving to 1.5.7-x can be a separate bump.
  • Size. The jar is 6.48 MiB (6,795,212 bytes). It is that large because it bundles the native zstd library for 17 OS/architecture combinations: darwin, freebsd, linux and windows. That is 15.1 MiB uncompressed, and it is what makes .zst decompression work without a system library. It is the same jar 15.8.0 distributed.

No configuration or API changes.

Tests

TikaExtractorTest gains three tests, each with a tiny fixture whose token appears nowhere else:

  • test_getTika_zst: extractor/zst/test.txt.zst (48 bytes)
  • test_getTika_tarzst: extractor/zst/test.tar.zst (106 bytes, a tar with one text member)
  • test_getTika_gz: extractor/gz/test.txt.gz (49 bytes), the gzip counterpart that has to pass with or without this change

Verification

  • mvn -pl fess-crawler test -Dtest=TikaExtractorTest without the pom.xml change: 50 tests, 2 failures (test_getTika_zst, test_getTika_tarzst). test_getTika_gz and the existing test_getTika_targz pass.
  • The same command with the change: 50 tests, 0 failures.
  • mvn dependency:tree -pl fess-crawler,fess-crawler-lasta: com.github.luben:zstd-jni:jar:1.5.6-1:runtime appears in both modules. Before the change it appears in neither.
  • mvn -pl fess-crawler test: 2,060 tests, 1 failure, unrelated to this change. Hc5HttpClientTest.test_doHead_accessTimeoutTarget failed while the machine was under load from other builds and passes when its class is run alone (32 tests, 0 failures).

…again

zstd-jni used to reach Fess through the OpenSearch jar. codelibs/fess#3439
dropped that jar, and zstd-jni went with it. Nothing failed loudly: Tika's
CompressorParser registers application/zstd only when
com.github.luben.zstd.ZstdInputStream can be loaded (CompressorParser.java:105),
and commons-compress declares zstd-jni optional. Since then a crawled .zst or
.tar.zst file is indexed without its text and without a warning, while .gz and
.tar.gz keep working.

The same missing class breaks fess-ds-box: box-java-sdk decodes zstd-encoded
responses in com.box.sdk.ZstdInterceptor with ZstdInputStream, and the plugin
does not bundle zstd-jni, so such a response ends in NoClassDefFoundError.

Declare zstd-jni 1.5.6-1 at runtime scope in fess-crawler, the version 15.8.0
shipped. Every zstd-jni member that commons-compress 1.28.0 and box-java-sdk
5.15.0 reference exists in it. fess-parent does not manage zstd-jni, so the
version is declared inline. The jar is 6.48 MiB because it carries native
libraries for 17 platforms.

TikaExtractorTest gains a .zst, a .tar.zst and a .gz fixture, each with a token
that appears nowhere else. The two zstd tests fail without the dependency; the
gzip test passes either way.
@marevol marevol added this to the 15.9.0 milestone Sep 15, 2026
@marevol marevol added bug dependencies Pull requests that update a dependency file labels Sep 15, 2026
@marevol
marevol merged commit 91d47ae into main Sep 15, 2026
1 check passed
marevol added a commit to codelibs/docker-fess that referenced this pull request Sep 16, 2026
codelibs/fess-crawler#211 put zstd-jni back into the distribution so that .zst
and .tar.zst files are extracted again. The jar is 6.5 MB because it bundles
its native library for 17 OS and architecture combinations, while a Linux
container only ever loads linux/<os.arch>, the resource zstd-jni looks up.

Every snapshot image, full and slim, now deletes the other libraries from the
jar right after installing Fess from the zip, deb or rpm, which leaves 390 KB
on arm64 and 466 KB on amd64. The architecture name comes from the image's own
java, and the build fails if the jar has no library under it rather than
producing an image that cannot extract those files. zip is installed for the
step and removed again, with the unzip it pulls in on Amazon Linux 2023. On
Noble and Amazon Linux 2023 the rewritten jar is given back to fess:fess, the
owner the packages set.

The README's Image Variants section mentions it.
marevol added a commit to codelibs/docker-fess that referenced this pull request Sep 16, 2026
)

codelibs/fess-crawler#211 put zstd-jni back into the distribution so that .zst
and .tar.zst files are extracted again. The jar is 6.5 MB because it bundles
its native library for 17 OS and architecture combinations, while a Linux
container only ever loads linux/<os.arch>, the resource zstd-jni looks up.

Every snapshot image, full and slim, now deletes the other libraries from the
jar right after installing Fess from the zip, deb or rpm, which leaves 390 KB
on arm64 and 466 KB on amd64. The architecture name comes from the image's own
java, and the build fails if the jar has no library under it rather than
producing an image that cannot extract those files. zip is installed for the
step and removed again, with the unzip it pulls in on Amazon Linux 2023. On
Noble and Amazon Linux 2023 the rewritten jar is given back to fess:fess, the
owner the packages set.

The README's Image Variants section mentions it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug dependencies Pull requests that update a dependency file

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant