build(fess): keep only the zstd-jni native library each image loads - #85
Merged
Merged
Conversation
codelibs/fess-crawler#211 put zstd-jni back into the distribution so that .zst and .tar.zst files are extracted again. The jar is 6.5 MB because it bundles its native library for 17 OS and architecture combinations, while a Linux container only ever loads linux/<os.arch>, the resource zstd-jni looks up. Every snapshot image, full and slim, now deletes the other libraries from the jar right after installing Fess from the zip, deb or rpm, which leaves 390 KB on arm64 and 466 KB on amd64. The architecture name comes from the image's own java, and the build fails if the jar has no library under it rather than producing an image that cannot extract those files. zip is installed for the step and removed again, with the unzip it pulls in on Amazon Linux 2023. On Noble and Amazon Linux 2023 the rewritten jar is given back to fess:fess, the owner the packages set. The README's Image Variants section mentions it.
marevol
force-pushed
the
feat/strip-zstd-jni-natives
branch
from
September 16, 2026 13:10
e289422 to
95b8278
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
codelibs/fess-crawler#211 put
zstd-jniback into the Fess distribution so that.zstand.tar.zstfiles are extracted again. The jar is 6.48 MiB (6,795,212 bytes) because it bundles its native library for 17 OS and architecture combinations (darwin, freebsd, linux and win), 15.1 MiB uncompressed. That is what the zip, deb and rpm need, since they are installed anywhere. A container is not: it only ever loads the library for its own architecture.Change
All six snapshot images (
snapshot,snapshot-noble,snapshot-al2023and their-slimvariants) now delete every other native library fromapp/WEB-INF/lib/zstd-jni-*.jar, right after Fess is installed from the zip, deb or rpm:Native.load()looks for the classpath resource/<os.name>/<os.arch>/libzstd-jni-<version>.so, so the name is read from the image's ownjavarather than mapped fromuname -m. That givesaarch64on arm64 andamd64on amd64, including under emulation.-XX:-UsePerfDatakeeps thatjavafrom leaving/tmp/hsperfdata_rootbehind in the slim images.zip -dwould delete every library and the image would build fine, but it could not extract.zstfiles.zipis installed for the step and removed again in the sameRUN. On Amazon Linux 2023 it pulls inunzip, which is removed too. On Alpine it joins theunzipthe image already adds and deletes. The installed package lists are unchanged.ziprewrites the jar as root:root, keeping its mode. On Noble and Amazon Linux 2023 the jar is chowned back tofess:fess, the owner the deb and rpm set. The Alpine images already chown the whole tree afterwards.Only the snapshot images change. The 15.8 and older directories are left alone.
Size
arm64, Fess 15.9.0-SNAPSHOT packages of 2026-09-15 21:36 UTC. The baseline is
mainbuilt from the same packages: 35e36fd (#86, the LibreOffice filters) forsnapshot-noble, and f4a523a for the rest, which #86 does not touch. Sizes are in bytes:snapshotsnapshot-noblesnapshot-al2023snapshot-slimsnapshot-slim-noblesnapshot-slim-al2023gzip is
docker save | gzip -6. Every image is 6.0 to 6.1 MiB smaller, and the gzip size drops by the same amount because the libraries were already compressed inside the jar. The jar itself goes from 6,795,212 bytes to 390,023 on arm64 and to 465,630 on amd64.Verification
arch=aarch64orarch=amd64and the matching-x linux/<arch>/*.linux/aarch64/libzstd-jni-1.5.6-1.soorlinux/amd64/libzstd-jni-1.5.6-1.so, andunzip -tfinds no errors. Owner and mode match the baseline image (fess:fess 644on Alpine,fess:fess 755from the deb and rpm).fessuser in every image, with the image's ownWEB-INF/libas its classpath. It extracts a.zst, a.tar.zstand a.gzfile (the fixtures from fess-crawler#211) through Tika, and checks thatCompressorParserlistsapplication/zstdand thatNative.load()succeeds. It passes in all nine new images and all six baseline images.linux/amd64on arm64, the same probe fails:UnsatisfiedLinkError: ... cannot find /linux/aarch64/libzstd-jni-1.5.6-1.so, and the.tar.zstfile then fails withNoClassDefFoundError: Could not initialize class com.github.luben.zstd.ZstdInputStreamNoFinalizer. The.gzfile still extracts.libc.so.6,libpthread.so.0), but they load on Alpine as well. The unmodified jar passes the probe oneclipse-temurin:21-jre-alpinewith and withoutgcompat, and so does the stripped jar in the Alpine images. This PR changes nothing there.snapshot-noblewas built again from 35e36fd and from this branch. The two builds still match,libreoffice-writer,libreoffice-calc,libreoffice-impressandunoconvare installed in both, and none of them depends on thezippackage that the step removes.snapshotimage (Alpine, arm64) against OpenSearch 3.8.0, with a file crawl config over a directory holding the three fixture files. After one run of the default crawler, each token is in thecontentof its own document:test.txt.zstandtest.tar.zstasapplication/zstd, including the tar member's text, andtest.txt.gzasapplication/gzip. Anonymous/api/v2/searchfinds each one. The crawler log has no extraction warning.