Skip to content

build(fess): keep only the zstd-jni native library each image loads - #85

Merged
marevol merged 1 commit into
mainfrom
feat/strip-zstd-jni-natives
Sep 16, 2026
Merged

marevol merged 1 commit into
mainfrom
feat/strip-zstd-jni-natives

Conversation

@marevol

@marevol marevol commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

codelibs/fess-crawler#211 put zstd-jni back into the Fess distribution so that .zst and .tar.zst files are extracted again. The jar is 6.48 MiB (6,795,212 bytes) because it bundles its native library for 17 OS and architecture combinations (darwin, freebsd, linux and win), 15.1 MiB uncompressed. That is what the zip, deb and rpm need, since they are installed anywhere. A container is not: it only ever loads the library for its own architecture.

Change

All six snapshot images (snapshot, snapshot-noble, snapshot-al2023 and their -slim variants) now delete every other native library from app/WEB-INF/lib/zstd-jni-*.jar, right after Fess is installed from the zip, deb or rpm:

arch=$("${JAVA_HOME}/bin/java" -XX:-UsePerfData -XshowSettings:properties -version 2>&1 | sed -n 's/^ *os\.arch = //p')
jar=$(ls /usr/share/fess/app/WEB-INF/lib/zstd-jni-*.jar)
zip -sf "${jar}" | grep -q "^ *linux/${arch}/libzstd-jni-.*\.so$" || (echo "zstd-jni has no native library for linux/${arch}" && exit 1)
zip -q -d "${jar}" '*.so' '*.dylib' '*.dll' -x "linux/${arch}/*"
  • Which library is kept. zstd-jni's Native.load() looks for the classpath resource /<os.name>/<os.arch>/libzstd-jni-<version>.so, so the name is read from the image's own java rather than mapped from uname -m. That gives aarch64 on arm64 and amd64 on amd64, including under emulation. -XX:-UsePerfData keeps that java from leaving /tmp/hsperfdata_root behind in the slim images.
  • Failing loudly. If the jar has no library for the architecture, the build stops. Without the check, zip -d would delete every library and the image would build fine, but it could not extract .zst files.
  • Tools. zip is installed for the step and removed again in the same RUN. On Amazon Linux 2023 it pulls in unzip, which is removed too. On Alpine it joins the unzip the image already adds and deletes. The installed package lists are unchanged.
  • Ownership. zip rewrites the jar as root:root, keeping its mode. On Noble and Amazon Linux 2023 the jar is chowned back to fess:fess, the owner the deb and rpm set. The Alpine images already chown the whole tree afterwards.
  • The README's Image Variants section gets one paragraph about it.

Only the snapshot images change. The 15.8 and older directories are left alone.

Size

arm64, Fess 15.9.0-SNAPSHOT packages of 2026-09-15 21:36 UTC. The baseline is main built from the same packages: 35e36fd (#86, the LibreOffice filters) for snapshot-noble, and f4a523a for the rest, which #86 does not touch. Sizes are in bytes:

Image Uncompressed before after gzip before after
snapshot 370,850,292 364,445,172 206,944,704 200,552,328
snapshot-noble 1,014,636,754 1,008,233,658 418,523,860 412,131,400
snapshot-al2023 907,195,606 900,827,329 350,617,284 344,254,828
snapshot-slim 334,457,469 328,052,349 173,684,034 167,292,849
snapshot-slim-noble 424,713,693 418,311,153 195,035,674 188,638,525
snapshot-slim-al2023 600,676,692 594,440,255 240,817,265 234,445,831

gzip is docker save | gzip -6. Every image is 6.0 to 6.1 MiB smaller, and the gzip size drops by the same amount because the libraries were already compressed inside the jar. The jar itself goes from 6,795,212 bytes to 390,023 on arm64 and to 465,630 on amd64.

Verification

  • Builds. All six variants built on arm64, and the three slim variants also built on amd64 (buildx, emulated). The build log shows arch=aarch64 or arch=amd64 and the matching -x linux/<arch>/*.
  • Jar contents. The jar copied out of each image has exactly one native entry, linux/aarch64/libzstd-jni-1.5.6-1.so or linux/amd64/libzstd-jni-1.5.6-1.so, and unzip -t finds no errors. Owner and mode match the baseline image (fess:fess 644 on Alpine, fess:fess 755 from the deb and rpm).
  • Extraction. A small probe runs as the fess user in every image, with the image's own WEB-INF/lib as its classpath. It extracts a .zst, a .tar.zst and a .gz file (the fixtures from fess-crawler#211) through Tika, and checks that CompressorParser lists application/zstd and that Native.load() succeeds. It passes in all nine new images and all six baseline images.
  • The probe catches a missing library. With a jar that keeps only linux/amd64 on arm64, the same probe fails: UnsatisfiedLinkError: ... cannot find /linux/aarch64/libzstd-jni-1.5.6-1.so, and the .tar.zst file then fails with NoClassDefFoundError: Could not initialize class com.github.luben.zstd.ZstdInputStreamNoFinalizer. The .gz file still extracts.
  • musl. The Linux libraries link against glibc (libc.so.6, libpthread.so.0), but they load on Alpine as well. The unmodified jar passes the probe on eclipse-temurin:21-jre-alpine with and without gcompat, and so does the stripped jar in the Alpine images. This PR changes nothing there.
  • Nothing else changes. Compared with the baseline, each new arm64 image has an identical installed-package list and an identical file list. After rebasing onto fix(noble): install the LibreOffice filters unoconv needs for Office thumbnails #86, snapshot-noble was built again from 35e36fd and from this branch. The two builds still match, libreoffice-writer, libreoffice-calc, libreoffice-impress and unoconv are installed in both, and none of them depends on the zip package that the step removes.
  • Crawling. Fess ran from the new snapshot image (Alpine, arm64) against OpenSearch 3.8.0, with a file crawl config over a directory holding the three fixture files. After one run of the default crawler, each token is in the content of its own document: test.txt.zst and test.tar.zst as application/zstd, including the tar member's text, and test.txt.gz as application/gzip. Anonymous /api/v2/search finds each one. The crawler log has no extraction warning.

@marevol marevol self-assigned this Sep 16, 2026
codelibs/fess-crawler#211 put zstd-jni back into the distribution so that .zst
and .tar.zst files are extracted again. The jar is 6.5 MB because it bundles
its native library for 17 OS and architecture combinations, while a Linux
container only ever loads linux/<os.arch>, the resource zstd-jni looks up.

Every snapshot image, full and slim, now deletes the other libraries from the
jar right after installing Fess from the zip, deb or rpm, which leaves 390 KB
on arm64 and 466 KB on amd64. The architecture name comes from the image's own
java, and the build fails if the jar has no library under it rather than
producing an image that cannot extract those files. zip is installed for the
step and removed again, with the unzip it pulls in on Amazon Linux 2023. On
Noble and Amazon Linux 2023 the rewritten jar is given back to fess:fess, the
owner the packages set.

The README's Image Variants section mentions it.
@marevol
marevol force-pushed the feat/strip-zstd-jni-natives branch from e289422 to 95b8278 Compare September 16, 2026 13:10
@marevol
marevol merged commit 7e9db75 into main Sep 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant