fix: shade protobuf-java, which Fess no longer ships - #3
Merged
Merged
Conversation
Building a Cloud Storage client fails on current Fess main with NoClassDefFoundError: com/google/protobuf/MessageOrBuilder, raised from DefaultStorageRetryStrategy.<clinit>. Both GcsStorageClient and fess-crawler's GcsClient build through it, so the storage page, uploads and gcs: crawling all break once the plugin is installed. protobuf-java reached the war through org.opensearch:opensearch, which codelibs/fess#3438 and codelibs/fess#3439 took out. Stop excluding it so the SDK's protobuf-java 4.33.6 is shaded. The comment claimed every excluded artifact is still in the war. That is no longer true of woodstox-core and stax2-api either; they left with codelibs/fess#3430. They stay excluded because only jackson-dataformat-xml uses them and neither GcsStorageClient nor the gcs: crawler client reaches it, and the comment now says so.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found while verifying 15.9.0. On current Fess main the plugin cannot build a Cloud Storage client:
StorageOptions.newBuilder()...build()is where both users of the SDK start:GcsStorageClient(storage.type=gcs) and fess-crawler'sGcsClient#init, which this plugin registers forgcs:URLs. With the plugin installed:GcsStorageClientand theNoClassDefFoundError. An upload is answered with success, but no file appears.gcs://file crawl indexes nothing; the failure URL records the error.GET /api/admin/storage/listanswers with status 0 and an empty list, so the failure does not show there at all.Could not initialize class com.google.cloud.storage.DefaultStorageRetryStrategy; the name of the missing class is gone.Cause
The shade's
artifactSetexcludescom.google.protobuf:protobuf-java, and the comment above the list says each entry is an artifact the war still ships. protobuf-java reached the war as a dependency oforg.opensearch:opensearch: codelibs/fess#3438 excluded it there and codelibs/fess#3439 removed the opensearch jar altogether. Neither the war nor this jar hascom.google.protobuf.*any more.Change
protobuf-javaexclude, so the SDK's protobuf-java 4.33.6 is shaded.The whole exclude list was re-checked against
mvn dependency:list -DincludeScope=runtimeon current Fess main. Apart from protobuf-java, two more entries are no longer in the war:woodstox-coreandstax2-api, which left with codelibs/fess#3430 (they came in through java-saml's xmlsec). They stay excluded, and the comment now says why:jackson-dataformat-xml, and the SDK uses that only fromXmlObjectParserandMultipartUploadHttpRequestManager, the machinery of the XML API'sMultipartUploadClient.GcsStorageClient,GcsClientCreator, fess-crawler'sGcsClientand itsgcs:URL handler never reachesjackson-dataformat-xml, and the storage round trip below passes without them.META-INF/servicesforjavax.xml.stream.XMLInputFactory,XMLOutputFactoryandXMLEventFactory. Shaded, it would become the StAX implementation of every Fess process that loads this jar.The remaining version-skewed entries are still in the war, at the war's versions: gson 2.11.0 (SDK: 2.13.2), grpc-api and grpc-context 1.70.0 (1.82.2), error_prone_annotations 2.36.0 (2.48.0), j2objc-annotations 3.0.0 (3.1).
Why shade it rather than put it back in the war
Removing the OpenSearch server jar and what it brought in was deliberate, and Fess itself does not use protobuf-java. The Cloud Storage SDK in this plugin does: its HTTP transport still loads protobuf types, starting with the default retry strategy. Only an installation that uses GCS needs it.
Compatibility
The jar grows by 1.80 MiB (15,651,606 → 17,542,777 bytes): 756 classes under
com/google/protobuf/and the 12google/protobuf/*.protofiles. Nothing is removed, nothing is duplicated, there is no module descriptor at the root, and the shade's overlap warnings are unchanged. (META-INF/versions/9/module-info.class, from jackson-dataformat-xml and jackson-datatype-jsr310, was already in the jar and is untouched; the manifest has noMulti-Releaseattribute.)Verification
mvn clean package: 1 test, 0 failures.dependency:copy-dependencies -DincludeScope=runtimeat codelibs/fess@5bd477e, 191 jars), Fess's own classes and the plugin jar:StorageOptions.newBuilder()...getService()fails withNoClassDefFoundError: com/google/protobuf/MessageOrBuilder;new GcsStorageClient(...)and fess-crawler'sGcsClient#init()then fail withCould not initialize class com.google.cloud.storage.DefaultStorageRetryStrategy;GcsStorageClientrunsensureBucketExists,isAvailable,uploadObject,listObjects,downloadObject(identical bytes, non-ASCII object name),deleteObjectandclosewithout error.com.google.protobufgroup is gone. The groups that remain were there before and are untouched: the gRPC transport, the OpenTelemetry SDK, App Engine URL Fetch, GraalVM and the slf4j 2 SPI, which belong to transports and integrations the pom already excludes, and stax2 through jackson-dataformat-xml.WEB-INF/libalone brought back the Storage page, the upload/list/download/delete round trip and thegcs://crawl; gRPC, OpenTelemetry and Woodstox were not needed.