build!: move the GCS client into the fess-lib-gcs plugin - #3424
Merged
Merged
Conversation
storage.type=gcs and gcs: crawling now need the fess-lib-gcs plugin. Installing it with
bin/fess-setup install plugin fess-lib-gcs restores both; without it the storage factory
reports the missing component by name and gcs: is not a file protocol, so neither fails
with a missing class.
This takes 29 jars and 16.9 MiB out of WEB-INF/lib for a backend most installations never
use, and stops the distribution tracking a Google SDK that releases far more often than
Fess does. Nothing is left behind in core except the name the plugin registers under:
StorageClientFactory has resolved storage clients as "<storage.type>StorageClient" since
the DI change, so fess_storage++.xml in the plugin is enough to reach GcsStorageClient,
and the endpoint detection that maps storage.googleapis.com to gcs stays here.
The exclusion on fess-crawler-lasta is what actually removes the jars. fess-crawler
declares google-cloud-storage at compile scope for GcsClient, which stays in that
library, so dropping the direct declaration here would have left the whole tree in the
war through the transitive path.
Three dependencies are declared to replace what the SDK used to supply:
- google-api-client and google-auth-library-oauth2-http are unused here. fess-ds-gsuite
declares both with scope provided and does not shade them, so it has been running on
the copies google-cloud-storage happened to bring. Removing them silently would break
a Google Drive crawl at run time, not at build time.
- httpclient has no caller here either, but leaving it to the resolver hands the decision
to io.rest-assured, a test dependency that brings 4.5.13 at a shallower depth than
fess-crawler's ${httpcomponents.version}. fess-ds-sharepoint compiles against the
latter with scope provided.
Three more pins close version splits the replacements opened, all of the shape that left
the GCS client calling a google-http-client method that did not exist:
google-auth-library-credentials (google-api-client wins the tie with 1.30.0 against the
1.50.0 the library it serves asks for), google-http-client-apache-v2 (the fifth member of
that family, and the only one that arrives at 1.45.2), and an exclusion of the javax
flavour of the annotation API that api-common pulls.
Apart from the GCS set, three artifacts change version, each back to what its remaining
consumer asks for: woodstox and stax2 to what xmlsec wants, grpc-api to what
google-http-client wants.
Verified against a running Fess with the plugin installed and a fake-gcs-server bucket:
the storage client resolves from the plugin and lists objects, gcs joins the file
protocols, and fess-crawler's GcsClient in WEB-INF/lib fetches an object using the SDK
that now lives in WEB-INF/plugin. Without the plugin, the storage factory raises
StorageException naming fess-lib-gcs and no crawler client is registered for gcs:.
This depends on the matching fess-crawler change; merging it first avoids a window where
crawler/client.xml still references a component whose SDK is gone.
marevol
force-pushed
the
feat/extract-gcs-lib
branch
from
September 9, 2026 21:06
4ddbdd0 to
b3db1ad
Compare
marevol
added a commit
to codelibs/fess-docs
that referenced
this pull request
Sep 9, 2026
codelibs/fess#3424 takes the Google Cloud Storage SDK out of the distribution, so gcs:// crawling and the gcs storage type now come from the fess-lib-gcs plugin. install/upgrade.rst gains a sixth incompatible change for 15.9 naming the plugin and both ways to install it, and the count in the section intro moves from five to six. The shipped crawler.file.protocols loses gcs as well, so its documented default becomes file,smb,smb1,ftp,s3 in install/upgrade.rst, config/crawler-advanced.rst (the property table and the example) and config/properties.rst. The pages where GCS is configured get the same short prerequisite note: the GCS Crawler section of config/crawler-advanced.rst, the GCS Settings of admin/storage-guide.rst and the GCS bucket example of admin/fileconfig-guide.rst. The gcs:// entry in the path list and the GCS storage type in admin/general-guide.rst now say the plugin is required. dev/plugin-architecture.rst lists the plugin types Fess derives from the artifact prefix and was missing fess-lib-*, which PluginHelper.ArtifactType has carried since #3418. The new plugin is the first of that type, so the row is added here. The ZIP figure in the Node.js section moves from 218.9 MiB to 204.7 MiB. The 14.2 MiB difference is measured rather than estimated: it is the compressed size of the 30 archive entries the change removes, less the three older jars it puts back. All seven languages. Three pages were left alone rather than translated as part of this change: ko/admin/storage-guide.rst has no storage server configuration section, ko/admin/fileconfig-guide.rst has no S3/GCS crawl examples, and config/properties.rst exists only in English. Each of the 42 files was parsed with docutils before and after the edit, with doctitle_xform=False the way Sphinx parses: no new system messages, no markup left unparsed in the rendered text, and the same number of top-level sections, which is what catches a mis-sized heading underline.
This was referenced Sep 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
storage.type=gcsandgcs:crawling now need thefess-lib-gcsplugin:Without it, the storage factory reports the missing component by name and
gcs:is not a file protocol — neither fails with a missing class.This takes 29 jars and 16.9 MiB out of
WEB-INF/libfor a backend most installations never use, and stops the distribution tracking a Google SDK that releases far more often than Fess does. Nothing is left behind in core except the name the plugin registers under:StorageClientFactoryhas resolved clients as<storage.type>StorageClientsince #3419, sofess_storage++.xmlin the plugin is enough to reachGcsStorageClient, and the endpoint detection that mapsstorage.googleapis.comtogcsstays here.The exclusion on
fess-crawler-lastais what actually removes the jars. fess-crawler declaresgoogle-cloud-storageat compile scope forGcsClient, which stays in that library, so dropping the direct declaration here alone would have left the whole tree in the war through the transitive path.The dependencies that had to replace it
Three are declared because something else was living off the SDK's transitive tree:
google-api-client2.7.2providedand does not shade itgoogle-auth-library-oauth2-http1.50.0httpclient${httpcomponents.version}io.rest-assured, a test dependency that brings 4.5.13 at a shallower depth than fess-crawler's request. fess-ds-sharepoint compiles against${httpcomponents.version}with scopeprovidedThree more pins close version splits the replacements opened, all of the shape that left the GCS client calling a
google-http-clientmethod that did not exist (#3420):google-auth-library-credentials—google-api-clientwins the nearest-wins tie with 1.30.0 against the 1.50.0 that the library loading it asks for.google-http-client-apache-v2— the fifth member of that family, and the only one that arrives at 1.45.2.api-commonpulls.Apart from the GCS set, three artifacts change version, each back to what its remaining consumer asks for: woodstox and stax2 to what xmlsec wants,
grpc-apito whatgoogle-http-clientwants.Verification
mvn test— 7696 tests, 0 failures.Against a built distribution with the plugin in
WEB-INF/plugin, an OpenSearch 3.8.0 container and a fake-gcs-server bucket:The last one is the point: fess-crawler's
GcsClientinWEB-INF/libfetched the object using the SDK that now lives inWEB-INF/plugin.Without the plugin, in the same setup:
Merge order
Depends on codelibs/fess-crawler#203, and stays red until that one is merged and its snapshot is published. The
Run Fessstep here fails on exactly the state this PR is fixing: the publishedfess-crawler-lastastill declaresgcsClient, and Lasta Di resolves the class named by a component while it parses the Di xml, so the webapp does not start once the SDK is gone.Build with Maven(the 7696 unit tests) passes;Run Fessis the only failing step andRun Integration Testis skipped after it.