Skip to content

build!: move the GCS client into the fess-lib-gcs plugin - #3424

Merged
marevol merged 1 commit into
masterfrom
feat/extract-gcs-lib
Sep 9, 2026
Merged

marevol merged 1 commit into
masterfrom
feat/extract-gcs-lib

Conversation

@marevol

@marevol marevol commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

storage.type=gcs and gcs: crawling now need the fess-lib-gcs plugin:

bin/fess-setup install plugin fess-lib-gcs

Without it, the storage factory reports the missing component by name and gcs: is not a file protocol — neither fails with a missing class.

This takes 29 jars and 16.9 MiB out of WEB-INF/lib for a backend most installations never use, and stops the distribution tracking a Google SDK that releases far more often than Fess does. Nothing is left behind in core except the name the plugin registers under: StorageClientFactory has resolved clients as <storage.type>StorageClient since #3419, so fess_storage++.xml in the plugin is enough to reach GcsStorageClient, and the endpoint detection that maps storage.googleapis.com to gcs stays here.

The exclusion on fess-crawler-lasta is what actually removes the jars. fess-crawler declares google-cloud-storage at compile scope for GcsClient, which stays in that library, so dropping the direct declaration here alone would have left the whole tree in the war through the transitive path.

The dependencies that had to replace it

Three are declared because something else was living off the SDK's transitive tree:

Artifact Who needs it
google-api-client 2.7.2 fess-ds-gsuite declares it provided and does not shade it
google-auth-library-oauth2-http 1.50.0 same
httpclient ${httpcomponents.version} nobody here, but leaving it to the resolver hands the decision to io.rest-assured, a test dependency that brings 4.5.13 at a shallower depth than fess-crawler's request. fess-ds-sharepoint compiles against ${httpcomponents.version} with scope provided

Three more pins close version splits the replacements opened, all of the shape that left the GCS client calling a google-http-client method that did not exist (#3420):

  • google-auth-library-credentialsgoogle-api-client wins the nearest-wins tie with 1.30.0 against the 1.50.0 that the library loading it asks for.
  • google-http-client-apache-v2 — the fifth member of that family, and the only one that arrives at 1.45.2.
  • an exclusion of the javax flavour of the annotation API that api-common pulls.

Apart from the GCS set, three artifacts change version, each back to what its remaining consumer asks for: woodstox and stax2 to what xmlsec wants, grpc-api to what google-http-client wants.

Verification

mvn test7696 tests, 0 failures.

Against a built distribution with the plugin in WEB-INF/plugin, an OpenSearch 3.8.0 container and a fake-gcs-server bucket:

PASS gcsStorageClient resolves -> org.codelibs.fess.lib.gcs.GcsStorageClient
PASS StorageClientFactory returns the plugin client
PASS isAvailable -> true
PASS listObjects sees the seeded prefix -> [docs]
PASS gcs is a valid file protocol -> file:,smb:,smb1:,ftp:,s3:,gcs:
PASS a client is registered for gcs: -> org.codelibs.fess.crawler.client.gcs.GcsClient
PASS the object is fetched through the crawler client -> status=200 body=...

The last one is the point: fess-crawler's GcsClient in WEB-INF/lib fetched the object using the SDK that now lives in WEB-INF/plugin.

Without the plugin, in the same setup:

storage error : org.codelibs.fess.exception.StorageException
                No storage client is registered as gcsStorageClient for storage.type=gcs.
                Install the plugin that provides it, such as fess-lib-gcs for gcs.
gcs is a valid file protocol: false (protocols: file:,smb:,smb1:,ftp:,s3:)
crawler client for gcs:: none

Merge order

Depends on codelibs/fess-crawler#203, and stays red until that one is merged and its snapshot is published. The Run Fess step here fails on exactly the state this PR is fixing: the published fess-crawler-lasta still declares gcsClient, and Lasta Di resolves the class named by a component while it parses the Di xml, so the webapp does not start once the SDK is gone.

DiXmlParseFailureException at /components[1]/include[19]
Caused by: java.lang.NoClassDefFoundError: com/google/cloud/NoCredentials
SEVERE: Exception starting filter [lastaPrepareFilter]
SEVERE: Context [] startup failed due to previous errors

Build with Maven (the 7696 unit tests) passes; Run Fess is the only failing step and Run Integration Test is skipped after it.

storage.type=gcs and gcs: crawling now need the fess-lib-gcs plugin. Installing it with
bin/fess-setup install plugin fess-lib-gcs restores both; without it the storage factory
reports the missing component by name and gcs: is not a file protocol, so neither fails
with a missing class.

This takes 29 jars and 16.9 MiB out of WEB-INF/lib for a backend most installations never
use, and stops the distribution tracking a Google SDK that releases far more often than
Fess does. Nothing is left behind in core except the name the plugin registers under:
StorageClientFactory has resolved storage clients as "<storage.type>StorageClient" since
the DI change, so fess_storage++.xml in the plugin is enough to reach GcsStorageClient,
and the endpoint detection that maps storage.googleapis.com to gcs stays here.

The exclusion on fess-crawler-lasta is what actually removes the jars. fess-crawler
declares google-cloud-storage at compile scope for GcsClient, which stays in that
library, so dropping the direct declaration here would have left the whole tree in the
war through the transitive path.

Three dependencies are declared to replace what the SDK used to supply:

- google-api-client and google-auth-library-oauth2-http are unused here. fess-ds-gsuite
  declares both with scope provided and does not shade them, so it has been running on
  the copies google-cloud-storage happened to bring. Removing them silently would break
  a Google Drive crawl at run time, not at build time.
- httpclient has no caller here either, but leaving it to the resolver hands the decision
  to io.rest-assured, a test dependency that brings 4.5.13 at a shallower depth than
  fess-crawler's ${httpcomponents.version}. fess-ds-sharepoint compiles against the
  latter with scope provided.

Three more pins close version splits the replacements opened, all of the shape that left
the GCS client calling a google-http-client method that did not exist:
google-auth-library-credentials (google-api-client wins the tie with 1.30.0 against the
1.50.0 the library it serves asks for), google-http-client-apache-v2 (the fifth member of
that family, and the only one that arrives at 1.45.2), and an exclusion of the javax
flavour of the annotation API that api-common pulls.

Apart from the GCS set, three artifacts change version, each back to what its remaining
consumer asks for: woodstox and stax2 to what xmlsec wants, grpc-api to what
google-http-client wants.

Verified against a running Fess with the plugin installed and a fake-gcs-server bucket:
the storage client resolves from the plugin and lists objects, gcs joins the file
protocols, and fess-crawler's GcsClient in WEB-INF/lib fetches an object using the SDK
that now lives in WEB-INF/plugin. Without the plugin, the storage factory raises
StorageException naming fess-lib-gcs and no crawler client is registered for gcs:.

This depends on the matching fess-crawler change; merging it first avoids a window where
crawler/client.xml still references a component whose SDK is gone.
@marevol
marevol force-pushed the feat/extract-gcs-lib branch from 4ddbdd0 to b3db1ad Compare September 9, 2026 21:06
@marevol
marevol merged commit 7fc20b7 into master Sep 9, 2026
2 of 3 checks passed
marevol added a commit to codelibs/fess-docs that referenced this pull request Sep 9, 2026
codelibs/fess#3424 takes the Google Cloud Storage SDK out of the distribution, so gcs://
crawling and the gcs storage type now come from the fess-lib-gcs plugin. install/upgrade.rst
gains a sixth incompatible change for 15.9 naming the plugin and both ways to install it, and
the count in the section intro moves from five to six.

The shipped crawler.file.protocols loses gcs as well, so its documented default becomes
file,smb,smb1,ftp,s3 in install/upgrade.rst, config/crawler-advanced.rst (the property table
and the example) and config/properties.rst.

The pages where GCS is configured get the same short prerequisite note: the GCS Crawler
section of config/crawler-advanced.rst, the GCS Settings of admin/storage-guide.rst and the
GCS bucket example of admin/fileconfig-guide.rst. The gcs:// entry in the path list and the
GCS storage type in admin/general-guide.rst now say the plugin is required.

dev/plugin-architecture.rst lists the plugin types Fess derives from the artifact prefix and
was missing fess-lib-*, which PluginHelper.ArtifactType has carried since #3418. The new
plugin is the first of that type, so the row is added here.

The ZIP figure in the Node.js section moves from 218.9 MiB to 204.7 MiB. The 14.2 MiB
difference is measured rather than estimated: it is the compressed size of the 30 archive
entries the change removes, less the three older jars it puts back.

All seven languages. Three pages were left alone rather than translated as part of this
change: ko/admin/storage-guide.rst has no storage server configuration section,
ko/admin/fileconfig-guide.rst has no S3/GCS crawl examples, and config/properties.rst exists
only in English.

Each of the 42 files was parsed with docutils before and after the edit, with
doctitle_xform=False the way Sphinx parses: no new system messages, no markup left unparsed
in the rendered text, and the same number of top-level sections, which is what catches a
mis-sized heading underline.
@marevol
marevol deleted the feat/extract-gcs-lib branch September 16, 2026 13:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant