build!: leave the s3: crawler client to the fess-storage-s3 plugin - #204
Merged
Merged
Conversation
S3Client and the s3: URL handler stay in this library, but the AWS SDK they need is 30 jars that only installations crawling S3 ever load. Fess ships it in the fess-storage-s3 plugin, which declares the s3Client component and registers it through crawlerClientCreator, the way fess-crawler-playwright registers playwrightClient and fess-storage-gcs registers gcsClient. The registration cannot stay here. clientFactory resolves its component references when the factory is created, not when a client is first used, so a reference to s3Client would make every CrawlerClientFactory fail to build wherever the SDK is absent -- which is every installation without the plugin. The class itself keeps compiling against the SDK and costs a few tens of KiB, so nothing moves out of this library. The gcs: comment this replaces still named fess-lib-gcs, which was renamed to fess-storage-gcs before it was published; both protocols now share one comment.
marevol
added a commit
to codelibs/fess-parent
that referenced
this pull request
Sep 10, 2026
…lished (#80) These two repositories are how one repository in this family sees another's change, and Maven's default updatePolicy for a snapshot is daily. A build whose local repository already resolved a -SNAPSHOT earlier the same day never asks again, so a CI job that caches ~/.m2 cannot see a sibling's snapshot published minutes ago. updatePolicy always removes that window. Measured, not assumed. codelibs/fess-crawler#204 merged at 02:48:54 and its snapshot deployed at 02:49:31. codelibs/fess#3428, which depends on it, was re-run at 02:59:29 -- ten minutes after the fix was published -- and failed with DiXmlParseFailureException: crawler/client.xml included by app.xml Caused by: NoClassDefFoundError: software/amazon/awssdk/auth/credentials/AwsCredentials because it built against the previous fess-crawler-lasta. The job log contains no request to maven.codelibs.org at all: not one "Downloading from codelibs.org.snapshot" line, while "Cache restored from key: Linux-maven-..." sits at the top. The published jar was correct the whole time. Reproduced and fixed locally on the same shape of cache. Before, the resolved fess-crawler-lasta-15.9.0-SNAPSHOT.jar declared one s3Client component; after installing this pom and resolving once, it declares none and names the plugin that took over, and the timestamped build the fix landed in appears in ~/.m2. The Fess branch's DI container then built clientFactory and mapped s3://bucket/key, using nothing but published artifacts. Every consumer inherits this, which is the point: the plugin repositories' CI installs this pom from source and then resolves org.codelibs.fess:fess from these repositories, so they are in the same window whenever core publishes. The cost is one metadata request per snapshot dependency per build. Offline builds skip it, and a release build has no snapshot dependencies to check.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First of three PRs that move the S3 storage backend out of the Fess distribution, following what codelibs/fess#3424 and #203 did for Google Cloud Storage. Merge this one first, and let its snapshot publish before the Fess side lands.
S3Clientand thes3:URL handler stay here and keep compiling against the AWS SDK; only the registration moves. The SDK itself leaves the war through an exclusion on the Fess side, and ships in the newfess-storage-s3plugin, which declares thes3Clientcomponent and registers it throughcrawlerClientCreator— the wayfess-crawler-playwrightregistersplaywrightClient.Why the registration cannot stay here
clientFactoryresolves its component references when the factory is created, not when a client is first used. A reference tos3Clienttherefore makes everyCrawlerClientFactoryfail to build wherever the SDK is absent, which is every installation without the plugin. That is the same reason thegcsClientregistration was removed in #203.Verified
A LastaDi container was booted on a plain JVM with
crawler/client.xml,crawler/mimetype.xmlandfess_storage.xml, on a class path built the way Fess builds one —WEB-INF/classes, then the war's runtime jars with the AWS SDK excluded, then the plugin jar.With this change,
clientFactorybuilds and mapss3://bucket/keytoorg.codelibs.fess.crawler.client.s3.S3Client— the class from this library, registered by the plugin.Without it, using the current
crawler/client.xmlagainst the same plugin:clientFactoryfails to build there, so every crawl dies — which is what makes the merge order a requirement rather than a preference, and why the plugin is paired with a Fess version.Also in this change
The
gcs:comment being replaced still namedfess-lib-gcs, which was renamed tofess-storage-gcsbefore it was ever published. Both protocols now share one comment.No test changes: the S3 tests here build their own container with
.singleton("s3Client", S3Client.class)rather than readingcrawler/client.xml, so none of them pins the registration.crawler/client.xmlis not read by any test in this repository.