Skip to content

refactor: compile against the forked OpenSearch package and drop the opensearch jar - #3439

Merged
marevol merged 2 commits into
mainfrom
fork/opensearch-migrate
Sep 14, 2026
Merged

marevol merged 2 commits into
mainfrom
fork/opensearch-migrate

Conversation

@marevol

@marevol marevol commented Sep 13, 2026

Copy link
Copy Markdown
Contributor

fesen-httpclient now carries the OpenSearch DTOs and query builders Fess needs under org.codelibs.fesen.opensearch.*, pruned from 8,578 classes to 3,751. This drops the org.opensearch:opensearch dependency and moves Fess onto them.

WEB-INF/lib goes from 229 jars / 146,843,781 bytes to 215 jars / 135,178,555 bytes — 11.67 MiB off every distribution. The 17.09 MiB server jar and its twelve small siblings leave, along with reactive-streams; fesen-httpclient grows from 0.44 MiB to 7.09 MiB. No org/opensearch/ class remains anywhere in the distribution — all 219 jars were swept.

All 13 Lucene modules stay at 10.5.0. The old exclusion block had been suppressing eight of them; the fork declares them directly, so they come back.

The change

264 files, 2,049 import lines. 142 of the 210 main files are checked-in ESFlute output, so 81% of the occurrences are generated code. src/main/webapp has none. Fess's own org.codelibs.fess.opensearch.* package is untouched — the rewrite was anchored so it could not match it.

Three things were not a rename:

  • QueryShardContext is gone from the fork — it was the bridge whose removal collapsed index/mapper/**, index/fielddata/** and search/internal/**. Four classes overrode a method taking it: KnnQueryBuilder, HybridQueryBuilder, StoredLtrQueryBuilder and DefaultQueryBuilder. All four bodies threw UnsupportedOperationException or delegated to one that did — these builders exist to be serialised to JSON, which still works — so the overrides are deleted. KnnQueryBuilderTest.test_doToQuery_isUnsupported called the removed method and could not compile; it is the only test removed.
  • The seeded Index Exporter job embeds org.opensearch.index.query.QueryBuilders in its script string. The seed is updated. See the upgrade note below.
  • Three plugin-facing core signatures move: IndexingHelper.deleteDocumentByQuery(SearchEngineClient, QueryBuilder), its four protected siblings, and LanguageHelper.createScript(Map, String) returning Script. Five plugin repositories follow them and ship together.

Two side effects of dropping the dependency, both fixed here:

  • SnakeYAML silently dropped from 2.6 to 2.5. ThemeManifest parses theme.yaml with it directly but never declared it — it was arriving through the opensearch jar, and without it jackson-dataformat-yaml decides the version. Now declared at the release it has been running.
  • The log4j2 rule silencing org.opensearch.bootstrap no longer names anything: the forked tree has no bootstrap package and the original jar is gone. Removed.

error_prone_annotations moves 2.45.0 → 2.36.0. Annotations only, unused at runtime, left alone.

Upgrade note for the persisted scheduled job

fess_config.scheduled_job/_id=index_exporter is reloaded at every webapp boot, but with OpType.CREATE, so an existing document conflicts and is skipped. The corrected seed therefore does not reach an upgraded installation, and Fess has no migration framework to hang a fix on.

The exposure is small and loud: the job ships available:false with an empty cron expression, so it never fires on its own. An administrator who had enabled it will see it fail with a ScriptEngineException in the job log rather than fail silently or corrupt anything. The fix is a one-line edit in Admin > Scheduler, replacing the package prefix. This is release-note material; rewriting the document at boot would silently overwrite a script an administrator may have customised.

Verification

  • mvn clean package and the full suite — 7,469 tests, 0 failures.
  • End to end against a real OpenSearch 3.8.0, from a virgin cluster: 40 indices created green, a crawl run through the crawler child process (which has its own classpath), search returning documents with highlighting, term and filter facet counts that match the indexed data, suggest indexing 27 documents and switching its alias, and _nodes/stats parsed with all 30 sections populated. Zero NoClassDefFoundError, ClassNotFoundException, NoSuchMethodError or ServiceConfigurationError across all 13 log files, and zero ERROR-level lines.

The facet result is the one that matters most: search/aggregations was cut from 1,042 classes to 334, and aggregation response parsers are only exercised at runtime.

Ordering

Requires codelibs/fesen-httpclient#43, codelibs/fess-suggest#98 and codelibs/fess-crawler#208 to be released first. The five plugin repositories that follow the core signatures above must be released with this, not after it.

…client

Fess shipped org.opensearch:opensearch (17.9 MB) purely for its request and
response types and its query builders; it never runs a node. Those classes are
now forked and pruned into org.codelibs.fesen.opensearch.* inside
fesen-httpclient, so the dependency and its exclusion block go away and every
org.opensearch import moves to the forked package. Apache Lucene stays as the
real org.apache.lucene and now arrives through fesen-httpclient, which keeps all
thirteen modules Fess resolved before. Fess's own org.codelibs.fess.opensearch.*
(DBFlute/ESFlute output) is unrelated and untouched.

Three things are not a plain rename:

- QueryShardContext is deliberately absent from the fork, so QueryBuilder no
  longer declares toQuery and AbstractQueryBuilder no longer declares doToQuery.
  The overrides in KnnQueryBuilder, HybridQueryBuilder, StoredLtrQueryBuilder and
  DefaultQueryBuilder only threw UnsupportedOperationException or delegated, so
  they are removed along with the now-unused imports. These builders exist to be
  serialised to JSON, which is unaffected. KnnQueryBuilderTest's
  test_doToQuery_isUnsupported covered a method that no longer exists and goes
  with it.

- The seeded Index Exporter job embeds a class name in its script body, so
  scheduled_job.bulk moves to org.codelibs.fesen.opensearch.index.query.
  QueryBuilders too. The startup reload is create-only, so an existing
  installation keeps its stale copy and needs a separate migration.

- IndexingHelper.deleteDocumentByQuery and LanguageHelper.createScript are part
  of the plugin-facing API and now take and return forked types; the plugins that
  call them migrate separately.

Removing the dependency drops opensearch-3.8.0.jar and its twelve siblings plus
reactive-streams, 17.84 MiB off the resolved runtime classpath (229 jars to 215).
Removing org.opensearch:opensearch took SnakeYAML off the resolved classpath at
2.6 and left jackson-dataformat-yaml to supply 2.5 instead. ThemeManifest parses
theme.yaml with SnakeYAML directly, so the version it gets should not be decided
by an unrelated Jackson module: declare it, at the release it has been running.

The log4j2 entry that silenced org.opensearch.bootstrap no longer names anything.
The forked tree has no bootstrap package - the node runtime was cut - and the
original jar is gone, so the logger cannot match. Remove it rather than leave a
rule pointing at a package that does not exist.

7469 tests pass.
@marevol marevol self-assigned this Sep 14, 2026
@marevol marevol added the task label Sep 14, 2026
@marevol marevol added this to the 15.9.0 milestone Sep 14, 2026
@marevol
marevol merged commit 925cd72 into main Sep 14, 2026
1 of 2 checks passed
marevol added a commit that referenced this pull request Sep 14, 2026
#3439 left two things behind.

`SearchEngineClient.pitSegments` overrides a `Client` method that the fork does
not have. The package rename was applied to every `org.opensearch.*` reference
without checking that each type survived: `PitSegmentsRequest` did not, so the
import resolves to nothing and the method is an `@Override` of nothing. Removed,
with the two imports it needed.

`AbstractDocumentSearcherTest` arrived from #3440 after the migration branch was
cut, so its six `org.opensearch.*` imports were never rewritten. They are now.

Nothing else in the tree still names `org.opensearch`.
marevol added a commit to codelibs/fess-ds-microsoft365 that referenced this pull request Sep 15, 2026
Every data store fails in Microsoft365Client's constructor on current
Fess main with NoClassDefFoundError: org/reactivestreams/Publisher, raised
while azure-identity's CredentialBuilderBase creates its ClientLogger.

reactive-streams is on the shade's include list, but it resolved as
provided under the provided org.opensearch:opensearch dependency, so the
shade never took it. The war used to supply it next to the opensearch jar
until codelibs/fess#3439 removed that jar.

Declare reactive-streams 1.0.4, the version reactor-core declares, as a
direct dependency so it resolves at compile scope and is shaded. Excluding
it from the opensearch dependency does not work (reactor-core's copy sits
under the same node), and dropping that dependency changes the resolved
versions of reactor-core, error_prone_annotations and jna.
marevol added a commit to codelibs/fess-storage-gcs that referenced this pull request Sep 15, 2026
Building a Cloud Storage client fails on current Fess main with
NoClassDefFoundError: com/google/protobuf/MessageOrBuilder, raised from
DefaultStorageRetryStrategy.<clinit>. Both GcsStorageClient and
fess-crawler's GcsClient build through it, so the storage page, uploads
and gcs: crawling all break once the plugin is installed.

protobuf-java reached the war through org.opensearch:opensearch, which
codelibs/fess#3438 and codelibs/fess#3439 took out. Stop excluding it so
the SDK's protobuf-java 4.33.6 is shaded.

The comment claimed every excluded artifact is still in the war. That is
no longer true of woodstox-core and stax2-api either; they left with
codelibs/fess#3430. They stay excluded because only jackson-dataformat-xml
uses them and neither GcsStorageClient nor the gcs: crawler client reaches
it, and the comment now says so.
marevol added a commit to codelibs/fess-storage-s3 that referenced this pull request Sep 15, 2026
Building an S3 client fails on current Fess main with
NoClassDefFoundError: org/reactivestreams/Publisher, raised from
DefaultS3BaseClientBuilder.finalizeServiceConfiguration. Both
S3StorageClient and fess-crawler's S3Client build through it, so the
storage page, the storage API and s3: crawling all break once the plugin
is installed.

The shade excluded org.reactivestreams:* on the assumption that the war
carries it with OpenSearch. It did until codelibs/fess#3439 dropped the
opensearch jar. Stop excluding it so the SDK's reactive-streams 1.0.4 is
shaded, and correct the comment. The remaining excludes (httpclient5,
httpcore5, httpcore5-h2, slf4j-api) are still shipped by the war.
marevol added a commit to codelibs/fess-docs that referenced this pull request Sep 15, 2026
…fig and plugin upgrades (#536)

An installation that upgrades from 15.8 by following install/upgrade.rst
still ends up with several things broken that the guide did not mention.
All seven languages of the development tree (15.9) are updated.

- Groovy: every job 15.8 seeded is stored as groovy and the startup
  seeding is create-only, so Default Crawler and the other bundled jobs
  fail without fess-script-groovy, mostly visible only in fess.log.
  The section now lists every kind of stored setting that 15.9
  evaluates as Groovy and what each does without the plugin, tells
  upgraders to look for the startup warning, and gives both ways out:
  install the plugin, or switch each kind to JavaScript with the field
  that selects its engine (Thumbnail Purger's 1000L literal needs an
  edit).
- Index Exporter: the stored script names
  org.opensearch.index.query.QueryBuilders, removed by
  codelibs/fess#3439; give the replacement script and fix the examples
  in config/admin-index-export.rst.
- Carried-over configuration: ZIP Step 3 copies fess_config.properties
  and fess.in.sh, and RPM keeps a modified fess_config.properties. List
  the keys whose 15.8.0 value now misbehaves (job.default.script,
  job.template.script, crawler.file.protocols, search_engine.http.url,
  the jcifs options, removed keys) and what a 15.8 fess.in.sh lacks.
  Say that DEB overwrites the file without prompting or a backup.
- jcifs 3.x property names (codelibs/fess#3433): SMB connect and session
  timeouts now take effect, 35 s to 60 s.
- The four keys removed by codelibs/fess#3435.
- Plugins: bin/fess-setup upgrade plugins and check; what a Playwright
  crawl configuration does without the plugin.
- Replace the ZIP size figure, which did not match the released
  fess-15.8.0.zip, with the Playwright driver bundle's share of it.

The startup warning, the Playwright crawler warning and path mappings
being left unapplied describe codelibs/fess#3451, #3445 and #3452.
@marevol
marevol deleted the fork/opensearch-migrate branch September 16, 2026 13:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant