Skip to content

fix(index): create the document index and its aliases in one request on first boot - #3472

Merged
marevol merged 1 commit into
mainfrom
fix/concurrent-first-boot-index-alias
Sep 23, 2026
Merged

marevol merged 1 commit into
mainfrom
fix/concurrent-first-boot-index-alias

Conversation

@marevol

@marevol marevol commented Sep 23, 2026

Copy link
Copy Markdown
Contributor

Why

Start several Fess instances that share the default index names at the same time against an OpenSearch cluster that has no Fess indices yet (for example, several replicas of one deployment brought up together). Since 15.8.0:

  • Each instance finds no fess.update alias, creates its own fess.<timestamp> index, and attaches fess.search and fess.update to it. Both aliases end up pointing at several document indices.

  • Writes through fess.update are rejected, because an alias that points at several indices without a write index cannot be written to. Searches through fess.search read every copy.

  • An instance started afterwards fails at startup:

    Can not configure method(open) of org.codelibs.fess.opensearch.client.SearchEngineClient, because [fess] IndexNotFoundException[no such index [fess]]
    

    open() finds more than one index behind the alias, falls back to the configuration name fess, and then reads the mapping of an index with that name, which does not exist.

The cause is the check-then-create sequence in SearchEngineClient#open(): check that the update alias exists, create the index, then add the aliases in a separate request. fess-suggest#100 fixed the same kind of race for the suggest settings indices.

What changes

  • On first boot the document index is created together with its aliases in one create-index request, and fess.update is marked is_write_index: true. OpenSearch applies index creations one at a time and refuses a second write index for an alias (illegal_state_exception: alias [fess.update] has more than one write index), so exactly one request succeeds. Every other instance then adopts the index behind the update alias and logs Using the document index created by another process.
  • createIndex logs a create that failed only because another process got there first (resource_already_exists_exception, or the second write index) at INFO rather than WARN with a stack trace. This also applies to the fixed-name config, user and log indices, which every instance booting together tries to create.
  • If the update alias already points at more than one index (a cluster left in that state by an earlier release), startup now logs a WARN naming the alias and the indices. Before, the only symptom was the no such index [fess] failure above.
  • The alias-name resolution in createAlias is extracted into resolveAliasName so the new code uses it too. Its behaviour is unchanged.

Nothing changes for an instance that finds the update alias in place. After a reindex from the maintenance screen, updateAlias swaps both aliases to the new index in one request, so the new index is the only index behind fess.update, as before. It does not carry the write-index flag, and it does not need it.

Verification

Unit tests: SearchEngineClientDocumentIndexTest (9 new). They cover the alias definitions sent with the create request, including configured alias names; that CreateIndexRequest turns them into {"fess.update":{"is_write_index":true}} and {"fess.search":{}}; the error classification; and the create / adopt / fail paths of the first-boot set-up. mvn test -Dtest='SearchEngineClient*Test,AdminMaintenance*Test' passes: 113 tests.

The refusal itself, checked directly against OpenSearch 3.8.0:

  • A second PUT <index> that declares the same alias with is_write_index: true is refused with HTTP 500 (has more than one write index).
  • Six such create requests sent in parallel: one succeeded and five were refused. The alias then pointed at exactly one index.

For the end-to-end check, I started one OpenSearch 3.8.0 node (ghcr.io/codelibs/fess-opensearch:3.8.0) and, each run, 4 Fess containers with the default index names at the same moment, then a 5th once the first four had settled. main and this branch were built into the same image, differing only in SearchEngineClient. After each run I read GET _alias/fess.*.

runs indices behind fess.update / fess.search first four started late (5th) instance started
main 3 4 / 4 in every run 10 / 12 0 / 3 (no such index [fess])
this branch 11 1 / 1 in every run (fess.update has is_write_index: true) 41 / 44 11 / 11

All five failures among the first four instances, on both sides, are the fess-suggest failure described under "Not addressed here". None of them came from the document index.

In the three branch runs at INFO level, the logs showed both refusal paths. Every non-winning instance logged Using the document index created by another process, after the second-write-index refusal or, where two instances generated the same timestamp name, resource_already_exists_exception.

Not addressed here

The runs above hit two more concurrent-first-boot problems, both in fess-suggest and not changed here:

  • fess.suggest / fess.suggest.update still ended up with two fess.suggest.<timestamp> indices in most runs, on main and on this branch alike. This is Suggester#createIndexIfNothing, which fess-suggest#100 left out of scope. The same create-with-write-alias approach would fit there.
  • In some runs, on main and on this branch alike, an instance failed in SuggestHelper.init with no_shard_available_action_exception on get [fess_suggest] (shard RECOVERING). The retry that fess-suggest#100 added ran and logged Settings index is not available yet, waiting for it, but its single second attempt failed as well.

Separately, per-instance index names on a fresh cluster depend on a fess-opensearch image that includes opensearch-configsync#18. That is a release step, not a code change.

…on first boot

When several Fess instances that share index names start at the same
time against a search engine with no document index yet, each of them
found no update alias, created its own fess.<timestamp> index, and then
attached fess.search and fess.update to it. The aliases ended up pointing
at several indices. Writes through fess.update fail on such an alias, and
an instance started afterwards died at startup with
"IndexNotFoundException[no such index [fess]]", because it could not pick
one index behind the alias.

On first boot the document index is now created together with its
aliases in a single create-index request, with fess.update marked as the
write index. The search engine applies index creations one at a time and
refuses a second write index for an alias, so only the first request
succeeds. Every other instance adopts the index behind the update alias.

A create request that fails only because another process created the
index first (resource_already_exists_exception, or the second write
index) is logged at INFO instead of WARN with a stack trace; this also
covers the fixed-name config, user and log indices. When the update alias
still points at more than one index, startup now logs that instead of
failing silently on a missing index named after the configuration.
@marevol marevol added this to the 15.9.0 milestone Sep 23, 2026
@marevol marevol self-assigned this Sep 23, 2026
@marevol
marevol merged commit c726630 into main Sep 23, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant