fix(index): create the document index and its aliases in one request on first boot - #3472
Merged
Merged
Conversation
…on first boot When several Fess instances that share index names start at the same time against a search engine with no document index yet, each of them found no update alias, created its own fess.<timestamp> index, and then attached fess.search and fess.update to it. The aliases ended up pointing at several indices. Writes through fess.update fail on such an alias, and an instance started afterwards died at startup with "IndexNotFoundException[no such index [fess]]", because it could not pick one index behind the alias. On first boot the document index is now created together with its aliases in a single create-index request, with fess.update marked as the write index. The search engine applies index creations one at a time and refuses a second write index for an alias, so only the first request succeeds. Every other instance adopts the index behind the update alias. A create request that fails only because another process created the index first (resource_already_exists_exception, or the second write index) is logged at INFO instead of WARN with a stack trace; this also covers the fixed-name config, user and log indices. When the update alias still points at more than one index, startup now logs that instead of failing silently on a missing index named after the configuration.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Start several Fess instances that share the default index names at the same time against an OpenSearch cluster that has no Fess indices yet (for example, several replicas of one deployment brought up together). Since 15.8.0:
Each instance finds no
fess.updatealias, creates its ownfess.<timestamp>index, and attachesfess.searchandfess.updateto it. Both aliases end up pointing at several document indices.Writes through
fess.updateare rejected, because an alias that points at several indices without a write index cannot be written to. Searches throughfess.searchread every copy.An instance started afterwards fails at startup:
open()finds more than one index behind the alias, falls back to the configuration namefess, and then reads the mapping of an index with that name, which does not exist.The cause is the check-then-create sequence in
SearchEngineClient#open(): check that the update alias exists, create the index, then add the aliases in a separate request. fess-suggest#100 fixed the same kind of race for the suggest settings indices.What changes
fess.updateis markedis_write_index: true. OpenSearch applies index creations one at a time and refuses a second write index for an alias (illegal_state_exception: alias [fess.update] has more than one write index), so exactly one request succeeds. Every other instance then adopts the index behind the update alias and logsUsing the document index created by another process.createIndexlogs a create that failed only because another process got there first (resource_already_exists_exception, or the second write index) at INFO rather than WARN with a stack trace. This also applies to the fixed-name config, user and log indices, which every instance booting together tries to create.no such index [fess]failure above.createAliasis extracted intoresolveAliasNameso the new code uses it too. Its behaviour is unchanged.Nothing changes for an instance that finds the update alias in place. After a reindex from the maintenance screen,
updateAliasswaps both aliases to the new index in one request, so the new index is the only index behindfess.update, as before. It does not carry the write-index flag, and it does not need it.Verification
Unit tests:
SearchEngineClientDocumentIndexTest(9 new). They cover the alias definitions sent with the create request, including configured alias names; thatCreateIndexRequestturns them into{"fess.update":{"is_write_index":true}}and{"fess.search":{}}; the error classification; and the create / adopt / fail paths of the first-boot set-up.mvn test -Dtest='SearchEngineClient*Test,AdminMaintenance*Test'passes: 113 tests.The refusal itself, checked directly against OpenSearch 3.8.0:
PUT <index>that declares the same alias withis_write_index: trueis refused with HTTP 500 (has more than one write index).For the end-to-end check, I started one OpenSearch 3.8.0 node (
ghcr.io/codelibs/fess-opensearch:3.8.0) and, each run, 4 Fess containers with the default index names at the same moment, then a 5th once the first four had settled.mainand this branch were built into the same image, differing only inSearchEngineClient. After each run I readGET _alias/fess.*.fess.update/fess.searchmainno such index [fess])fess.updatehasis_write_index: true)All five failures among the first four instances, on both sides, are the fess-suggest failure described under "Not addressed here". None of them came from the document index.
In the three branch runs at INFO level, the logs showed both refusal paths. Every non-winning instance logged
Using the document index created by another process, after the second-write-index refusal or, where two instances generated the same timestamp name,resource_already_exists_exception.Not addressed here
The runs above hit two more concurrent-first-boot problems, both in fess-suggest and not changed here:
fess.suggest/fess.suggest.updatestill ended up with twofess.suggest.<timestamp>indices in most runs, onmainand on this branch alike. This isSuggester#createIndexIfNothing, which fess-suggest#100 left out of scope. The same create-with-write-alias approach would fit there.mainand on this branch alike, an instance failed inSuggestHelper.initwithno_shard_available_action_exceptiononget [fess_suggest](shardRECOVERING). The retry that fess-suggest#100 added ran and loggedSettings index is not available yet, waiting for it, but its single second attempt failed as well.Separately, per-instance index names on a fresh cluster depend on a
fess-opensearchimage that includes opensearch-configsync#18. That is a release step, not a code change.