fix: stop concurrent flushes and the updater from skipping stored files

[marevol]

Summary

When several clients store files with POST /_configsync/file and then call POST /_configsync/flush at the same time, a flush can return before the files its caller stored are written, and a few files are never written at all, not even by the scheduled updater. Every store and flush request still returns 200.

Found while verifying Fess 15.9.0. Several Fess instances with their own index names and dictionary prefix, started at the same time against one OpenSearch cluster, each upload their dictionary files, flush, and create their indices. All but one instance then fail to create the document index with IOException while reading mappings_path: file not readable (or stopwords_path), and OpenSearch logs NoSuchFileException for files under that instance's own dictionary prefix.

Cause

ConfigSyncService keeps one lastChecked watermark per node (ConfigSyncService.java:145), used by both the flush transport handler (:715) and the scheduled ConfigFileUpdater (:632). ConfigFileWriter#execute (:651-659) searches for files with @timestamp >= lastChecked and sets lastChecked to the current time before its search has started:

1. Flush A moves the watermark, then scrolls through everything stored since the old watermark, one file per scroll request with the default configsync.scroll_size. Flush B, started a moment later, searches from A's new watermark, finds none of the files B's client stored, and returns while A is still writing them.
2. store() takes a file's @timestamp (:395) before the file is indexed. A flush or updater run that takes its own timestamp after that, but searches before the file is searchable, moves the watermark past a file it never saw, and no later run selects it. The file stays missing on that node until the node restarts.

Fix

• A flush reads every stored file and leaves the watermark alone. Stores use RefreshPolicy.IMMEDIATE, so a file whose store request returned before the flush was sent is searchable and is on disk on every data node when the flush returns, whatever other flushes, updater runs or node clocks do.
• The updater still reads incrementally, but only a completed run advances the watermark, and only to five minutes before that run started. A file stamped before a run started but searchable only after the run's search is read by the next run; the margin also covers clock differences between the node that stamps a file and the nodes that write it. A failed or terminated run leaves the watermark where it was.
• lastChecked is now an AtomicLong; it was a plain field written from transport and scheduler threads.

Reading a file again is safe: updateConfigFile writes a file only when it is missing or older than the stored @timestamp (:580), so files already on disk are not rewritten.

Compatibility

• A flush now reads all stored files, not only those stored since the last run: one scroll request per file on each data node with the default configsync.scroll_size of 1. Flushes are explicit requests, so this cost is paid only when a client asks for it.
• A flush also writes back a file that was removed from a node's config directory; before, only the first updater run after a node restart did.
• Each updater run checks the files stored in the last five minutes again; they are not rewritten.

No settings or REST API change.

Verification

• ConfigSyncPluginTest#test_configFiles_concurrentFlush (new): three clients, each talking to a different node, store 50 files and flush at the same time, for three rounds; after its own flush each client checks all of its files on all three nodes. On main it fails (for example 16 and 18 of 50 files missing on two nodes); with this change it passes.
• ConfigSyncPluginTest#test_configFiles_storedBeforeUpdaterRun (new): indexes a file whose @timestamp is older than updater runs that have already completed, as a store that became searchable late would be. On main no node ever writes it; with this change every node writes it on its next run.
• mvn package: 68 tests, 0 failures, 0 errors.
• The plugin built from this branch, installed into OpenSearch 3.8.0 in place of the released 3.8.0 plugin, with three writers that each store 120 files and flush at the same time, three rounds. Released plugin: in every round two of the three writers had only 26 to 68 of their 120 files after their own flush, and 4 of 1,440 files were still missing after the updater had run. This branch: 120 of 120 after every flush, and 1,440 of 1,440.
• Three Fess instances (built from main) with separate index names and dictionary prefixes, started at once against a fresh OpenSearch 3.8.0 cluster, five runs each. Released plugin: all three instances came up in 1 of 5 runs, 9 of 15 instances in total; every failed instance had file not readable for one of its own dictionary files. This branch: all three came up in 5 of 5 runs, 15 of 15 instances, and no such error.