benchmarks and perf improvement

README.md

@@ -600,7 +600,7 @@ Schema, table, column, identifier, and relationship names may contain environmen
 - `core`: runtime ORM, H2 cache, PostgreSQL synchronization, and Redis integration
 - `processor`: Java annotation processor that creates builders and query builders
 - `intellij-plugin`: IntelliJ IDEA awareness for the generated API
-- `benchmark`: JMH benchmarks
+- `benchmark`: JMH microbenchmarks and container-backed Minecraft workloads; see [`benchmark/README.md`](benchmark/README.md)
 - `utils`: shared internal utilities
 
 The project is currently published as a snapshot. Expect API and behavior changes between snapshot versions.

benchmark/PERFORMANCE.md

@@ -0,0 +1,80 @@
+# Current performance baseline
+
+This document records measurements of the current Static Data implementation. Results are machine-specific and should be compared on the same idle host, JVM, benchmark parameters, and commit.
+
+## Read throughput
+
+`ReadThroughputBenchmark` reports explicit operations per second for production read paths. The following results used Java 21, eight reader threads, and a retained, prewarmed 100-player set. Each player has one settings reference and eight friends.
+
+A compound player read performs a player instance lookup, resolves the settings reference and underlying settings instance, then reads the settings priority and player name.
+
+| Read operation | Throughput |
+| --- | ---: |
+| Instance-cache lookup | 25.94 ± 6.07 million ops/s |
+| Player lookup + persistent value | 12.79 ± 1.11 million ops/s |
+| Player lookup + settings reference | 12.35 ± 1.65 million ops/s |
+| Compound player read | 5.72 ± 0.46 million ops/s |
+| Friend collection | 0.497 ± 0.040 million ops/s |
+| Complete 100-player scan | 3,777 ± 258 scans/s |
+
+Working-set retention is parameterized. With eight readers selecting across 1,000 players, compound throughput was 4.33 ± 0.77 million reads/s when the complete set was retained and prewarmed. It was 3.94 ± 0.50 million reads/s when only the 100-player tick set was retained and the other 900 entries could be reclaimed between iterations.
+
+## End-to-end player scan
+
+`StaticDataBenchmark` uses the current DataManager, H2 mirror, PostgreSQL 16.2, and Redis 7.4.1. It creates 32 players with settings and friends, then runs one simulated Minecraft server thread alongside four cache-reader threads.
+
+| Operation | Result |
+| --- | ---: |
+| Hot `DataManager.getInstance()` hit | 0.06 ± 0.02 µs/op |
+| Complete 32-player scan under asynchronous load | 538.22 ± 101.91 µs/op |
+
+The modeled scan consumes about 0.54 ms, or 1.1% of a 50 ms tick budget, on the measurement host. It is a regression workload, not a production TPS prediction.
+
+## Cross-container load
+
+`CrossContainerLoadBenchmark` uses 100 players with eight friends and one settings reference per player. Peer sessions update PostgreSQL rows across disjoint partitions of the configured write set. Each committed update traverses the PostgreSQL trigger, `NOTIFY`, the full Static Data listener, H2 application, cache invalidation, and subsequent reads. Seven additional listener connections model notification fan-out.
+
+The mixed-load runs use 50 writes/s per peer session:
+
+| Remote load | Tick mean | Tick p95 | Tick p99 | Async-read p50 | Async-read p99 |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| Local-only control | 1.65 ms | 2.30 ms | 3.21 ms | 1.4 µs | 2.8 µs |
+| 1 peer writer, 50 writes/s | 2.24 ms | 3.33 ms | 4.80 ms | 1.6 µs | 6.5 µs |
+| 8 peer writers, 400 writes/s | 2.42 ms | 3.49 ms | 4.66 ms | 1.7 µs | 4.3 µs |
+
+At 400 remote writes/s, the modeled tick p99 consumes about 9.3% of a 50 ms tick budget on this host.
+
+The isolated four-peer measurements are:
+
+| Distributed operation | Mean | p50 | p95 | p99 |
+| --- | ---: | ---: | ---: | ---: |
+| PostgreSQL write/trigger/commit round trip | 1.13 ms | 1.07 ms | 1.72 ms | 2.22 ms |
+| Update request to visibility in the listening cache | 1.34 ms | 1.30 ms | 1.89 ms | 2.33 ms |
+
+These cover the distributed pipeline end to end. They do not isolate network delivery, listener work, H2 application, and cache invalidation into separate timings.
+
+## Controlled high-read load
+
+The controlled workload uses eight reader threads sharing one aggregate rate limiter. In parallel, one thread continuously scans all 100 players, eight peer sessions target a combined 400 writes/s, and eight PostgreSQL listeners receive invalidations.
+
+| Compound-read target | Achieved | Tick mean | Tick p95 | Tick p99 |
+| ---: | ---: | ---: | ---: | ---: |
+| 100,000/s | 98,568/s | 2.09 ms | 2.88 ms | 3.56 ms |
+| 250,000/s | 246,294/s | 2.17 ms | 3.13 ms | 4.33 ms |
+| 500,000/s | 496,101/s | 2.15 ms | 3.01 ms | 4.17 ms |
+
+The paced writers achieved about 400 updates/s and the seven notification-only peers observed about 2,800 callbacks/s, confirming the expected notification fan-out. No throughput cliff appeared by 500,000 compound reads/s on the measurement host.
+
+The JMH score for a controlled reader includes its wait for the next permit. Use the emitted achieved-rate line for controlled workloads and `ReadThroughputBenchmark` for saturation capacity.
+
+## Current slow paths
+
+1. `PersistentManyToManyCollectionImpl.getIds()` constructs SQL and executes an H2 join query for every collection read. This is the clearest measured read bottleneck. A membership cache needs dependency or generation invalidation that also handles remotely inserted join rows.
+2. `ReferenceImpl.getReferencedColumnValuePairs()` creates query inputs and identifier objects on cached reference reads. A cached per-reference lookup key could reduce allocation if it is invalidated when holder ID or linking columns change.
+3. A complete player scan repeatedly resolves relationships and collection members. Consumers that read the same projection several times during one tick may benefit from a server-layer per-tick snapshot.
+
+The benchmarks intentionally retain these production paths. Add a focused benchmark before optimizing one, then compare measurements on the same machine and commit range.
+
+## Scope and limitations
+
+The container-backed results use local Docker networking. Notification-only peers do not instantiate complete DataManager and H2 stacks. The suite does not model other plugins, Minecraft engine work, WAN latency, or database hosts under unrelated load. The continuously repeated tick scan is a contention stress workload rather than a 20 Hz scheduler.

benchmark/README.md

@@ -0,0 +1,91 @@
+# Static Data benchmarks
+
+The benchmark module contains three layers:
+
+- `StaticDataBenchmark` is an integration benchmark backed by Testcontainers PostgreSQL and Redis plus Static Data's H2 cache. Its grouped workload models one Minecraft server thread resolving 32 players, settings references, and friend collections while four asynchronous workers resolve cached players.
+- `ReadThroughputBenchmark` reports explicit operations/second for individual production read paths and a complete configurable player scan. It defaults to eight reader threads, 100 retained players, and a fully warmed working set.
+- `CrossContainerLoadBenchmark` models one listening Static Data container and peer containers connected to the same PostgreSQL database. Peer writers use persistent database sessions with distinct application names, so writes traverse the real PostgreSQL trigger, `NOTIFY`, Static Data listener, H2 mirror update, cache invalidation, and subsequent local-read path. Additional notification-only listeners model PostgreSQL fan-out without incorrectly sharing a single H2 mirror between simulated containers.
+
+`CrossContainerLoadBenchmark` supplies these scenarios:
+
+- `localOnly`: one 100-player tick scan plus four local asynchronous readers, used as the control.
+- `readHeavyCrossContainer`: the same readers plus one peer writer, paced to 50 writes/s by default.
+- `writeHeavyCrossContainer`: the same readers plus eight peer writers, paced to a combined 400 writes/s by default.
+- `controlledReadLoad`: one tick scanner plus eight readers sharing an explicit aggregate target of 250,000 compound reads/s by default.
+- `controlledMixedLoad`: the controlled readers plus eight peer writers (400 writes/s total by default) and the configured notification listeners.
+- `remoteWriteRoundTrip`: four unpaced peer sessions measuring PostgreSQL update/trigger/commit latency.
+- `remoteUpdatePropagation`: four peer sessions measuring the complete update-request-to-local-cache-visibility latency.
+
+One compound read performs a player instance lookup, resolves its settings reference (including the settings instance lookup), and reads the settings priority and player name. Thus, 250,000 compound reads/s represents roughly one million public API-level lookup/value operations per second.
+
+The controlled-reader and mixed-load writer scores include intentional pacing and should not be interpreted as operation latency. The load-rate lines emitted after every iteration are the authoritative achieved read/write rates and also report notification fan-out plus Static Data's rolling H2 counters. Use `ReadThroughputBenchmark` for maximum read throughput and `remoteWriteRoundTrip` for database-write latency. The tick loop runs continuously rather than at 20 Hz, making it a contention stress test rather than a literal server scheduler.
+
+Docker must be running for the benchmark suite.
+
+Run the end-to-end Minecraft workload:
+
+```powershell
+.\gradlew.bat :benchmark:jmh -PjmhIncludes='.*StaticDataBenchmark.*'
+```
+
+Report maximum throughput for every production read path:
+
+```powershell
+.\gradlew.bat :benchmark:jmh -PjmhIncludes='.*ReadThroughputBenchmark.*'
+```
+
+Run the cross-container workloads:
+
+```powershell
+.\gradlew.bat :benchmark:jmh -PjmhIncludes='.*CrossContainerLoadBenchmark.*'
+```
+
+Run only the write-heavy scenario:
+
+```powershell
+.\gradlew.bat :benchmark:jmh -PjmhIncludes='.*CrossContainerLoadBenchmark.writeHeavy.*'
+```
+
+Run a controlled 100k/250k/500k compound-read matrix while eight peers write at a combined 400 writes/s:
+
+```powershell
+.\gradlew.bat :benchmark:jmh -PjmhIncludes='.*CrossContainerLoadBenchmark.controlledMixed.*' "-PjmhParams=targetReadsPerSecond=100000,250000,500000"
+```
+
+The container-backed states default to eight PostgreSQL notification listeners, 100 database players, a 100-player hot set, 100 players scanned per tick operation, eight friends per player, 50 writes/s per peer writer, peer writes partitioned across 100 players, and a fully retained/prewarmed read set. Each peer owns a disjoint slice so propagation tests cannot overwrite a value before its writer observes it. Override JMH parameters without editing source:
+
+```powershell
+.\gradlew.bat :benchmark:jmh -PjmhIncludes='.*CrossContainerLoadBenchmark.writeHeavy.*' "-PjmhParams=listenerCount=16;playerCount=1000;hotPlayerCount=500;playersPerTick=250;remoteWritePlayerCount=500;remoteWritesPerSecond=100"
+```
+
+Multiple values create a parameter matrix, for example `"-PjmhParams=listenerCount=1,4,8,16;remoteWritesPerSecond=10,50,100"`. `hotPlayerCount`, `playersPerTick`, and `remoteWritePlayerCount` must not exceed `playerCount`; `remoteWritePlayerCount` must also be at least the scenario's peer-writer count (eight for the heavy groups). A read or write rate of zero disables its pacing and runs it at saturation.
+
+To expose weak-cache misses and rehydration instead of measuring only a permanently hot online-player set, use a larger read set and disable full retention/prewarming:
+
+```powershell
+.\gradlew.bat :benchmark:jmh -PjmhIncludes='.*ReadThroughputBenchmark.compoundPlayerRead.*' "-PjmhParams=playerCount=1000;hotPlayerCount=1000;playersPerTick=100;warmReadWorkingSet=true,false"
+```
+
+Build a reader scaling curve by running the same command with `-PjmhThreads=1`, `4`, `8`, `16`, and `32`. The following optional project properties let CI or a local investigation make any scenario longer without editing annotations:
+
+- `jmhThreads`: override the benchmark's reader-thread count.
+- `jmhWarmupIterations`: number of warmup iterations.
+- `jmhIterations`: number of measured iterations.
+- `jmhTime`: duration of each iteration, such as `2s` or `60s`.
+- `jmhForks`: independent benchmark JVM count.
+
+For example, this is a five-minute measured soak at 500,000 compound reads/s plus 400 remote writes/s:
+
+```powershell
+.\gradlew.bat :benchmark:jmh -PjmhIncludes='.*CrossContainerLoadBenchmark.controlledMixed.*' "-PjmhParams=playerCount=1000;hotPlayerCount=1000;playersPerTick=250;targetReadsPerSecond=500000;warmReadWorkingSet=false" -PjmhWarmupIterations=2 -PjmhIterations=5 -PjmhTime=60s
+```
+
+Run every benchmark:
+
+```powershell
+.\gradlew.bat :benchmark:jmh
+```
+
+Machine-readable results are written to `benchmark/build/reports/jmh/results.json`; the complete console-style report is written to `benchmark/build/reports/jmh/human.txt`. Each invocation replaces these files, so copy them elsewhere when comparing separate runs.
+
+See [`PERFORMANCE.md`](PERFORMANCE.md) for the current baseline, measured slow paths, and benchmark limitations.

benchmark/build.gradle

@@ -22,6 +22,7 @@ dependencies {
     implementation 'net.staticstudios:static-utils:1.0.6-SNAPSHOT'
     implementation("org.testcontainers:postgresql:1.19.8")
     implementation("com.redis:testcontainers-redis:2.2.2")
+    implementation("com.impossibl.pgjdbc-ng:pgjdbc-ng:0.8.9")
     implementation("org.slf4j:slf4j-log4j12:2.0.16")
 }
 
@@ -33,15 +34,67 @@ tasks.named('jmh') {
     jvmArgs = [
             '-Xms1g',
             '-Xmx1g',
-            '-XX:+AlwaysPreTouch',
-            '-Djmh.ignoreLock=true'
+            '-XX:+AlwaysPreTouch'
     ]
 }
 
+jmh {
+    def configuredIncludes = project.findProperty('jmhIncludes')
+    if (configuredIncludes != null) {
+        includes = [configuredIncludes.toString()]
+    }
+
+    def configuredParams = project.findProperty('jmhParams')
+    if (configuredParams != null) {
+        benchmarkParameters = configuredParams.toString()
+                .split(';')
+                .collectEntries { entry ->
+                    def parts = entry.split('=', 2)
+                    if (parts.length != 2 || parts[0].isBlank() || parts[1].isBlank()) {
+                        throw new GradleException("Invalid jmhParams entry '${entry}'; expected name=value1,value2")
+                    }
+                    def values = project.objects.listProperty(String)
+                    values.set(parts[1].split(',').toList())
+                    [(parts[0]): values]
+                }
+    }
+
+    def configuredThreads = project.findProperty('jmhThreads')
+    if (configuredThreads != null) {
+        threads = Integer.parseInt(configuredThreads.toString())
+    }
+
+    def configuredIterations = project.findProperty('jmhIterations')
+    if (configuredIterations != null) {
+        iterations = Integer.parseInt(configuredIterations.toString())
+    }
+
+    def configuredWarmupIterations = project.findProperty('jmhWarmupIterations')
+    if (configuredWarmupIterations != null) {
+        warmupIterations = Integer.parseInt(configuredWarmupIterations.toString())
+    }
+
+    def configuredIterationTime = project.findProperty('jmhTime')
+    if (configuredIterationTime != null) {
+        timeOnIteration = configuredIterationTime.toString()
+    }
+
+    def configuredForks = project.findProperty('jmhForks')
+    if (configuredForks != null) {
+        fork = Integer.parseInt(configuredForks.toString())
+    }
+
+    failOnError = true
+    forceGC = true
+    resultFormat = 'JSON'
+    resultsFile = layout.buildDirectory.file('reports/jmh/results.json').get().asFile
+    humanOutputFile = layout.buildDirectory.file('reports/jmh/human.txt').get().asFile
+}
+
 java {
     targetCompatibility = JavaVersion.VERSION_21
     sourceCompatibility = JavaVersion.VERSION_21
     toolchain {
         languageVersion = JavaLanguageVersion.of(21)
     }
-}
+}