Add a batch read/write API to LocalDictionary - #28
Merged
Merged
Conversation
Every indexer, ContainsKey and TryGetValue call is its own SQL round trip, so a
caller looping over keys pays a round trip per key. At 100 000 keys that is ~92 s
of reads and ~78 s of writes against a warm database, which is what the fee run in
MMS-8327 was spending most of its time on.
PersistentBlobCache already had the batch primitives - Get(IEnumerable<string>) and
Insert(IDictionary<string, byte[]>), chunking at 950 keys per statement - but
LocalDictionary exposed no way to reach them. This adds:
IBulkDictionary<T> GetRange(keys) / SetRange(items), a separate interface
so a consumer holding only the interface can test for it
LocalDictionary<T> implements it, routing to the batch primitives
DictionaryExtensions the same two methods on IDictionary<string, T>, taking
the batch path when available and looping otherwise
In blocks of 1 000 the reads drop to 1.27 s. The writes drop to 75.94 s, of which
about 74 s is JSON serialization rather than SQLite - that half is fixed by
Xrm.Json.Serialization 1.2026.9.0, so the nuspec floor moves to it. Together they
turn the pass from ~170 s into a few seconds.
Contract notes: GetRange omits keys it did not find rather than returning a
placeholder, matching TryGetValue per key, and duplicate keys collapse. The backend
cannot distinguish a miss from a stored empty blob, so both read as a miss - the
same as ContainsKey and TryGetValue already do. SetRange replaces existing keys
like the indexer setter.
13 tests cover the fast path, the loop fallback, duplicate keys, missing keys, the
2 000-key chunk boundary, agreement with per-key TryGetValue, CRM attribute round
trips, replacement and null arguments. Suite is 62 tests, all passing.
Updating the Xrm.Json.Serialization package to 1.2026.9 let Visual Studio rewrite the binding redirects, which narrowed the unification range added for MMS-8327. Three related corrections: - Restore oldVersion="0.0.0.0-3.0.0.0" on the SQLitePCLRaw.core and SQLitePCLRaw.batteries_v2 redirects in both app.config files. VS only ever generates an up-to-installed range, so a package update silently narrows this back to 0.0.0.0-2.1.11.2622 and removes the headroom that keeps a 1.x or 3.x reference from surfacing as a TypeLoadException. Xrm.Persistent.Collections/app.config is documentary for consumers; the test project's copy is the one that applies at runtime. - Normalize the nuspec dependency on Xrm.Json.Serialization to 1.2026.9, matching how NuGet normalizes and restores it. - Drop SQLitePCLRaw.provider.e_sqlite3. Reading the assembly references out of the built DLLs shows the managed chain is SQLite-net -> batteries_v2 + core, and batteries_v2 -> core + provider.dynamic_cdecl. Nothing references provider.e_sqlite3, which belongs to the unused bundle_e_sqlite3; bundle_green is what supplies the initialisation path. Removed from both packages.config files, both Reference blocks, and the CopySQLitePclRawAssemblies target, whose literal Include would otherwise have failed the Copy task. VS also added a System.ValueTuple redirect, which is a genuine dependency here and is kept. Verified with a clean rebuild of Debug/AnyCPU and of x64/Release, which is the configuration the nuspec packs from. Both outputs carry exactly one version of each assembly - SQLite-net 1.9.172.0 and batteries_v2, core and provider.dynamic_cdecl all at 2.1.11.2622 - with no provider.e_sqlite3. 62 of 62 tests pass in both configurations. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Updated AssemblyFileVersion in AssemblyInfo.cs from 2.2026.9.7 to 2.2026.9.8 for a new build or patch release. AssemblyVersion remains unchanged.
The nuspec <releaseNotes> block still held the entire 2.2026.9.7 security-release text. NuGet publishes it verbatim to nuget.org and a published version cannot be re-pushed, so this had to be correct before the tag rather than after it. Every headline claim in it was wrong for this release: it opened "Security release. No API changes", named 2.2026.3.1 as the previous version, and closed with "No measurable change is expected or claimed" - against a release that adds IBulkDictionary<T>, follows 2.2026.9.7, and takes reads from ~92 s to 1.27 s. It also re-announced the CVE-2025-6965 fix and the concurrency work as if new. Replaced with notes for what this release actually contains: the batch API and its contract, the measured numbers, the Xrm.Json.Serialization floor raise, the provider.e_sqlite3 removal, and a consumer note that no binding redirect change is required. CHANGELOG.md already carried a 2.2026.9.8 section covering the API. Added the dependency-graph half from b7f9a67 - the restored 0.0.0.0-3.0.0.0 redirect ceiling and why Visual Studio narrows it, the kept System.ValueTuple redirect, and the provider.e_sqlite3 removal with the assembly-reference chain that shows nothing referenced it - plus a Verified section matching the format of the 2.2026.9.7 entry. Normalized the serializer floor to 1.2026.9, which is how NuGet stores and restores the published 1.2026.9.0. README.md: the Key Dependencies table still listed Xrm.Json.Serialization 1.2026.3.1 and SQLitePCLRaw.bundle_e_sqlite3 2.1.10, a package dropped in 2.2026.9.7. It now lists the four SQLitePCLRaw packages actually declared. Footer test count 43 -> 62, and it now names the release and the assembly version, which differ by design. Verified by packing the nuspec locally: the notes come through as written, with all eight dependency floors unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Every indexer,
ContainsKeyandTryGetValuecall is its own SQL round trip, so a callerlooping over keys pays a round trip per key. At 100 000 keys that is ~92 s of reads and ~78 s
of writes against a warm database — the profile the MMS-8327 fee run was stuck in.
PersistentBlobCachealready had the batch primitives,Get(IEnumerable<string>)andInsert(IDictionary<string, byte[]>), chunking at 950 keys per statement.LocalDictionaryjust exposed no way to reach them.
How
IBulkDictionary<T>GetRange(keys)/SetRange(items). A separate interface fromIDictionary, so a consumer holding only the interface can test for it and fall back.LocalDictionary<T>DictionaryExtensionsIDictionary<string, T>— batch path when the target supports it, per-key loop when it does notThe extension pair is what lets a caller that holds an
IDictionary<string, T>, not knowingwhether it is in memory or persistent, get the batch behaviour without a type check.
Numbers
100 000 keys holding
IList<Entity>of five attributes each, .NET Framework 4.8 x64, warmdatabase:
GetRange/SetRange, blocks of 1 000Batching alone is ~2.2x end to end (170 s -> 77 s) and all but eliminates the read cost. The
write side turned out not to be SQLite at all — 74 of those 76 seconds were JSON
serialization, which is what the serializer upgrade fixes. For reference, raw SQLite against
the real
CacheItemschema is ~12 s for the same rows in blocks of 1 000 and ~2 s in onetransaction.
This raises the
Xrm.Json.Serializationfloor from 1.2026.3.1 to 1.2026.9. NuGet resolveslowest-applicable, so the floor has to move or downstream projects keep restoring the slow
version and get half the fix.
Contract notes
GetRangeomits keys it did not find rather than returning a placeholder, matchingTryGetValueper key. Duplicate keys in the input collapse to one entry, in both the fastpath and the fallback.
empty blob" are indistinguishable at that layer.
GetRangetreats both as a miss — the sameas
ContainsKeyandTryGetValuealready do.SetRangereplaces existing keys like the indexer setter, and does not throw on a key thatalready exists.
Dependency graph (second commit,
b7f9a67)Taking
Xrm.Json.Serialization 1.2026.9let Visual Studio regenerate the binding redirects,which narrowed the unification range added for MMS-8327 — VS only ever writes an
up-to-installed range, so
SQLitePCLRaw.coreandSQLitePCLRaw.batteries_v2came back as0.0.0.0-2.1.11.2622. That silently removes the headroom that keeps a stray 1.x or 3.xreference from surfacing as the
TypeLoadExceptionthis ticket exists to eliminate. Restoredto
0.0.0.0-3.0.0.0in bothapp.configfiles. Only the test project's copy applies atruntime — a library's
app.configis inert, and the one next to the library is documentaryfor consumers.
Three smaller corrections in the same commit:
The nuspec dependency on
Xrm.Json.Serializationreads1.2026.9, matching how NuGetnormalizes and restores it (
1.2026.9.0->1.2026.9).SQLitePCLRaw.provider.e_sqlite3removed. Reading the assembly-reference tables out ofthe built DLLs gives the actual managed chain:
Nothing references
provider.e_sqlite3— it belongs tobundle_e_sqlite3, which thisproject does not use;
bundle_greenis what supplies the initialisation path. Removed fromboth
packages.configfiles, bothReferenceblocks, and from theCopySQLitePclRawAssembliestarget, whose literal (non-wildcard)Includewould otherwisehave failed the
Copytask on a missing file.VS also added a
System.ValueTupleredirect, which is a real dependency here. Kept.Two things deliberately not changed:
sqlite-net-pclstays at 1.9.172. 1.11.285 publishes no .NET Framework dependencygroup at all; its
.NETStandard2.0group wantsSQLitePCLRaw.coreandprovider.e_sqlite33.0.3 plus a newSourceGear.sqlite3package. Taking it moves theassembly identity to
3.0.0.0, dropsbundle_greenfrom the graph, and invalidates every2.1.11.2622redirect andReference Version=in both this repo and JOE. That is the sameclass of failure as previously seen, so it belongs in its own ticket, after JOE is stable, with
JOE's redirects updated in the same change.
SQLitePCLRaw.lib.e_sqlite3stays at 2.1.13 whilecore,bundle_greenand theproviders sit at 2.1.11. That is not skew: 2.1.13 is the CVE-2025-6965 floor, and the
package ships native assets only — no managed assembly, so no binding identity to unify.
Tests
13 tests in
BulkDictionaryTests: the interface fast path, the loop fallback, duplicate keycollapsing, missing keys, the 2 000-key chunk boundary, agreement with per-key
TryGetValue,CRM attribute round trips, replacement semantics, empty input and null argument rejection.
Suite: 62 tests, 0 failed (49 before), verified in both configurations after a clean
rebuild — Debug/AnyCPU in 1.60 s and x64/Release, which is what the nuspec packs from, in
2.38 s.
Both output trees carry exactly one version of each assembly, with no
provider.e_sqlite3:The library's own
bincorrectly has no nativee_sqlite3.dll— that reaches consumersthrough the
lib.e_sqlite3nuget dependency, not thelibfolder.Known, pre-existing, not addressed here
Xrm.Persistent.Collections.Tests.csprojdeclaresSystem.Buffers, Version=4.0.3.0andSystem.Runtime.CompilerServices.Unsafe, Version=6.0.0.0in itsReferenceattributes whilethe HintPaths point at packages whose assemblies are
4.0.5.0and6.0.3.0, so every buildemits MSB3277. The redirects cover it at runtime, which is why the suite is green. The main
project has the correct values. Test-project only, predates this branch.