Skip to content

Add a batch read/write API to LocalDictionary - #28

Merged
imranakram merged 4 commits into
masterfrom
feature/MMS-8327-bulk-dictionary
Sep 7, 2026
Merged

imranakram merged 4 commits into
masterfrom
feature/MMS-8327-bulk-dictionary

Conversation

@imranakram

Copy link
Copy Markdown
Owner

What

Every indexer, ContainsKey and TryGetValue call is its own SQL round trip, so a caller
looping over keys pays a round trip per key. At 100 000 keys that is ~92 s of reads and ~78 s
of writes against a warm database — the profile the MMS-8327 fee run was stuck in.

PersistentBlobCache already had the batch primitives, Get(IEnumerable<string>) and
Insert(IDictionary<string, byte[]>), chunking at 950 keys per statement. LocalDictionary
just exposed no way to reach them.

How

IBulkDictionary<T> GetRange(keys) / SetRange(items). A separate interface from IDictionary, so a consumer holding only the interface can test for it and fall back.
LocalDictionary<T> implements it, routing both members to the batch primitives
DictionaryExtensions the same two methods as extensions on IDictionary<string, T> — batch path when the target supports it, per-key loop when it does not

The extension pair is what lets a caller that holds an IDictionary<string, T>, not knowing
whether it is in memory or persistent, get the batch behaviour without a type check.

Numbers

100 000 keys holding IList<Entity> of five attributes each, .NET Framework 4.8 x64, warm
database:

Per-key loop GetRange / SetRange, blocks of 1 000
Reads ~92 s 1.27 s
Writes ~78 s 75.94 s -> ~3 s with Xrm.Json.Serialization 1.2026.9

Batching alone is ~2.2x end to end (170 s -> 77 s) and all but eliminates the read cost. The
write side turned out not to be SQLite at all — 74 of those 76 seconds were JSON
serialization, which is what the serializer upgrade fixes. For reference, raw SQLite against
the real CacheItem schema is ~12 s for the same rows in blocks of 1 000 and ~2 s in one
transaction.

This raises the Xrm.Json.Serialization floor from 1.2026.3.1 to 1.2026.9. NuGet resolves
lowest-applicable, so the floor has to move or downstream projects keep restoring the slow
version and get half the fix.

Contract notes

  • GetRange omits keys it did not find rather than returning a placeholder, matching
    TryGetValue per key. Duplicate keys in the input collapse to one entry, in both the fast
    path and the fallback.
  • The backend returns an empty buffer for a key it did not find, so absence and "stored an
    empty blob" are indistinguishable at that layer. GetRange treats both as a miss — the same
    as ContainsKey and TryGetValue already do.
  • SetRange replaces existing keys like the indexer setter, and does not throw on a key that
    already exists.

Dependency graph (second commit, b7f9a67)

Taking Xrm.Json.Serialization 1.2026.9 let Visual Studio regenerate the binding redirects,
which narrowed the unification range added for MMS-8327 — VS only ever writes an
up-to-installed range, so SQLitePCLRaw.core and SQLitePCLRaw.batteries_v2 came back as
0.0.0.0-2.1.11.2622. That silently removes the headroom that keeps a stray 1.x or 3.x
reference from surfacing as the TypeLoadException this ticket exists to eliminate. Restored
to 0.0.0.0-3.0.0.0 in both app.config files. Only the test project's copy applies at
runtime — a library's app.config is inert, and the one next to the library is documentary
for consumers.

Three smaller corrections in the same commit:

  • The nuspec dependency on Xrm.Json.Serialization reads 1.2026.9, matching how NuGet
    normalizes and restores it (1.2026.9.0 -> 1.2026.9).

  • SQLitePCLRaw.provider.e_sqlite3 removed. Reading the assembly-reference tables out of
    the built DLLs gives the actual managed chain:

    SQLite-net.dll   -> batteries_v2, core
    batteries_v2.dll -> core, provider.dynamic_cdecl
    core.dll         -> (nothing)
    

    Nothing references provider.e_sqlite3 — it belongs to bundle_e_sqlite3, which this
    project does not use; bundle_green is what supplies the initialisation path. Removed from
    both packages.config files, both Reference blocks, and from the
    CopySQLitePclRawAssemblies target, whose literal (non-wildcard) Include would otherwise
    have failed the Copy task on a missing file.

  • VS also added a System.ValueTuple redirect, which is a real dependency here. Kept.

Two things deliberately not changed:

  • sqlite-net-pcl stays at 1.9.172. 1.11.285 publishes no .NET Framework dependency
    group at all; its .NETStandard2.0 group wants SQLitePCLRaw.core and
    provider.e_sqlite3 3.0.3 plus a new SourceGear.sqlite3 package. Taking it moves the
    assembly identity to 3.0.0.0, drops bundle_green from the graph, and invalidates every
    2.1.11.2622 redirect and Reference Version= in both this repo and JOE. That is the same
    class of failure as previously seen, so it belongs in its own ticket, after JOE is stable, with
    JOE's redirects updated in the same change.
  • SQLitePCLRaw.lib.e_sqlite3 stays at 2.1.13 while core, bundle_green and the
    providers sit at 2.1.11. That is not skew: 2.1.13 is the CVE-2025-6965 floor, and the
    package ships native assets only — no managed assembly, so no binding identity to unify.

Tests

13 tests in BulkDictionaryTests: the interface fast path, the loop fallback, duplicate key
collapsing, missing keys, the 2 000-key chunk boundary, agreement with per-key TryGetValue,
CRM attribute round trips, replacement semantics, empty input and null argument rejection.

Suite: 62 tests, 0 failed (49 before), verified in both configurations after a clean
rebuild — Debug/AnyCPU in 1.60 s and x64/Release, which is what the nuspec packs from, in
2.38 s.

Both output trees carry exactly one version of each assembly, with no provider.e_sqlite3:

SQLite-net                           1.9.172.0
SQLitePCLRaw.batteries_v2            2.1.11.2622
SQLitePCLRaw.core                    2.1.11.2622
SQLitePCLRaw.provider.dynamic_cdecl  2.1.11.2622
e_sqlite3.dll                        (native, from lib.e_sqlite3 2.1.13)

The library's own bin correctly has no native e_sqlite3.dll — that reaches consumers
through the lib.e_sqlite3 nuget dependency, not the lib folder.

Known, pre-existing, not addressed here

Xrm.Persistent.Collections.Tests.csproj declares System.Buffers, Version=4.0.3.0 and
System.Runtime.CompilerServices.Unsafe, Version=6.0.0.0 in its Reference attributes while
the HintPaths point at packages whose assemblies are 4.0.5.0 and 6.0.3.0, so every build
emits MSB3277. The redirects cover it at runtime, which is why the suite is green. The main
project has the correct values. Test-project only, predates this branch.

imranakram and others added 4 commits September 7, 2026 14:03
Every indexer, ContainsKey and TryGetValue call is its own SQL round trip, so a
caller looping over keys pays a round trip per key. At 100 000 keys that is ~92 s
of reads and ~78 s of writes against a warm database, which is what the fee run in
MMS-8327 was spending most of its time on.

PersistentBlobCache already had the batch primitives - Get(IEnumerable<string>) and
Insert(IDictionary<string, byte[]>), chunking at 950 keys per statement - but
LocalDictionary exposed no way to reach them. This adds:

  IBulkDictionary<T>       GetRange(keys) / SetRange(items), a separate interface
                           so a consumer holding only the interface can test for it
  LocalDictionary<T>       implements it, routing to the batch primitives
  DictionaryExtensions     the same two methods on IDictionary<string, T>, taking
                           the batch path when available and looping otherwise

In blocks of 1 000 the reads drop to 1.27 s. The writes drop to 75.94 s, of which
about 74 s is JSON serialization rather than SQLite - that half is fixed by
Xrm.Json.Serialization 1.2026.9.0, so the nuspec floor moves to it. Together they
turn the pass from ~170 s into a few seconds.

Contract notes: GetRange omits keys it did not find rather than returning a
placeholder, matching TryGetValue per key, and duplicate keys collapse. The backend
cannot distinguish a miss from a stored empty blob, so both read as a miss - the
same as ContainsKey and TryGetValue already do. SetRange replaces existing keys
like the indexer setter.

13 tests cover the fast path, the loop fallback, duplicate keys, missing keys, the
2 000-key chunk boundary, agreement with per-key TryGetValue, CRM attribute round
trips, replacement and null arguments. Suite is 62 tests, all passing.
Updating the Xrm.Json.Serialization package to 1.2026.9 let Visual Studio
rewrite the binding redirects, which narrowed the unification range added
for MMS-8327. Three related corrections:

- Restore oldVersion="0.0.0.0-3.0.0.0" on the SQLitePCLRaw.core and
  SQLitePCLRaw.batteries_v2 redirects in both app.config files. VS only
  ever generates an up-to-installed range, so a package update silently
  narrows this back to 0.0.0.0-2.1.11.2622 and removes the headroom that
  keeps a 1.x or 3.x reference from surfacing as a TypeLoadException.
  Xrm.Persistent.Collections/app.config is documentary for consumers; the
  test project's copy is the one that applies at runtime.
- Normalize the nuspec dependency on Xrm.Json.Serialization to 1.2026.9,
  matching how NuGet normalizes and restores it.
- Drop SQLitePCLRaw.provider.e_sqlite3. Reading the assembly references
  out of the built DLLs shows the managed chain is SQLite-net ->
  batteries_v2 + core, and batteries_v2 -> core +
  provider.dynamic_cdecl. Nothing references provider.e_sqlite3, which
  belongs to the unused bundle_e_sqlite3; bundle_green is what supplies
  the initialisation path. Removed from both packages.config files, both
  Reference blocks, and the CopySQLitePclRawAssemblies target, whose
  literal Include would otherwise have failed the Copy task.

VS also added a System.ValueTuple redirect, which is a genuine dependency
here and is kept.

Verified with a clean rebuild of Debug/AnyCPU and of x64/Release, which is
the configuration the nuspec packs from. Both outputs carry exactly one
version of each assembly - SQLite-net 1.9.172.0 and batteries_v2, core and
provider.dynamic_cdecl all at 2.1.11.2622 - with no provider.e_sqlite3.
62 of 62 tests pass in both configurations.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Updated AssemblyFileVersion in AssemblyInfo.cs from 2.2026.9.7 to 2.2026.9.8 for a new build or patch release. AssemblyVersion remains unchanged.
The nuspec <releaseNotes> block still held the entire 2.2026.9.7
security-release text. NuGet publishes it verbatim to nuget.org and a
published version cannot be re-pushed, so this had to be correct before
the tag rather than after it. Every headline claim in it was wrong for
this release: it opened "Security release. No API changes", named
2.2026.3.1 as the previous version, and closed with "No measurable change
is expected or claimed" - against a release that adds IBulkDictionary<T>,
follows 2.2026.9.7, and takes reads from ~92 s to 1.27 s. It also
re-announced the CVE-2025-6965 fix and the concurrency work as if new.

Replaced with notes for what this release actually contains: the batch
API and its contract, the measured numbers, the Xrm.Json.Serialization
floor raise, the provider.e_sqlite3 removal, and a consumer note that no
binding redirect change is required.

CHANGELOG.md already carried a 2.2026.9.8 section covering the API. Added
the dependency-graph half from b7f9a67 - the restored 0.0.0.0-3.0.0.0
redirect ceiling and why Visual Studio narrows it, the kept
System.ValueTuple redirect, and the provider.e_sqlite3 removal with the
assembly-reference chain that shows nothing referenced it - plus a
Verified section matching the format of the 2.2026.9.7 entry. Normalized
the serializer floor to 1.2026.9, which is how NuGet stores and restores
the published 1.2026.9.0.

README.md: the Key Dependencies table still listed Xrm.Json.Serialization
1.2026.3.1 and SQLitePCLRaw.bundle_e_sqlite3 2.1.10, a package dropped in
2.2026.9.7. It now lists the four SQLitePCLRaw packages actually
declared. Footer test count 43 -> 62, and it now names the release and
the assembly version, which differ by design.

Verified by packing the nuspec locally: the notes come through as
written, with all eight dependency floors unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@imranakram
imranakram merged commit 615d5eb into master Sep 7, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant