The current architecture holds the entire dataset in memory as EntityCollection objects stored in ShuffleBlocks (a Dictionary<string, EntityCollection>). This causes two scaling problems:
Memory pressure - Exporting/importing 100k+ records loads all entities into RAM simultaneously. Each entity with 50 attributes can consume several KB, so large datasets can exhaust available memory.
O(n²) record matching on import - GetMatchingRecordsFromPreRetrieved() in ShuffleDataImport.cs compares every imported record against every existing record using string-converted attribute values. For 1000 imports × 5000 existing records × 5 match attributes, this produces ~25 million string conversions.
Proposal
Integrate Xrm.Persistent.Collections to back EntityCollection storage with SQLite. This provides:
Benefits
Bounded memory - only the active working set is in RAM; the rest lives on disk in SQLite
Indexed matching - create SQLite indexes on match attributes for O(log n) lookups instead of O(n²) scans
Chunked processing - natural support for processing records in batches (pairs well with ExecuteMultipleRequest batching from #15)
Crash recovery - SQLite state survives process interruption; long-running imports can resume
The current architecture holds the entire dataset in memory as
EntityCollectionobjects stored inShuffleBlocks(aDictionary<string, EntityCollection>). This causes two scaling problems:Memory pressure - Exporting/importing 100k+ records loads all entities into RAM simultaneously. Each entity with 50 attributes can consume several KB, so large datasets can exhaust available memory.
O(n²) record matching on import -
GetMatchingRecordsFromPreRetrieved()inShuffleDataImport.cscompares every imported record against every existing record using string-converted attribute values. For 1000 imports × 5000 existing records × 5 match attributes, this produces ~25 million string conversions.Proposal
Integrate Xrm.Persistent.Collections to back EntityCollection storage with SQLite. This provides:
Benefits
Bounded memory - only the active working set is in RAM; the rest lives on disk in SQLite
Indexed matching - create SQLite indexes on match attributes for O(log n) lookups instead of O(n²) scans
Chunked processing - natural support for processing records in batches (pairs well with ExecuteMultipleRequest batching from #15)
Crash recovery - SQLite state survives process interruption; long-running imports can resume