Cache compiled protocols on disk across processes

[u9g]

Draft: depends on ProtoDef-io/node-protodef#175 being merged and released (uses the new compileProtoDefSync({ cacheFile }) / loadCompiledProtoDefSync).

Problem

Every process recompiles every protocol state from minecraft-data JSON: load protocol.json, walk it in addProtocol, generate code, eval it. The eval'd result dies with the process and is invisible to V8's compile cache. For a full 26.1 client (4 states × both directions) that's ~340ms of protocol setup per process on an idle M-series Mac — paid again by every bot, every test child process, every CLI invocation.

Change

Pass a cacheFile to protodef keyed by the nmp + protodef + minecraft-data versions, protocol key, and a hash of customPackets — everything that determines the generated code — so a stale cache file is simply unreachable and no explicit invalidation exists. On a hit:

• mcData.protocol is never loaded and addProtocol never walks it (those cost as much as compilation itself)
• protodef requires the cached module with the V8 compile cache enabled, so the parse is also amortized across processes

Protocol setup for the same 26.1 client drops to ~170ms with warm caches. First run in a clean tmpdir is unchanged (compile + write).

Cache location: os.tmpdir()/node-minecraft-protocol-cache, override with NMP_PROTOCOL_CACHE_DIR, disable with NMP_PROTOCOL_CACHE_DIR=0. Any read/write failure falls back silently to in-process compilation.

Ordering note

The nbt types are now registered before the cache attempt (their natives are needed to load a cached protocol) and again after addProtocol on a miss. This preserves the previous type precedence (the nbt schemas override types the protocol declares native) — verified by diffing every compiled type function (246 types × read/write/sizeOf) against the old ordering: identical modulo protodef's nondeterministic generated variable names.

Verification

• test/packetTest.js: 5199 passing on both the cold (cache-writing) and warm (cache-loading) paths; test/cyclePacketTest.js: 1014 passing; standard clean.
• Cached and uncached serializers produce byte-identical packets.

[extremeheat]

I don't think this is a good idea

Not only is the code hacky, but it creates a dependence on file system writes (issues for prismarine-viewer, prismarine-web-client, other browser based tools) but also creates issues with stale caches, cache invalidation and the program unexpectedly writing to disk on the machine without obvious means to reclaim the space

What you are likely looking for instead is an in-memory cache, something that there should already be

[u9g]

I don't think this is a good idea

Not only is the code hacky, but it creates a dependence on file system writes (issues for prismarine-viewer, prismarine-web-client, other browser based tools) but also creates issues with stale caches, cache invalidation and the program unexpectedly writing to disk on the machine without obvious means to reclaim the space

What you are likely looking for instead is an in-memory cache, something that there should already be

My goal was to optimize runExample which runs out of process so I can’t do an in memory cache, afaik.

[extremeheat]

340ms is still a fraction of a second only done once/version, I doubt that's what's causing the test delay. >300ms seems rather slow and that itself should be investigated but even despite that for 20 versions it's about 6 seconds in total if not being done in sequence/JITable

[u9g]

340ms is still a fraction of a second only done once/version, I doubt that's what's causing the test delay. >300ms seems rather slow and that itself should be investigated but even despite that for 20 versions it's about 6 seconds in total if not being done in sequence/JITable

Its done for each runExample (which we have many of) serially. The goal is to shed as many seconds as possible off the test suite but if this isn’t a good idea, it’s fine we can drop this one.

[extremeheat]

Since the protocol files can get big (we have 3 sets of the read, write and sizeof protocols), writing to and retrieving from the disk takes much more time than reading from memory (like a 1000x difference)