13 NodeTypes are parked at CompileError on memex-cloud (2 on memex) — configuration lambdas reference types their source query never includes

[rbuergi]

Found while stabilising prod on 2026-08-17. Not a new regression — a standing condition that nothing has been ticketing (see the log-watcher issue filed alongside this).

What is broken

portal	NodeTypes parked at CompileError
memex-cloud (memex.meshweaver.cloud)	13
memex (memex.systemorph.com)	2 — BinaryClickerV2/BinaryToggle, Edu/Course
atioz (customer prod)	0 ✅

The 13, with the first error of each:

node	error
Northwind/Product	CS0246 Supplier, Category not found
Northwind/AnalyticsCatalog	CS0103 OrderViews, SalesViews do not exist
FutuRe/{AmericasIns,EuropeRe,AsiaRe}/LineOfBusiness	CS0246 LineOfBusiness; CS1061 AddLineOfBusinessLayoutAreas
FutuRe/{AmericasIns,EuropeRe,AsiaRe}/TransactionMapping	CS0246 TransactionMapping
Doc/DataMesh/SocialMedia/{Post,Profile}	CS0246 SocialMediaPost/SocialMediaProfile, Platform
SocialMedia/{Post,Profile}	same
Doc/Architecture/BusinessRules/Cession	CS0246 CessionData, Cashflow

All are under the MeshWeaver/samples/Graph/Data/** import.

One shape, three causes — and they need different fixes

Every one of these is "the configuration lambda references a type its source query never included". The compiler says so explicitly for the worst case:

Matched Code nodes (0):
  (none) — the configuration lambda cannot reference types because no source files were
  included. Check that your Source Code nodes exist and that the NodeType's `sources` list
  points at them.

But the reason the query comes up short differs, and lumping them would produce one wrong fix:

1. Stale orphans. samples/Graph/Data/SocialMedia/** does not exist in the MeshWeaver repo at all — SocialMedia moved to its own repo. The mesh still holds the NodeTypes, so the import adds and updates but never retires what the source tree dropped. These should be deleted, not fixed.
2. Sources that are not under the node's own Source/. FutuRe/* and Doc/Architecture/BusinessRules/Cession have zero .cs files under them in the repo — their types live elsewhere (shared/sibling), and the default query (namespace:<node>/Source scope:subtree nodeType:Code) cannot reach them. Needs a sources list, or the content moved.
3. Sources that exist and still do not resolve. Northwind/Product and AnalyticsCatalog do ship Source/*.cs with correct // <meshweaver> headers and no NodeType: override, so they should import as Code — yet Supplier, Category, OrderViews, SalesViews are missing while other types in the same lambda resolve. This is the one to investigate first: a partial source set means the import dropped specific files, which is the same silent-skip family as A UTF-8 BOM makes the node-repo parser skip a file — PensionFund loses 62 of 72 files and gates ZERO NodeTypes, silently #1767.

🚨 Ruled out: it is not the #1767 BOM defect — 0 of 28 .cs under the failing Northwind nodes carry a BOM.

Why it matters, stated accurately

The failures are contained: CompileWatcher parks a failed NodeType and serves the cached error without recompiling, so there is no retry storm (the log says so in as many words). What it costs is real anyway — every page backed by these types is broken for users, and the parked state is invisible unless someone reads pod logs.

Verification

kubectl -n memex-cloud logs <portal-pod> | grep "PARKED after compile failure", on a pod started fresh at 19:26 UTC, so this is the current state and not history.

[rbuergi]

Root cause of "case 3" — and it is not a dropped file

No file was dropped by the importer. The mesh was never asked.

The 32 NodeType files under samples/Graph/Data were produced by exporting a running mesh, so each carries that machine's runtime compile record next to the authored definition: compilationStatus, compilationError, Roslyn compilationDiagnostics, compiledSources/currentSourceVersions, assembly coordinates, release-request timestamps.

Exactly eight carry "compilationStatus": "Error" — and they are exactly the eight non-SocialMedia rows in the table above:

samples/Graph/Data/Northwind/Product.json
samples/Graph/Data/Northwind/AnalyticsCatalog.json
samples/Graph/Data/FutuRe/{AmericasIns,EuropeRe,AsiaRe}/LineOfBusiness.json
samples/Graph/Data/FutuRe/{AmericasIns,EuropeRe,AsiaRe}/TransactionMapping.json

The compilationError and Matched Code nodes (0) in each is the transcript of a compile that ran on 2026-06-26, committed into the file. Two proofs it cannot describe the sources it ships beside:

• AnalyticsCatalog's error names OrderViews, SalesViews, SupplierViews, ProductViews. Those identifiers occur nowhere in the configuration in the same file, which calls AddOrderLayoutAreas() / AddSalesLayoutAreas() / AddSupplierLayoutAreas() / AddProductLayoutAreas() — all of which exist under AnalyticsCatalog/Source/. The configuration had already been corrected; the correction could never take effect.
• Product's error points at lines 200 and 204 of a source set that is 181 lines long.

Why the state is unrecoverable, not merely stale

Every automatic path that could re-drive a compile is closed to such a node:

path	requires	a baked-Error node
first-build kickoff	CompilationStatus is null	✗ it is Error — reads as "already attempted here"
recovery kickoff	CompilationStatus == Compiling	✗
framework-stale kickoff	LatestAssemblyCollection + LatestAssemblyPath + CompiledFrameworkVersion	✗ ApplyCompileFailure stamps none of them
release-request watcher	RequestedReleaseAt > LastReleaseRequestHandledAt	✗ the export baked the two equal

The 24 files that baked Ok escaped only by accident: they also carry assembly coordinates plus a compiledFrameworkVersion MVID, which never matches another deployment, so the framework-stale kickoff rebuilds them. The Ok bake is the same defect; it is merely self-healing.

PR #1792 strips the runtime-owned members from all 32 files (byte-exact elsewhere) and adds ShippedNodeTypeStateTest, whose banned set is derived from NodeTypeDefinition by reflection over the control plane's naming convention so a member added later is banned the day it is added. Verified on memex-cloud that omitting the state is the supported shape: Doc/Architecture/BusinessRules/Cession ships no compile state in its file and its live node holds a full, current compile record.

---

The measurements correct three rows of the table

Read live off memex-cloud (get on each node, so this is the node's own record, not a log line):

node	measured now
Doc/DataMesh/SocialMedia/Post	Ok, v804, compiled 19:24:44, 3 sources present
Doc/DataMesh/SocialMedia/Profile	Ok, v664, compiled 19:24:44, 2 sources present
SocialMedia/Post	Ok, v1177, compiled 15:43:50, 9 sources present
SocialMedia/Profile	Ok, v471, 10 sources present (uses sources: ["namespace:Source scope:subtree", "shared=@SocialMedia/Post/Source"])
Doc/Architecture/BusinessRules/Cession	Ok, v671, lsp_diagnostics_for_node returns ok:true — warnings only

So "case 1 — stale orphans that want deleting" does not hold. SocialMedia/{Post,Profile} are the live plugin's types with their sources in the mesh, and Doc/DataMesh/SocialMedia/{Post,Profile} compile from Doc/DataMesh/SocialMedia/*/Source/*, which exist. Cession is likewise fine — its four Source/* Code nodes are present (CessionData declares Cashflow, ExcessOfLossLayer, CededCashflow). Deleting any of these would have destroyed working content.

Those five rows were true of some pod, not of the mesh at the time they were written down — which is the same shape as the bug itself: a compile verdict outliving the thing it described.

Northwind/{Product,AnalyticsCatalog} are not on memex-cloud at all — Memex.Portal.Distributed syncs only Doc, Agent, Model, Harness, Skill, and search nodeType:NodeType namespace:MeshWeaver scope:descendants returns 19 types under ACME / FutuRe / Systemorph and no Northwind. Their baked Error is still a real defect for every deployment that does load them (the dev Monolith, the test fixtures) — fixed by #1792.

---

What is genuinely broken, and why I stopped short of fixing it

MeshWeaver/samples/Graph/Data/FutuRe/{AmericasIns,EuropeRe,AsiaRe}/{LineOfBusiness,TransactionMapping} — six nodes, and this one is live, not stale:

• compilationStatus: Error, currentSourceVersions: {} — the live sources watcher measured zero sources.
• lastCompileStartedAt 2026-08-17T19:40:20, one second after requestedReleaseAt 19:40:19. A real compile ran today and really failed.
• search namespace:.../AmericasIns/LineOfBusiness/Source scope:subtree → 0; the group-level .../FutuRe/LineOfBusiness/Source → 2 Code nodes.

Each declares WithContentType<LineOfBusiness>() (+ .AddLineOfBusinessLayoutAreas()) while the types live in the group-level node's Source/.

I did not "just delete them", and the reason matters. They are not dead weight — they are the per-business-unit browse pages, and that is tested: FutuReAnalysisTest.EuropeRe_LineOfBusiness_Search_ShouldReturn8LoBs asserts FutuRe/EuropeRe/LineOfBusiness renders the NodeType search shell with HiddenQuery scoped to namespace:FutuRe/EuropeRe/LineOfBusiness, and explicitly asserts it is not the instance fallback (path:). Deleting the node removes a tested behaviour.

Nor is the obvious repair safe: pointing their sources at the group-level Source would compile the same LineOfBusiness record into four separate collectible assemblies, giving four CLR types with the same $type discriminator — the silent-empty-render trap.

So the fix is a design call I don't want to guess at:

1. drop WithContentType<…> from the regional nodes and let them be pure browse pages over instances of the group-level type (needs confirming that the Search shell doesn't require a compiled content type — it renders correctly today while the node is parked, which suggests it doesn't); or
2. keep one type and have the regional pages be instances/views of it, as every other container in the FutuRe tree already is (FutuRe/AsiaRe.json is nodeType: FutuRe/BusinessUnit, FutuRe/AmericasIns/Analysis.json is nodeType: FutuRe/LocalAnalysis — the six are the only nodes in that tree that declare a type instead of being one); or
3. a first-class "same types, one assembly, many browse scopes" mechanism, if that is what the shape actually wants.

Which of those do you want?

---

Platform gap worth its own issue

A NodeType that has never compiled successfully and settles at Error is unreachable by every automatic re-drive — so a fix to the cause can never reach the node. That is not only how the baked state persists; it is why the AnchorIncludePath fix (which was written for "memex-cloud 2026-08-12 parked 15 NodeTypes (FutuRe, Cession, SocialMedia, Northwind/AnalyticsCatalog)" — near enough the same list) left them parked after it shipped.

The framework-stale kickoff's own Where already reads CompilationStatus is Ok or Error, so covering Error is clearly intended; it cannot, because HasStaleFrameworkBuild requires assembly coordinates that ApplyCompileFailure never writes. Stamping CompiledFrameworkVersion on failure (or an equivalent "which framework produced this verdict" marker) would let a framework bump retry a failed type exactly once — which is already the treatment a stale successful build gets.

I have not written that: it changes compile-control-plane behaviour and wants its own review.

[rbuergi]

Re-measured on prod today — 13 → 6, and PR #2030 takes the rest

Verified against memex.meshweaver.cloud (serving 3.0.0-rc6.ci.4780) with GetDiagnostics per type, 2026-08-21. Seven of the thirteen are gone; six are still Error.

the 13	today	which cause (from this issue)
FutuRe/{AmericasIns,EuropeRe,AsiaRe}/LineOfBusiness	Error ❌	2 — sources not under the node's own Source/
FutuRe/{AmericasIns,EuropeRe,AsiaRe}/TransactionMapping	Error ❌	2
Doc/DataMesh/SocialMedia/{Post,Profile}	Ok ✅ (compiled 20:11:24Z)	1
SocialMedia/{Post,Profile}	Ok ✅ (20:19:40Z / 20:17:26Z)	1
Doc/Architecture/BusinessRules/Cession	Ok ✅ (20:07:50Z)	2
Northwind/{Product,AnalyticsCatalog}	node no longer exists ✅ (namespace:Northwind scope:subtree → 0)	1

Cause 3 was the interesting one, and it is answered

This issue flagged Northwind/Product / AnalyticsCatalog as "sources exist and still do not resolve — investigate first". They are gone now, and ShippedNodeTypeStateTest records what they actually were: the tree shipped 32 NodeType files exported from a running mesh, carrying that machine's compile record. Eight had "compilationStatus": "Error" baked in from 2026-06-26, and every deployment adopted the verdict verbatim — unreachable by every automatic re-drive path. That is why AnalyticsCatalog's committed error named OrderViews/SalesViews, identifiers that occur nowhere in the configuration it shipped alongside: the configuration had been corrected and the correction could never take effect.

So cause 3 was not a silent-skip sibling of #1767 at all — it was a stale imported verdict, and stripping it is what let the other seven compile for real.

What is left, and why it survived the cleanup

The six FutuRe types now fail for a real reason — the live compiler's own report, not a baked-in one:

CS0246 The type or namespace name 'LineOfBusiness' could not be found
--- Source discovery ---
  - namespace:…/AmericasIns/LineOfBusiness/Source scope:subtree nodeType:Code
Matched Code nodes (0):
  (none) — the configuration lambda cannot reference types because no source files were included.

They are copy-pastes of their parents: identical WithContentType<LineOfBusiness>() lambda, none of the parent's Source/*.cs. The parent FutuRe/LineOfBusiness compiles Ok on the same mesh because it owns its Source/.

And nothing is typed by any of the six. Every instance beneath them names the parent — 8+8 LoB instances as FutuRe/LineOfBusiness, 14+13 mappings as FutuRe/TransactionMapping, AsiaRe has none at all — and a repo-wide scan for "nodeType": "<any of the six>" returns 0 files. They were dead weight that could only ever park.

PR #2030

Retires the six as Markdown nodes (keeping the governance prose and their children), removes three source-less duplicates under samples/Graph/Data/Doc/ that can only clobber the documentation tree's good copies, and adds ShippedNodeTypeSourceResolutionTest — a shipped NodeType must not name a content type that neither its own Source/ nor the framework declares. Proven non-vacuous: reverting only the data half fails the compiled test naming all nine.

One thing this issue is right about that the guard cannot fix: the mesh keeps what the source tree drops. Merging #2030 will not clear the six parked nodes on memex-cloud — they will still be there after the sync until they are retyped or removed on the instance. That is a live-mesh write, so I have not made it; flagging it for whoever deploys.