Tier 9: stabilisation, and the 1.0 release - #83
Merged
Merged
Conversation
The three shims raised ManifestFormatException when the engine failed, whose remedy is to re-export the document. On a bundle whose manifest parsed and whose hash matched, that advice rebuilds something already correct and leaves the real cause unexamined. VoltaCast made it obvious: a provably good bundle failing at execution, and the message pointing at the export. RuntimeExecutionException names which of the three engines failed and carries the shim's status and the runtime's own error code as numbers. The shim already formatted that code into its message, so it is pulled back out rather than left inside a string a reader has to search. The shared client gains the engine's name, because what the three shims share is the C ABI rather than an implementation, and without it a failure could only say the native call failed, which is true of all three. The exhaustive switch over the sealed hierarchy caught the addition, which is what sealing it is for. Closes #79
Every model committed here returned a single tensor. The runtime, the gate and the generator all handle lists throughout, so the code that keeps a second output in the right order looked finished; it had simply never been executed. The gate's loop over the outputs had only ever taken one trip, and the emitter had never written an index other than zero. testdata/multi_io is the fixture that runs it. Two inputs, two outputs, through ExecuTorch, LiteRT and ONNX Runtime, because each shim converts its own runtime's output list into ours and one of them keeping the pair straight says nothing about the other two. Both inputs share a shape and so do both outputs. That is the whole design rather than a convenience: a pair that differed in shape would be caught on the way past by the shape check, and every assertion would then pass for a reason that has nothing to do with ordering. What is left to tell these tensors apart is the name and the numbers, which is what the generated API and the golden replay actually rely on. The gate half runs without an engine, so it sits in fluttorch_test where CI reaches it rather than only on a machine with the native libraries built. The perturbation there moves the second output alone and asserts the report names it and not the first. VoltaCast already covered two inputs and is left alone: its model directory is vendored verbatim from upstream, and a second head there would make that sentence false.
Above a size torch.onnx decides on its own, the weights leave the graph and land beside it. The export refused that, for two stated reasons: the weight hash covered the artifact, so a bundle could pass every check while carrying none of the numbers, and the runtime takes bytes and had no way to reach a file next to them. The first reason is answered here. A manifest can name the parts an artifact references and cannot be loaded without, and weight_hash is computed over the artifact and every part together, each contributing its name and its length as well as its bytes. Concatenation alone would be ambiguous, and a hash whose meaning depends on where you cut it commits to nothing. With no parts the digest is the artifact's and nothing else, byte for byte what every manifest already declares, so this re-exports nothing. The schema version rises to 2, and only for a manifest that actually carries parts. Every field added to this schema so far has been additive, meaning a reader that did not know it carried on doing what it did before. This one is not: the decoder ignores keys it does not recognise, by design, so a reader without parts would load the structure alone and answer nonsense from a graph with no numbers in it. The version is the only thing that can stop it. The second reason, the loader, is the next commit.
The Dart side of what the exporter started writing. A manifest can carry parts, verifyArtifact takes them, and the digest is computed over the artifact and each part with its name and its length framed in, matching bundle_digest byte for byte. VoltaCast through torch.onnx is the export that produces one, and it is committed: 506 kB of graph and 3.4 MB of weights in a file beside it. That makes the cross-language check a real one rather than a fixture written to agree with itself. If either side ever frames a length differently, the bundle stops loading, and the test says which side moved. currentSchemaVersion rises to 2 because this build can now read parts. An export without them still declares 1, so nothing that already shipped stops being loadable, and a build from before this refuses a bundle it would otherwise load the structure of and answer from. Three failures rather than one, because they have three fixes. A part that did not arrive is a deployment that shipped the artifact alone. A part whose length is short is a copy that got cut. A part whose hash moved is a stale export. Folding them together would send every one of them to re-export a model that was fine. The loader is still to come: nothing yet hands these bytes to an engine.
ft_load_parts takes the artifact and the files it references, and the header declares it once for all three bindings because they implement one seam and a second signature would be a second seam. ONNX Runtime resolves external data against the directory a model was read from, and a buffer has no directory. AddExternalInitializersFromFilesInMemory registers the names against the bytes instead, so nothing is looked up on disk and the session never needs a path. That is what the issue thought was closed, and it is open. The other two refuse. A .pte and a .tflite each carry their own weights, so a bundle arriving at either with parts was exported for a different engine, and the refusal says which. ExecuTorch does have a shape for this in its .ptd data file and nothing here writes one yet; when something does, this is where it gets answered rather than where it gets discovered. Refusing is the point. Loading the graph and leaving the weights behind gives a session that parses, declares every shape the manifest promised, runs, and answers from nothing. ft_load stays, delegating with no parts, so a caller that does not know which shape it holds does not have to ask.
The last stretch of it: the Dart side hands the parts to the binding, the gate reads them from beside the manifest, and the model that made this necessary runs. VoltaCast through ONNX Runtime is 506 kB of graph and 3.4 MB of weights in a file next to it. It loads, and its goldens hold against references captured from the source model before any of this was split apart. That last clause is the one that matters: loading was never the hard part, because a graph without its weights loads too, declares the same shapes, and answers. The FFI keeps one loader rather than two. ft_load_parts delegates when there are none, so binding both would have been two paths to drift apart, and the one that stayed would have been the one nothing exercised. A binding that cannot resolve external data names that as what it lacks, not "this load". Two absences that read the same in a log are two absences nobody can tell apart at three in the morning.
Every bound in the gate was a starting point, written before any model had been measured and documented as such. tool/measure_tolerances.dart now produces the numbers, and every entry cites what it was measured on. The matrix runs on a model that can go wrong. Two linear layers, 4 to 8 to 3, has nowhere for two delegates to disagree, so the old matrix came out ordered the way arithmetic says it should and proved the measurement worked rather than that it was worth taking. testdata/matrix has convolutions, a foldable BatchNorm2d, a GroupNorm that reduces at run time and a softmax, exported once per backend and once per recipe. The measurement said something I did not expect. The two-layer model drifts further than the convolutional one on every backend, by up to eight times, and it is the simpler network: its outputs land near 9.4 while the other ends in a softmax that pins its own into [0, 1], and a relative error measured against a small output reads smaller for the same rounding. So the bounds are set by the model with the large outputs. Narrowing float16 to what testdata/matrix alone justifies was tried, at 5e-3, and it failed testdata/coreml on the first run. A bound derived from the better-behaved of two committed fixtures is not a measured bound, it is a measurement of which fixture was picked. int8-dynamic widens rather than narrows. It measures 4.1e-2 against a bound of 5e-2, and 1.2 times is a coincidence rather than a margin. startingPointFor becomes boundFor. The old name was honest about what those numbers were and would have stopped being honest here.
Four surfaces here carry a compatibility promise and only one of them is a Dart API, which is why SemVer on the package version answers none of them on its own. The manifest is a document two implementations parse in two languages, and schema_version rather than the package version decides whether an older reader may proceed. The C header is an ABI three bindings implement and a consumer may link from a prebuilt library, where struct field order is part of the contract in a way that does not fail to compile. The generated Dart is code somebody else commits, so a change to what the emitter writes shows up as a diff in their repository with nothing of theirs changed. And the tolerances decide whether a build is green, so narrowing one turns a passing build red without a line of caller code moving. The numbers a model produces are not covered, and that is the answer most likely to be wanted. The arithmetic belongs to the delegate that runs it. Two minor releases before a deprecated API is removed, because that is ExecuTorch's policy and a binding cannot outlive a symbol the engine below it has removed. Promising longer would be promising something this project does not control.
Three costs, because three different claims get made about this library and each has its own way of being wrong. Codegen is paid once per manifest at build time, load once per model per process, and run on every inference. Codegen is about a millisecond and flat across models that differ by three orders of magnitude in weight size, which is expected rather than impressive: the emitter reads specs and writes text and never touches the artifact. What it supports is only that the emitter is not why a build is slow. Load tracks artifact size at roughly 6.5 ms per megabyte. VoltaCast costs 22 ms, once per process rather than once per screen, which is worth knowing before deciding whether to load a model at startup or on first use. The run numbers invert the advice you might expect. runInto saves four to seven microseconds over run, and the saving does not scale with the model, so it is three times faster on the two-layer model and under two per cent on the convolutional one. Reach for it on something small called at frame rate. Resident memory was measured and is not published. Dart exposes no allocation count, and the RSS delta over a thousand runs came back negative as often as positive: it records when the collector ran, not what the loop allocated. A column of noise in a published table is worse than a missing one, because a reader cannot tell which they are looking at. The README status block said the tolerances were unmeasured and the bindings executed nothing. Both had stopped being true.
The API is frozen and STABILITY.md says what that covers. Four surfaces here carry a compatibility promise and only one of them is a Dart API, so each has its own rule: the manifest is a document two implementations parse, the C header is an ABI three bindings implement, the generated Dart is code somebody else commits, and the tolerances decide whether a build is green. The tolerances are measured against two models rather than started from, and what they measured changed the answer. The two-layer model drifts up to eight times further than the convolutional one and it is the simpler network, so the bounds are set by the model with the large outputs rather than by the better-behaved fixture. The rest of the tier: an artifact can be more than one file and the hash still covers the numbers, two inputs and two outputs are kept apart by every engine, the parity matrix runs on a model that can go wrong, and what a run costs is published with the tool that reproduces it. One release rather than two. The board carried the work at 0.9.0 and the release item at 1.0.0, and the tier ships as a single tag.
boundFor left a caller behind in the Flutter spike, which is the one place plain Dart tooling cannot parse, so the local analyze that covers packages/ could not see it. CI could. The rename was scoped to packages/ and the repo is wider than that. The stale references were not only the identifier. docs/tolerance.md still carried a section headed "Starting points, not measurements" pointing at issue 60 as open, and docs/ci-parity-gate.md still told a reader that every number in the table should be replaced by one they measured themselves. Both had become false in the commit that measured them, and a document promising the opposite of what the code does is worse than one that says nothing. What replaces them says which two models the bounds came from and why they disagree, because that difference is the useful part: the simpler model drifts further, and a reader who does not know that will measure the wrong one and trust the answer. The exception is named rather than glossed. int4-weight-only is still a starting point because nothing here exports int4.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #79, #62, #65, #61, #60, #33, #34, #35.
The tier that turns the gate from something that works into something whose
numbers can be argued with.
What the measurements changed
Two of these items were expected to confirm what was already there and did not.
int8-dynamicwas measured at4.1e-2against a bound of5e-2. That is not amargin, it is a coincidence, and a build on other hardware had every chance of
tipping a committed fixture over it. The bound widens rather than narrows.
Narrowing
float16to what the convolutional model alone would justify wastried, at
5e-3, and it failed the two-layer fixture on the first run. Thetwo-layer model drifts up to eight times further on every backend and it is the
simpler network: its outputs land near
9.4while the other ends in a softmaxthat pins its own into
[0, 1], and a relative error is measured against theoutput while the rounding happened on intermediates. The bounds are set by the
model with the large outputs.
tool/measure_tolerances.dartreproduces all of it.What was covered rather than fixed
Every model committed here returned a single tensor, so the code keeping a
second one in the right order had been read and never executed.
testdata/multi_ioruns it through all three engines, because each shim converts its own runtime's
output list into ours. Its two inputs share a shape and so do its two outputs,
which removes the shape check as a safety net and leaves ordering as the only
thing that can be right.
What the issue got wrong
#65 said ONNX Runtime could not reach a weight file beside the graph because
ft_loadtakes a buffer and external data resolves against a path.AddExternalInitializersFromFilesInMemoryregisters the names against the bytes,so nothing is looked up on disk. An artifact can now be more than one file: the
manifest names the parts,
weight_hashcovers them, andschema_versionrisesto 2 only for a manifest that carries any, because a reader without them would
load 506 kB of structure and answer from a graph with no numbers in it.
VoltaCast through ONNX Runtime is the proof. It loads with its weights in a
separate file and its goldens hold.
The rest
normalisation instead of two linear layers that had nowhere to disagree.
STABILITY.md: four surfaces carry a compatibility promise and only one is aDart API.
docs/benchmarks.md:runIntosaves four to seven microseconds overrunand the saving does not scale with the model, which inverts the advice you
might expect.
Verification
dart analyze --fatal-warningsclean, all six Dart suites and the typed APIexample green, 118 Python tests,
ruffclean,audit-library.shat standard.The three native shims were rebuilt and the on-device suites re-run against
them.
The tag and the pub.dev publish are not in this branch.