Skip to content
16 changes: 16 additions & 0 deletions corpus/accepted.json
Original file line number Diff line number Diff line change
Expand Up @@ -293,6 +293,22 @@
"reason": "The cloud refuses DeleteSecurityGroup with 409 ResourceConflict for about ninety seconds after the machine that wore the group is terminated, and accepts on the thirty-first attempt; this emulator releases the group the moment the machine is terminated and answers 200 at once. A REPLAY CANNOT GRADE THE DIFFERENCE EITHER WAY: the recording is thirty refusals then an acceptance, and a corpus has no ninety seconds in it, since the sanitiser normalises every timestamp to one second apart. Reproducing the delay would make every stack teardown and every conformance run wait it out, which is a product decision rather than a defect to patch. #380 carries it, and internal/providers/outscale/securitygroups.go records the measurement where the skip is.",
"issue": "https://github.com/stephrobert/feint/issues/380"
},
{
"file": "outscale/oapi-cli-lifecycle.jsonl",
"operation": "osc/Client.ReadKeypairs",
"kind": "status",
"path": "",
"reason": "NOT A DIVERGENCE OF THE EMULATOR, and the recorded REQUEST is what is wrong. oapi-cli sent Filters.KeypairNames as the array FiltersKeypair declares (contracts/outscale.json); the proxy's redaction replaced the whole value with one string, because KeypairNames matches the \"key\" carrier \u2014 a price internal/proxy/redact.go names as paid knowingly. So the corpus holds {\"Filters\":{\"KeypairNames\":\"REDACTED-17\"}} and the replay reissues it verbatim. Nothing could see it: until 2026-08-28 the Outscale pack read an undecodable filter as an absent one and answered 200 with the whole inventory, so two silent defects cancelled out and this gate passed by accident. #566's type gate refuses the value with a 400, and the absent Keypairs finding on the same exchange is that 400's body. redactValue now keeps a list of scalars a list (TestARedactedListOfScalarsStaysAList), so a recording made after that change carries the array and needs neither entry \u2014 these two cover the two lines recorded before it, and the staleness rule deletes them the day this corpus is recorded again.",
"issue": "https://github.com/stephrobert/feint/issues/566"
},
{
"file": "outscale/oapi-cli-lifecycle.jsonl",
"operation": "osc/Client.ReadKeypairs",
"kind": "absent",
"path": "Keypairs",
"reason": "NOT A DIVERGENCE OF THE EMULATOR, and the recorded REQUEST is what is wrong. oapi-cli sent Filters.KeypairNames as the array FiltersKeypair declares (contracts/outscale.json); the proxy's redaction replaced the whole value with one string, because KeypairNames matches the \"key\" carrier \u2014 a price internal/proxy/redact.go names as paid knowingly. So the corpus holds {\"Filters\":{\"KeypairNames\":\"REDACTED-17\"}} and the replay reissues it verbatim. Nothing could see it: until 2026-08-28 the Outscale pack read an undecodable filter as an absent one and answered 200 with the whole inventory, so two silent defects cancelled out and this gate passed by accident. #566's type gate refuses the value with a 400, and the absent Keypairs finding on the same exchange is that 400's body. redactValue now keeps a list of scalars a list (TestARedactedListOfScalarsStaysAList), so a recording made after that change carries the array and needs neither entry \u2014 these two cover the two lines recorded before it, and the staleness rule deletes them the day this corpus is recorded again.",
"issue": "https://github.com/stephrobert/feint/issues/566"
},
{
"file": "outscale/oapi-cli-lifecycle.jsonl",
"operation": "osc/Client.ReadLoadBalancers",
Expand Down
45 changes: 45 additions & 0 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -152,6 +152,51 @@ without the second, "not done yet" and "out of scope" become the same thing.
A change that makes any of the three inoperative is a bad change, even if it
simplifies the code.

### A decline has to be legible on the wire, not only in `coverage/`

`Declined()` and the coverage artefacts are honest about what a pack does not
serve, and they are read by nobody at run time. A **client** meets the refusal
instead, and it must be able to tell "feint does not serve this operation" from
"the cloud has nothing for you" — otherwise a decline reads as an empty answer
and a green run proves something it did not (#477: three refusals of
`GET /v2/reverse-dns/elastic-ip/{id}` inside a stack that applied seven
resources, planned empty and destroyed seven, with nothing anywhere saying so).

Each pack answers in its own dialect, and each spelling was decided by what its
clients do rather than by a house style:

| pack | status | body | why that one |
|---|---|---|---|
| Scaleway | `501` | `{"type":"not_emulated", "message": …}` | a type their SDK does not map, so `errors.As(&ResourceNotFoundError{})` cannot agree that a resource is missing; their SDK has no retry policy, so the status costs nothing |
| Outscale | `404` | `{"Errors":[{"Code":"", "Type":"OperationNotEmulated", …}]}` | their envelope has a `Type` to put the marker in, and `501` costs `oapi-cli` 12 seconds of backed-off retries — measured |
| Exoscale | `404` | `{"message": …}` | their envelope has nowhere to put a marker, and `501` **fails `exo compute instance create`** — measured |

**No status and no body carries the marker for every operation, and that is a
measurement rather than a preference.** Exoscale is the case that proves it:
`exo compute instance create` calls `GET /v2/reverse-dns/instance/{id}` after
every create and treats anything but a `404` as fatal, so the status a program
could branch on is the status that breaks the client. Their error envelope
requires `message` and declares no code field, so a marker in the body would be
an invented format (rule 4). Both doors are shut, and `404` on a declined
operation is then the same bytes as "this elastic IP has no reverse record".

So the marker is **out of band**, set once in the shared layer for every pack:

```text
X-Feint-Not-Emulated: exoscale
```

`emulator.handleUnrouted` sets it on every unrouted refusal, with the name of
the pack that owns the URL space — read from this process's mount table, never
from the path the client sent. It sits beside `X-Feint-Fault` and
`X-Feint-Probe`, headers no real cloud sends, and no client can trip over it.

Two things it deliberately does not do. It does not mark a `404` that is an
ordinary missing object — only a refusal — or the marker would mean nothing.
And it does not tell an **operator**: a header is invisible to Terraform and to
a human reading an apply, so a run can still be green while three declined
operations were called (#477's remaining half).

Those three are the first links of a longer chain — the contract, the two
witnesses that drive it, the recordings that look in the omission direction, the
seven-axis evidence record, and the versioned surface a pipeline reads.
Expand Down
4 changes: 2 additions & 2 deletions docs/limits-acks.json
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
"A Scaleway server's root volume type: what is writable, and what is not": "2026-08-27",
"A VPC created without `enable_routing` answered `routing_enabled=false` where the real cloud answers `true` (#497, lifted 2026-08-27)": "2026-08-27",
"A declared query parameter is served or refused, never dropped — and `labels` is the refused one": "2026-08-27",
"A machine's route out: which shapes reach a package repository (#507)": "2026-08-27",
"A machine's route out: which shapes reach a package repository (#507)": "2026-08-28",
"A public address is the provider's value, made to answer on the host": "2026-08-27",
"A run presented as local can still reach the real cloud (#280)": "2026-08-27",
"An API reboot used to log `Failed to add route: file exists` for its own public /32 (#498, lifted 2026-08-27)": "2026-08-27",
Expand All @@ -15,7 +15,7 @@
"An Outscale load balancer distributes packets inside its network, and nowhere else": "2026-08-27",
"An Outscale machine owns a root volume, and that volume holds no bytes": "2026-08-27",
"Exoscale has one zone per process, and the reason is the client": "2026-08-27",
"Fourteen ERROR lines over fifteen stack replays are one documented refusal, logged at the wrong level (#474)": "2026-08-27",
"An ERROR is a failure and a WARN is a decline, and one refusal was on the wrong side (#474)": "2026-08-28",
"Identifiers are not checked against anything": "2026-08-27",
"Lifecycle transitions are immediate": "2026-08-27",
"Managed Kubernetes is not emulated, and a CRUD-only version is refused (#283)": "2026-08-27",
Expand Down
103 changes: 84 additions & 19 deletions docs/limits.md
Original file line number Diff line number Diff line change
Expand Up @@ -3570,7 +3570,32 @@ replayed green for this record on 2026-08-27 — and
`tools/falsify/specs/lifecycle-tells-the-truth.json` replays them with the
guard neutralised in both directions, the refusal and the acceptance.

## Fourteen ERROR lines over fifteen stack replays are one documented refusal, logged at the wrong level (#474)
## An ERROR is a failure and a WARN is a decline, and one refusal was on the wrong side (#474)

**Fixed on 2026-08-28.** The refusal below is logged at WARN, and the rule the
measurement suggested is now written where the refusal is
(`Binding.refuseUnknownImage`) and held by
`TestADocumentedRefusalIsAWarningAndAFailureStaysAnError`:

> An **ERROR** is something this emulator did not do that it was built to do.
> A **WARN** is something it deliberately declines and documents, where the API
> answer stays honest.

The half the issue had not measured is measured now: of the **48 ERROR sites
under `internal/` on 2026-08-28**, that one call was the only one on the wrong
side of the line. Its own neighbours are failures and stay ERROR — a start the
driver refused, an image build that could not fetch its source, a pack that
declares no interface plan (`plan.go` says why that one is not a decline). The
test asserts both directions for exactly that reason: a change that made every
refusal a warning would pass its first half and quiet the lines the log exists
for. Nothing else moved — the boot is still refused, the machine still reads
back its `FailedState`, and the refusal still names the identifier, the reason,
the consequence and both gestures.

What follows is the measurement that established it, kept because the reasoning
is the reusable part.

### The measurement, 2026-08-25

Replaying the fifteen surveyed stacks under a machine runtime (`main@72d861d`,
`--vm incus-ovn`, logs of 2026-08-25), five runs printed `level=ERROR` —
Expand All @@ -3592,14 +3617,11 @@ it is the one that follows the rule. The run that printed five of these ERRORs
was a **success**: ztiac applied 54 of 54, matched its reference exactly, and
destroyed 54 cleanly.

What to do with it: do not grade an emulator log by `grep ERROR` alone — under
these replays it finds fourteen lines about a documented behaviour and nothing
about the run that really failed. What would lift it: logging this refusal at
WARN, where its balancer sibling already is; the distinction, if a rule is
wanted, is that an ERROR is something the emulator did not do that it was
built to do, and a WARN is something it deliberately declines and documents
while the API answer stays honest. Not measured: whether any other ERROR site
is in the same position — the fifteen replays surfaced only this one.
The lesson that outlives the fix: do not grade an emulator log by `grep ERROR`
alone, and do not level a limit like an incident. Under those replays the grep
found fourteen lines about a documented behaviour and nothing about the run
that really failed, which is precisely how a team learns to skip a log's
errors.

## `feint images resolve` can print a `FEINT_BOOT_IMAGES` line that cannot boot (#476)

Expand Down Expand Up @@ -3662,16 +3684,59 @@ recovered, printed, and the process exits 0.

Measured: the exact command, rc=0, the ACL deleted, the stack, the
reproduction under `--vm off` as under `--vm incus-ovn`, and no ERROR on the
emulator's side. Deduced, not measured on a real account: the real cloud would
provoke the same panic — nothing in the faulty path depends on the emulator's
answer.

What to do with it: nothing, here. This is not a divergence and there is
nothing to fix in the emulator; the line is one of noise in every conformance
log, tolerated because delete stderr passes through and rc is 0. Whoever meets
it in a log: #505 is the reference to point at. What would lift it: the
upstream fix in scaleway-cli — one type in the assertion, to be reported
through `scw feedback bug`; the line disappears when a fixed `scw` ships.
emulator's side.

### The one question worth asking, and its answer: no

**Could this emulator answer something else, without lying, that does not
trigger the fault?** That is the only version of #505 that would be a fix
rather than a note, so it was answered by reading the two functions and then by
experiment, on 2026-08-28.

Read (`scaleway-cli/v2@v2.56.3` and `scaleway-sdk-go` in the module cache):

- `ZonedAPI.DeleteACL` ends on `s.client.Do(scwReq, nil, opts...)` — the
response is decoded into `nil`. **No body this emulator sends reaches the
faulty path**, so the shape of the answer is not a lever at all.
- `lbACLDelete`'s `Run` returns `&core.SuccessResult{Resource: "acl", Verb:
"delete"}` unconditionally whenever that call returns a nil error. So *any*
truthful success produces the value the interceptor then dereferences.
- The interceptor is installed on all four ACL verbs, and its pre-fetch is
guarded by `argsI.(*lb.ZonedAPIDeleteCertificateRequest)` — never true here,
so `getACL` is nil on every path.

Measured, with the emulator's own fault injection, which is what makes this an
experiment rather than a second reading:

| what the emulator answers | rc | panic on stderr | the ACL |
|---|---|---|---|
| 204, as it does (`DeleteACL` succeeds) | 0 | yes | deleted |
| 500, via `PUT /_feint/faults` | 1 | **no** | **survives** |

And the three sibling verbs — `acl get`, `acl update`, `acl create` — carry the
same interceptor, exit 0 with empty stderr, and never panic: they answer an
`*lb.ACL` rather than a `*core.SuccessResult`, so they never reach the branch.
That locates the fault exactly at "the runner returned a success", and nowhere
near the emulator.

So the emulator's only lever is to **fail a delete that worked**, which loses
the resource and lies about it — the one thing this project exists not to do.
Option (1) is disproved, and #505 closes as documentation rather than as a fix.

What to do with it: nothing, here. This is not a divergence; the line is noise
in every conformance log, tolerated because delete stderr passes through and rc
is 0. Not filtered either, which would be a workaround hiding a real upstream
defect: it is tolerated *and* the suite now asserts what makes tolerating it
honest — `scw-cli.sh` reads the ACL list back after the delete, so "rc=0 with a
panic on stderr" is measured rather than trusted. Whoever meets the line in a
log: #505 is the reference to point at. What would lift it: the upstream fix in
scaleway-cli — one type in the assertion, to be reported through
`scw feedback bug`; the line disappears when a fixed `scw` ships.

Still not measured, and it does not change the verdict: what a real account
answers. Nothing in the faulty path reads a response, so the same panic is
expected there — but that sentence is a deduction from the source above, not a
measurement, and it is written here as one.

## The Exoscale stack's second plan is not empty: two per-id outputs read back null at apply time (#520)

Expand Down
49 changes: 48 additions & 1 deletion internal/core/emulator/emulator.go
Original file line number Diff line number Diff line change
Expand Up @@ -519,6 +519,16 @@ func checkSpaces(packs []Pack) error {
return nil
}

// NotEmulatedHeader marks an answer that is this emulator refusing an operation
// rather than a cloud answering, and names the pack whose URL space the request
// landed in. No real cloud sends it — the same trade FaultHeader documents.
//
// It exists because a status cannot carry that fact for every operation: for
// one whose real 404 means "this object has no such record", a louder refusal
// fails a client the real cloud would have served. See handleUnrouted for the
// measurement (#477).
const NotEmulatedHeader = "X-Feint-Not-Emulated"

// handleUnrouted answers a request no route claimed, in the dialect of whichever
// pack owns the URL space it landed in.
//
Expand All @@ -527,17 +537,54 @@ func checkSpaces(packs []Pack) error {
// is right for a request that belongs to no provider at all.
func (s *Server) handleUnrouted(w http.ResponseWriter, r *http.Request) {
best, bestLen := (Unrouted)(nil), -1
bestName := ""
for _, p := range s.packs {
unrouted, ok := p.(Unrouted)
if !ok {
continue
}
for _, prefix := range unrouted.Prefixes() {
if len(prefix) > bestLen && strings.HasPrefix(r.URL.Path, prefix) {
best, bestLen = unrouted, len(prefix)
best, bestLen, bestName = unrouted, len(prefix), p.Name()
}
}
}
// The marker, out of band, and here rather than in each pack (#477).
//
// A refusal a program cannot tell from an empty answer is a refusal that
// reads as success. Replaying the register's best Exoscale stack — seven
// resources applied, empty second plan, seven destroyed — the recorder
// showed three refusals of an operation that pack declines, and nothing
// anywhere said so: a bare 404 is also what the cloud answers for an
// elastic IP with no reverse record, so the refusal and the ordinary empty
// answer were the same bytes.
//
// It could not be fixed in the status, and that was measured rather than
// argued. Answering 501, as the Scaleway pack does for its own space, is
// legible and costs exo 1.95.1 nothing in latency (22, 21 and 19 ms against
// the 19 ms of a served route) — and it breaks `exo compute instance
// create`, which calls GET /v2/reverse-dns/instance/{id} after every create
// and treats anything but a 404 as fatal. Measured on 2026-08-28: the
// exo-cli leg failed at "instance create rejected" with 501 and passes with
// 404. That is the polar star inverted — a refusal loud enough to fail a
// client the real cloud would have served — and it is why the status of an
// operation whose real 404 means "nothing here" cannot carry the marker.
//
// So the marker goes where no client can trip over it, and where every pack
// gets it at once: a header in this emulator's own namespace, beside
// X-Feint-Fault and X-Feint-Probe, which faults.go already documents as
// headers no real cloud sends. Its value is the pack that owns the URL
// space, read out of this process's own mount table — never the path the
// client chose, for the reason the log below gives.
//
// What it does NOT do, and what the packs' own bodies still have to: tell an
// operator. A header is invisible to Terraform and to a human reading an
// apply. That half is #477's open remainder.
//
// TestAnUnroutedAnswerCarriesTheNotEmulatedHeader fails without this.
if bestName != "" {
w.Header().Set(NotEmulatedHeader, bestName)
}
// Logged like any other request, and this is the line the log exists for:
// a client walking a plan meets one route nobody mounted, the whole apply
// dies, and no counter anywhere records which one it was. The emulator's own
Expand Down
Loading
Loading