You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Found by the Go executor's end-to-end suite in #928. The client works around it;
the next executor client written against the contract will hit it too.
The asymmetry
Whether the scheduler checks an executor's frames for a lease comes off the
incoming hello (crates/flexiq-core/src/worker/remote.rs:940):
leases: capabilities.iter().any(|cap| cap == CAP_LEASE),
Whether it advertiseslease back in hello_ack comes off its own state at
that instant (remote.rs:1024):
The book is installed when the scheduler role starts. An executor that attaches
before that gets an acknowledgement with no lease in it — and then dispatches
that do carry one, because by then the book exists.
frame_is_current reads a lease-less frame from such an executor as stale
(remote.rs:1058):
None => !executor.leases,
What that costs
An executor that follows the contract literally — "a client MUST NOT send a
frame for a behaviour it did not advertise", and hello_ack.capabilities is
what the scheduler will do on its behalf — sends no lease, and has every frame
about every job dropped. Successes, failures, progress, logs. The job is never
settled and waits for the reaper.
The server's own log makes it worse by being confidently wrong:
executor go-e2e-happy sent a result for job 01a09040-… under a lease that is no
longer current; refusing it — the job was re-dispatched while that attempt was
still running
Nothing was re-dispatched. The attempt was the first and only one.
Reproducing
Start a server with FLEXIQ_GRPC_LISTEN set and attach an executor that
advertises lease in the window before the scheduler role installs its book —
the Go e2e harness hits it on most runs, because the executor attaches while the
server is still starting up. Gate the lease echo on hello_ack and every job
hangs.
Options
Set Executor.leases from the acknowledgement rather than from hello — so
the check and the advertisement cannot disagree. Narrow, and makes the
contract sentence true as written.
Install the lease book before the door accepts a stream, so the window
does not exist. Cleaner if the ordering allows it.
State the rule the other way in the contract: the lease on the dispatch is
what puts it in force for that attempt, and an echo is never wrong. This is
what feat: a Go executor client over the dispatch door #928's client does, and it is safe against either server behaviour — but
it means the general "send no frame for a behaviour that was not advertised"
rule has a documented exception, which is worth saying out loud rather than
leaving each client to discover.
Whichever is chosen, report_stale's message should stop asserting a
re-dispatch it has not checked for. A lease-less frame from a lease-advertising
executor is a different fault from a superseded one, and reads nothing like it.
Found by the Go executor's end-to-end suite in #928. The client works around it;
the next executor client written against the contract will hit it too.
The asymmetry
Whether the scheduler checks an executor's frames for a lease comes off the
incoming
hello(crates/flexiq-core/src/worker/remote.rs:940):Whether it advertises
leaseback inhello_ackcomes off its own state atthat instant (
remote.rs:1024):The book is installed when the scheduler role starts. An executor that attaches
before that gets an acknowledgement with no
leasein it — and then dispatchesthat do carry one, because by then the book exists.
frame_is_currentreads a lease-less frame from such an executor as stale(
remote.rs:1058):What that costs
An executor that follows the contract literally — "a client MUST NOT send a
frame for a behaviour it did not advertise", and
hello_ack.capabilitiesiswhat the scheduler will do on its behalf — sends no lease, and has every frame
about every job dropped. Successes, failures, progress, logs. The job is never
settled and waits for the reaper.
The server's own log makes it worse by being confidently wrong:
Nothing was re-dispatched. The attempt was the first and only one.
Reproducing
Start a server with
FLEXIQ_GRPC_LISTENset and attach an executor thatadvertises
leasein the window before the scheduler role installs its book —the Go e2e harness hits it on most runs, because the executor attaches while the
server is still starting up. Gate the lease echo on
hello_ackand every jobhangs.
Options
Executor.leasesfrom the acknowledgement rather than from hello — sothe check and the advertisement cannot disagree. Narrow, and makes the
contract sentence true as written.
does not exist. Cleaner if the ordering allows it.
what puts it in force for that attempt, and an echo is never wrong. This is
what feat: a Go executor client over the dispatch door #928's client does, and it is safe against either server behaviour — but
it means the general "send no frame for a behaviour that was not advertised"
rule has a documented exception, which is worth saying out loud rather than
leaving each client to discover.
Whichever is chosen,
report_stale's message should stop asserting are-dispatch it has not checked for. A lease-less frame from a lease-advertising
executor is a different fault from a superseded one, and reads nothing like it.