Skip to content

grpc: an executor that honours the lease acknowledgement loses every frame #932

Description

@kartikeya-27

Found by the Go executor's end-to-end suite in #928. The client works around it;
the next executor client written against the contract will hit it too.

The asymmetry

Whether the scheduler checks an executor's frames for a lease comes off the
incoming hello (crates/flexiq-core/src/worker/remote.rs:940):

leases: capabilities.iter().any(|cap| cap == CAP_LEASE),

Whether it advertises lease back in hello_ack comes off its own state at
that instant (remote.rs:1024):

if self.lease_book().is_some() {
    capabilities.push(CAP_LEASE.to_string());
}

The book is installed when the scheduler role starts. An executor that attaches
before that gets an acknowledgement with no lease in it — and then dispatches
that do carry one, because by then the book exists.

frame_is_current reads a lease-less frame from such an executor as stale
(remote.rs:1058):

None => !executor.leases,

What that costs

An executor that follows the contract literally — "a client MUST NOT send a
frame for a behaviour it did not advertise"
, and hello_ack.capabilities is
what the scheduler will do on its behalf — sends no lease, and has every frame
about every job dropped
. Successes, failures, progress, logs. The job is never
settled and waits for the reaper.

The server's own log makes it worse by being confidently wrong:

executor go-e2e-happy sent a result for job 01a09040-… under a lease that is no
longer current; refusing it — the job was re-dispatched while that attempt was
still running

Nothing was re-dispatched. The attempt was the first and only one.

Reproducing

Start a server with FLEXIQ_GRPC_LISTEN set and attach an executor that
advertises lease in the window before the scheduler role installs its book —
the Go e2e harness hits it on most runs, because the executor attaches while the
server is still starting up. Gate the lease echo on hello_ack and every job
hangs.

Options

  1. Set Executor.leases from the acknowledgement rather than from hello — so
    the check and the advertisement cannot disagree. Narrow, and makes the
    contract sentence true as written.
  2. Install the lease book before the door accepts a stream, so the window
    does not exist. Cleaner if the ordering allows it.
  3. State the rule the other way in the contract: the lease on the dispatch is
    what puts it in force for that attempt, and an echo is never wrong. This is
    what feat: a Go executor client over the dispatch door #928's client does, and it is safe against either server behaviour — but
    it means the general "send no frame for a behaviour that was not advertised"
    rule has a documented exception, which is worth saying out loud rather than
    leaving each client to discover.

Whichever is chosen, report_stale's message should stop asserting a
re-dispatch it has not checked for. A lease-less frame from a lease-advertising
executor is a different fault from a superseded one, and reads nothing like it.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Opportunisticarea/wireNetwork contract: proto, gRPC, JSON facadebugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions