`/readyz` reports ready before flags are evaluable

[aepfli]

Summary

flagd's readiness probe returns 200 while the flag store is still empty, so an evaluation made immediately after /readyz first succeeds gets FLAG_NOT_FOUND for a flag the configuration plainly defines. Measured at 100% of starts with a realistic flag set, with a window of ~14ms (valid 30KB payload) to ~51ms (30KB payload containing entries that fail schema validation).

To be precise about what is and is not being claimed: the monitoring docs say 200 means "all sync providers at least have one successful data sync", and the code literally satisfies that — the sync provider really has fetched and delivered the data. So this is not code violating its documented contract. The claim is that those semantics are the wrong ones for a readiness probe, because "delivered" is not "applied".

Mechanism

core/pkg/sync/file/filepath_sync.go:

func (fs *Sync) Sync(ctx context.Context, dataSync chan<- sync.DataSync) error {
	...
	fs.sendDataSync(ctx, dataSync)   // only pushes onto the channel
	fs.setReady(true)                // IsReady() is now true -> /readyz 200

sendDataSync ends in dataSync <- sync.DataSync{...}, and in flagd/pkg/runtime/runtime.go that channel is

dataSync := make(chan sync.DataSync, len(r.Syncs))

buffered to exactly the number of sources. So every source can hand its payload to the buffer and call setReady(true) without a single payload having been received. Runtime.isReady() only asks each sync IsReady(), so /readyz flips to 200 at that point.

What actually makes a flag evaluable happens afterwards, on the separate goroutine draining the channel:

case data := <-dataSync:
	r.updateAndEmit(data)     // -> r.Evaluator.SetState(payload) : parse, validate, swap store

The window is therefore the cost of parsing and validating the payload, which is why it scales with flag-set size and is invisible in a trivial test fixture.

Reproduction

No provider and no test harness involved — plain flagd, one file source, tight-polling /readyz and evaluating over OFREP the instant it returns 200.

flags.json — 201 trivial, schema-valid flags (~30KB), generated with:

import json
flags = {"flag-%03d" % i: {"state": "ENABLED",
                           "variants": {"on": True, "off": False},
                           "defaultVariant": "on"} for i in range(200)}
flags["boolean-flag"] = {"state": "ENABLED",
                         "variants": {"on": True, "off": False},
                         "defaultVariant": "on"}
json.dump({"flags": flags}, open("flags.json", "w"), indent=2)

docker run -d --name flagd-repro -p 8014:8014 -p 8016:8016 \
  -v "$PWD/flags.json:/flags.json:ro" \
  ghcr.io/open-feature/flagd:v0.16.0 start --uri file:/flags.json

# poll /readyz in a tight loop until 200, then immediately:
curl -s -X POST -H 'Content-Type: application/json' -d '{}' \
  http://localhost:8016/ofrep/v1/evaluate/flags/boolean-flag
# {"key":"boolean-flag","errorCode":"FLAG_NOT_FOUND","errorDetails":"flag `boolean-flag` does not exist","metadata":{}}

The curl above has to win a ~14ms race against a process start, so it is unreliable by hand. This runs the whole thing in a loop and reports a rate:

flagd_repro.go — stdlib only, go run flagd_repro.go 15

// Minimal reproduction: flagd's /readyz reports 200 before flags are evaluable.
package main

import (
	"bytes"
	"fmt"
	"io"
	"net/http"
	"os"
	"os/exec"
	"sort"
	"strconv"
	"time"
)

const (
	image     = "ghcr.io/open-feature/flagd:v0.16.0"
	container = "flagd-readyz-repro"
	flagKey   = "boolean-flag"

	readyzURL = "http://localhost:8014/readyz"
	ofrepURL  = "http://localhost:8016/ofrep/v1/evaluate/flags/" + flagKey
)

func main() {
	runs := 20
	if len(os.Args) > 1 {
		runs, _ = strconv.Atoi(os.Args[1])
	}

	client := &http.Client{Timeout: 2 * time.Second}
	var windows []time.Duration
	raced := 0

	for i := 1; i <= runs; i++ {
		stopContainer()
		if err := startContainer(); err != nil {
			fmt.Printf("%-4d could not start flagd: %v\n", i, err)
			continue
		}

		// Tight-poll /readyz so we observe the 200 the instant it happens.
		for {
			resp, err := client.Get(readyzURL)
			if err == nil {
				resp.Body.Close()
				if resp.StatusCode == http.StatusOK {
					break
				}
			}
		}
		ready := time.Now()

		notFound := 0
		var window time.Duration
		for {
			body, err := evaluate(client)
			elapsed := time.Since(ready)
			if err == nil && !bytes.Contains(body, []byte("FLAG_NOT_FOUND")) {
				window = elapsed
				break
			}
			notFound++
			if elapsed > 10*time.Second {
				window = elapsed
				break
			}
		}

		status := "ok"
		if notFound > 0 {
			status = "RACED"
			raced++
			windows = append(windows, window)
		}
		fmt.Printf("%-4d %-6s flag_not_found_responses=%-5d evaluable_after=%v\n", i, status, notFound, window)
	}
	stopContainer()

	fmt.Printf("\nflagd %s\n", image)
	fmt.Printf("runs=%d  runs_where_/readyz_200_preceded_evaluability=%d (%.0f%%)\n",
		runs, raced, 100*float64(raced)/float64(runs))
	if len(windows) > 0 {
		sort.Slice(windows, func(a, b int) bool { return windows[a] < windows[b] })
		fmt.Printf("window: min=%v median=%v max=%v\n",
			windows[0], windows[len(windows)/2], windows[len(windows)-1])
	}
}

func evaluate(c *http.Client) ([]byte, error) {
	resp, err := c.Post(ofrepURL, "application/json", bytes.NewReader([]byte("{}")))
	if err != nil {
		return nil, err
	}
	defer resp.Body.Close()
	return io.ReadAll(resp.Body)
}

func startContainer() error {
	wd, _ := os.Getwd()
	cmd := exec.Command("docker", "run", "-d", "--name", container,
		"-p", "8014:8014", "-p", "8016:8016",
		"-v", wd+"/flags.json:/flags.json:ro",
		image, "start", "--uri", "file:/flags.json")
	out, err := cmd.CombinedOutput()
	if err != nil {
		return fmt.Errorf("%v: %s", err, out)
	}
	return nil
}

func stopContainer() {
	_ = exec.Command("docker", "rm", "-f", container).Run()
}

Results — flagd v0.16.0, OFREP on 8016, Docker on Linux

payload	runs	runs where /readyz 200 preceded evaluability	window (min / median / max)
1 flag, ~200 bytes	20	0 (0%)	not observable
201 valid flags, 30KB	15	15 (100%)	11.9ms / 13.8ms / 23.0ms
flagd-testbed's own flag set, 30KB, contains entries that fail schema validation	20	20 (100%)	43.5ms / 51.4ms / 85.5ms

The single-flag row is the control: the race is structural, but the window is sub-millisecond when there is nothing to parse — which is why a minimal fixture will not show it, and probably why this has gone unnoticed.

It is not protocol-specific. The flag evaluation port (8013) races identically; OFREP is just the easiest way to observe it.

Why it matters

Kubernetes. /readyz on the management port is the documented readiness probe. A pod therefore passes readiness and starts taking traffic before it can resolve a flag, and every request in that window gets FLAG_NOT_FOUND — which providers surface as the code default, silently. On a rollout that is a burst of wrong-but-plausible flag values rather than an error anyone notices. The window grows with flag-set size, so the larger the deployment, the worse it gets.

Conformance testing. This is how it was found. The OpenFeature provider conformance suite (open-feature/spec#417) drives flagd-testbed, whose /start waited on /readyz. Providers that block during their own initialisation absorb the window and never see it; a stateless provider such as OFREP evaluates the instant the backend reports ready, and fails essentially every scenario — which reads as a catastrophically broken provider rather than as a racing backend. Worked around on the testbed side in open-feature/flagd-testbed#394, but that fix is a workaround for this.

Suggested direction

Gate readiness on the store having been populated rather than on the payload having been handed over — e.g. have the runtime mark readiness once updateAndEmit has applied at least one payload per source, rather than deriving it from sync.IsReady() alone.

If the current semantics are intentional, then the documentation is the thing to change: it should say plainly that 200 does not mean flags are servable, and that /readyz is consequently not suitable as a Kubernetes readiness probe. I would expect most users to read the current wording as the opposite.

Version

• flagd v0.16.0 (ghcr.io/open-feature/flagd:v0.16.0, built 2026-06-01)
• Mechanism also present on main at 6fdf338