From de2dd0b58e097bae6a8c6455f170b9a3ece641a5 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Fri, 11 Sep 2026 14:31:49 +0200 Subject: [PATCH 1/2] feat: scenarios for type mismatch across the value types Nothing in the provider harness checks that asking for a flag through the wrong accessor is refused. The evaluator harness checks one row -- wrong-flag as an integer -- and that is the whole of it across both suites. A coerced value is the worst failure mode a flag has: the application receives something plausible and no signal that anything went wrong. Worth asking about directly rather than assuming, because it is not hypothetical. Requesting boolean-flag as a Float through openfeature-flagd-core returns 1.0 with reason STATIC and no error code, and it went unnoticed until a suite asked (open-feature/python-sdk-contrib#417). The same quirk in the integer direction was open-feature/python-sdk#619. Both exist because Python's bool is a subclass of int, so isinstance(True, int) is True and a naive check admits a boolean -- which is exactly the kind of thing one language gets wrong and the others do not. So the two boolean-to-number rows are the point of this, and the rest of the matrix is here because a matrix with holes in it invites the same surprise somewhere else. Numeric coercion is deliberately absent. Whether 0.5 may be narrowed to an integer is unsettled in the specification (open-feature/spec#430), and flagd's own ADR permits coercion when it is lossless, so an integer/float row would assert a rule that does not exist yet. "Is a string a boolean?" needs no such defence, which is the line the OpenFeature provider conformance suite draws in the same matrix (open-feature/spec#423). No new flags: every key used here is already in flags/ and in evaluator/flags/testkit-flags.json. The provider-harness feature is tagged @type-mismatch at the feature level so an implementation that cannot pass it yet can exclude the file and migrate, which is what the tagging scheme in the README is for. Signed-off-by: Simon Schrottner --- evaluator/gherkin/errors.feature | 30 +++++++++++++++++++++ gherkin/errors.feature | 46 ++++++++++++++++++++++++++++++++ 2 files changed, 76 insertions(+) create mode 100644 gherkin/errors.feature diff --git a/evaluator/gherkin/errors.feature b/evaluator/gherkin/errors.feature index 2ff2f54..19f55ac 100644 --- a/evaluator/gherkin/errors.feature +++ b/evaluator/gherkin/errors.feature @@ -16,3 +16,33 @@ Feature: Evaluator error handling Given a Integer-flag with key "wrong-flag" and a fallback value "13" When the flag was evaluated with details Then the error-code should be "TYPE_MISMATCH" + + @type-mismatch + Scenario Outline: Type mismatch across the value types + # Numeric coercion is deliberately absent: whether 0.5 may be narrowed to an integer is + # unsettled (https://github.com/open-feature/spec/issues/430) and flagd's own ADR permits + # coercion when it is lossless, so an integer/float row would assert a rule that does not + # exist yet. + Given a -flag with key "" and a fallback value "" + When the flag was evaluated with details + Then the resolved details value should be "" + And the error-code should be "TYPE_MISMATCH" + + Examples: a boolean flag requested as a number + # These two catch a language where the boolean type is a subtype of the integer type, as + # it is in Python: isinstance(True, int) is True, so a naive check admits a boolean and + # the caller is handed 1 or 1.0 with no error. + | key | type | default | + | boolean-flag | Integer | 1 | + | boolean-flag | Float | 0.1 | + + Examples: a string flag requested as something else + | key | type | default | + | string-flag | Boolean | false | + | string-flag | Integer | 1 | + + Examples: a numeric flag requested as a non-numeric type + | key | type | default | + | integer-flag | Boolean | false | + | integer-flag | String | fallback | + | float-flag | Boolean | false | diff --git a/gherkin/errors.feature b/gherkin/errors.feature new file mode 100644 index 0000000..3abbeb5 --- /dev/null +++ b/gherkin/errors.feature @@ -0,0 +1,46 @@ +@rpc @in-process @file @type-mismatch +Feature: flagd type mismatch handling + + # Validates that asking for a flag through the wrong accessor returns the caller's default + # with error code TYPE_MISMATCH, rather than a coerced value. A coerced value is the worst + # failure mode a flag has: the application receives something plausible and no signal that + # anything went wrong. + # + # Numeric coercion is deliberately absent. Whether 0.5 may be narrowed to an integer is + # unsettled in the specification (https://github.com/open-feature/spec/issues/430), and + # flagd's own ADR permits coercion when it is lossless -- so an integer/float row here would + # assert a rule that does not exist yet. "Is a string a boolean?" has no such defence. + # + # It's associated with the flags configured in flags. + + Scenario Outline: Requesting the wrong type returns the default + Given an option "cache" of type "CacheType" with value "disabled" + And a stable flagd provider + And a -flag with key "" and a default value "" + When the flag was evaluated with details + Then the resolved details value should be "" + And the reason should be "ERROR" + And the error-code should be "TYPE_MISMATCH" + + Examples: a string flag requested as something else + | key | type | default | + | string-flag | Boolean | false | + | string-flag | Integer | 1 | + | string-flag | Float | 0.1 | + | wrong-flag | Boolean | false | + + Examples: a boolean flag requested as something else + # The two numeric rows are the ones that catch a language where the boolean type is a + # subtype of the integer type, as it is in Python: isinstance(True, int) is True, so a + # naive check admits a boolean and the caller is handed 1 or 1.0 with no error. + | key | type | default | + | boolean-flag | String | fallback | + | boolean-flag | Integer | 1 | + | boolean-flag | Float | 0.1 | + + Examples: a numeric flag requested as a non-numeric type + | key | type | default | + | integer-flag | Boolean | false | + | integer-flag | String | fallback | + | float-flag | Boolean | false | + | float-flag | String | fallback | From d661bfdaa30788e79044526dc3a3dbde526fab2d Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sun, 13 Sep 2026 00:16:30 +0200 Subject: [PATCH 2/2] feat: complete the type mismatch matrix in the evaluator suite The evaluator suite's matrix had three holes the provider suite does not: boolean-flag as String, string-flag as Float, and float-flag as String. CodeRabbit flagged the last of those; the other two are the same oversight. "A matrix with holes in it invites the same surprise somewhere else" is the reason the outline exists at all, so both suites should ask the same questions of the same flags. They now do -- the two row sets are identical. Renames the first Examples block from "requested as a number" to "requested as something else", since it is no longer only numbers. The comment about Python's bool being a subclass of int still applies to the two numeric rows and now says so. No new flags: boolean-flag, string-flag and float-flag are all already in evaluator/flags/testkit-flags.json. Signed-off-by: Simon Schrottner --- evaluator/gherkin/errors.feature | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/evaluator/gherkin/errors.feature b/evaluator/gherkin/errors.feature index 19f55ac..c629971 100644 --- a/evaluator/gherkin/errors.feature +++ b/evaluator/gherkin/errors.feature @@ -28,21 +28,24 @@ Feature: Evaluator error handling Then the resolved details value should be "" And the error-code should be "TYPE_MISMATCH" - Examples: a boolean flag requested as a number - # These two catch a language where the boolean type is a subtype of the integer type, as - # it is in Python: isinstance(True, int) is True, so a naive check admits a boolean and - # the caller is handed 1 or 1.0 with no error. - | key | type | default | - | boolean-flag | Integer | 1 | - | boolean-flag | Float | 0.1 | + Examples: a boolean flag requested as something else + # The two numeric rows catch a language where the boolean type is a subtype of the integer + # type, as it is in Python: isinstance(True, int) is True, so a naive check admits a + # boolean and the caller is handed 1 or 1.0 with no error. + | key | type | default | + | boolean-flag | String | fallback | + | boolean-flag | Integer | 1 | + | boolean-flag | Float | 0.1 | Examples: a string flag requested as something else | key | type | default | | string-flag | Boolean | false | | string-flag | Integer | 1 | + | string-flag | Float | 0.1 | Examples: a numeric flag requested as a non-numeric type | key | type | default | | integer-flag | Boolean | false | | integer-flag | String | fallback | | float-flag | Boolean | false | + | float-flag | String | fallback |