Support Microsoft Testing Platform simple test filters + wildcards - #145
Open
mattleibow wants to merge 3 commits into
Open
Support Microsoft Testing Platform simple test filters + wildcards#145mattleibow wants to merge 3 commits into
mattleibow wants to merge 3 commits into
Conversation
Add the eight xUnit v3 Microsoft Testing Platform 'simple' filter switches (--filter-class/-not-class, --filter-method/-not-method, --filter-namespace/-not-namespace, --filter-trait/-not-trait) to the device-runners CLI test command, translating them into the existing --filter expression that the on-device TestCaseFilter evaluates. Also add '*' wildcard support to the equals family (= / !=) of the on-device evaluator, a superset of the VSTest grammar (no-wildcard filters are unchanged), so the simple filters map faithfully and 'dotnet test --filter "ClassName=Calc*"' now works too. - TestCaseFilter: '*' glob via anchored regex for = / != conditions - MtpFilterTranslator: maps the 8 switches into a combined --filter expression (same-kind OR, cross-kind AND, not- exclusions, structural-char escaping) - BaseTestCommandSettings: 8 repeatable options, mutual-exclusion validation, GetEffectiveFilter() feeding DEVICE_RUNNERS_FILTER (env/arg/WASM query string) - Tests for the translator, env-var wiring, validation, and wildcard matching - Docs: CLI filtering section, dotnet test wildcard note, xUnit v3 MTP parity and a --filter-query explainer (explained, not implemented) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
From GPT-5.5 and Opus 4.8 reviews of #145: - TestCaseFilter: build the wildcard regex with RegexOptions.NonBacktracking so a pathological filter (many '*' segments) cannot trigger catastrophic backtracking; matching is now guaranteed linear-time (GPT-5.5, medium). - Windows loose-MSIX launch path: forward GetEffectiveFilter(settings) instead of the raw settings.Filter, so --filter-class/--filter-* are no longer silently dropped on that path (GPT-5.5, medium). - MtpFilterTranslator.ValidateTraits + Validate(): reject malformed trait filters (missing '=' or empty name) instead of silently reinterpreting a nameless trait like '=Fast' as a FullyQualifiedName filter (Opus 4.8, low). - Tests: multi-wildcard and no-match-many-segment matching, trait-format validation (well-formed, missing separator, empty name, non-trait), and a malformed-trait Validate() rejection. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Adds Microsoft Testing Platform (xUnit v3) “simple” filter switches to the DeviceRunners CLI by translating them into the existing --filter expression, and extends the on-device TestCaseFilter evaluator to support * wildcards for =/!= comparisons (with docs + tests).
Changes:
- Implemented
MtpFilterTranslator+ CLI settings/options/validation to support the 8 MTP simple filter switches (including mutual exclusivity with raw--filter). - Added
*wildcard support toTestCaseFilterfor=/!=via anchored, case-insensitive regex matching. - Added/updated unit tests and documentation to cover the new filter switches and wildcard behavior.
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| test/DeviceRunners.VisualRunners.Tests/Testing/TestCaseFilterTests.cs | Adds wildcard-matching test coverage for TestCaseFilter, including traits and negation. |
| test/DeviceRunners.Cli.Tests/Commands/MtpFilterTranslatorTests.cs | New unit tests validating translation semantics (OR/AND rules, negation, escaping, traits). |
| test/DeviceRunners.Cli.Tests/Commands/AppEnvironmentVariableTests.cs | Extends CLI tests to ensure simple filters translate into DEVICE_RUNNERS_FILTER and validate exclusivity. |
| src/DeviceRunners.VisualRunners/Filtering/TestCaseFilter.cs | Implements * wildcard support for equals/not-equals evaluation using RegexOptions.NonBacktracking. |
| src/DeviceRunners.Cli/Commands/Windows/TestCommand.cs | Uses the resolved “effective filter” (raw or translated) when launching Windows apps. |
| src/DeviceRunners.Cli/Commands/Wasm/WasmTestCommand.cs | Uses the resolved “effective filter” when constructing the WASM test URL query string. |
| src/DeviceRunners.Cli/Commands/MtpFilterTranslator.cs | New translator for MTP simple filters into DeviceRunners’ existing filter grammar, including escaping. |
| src/DeviceRunners.Cli/Commands/BaseTestCommand.cs | Adds CLI options for the 8 simple filters, validation (mutual exclusivity + trait format), and effective-filter resolution. |
| docs/articles/xunit-v3-support.md | Documents MTP simple-filter parity and clarifies --filter-query is not implemented. |
| docs/articles/using-dotnet-test.md | Documents wildcard behavior for =/!= and provides wildcard examples. |
| docs/articles/using-devicerunners-cli.md | Adds a “Filtering Tests” section describing both --filter and the MTP simple filters + combination rules. |
Honor the documented backslash-escape contract for the new '*' wildcard: the tokenizer previously unescaped '\*' to a bare '*', which was then treated as a wildcard, so a literal '*' could never be matched with '='/'!='. Carry an escaped star through tokenization as a private-use sentinel so an unescaped '*' wildcards while '\*' matches a literal '*'. Direct comparisons collapse the sentinel back to '*'; the wildcard regex builder does the same per literal segment. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds support for the eight xUnit v3 Microsoft Testing Platform (MTP) "simple" test-filter switches to the
device-runnersCLI, and adds*wildcard support to the on-device filter evaluator.This was triggered by reports of a new
--filter-classarg. That switch is part of MTP's typed "simple" filter family (an alternative to the VSTest-style--filterexpression). This PR brings all 8 of them to DeviceRunners and makes the matching faithful via real wildcard support.What changed
On-device evaluator (
TestCaseFilter)*wildcard support to the equals family (=/!=) via an anchored, case-insensitive regex.*matches zero or more characters.*matches exactly as before (parity preserved).RegexOptions.NonBacktrackingfor guaranteed linear-time matching (no catastrophic-backtracking / ReDoS risk on adversarial patterns).dotnet test --filter "ClassName=Calc*"now works too.CLI (
device-runners <platform> test)--filter-class/--filter-not-class,--filter-method/--filter-not-method,--filter-namespace/--filter-not-namespace,--filter-trait/--filter-not-trait.MtpFilterTranslatormaps them into the existing--filterexpression: same-kind values OR together, different kinds AND together,not-variants exclude (AND-ed negations), structural chars escaped,*left intact.--filterand the simple filters are mutually exclusive (validation error if both supplied, mirroring xUnit).--filter-traitvalues (missing=separator or empty trait name) are rejected with a clear validation error rather than being silently reinterpreted.DEVICE_RUNNERS_FILTERenv var / MSIX arg / WASM query string), including the Windows loose-MSIX launch path.Filter mapping
--filter-classClassName=--filter-methodFullyQualifiedName=--filter-namespaceNamespace=--filter-trait name=valuename=value--filter-not-*!=) variantsTests
MtpFilterTranslatorunit tests (per-kind OR, cross-kind AND,not-exclusion, wildcard pass-through, trait split, escaping, empty→null, trait validation).DEVICE_RUNNERS_FILTERbuilt from simple filters, precedence, mutual-exclusion, malformed-trait rejection).TestCaseFilterTests(starts-with / ends-with / contains / mid-string, multi-segment,!=negation, trait values, case-insensitivity, exact-match parity).Docs
using-devicerunners-cli.md: new Filtering Tests section (raw--filter+ the 8 simple switches, combine rules).using-dotnet-test.md:*wildcard note + examples.xunit-v3-support.md: MTP-parity table and a--filter-queryexplainer (explained only — not implemented).Scoping note
Android delivers filters via build-time MSBuild env-baking, so
device-runners android test(which launches an already-installed app) doesn't re-apply these switches — Android filtering goes through thedotnet test --filterpath (which now has wildcards). This is documented. Wiring the simple switches into the MSBuild/Android path can be a follow-up if full parity there is desired.Review feedback addressed
Two-model review (GPT-5.5 + Opus 4.8). Fixes in
340c47d:RegexOptions.NonBacktracking(linear-time, no catastrophic backtracking).--filter-traitentry with no=or an empty name now fails validation instead of being silently treated as aFullyQualifiedNamefilter.