Skip to content

Support robust comparison script filtering by comparison status - #464

Open
oleksandr-pavlyk wants to merge 1 commit into
NVIDIA:mainfrom
oleksandr-pavlyk:nvbench-compare-filter-by-status
Open

Support robust comparison script filtering by comparison status#464
oleksandr-pavlyk wants to merge 1 commit into
NVIDIA:mainfrom
oleksandr-pavlyk:nvbench-compare-filter-by-status

Conversation

@oleksandr-pavlyk

Copy link
Copy Markdown
Collaborator

Deprecate --threshold-diff, support --status <TOKEN1>,<TOKEN2>,... filter.

Robust comparison does not take the threshold value into account, so document it as deprecated, emit warning if user specifies it as CLI argument.

Support filtering of displayed rows by comparison status. This permits an easy way to select filters with status of interests.

…lter

Robust comparison does not take the threshold value into account, so
document it as deprecated, emit warning if user specifies it as CLI
argument.

Support filtering of displayed rows by comparison status. This permits
an easy way to select filters with status of interests.
@oleksandr-pavlyk

Copy link
Copy Markdown
Collaborator Author

@coderabbitai full review

@coderabbitai

coderabbitai Bot commented Aug 27, 2026

Copy link
Copy Markdown
✅ Action performed

Full review finished.

@coderabbitai

coderabbitai Bot commented Aug 27, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 2185fe18-79e9-4b09-a5ac-c0dabb236165

📥 Commits

Reviewing files that changed from the base of the PR and between 65d36b3 and 5cf207f.

📒 Files selected for processing (3)
  • docs/nvbench_compare_robust.md
  • python/scripts/nvbench_compare_robust.py
  • python/test/test_nvbench_compare_robust.py

Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review.


📝 Walkthrough

Summary by CodeRabbit

  • New Features

    • Added a --status filter with aliases for comparison statuses, including not-unknown.
    • Status filters now control displayed table and bulk-debug results.
  • Bug Fixes

    • Plot output remains unaffected by table status filtering.
  • Deprecation

    • --threshold-diff is retained for compatibility, but its value is ignored and a warning directs users to --status.
  • Documentation

    • Updated command-line documentation with status filtering behavior and accepted aliases.

Walkthrough

Changes

Status filtering and threshold compatibility

Layer / File(s) Summary
Status parsing and row selection
python/scripts/nvbench_compare_robust.py, python/test/test_nvbench_compare_robust.py
The tool parses named and display status aliases, supports not-unknown, and filters displayed table and bulk-debug rows by status.
CLI options and compatibility behavior
python/scripts/nvbench_compare_robust.py, python/test/test_nvbench_compare_robust.py
The CLI adds --status. It forwards the parsed filter to compare_benches(). --threshold-diff remains accepted, but its value is ignored and triggers a warning.
Output validation and documentation
python/test/test_nvbench_compare_robust.py, docs/nvbench_compare_robust.md
Tests cover status parsing, output selection, plot-along behavior, metadata validation, and CLI errors. Documentation describes status filters and deprecated threshold handling.

Merge Risk: ⚪ Minimal · up to 5cf20

The PR adds comparison-status filtering and deprecates the threshold option without any actionable merge-blocking risk remaining.


Comment @coderabbitai help to get the list of available commands.

@oleksandr-pavlyk oleksandr-pavlyk changed the title Support robust comparison to filter by comparison status Support robust comparison script filtering by comparison status Aug 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant