Skip to content

refactor: migrate vulnerability and license matching end-to-end to scalibr - #2935

Merged
another-rex merged 14 commits into
google:mainfrom
another-rex:migrate-scalibr-e2e
Jul 30, 2026
Merged

refactor: migrate vulnerability and license matching end-to-end to scalibr#2935
another-rex merged 14 commits into
google:mainfrom
another-rex:migrate-scalibr-e2e

Conversation

@another-rex

@another-rex another-rex commented Jul 22, 2026

Copy link
Copy Markdown
Collaborator

This PR migrates scanning, filtering, and matching in osv-scanner to use osv-scalibr end-to-end:

This is essentially the first part of #2934 split out into it's own PR.

  • Matching Offloaded to Scalibr Enrichers: Removed legacy vulnerability and license matchers from osv-scanner, delegating them to osv-scalibr enrichers.
  • Filtering Annotator Plugin: Implemented package filtering as a scalibr annotator plugin (osv-scanner/filter) to handle unscannable packages, container-specific package exclusions, and ignore rules.
  • Client Factories: Introduced ClientFactories implementing scalibrconfig.ClientFactories to inject User-Agent headers and pool client connections.
    • This replaces usages in the fix and scan subcommands.
  • PURL Type Resolution: Updated osvscannerjson extractor to map ecosystem names to valid PURL types (golang, gem, cargo, npm, etc.).
  • Model & Utility Cleanups: Cleaned up imodels package and muffled asynchronous database loading logs in testlogger.

agy

Comment thread cmd/osv-scanner/internal/testcmd/vcr.go Fixed
Comment thread cmd/osv-scanner/internal/testcmd/vcr.go Fixed
Comment thread cmd/osv-scanner/internal/testcmd/vcr.go Fixed
Comment thread cmd/osv-scanner/internal/testcmd/vcr.go Fixed
Comment thread cmd/osv-scanner/internal/testcmd/vcr.go Fixed
Comment thread cmd/osv-scanner/internal/testcmd/vcr.go Fixed
…alibr

This change completes the migration to use osv-scalibr end-to-end for scanning, filtering, and matching:
- Remove vulnerability and license matchers from osv-scanner, delegating them to scalibr enrichers.
- Move package filtering to a scalibr annotator plugin (osv-scanner/filter).
- Implement scalibr ClientFactories for HTTP client and gRPC connection management.
- Update osvscannerjson extractor to map ecosystem names to valid PURL types.
- Clean up imodels and test logger handlers.

agy
@another-rex another-rex reopened this Jul 23, 2026
Update VCR recorder rules in testcmd to exclude binary file extensions (.zip, .gz, .db, etc.) from being recorded into cassettes, and update test snapshots accordingly.

agy
@codecov-commenter

codecov-commenter commented Jul 23, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 84.56660% with 73 lines in your changes missing coverage. Please review.
✅ Project coverage is 79.37%. Comparing base (83c195f) to head (38dabd0).
⚠️ Report is 2 commits behind head on main.

Files with missing lines Patch % Lines
internal/scalibr/client_factories.go 71.42% 10 Missing and 6 partials ⚠️
cmd/osv-scanner/internal/testcmd/vcr.go 47.82% 12 Missing ⚠️
...brextract/language/osv/osvscannerjson/extractor.go 52.00% 12 Missing ⚠️
pkg/osvscanner/scan.go 91.05% 10 Missing and 1 partial ⚠️
pkg/osvscanner/osvscanner.go 84.48% 7 Missing and 2 partials ⚠️
internal/scalibrannotator/filter/filter.go 93.58% 4 Missing and 1 partial ⚠️
internal/scalibrplugin/presets.go 20.00% 3 Missing and 1 partial ⚠️
cmd/osv-scanner/fix/command.go 93.10% 1 Missing and 1 partial ⚠️
cmd/osv-scanner/mcp/command.go 0.00% 1 Missing ⚠️
internal/testlogger/handler.go 87.50% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2935      +/-   ##
==========================================
+ Coverage   78.82%   79.37%   +0.54%     
==========================================
  Files         122      118       -4     
  Lines        8360     8059     -301     
==========================================
- Hits         6590     6397     -193     
+ Misses       1383     1306      -77     
+ Partials      387      356      -31     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…d Body

Update sortCassetteInteractions in testcmd/vcr.go to sort interactions belonging to the same test (X-Test-Name) secondary by Method, URL, and Body.

agy
…reachArchive

Skip the jars_can_be_scanned_with_call_analysis subtest using testutility.Skip in TestCommand_JavareachArchive.

agy
@another-rex
another-rex force-pushed the migrate-scalibr-e2e branch from f0e40f1 to ea396ef Compare July 24, 2026 03:25
@another-rex
another-rex force-pushed the migrate-scalibr-e2e branch 2 times, most recently from 0065c21 to 346946e Compare July 24, 2026 04:47
Comment thread cmd/osv-scanner/scan/image/testdata/stuff.py Outdated
Comment thread cmd/osv-scanner/internal/testcmd/vcr.go Outdated
Comment thread internal/scalibr/client_factories.go Outdated
Comment thread internal/scalibr/client_factories.go Outdated
Comment on lines +78 to +79
func (c *ClientFactories) GoogleHTTPClient(_ context.Context, _ ...string) (*http.Client, error) {
return nil, nil

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this expected to be called by scalibr?
If not, should this panic or return an error (presumably if it doesn't return an error, it will panic anyway when something tries to use the nil client?)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess not. I have this returning an error now. Someone could probably configure osv-scanner to call osv-scalibr in a way that would try to get the GoogleHTTPClient, but it's probably fine to just error in those cases.

Comment thread internal/scalibrannotator/filter/filter.go Outdated
@another-rex
another-rex requested a review from michaelkedar July 29, 2026 03:11
- Update github.com/google/osv-scalibr to latest main commit
- Set DisableGoogleAuth: true in PluginConfig to prevent unnecessary Google auth attempts and log warnings
- Restore error return in ClientFactories.GoogleHTTPClient

TAG=agy
CONV=5862a67d-fb12-45b9-adc3-1c45d4a4783d
@another-rex
another-rex enabled auto-merge (squash) July 29, 2026 05:45
@another-rex
another-rex disabled auto-merge July 30, 2026 01:39
@another-rex
another-rex merged commit 22613e7 into google:main Jul 30, 2026
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants