culane_metric: let a prediction claim a free lane instead of dropping it - #8
Open
egeboy35 wants to merge 1 commit into
Open
culane_metric: let a prediction claim a free lane instead of dropping it#8egeboy35 wants to merge 1 commit into
egeboy35 wants to merge 1 commit into
Conversation
The module docstring promises a "greedily one-to-one match". _match offers
each prediction only its argmax ground-truth lane:
for i in np.argsort(-iou.max(axis=1)):
j = int(iou[i].argmax())
if j not in used_g and iou[i, j] >= iou_thresh:
so when that lane is already taken the prediction is discarded -- including
where the same IoU matrix says it clears the threshold against a lane nothing
has claimed. Greedy one-to-one takes that pair.
Found by sweeping lane spacing against prediction offset at the class's own
defaults (320x800, width 15 px). Ground truth at x=300 and x=306, predictions
at x=300 and x=301:
IoU g0 g1
p0 1.000* 0.478
p1 0.889* 0.545* * clears the 0.5 threshold
p0 takes g0. p1's argmax is also g0, now used, so p1 is dropped -- while
IoU(p1, g1) = 0.545 and g1 is free.
_match tp=1 fp=1 fn=1
greedy one-to-one tp=2 fp=0 fn=0
The missed pair is charged twice: once as a false positive, once as a false
negative. Across lane gaps 4..17 px and prediction offsets in 0.5 px steps,
29 of 294 combinations under-count.
Over 400 random four-lane frames through the public CULaneF1 class, where a
quarter of the predictions are pulled toward a neighbouring lane:
_match tp=1218 fp=382 fn=382 F1=0.7612
greedy one-to-one tp=1236 fp=364 fn=364 F1=0.7725
Both matchers see the same IoU matrices, so the difference is the matching
rule alone. F1 is understated by 0.0113 on that sample, and the direction is
fixed: the previous rule can never count more.
The change walks the matrix best-pair-first and skips a pair only when the
prediction or the lane is already used -- the docstring's rule, applied to the
whole matrix rather than to one column per row. It stays pure numpy; the loop
breaks at the first sub-threshold pair because the order is sorted.
One thing I am not claiming: greedy is not always the maximum matching. It can
commit a high pair that blocks two lower ones. On the 40-frame sample in the
tests greedy takes 129 where an exhaustive search takes 130. Closing that gap
means Hungarian assignment, which would pull scipy into a module whose first
line advertises "pure numpy (no cv2, no external eval binary)", and would
depart from the rule the docstring states. If you would rather have the
optimum I am happy to send that instead -- it is a small change on top of this
one.
Adds ngperception/lane/tests/test_culane_metric.py -- 12 tests, no dataset, no
model, no GPU. Against this branch: 12 passed. Against the file as it stands on
main: 3 failed, 9 passed --
FAILED test_a_prediction_is_not_dropped_while_a_lane_it_matches_is_free
FAILED test_the_sweep_that_found_it_reports_no_under_counting
FAILED test_the_class_agrees_with_an_independent_greedy_matcher (124 vs 129)
The 9 that pass either way are the properties this must not break: one lane per
prediction, one prediction per lane, counts adding up, the threshold respected,
the empty cases, and a perfect prediction scoring 1.0.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The module docstring promises a "greedily one-to-one match".
_matchoffers each prediction only its argmax ground-truth lane:So when that lane is already taken the prediction is discarded — including where the same IoU matrix says it clears the threshold against a lane nothing has claimed. Greedy one-to-one takes that pair.
Measured
Found by sweeping lane spacing against prediction offset at the class's own defaults (320×800, width 15 px). Ground truth at x=300 and x=306, predictions at x=300 and x=301:
p0takesg0.p1's argmax is alsog0, now used, sop1is dropped — whileIoU(p1, g1) = 0.545andg1is free.The missed pair is charged twice: once as a false positive, once as a false negative. Across lane gaps 4–17 px and prediction offsets in 0.5 px steps, 29 of 294 combinations under-count.
Over 400 random four-lane frames through the public
CULaneF1class, where a quarter of the predictions are pulled toward a neighbouring lane:Both matchers see the same IoU matrices, so the difference is the matching rule alone — not the geometry, the threshold or the model. F1 is understated by 0.0113 on that sample, and the direction is fixed: the previous rule can never count more.
The change
Walk the matrix best-pair-first and skip a pair only when the prediction or the lane is already used — the docstring's rule applied to the whole matrix rather than to one column per row. It stays pure numpy, and the loop breaks at the first sub-threshold pair because the order is sorted.
What I am not claiming
Greedy is not always the maximum matching. It can commit a high pair that blocks two lower ones. On the 40-frame sample in the tests, greedy takes 129 where an exhaustive search takes 130.
Closing that gap means Hungarian assignment, which would pull scipy into a module whose first line advertises "pure numpy (no cv2, no external eval binary)", and would depart from the rule the docstring states. If you would rather have the optimum I am happy to send that instead — it is a small change on top of this one.
Tests
Adds
ngperception/lane/tests/test_culane_metric.py— 12 tests, no dataset, no model, no GPU.Against this branch: 12 passed. Against the file as it stands on
main: 3 failed, 9 passed:The 9 that pass either way are the properties this must not break: one lane per prediction, one prediction per lane, counts adding up, the threshold respected, the empty cases, and a perfect prediction scoring 1.0.