Skip to content

culane_metric: let a prediction claim a free lane instead of dropping it - #8

Open
egeboy35 wants to merge 1 commit into
lkk688:mainfrom
egeboy35:fix/culane-match-uses-free-lanes
Open

culane_metric: let a prediction claim a free lane instead of dropping it#8
egeboy35 wants to merge 1 commit into
lkk688:mainfrom
egeboy35:fix/culane-match-uses-free-lanes

Conversation

@egeboy35

Copy link
Copy Markdown

The module docstring promises a "greedily one-to-one match". _match offers each prediction only its argmax ground-truth lane:

for i in np.argsort(-iou.max(axis=1)):
    j = int(iou[i].argmax())
    if j not in used_g and iou[i, j] >= iou_thresh:

So when that lane is already taken the prediction is discarded — including where the same IoU matrix says it clears the threshold against a lane nothing has claimed. Greedy one-to-one takes that pair.

Measured

Found by sweeping lane spacing against prediction offset at the class's own defaults (320×800, width 15 px). Ground truth at x=300 and x=306, predictions at x=300 and x=301:

IoU     g0       g1
  p0   1.000*   0.478
  p1   0.889*   0.545*        * clears the 0.5 threshold

p0 takes g0. p1's argmax is also g0, now used, so p1 is dropped — while IoU(p1, g1) = 0.545 and g1 is free.

_match              tp=1  fp=1  fn=1
greedy one-to-one   tp=2  fp=0  fn=0

The missed pair is charged twice: once as a false positive, once as a false negative. Across lane gaps 4–17 px and prediction offsets in 0.5 px steps, 29 of 294 combinations under-count.

Over 400 random four-lane frames through the public CULaneF1 class, where a quarter of the predictions are pulled toward a neighbouring lane:

_match              tp=1218  fp=382  fn=382   F1=0.7612
greedy one-to-one   tp=1236  fp=364  fn=364   F1=0.7725

Both matchers see the same IoU matrices, so the difference is the matching rule alone — not the geometry, the threshold or the model. F1 is understated by 0.0113 on that sample, and the direction is fixed: the previous rule can never count more.

The change

Walk the matrix best-pair-first and skip a pair only when the prediction or the lane is already used — the docstring's rule applied to the whole matrix rather than to one column per row. It stays pure numpy, and the loop breaks at the first sub-threshold pair because the order is sorted.

What I am not claiming

Greedy is not always the maximum matching. It can commit a high pair that blocks two lower ones. On the 40-frame sample in the tests, greedy takes 129 where an exhaustive search takes 130.

Closing that gap means Hungarian assignment, which would pull scipy into a module whose first line advertises "pure numpy (no cv2, no external eval binary)", and would depart from the rule the docstring states. If you would rather have the optimum I am happy to send that instead — it is a small change on top of this one.

Tests

Adds ngperception/lane/tests/test_culane_metric.py — 12 tests, no dataset, no model, no GPU.

Against this branch: 12 passed. Against the file as it stands on main: 3 failed, 9 passed:

FAILED test_a_prediction_is_not_dropped_while_a_lane_it_matches_is_free
FAILED test_the_sweep_that_found_it_reports_no_under_counting
FAILED test_the_class_agrees_with_an_independent_greedy_matcher   (124 vs 129)

The 9 that pass either way are the properties this must not break: one lane per prediction, one prediction per lane, counts adding up, the threshold respected, the empty cases, and a perfect prediction scoring 1.0.

pytest DeepDataMiningLearning/ngperception/lane/tests

The module docstring promises a "greedily one-to-one match". _match offers
each prediction only its argmax ground-truth lane:

    for i in np.argsort(-iou.max(axis=1)):
        j = int(iou[i].argmax())
        if j not in used_g and iou[i, j] >= iou_thresh:

so when that lane is already taken the prediction is discarded -- including
where the same IoU matrix says it clears the threshold against a lane nothing
has claimed. Greedy one-to-one takes that pair.

Found by sweeping lane spacing against prediction offset at the class's own
defaults (320x800, width 15 px). Ground truth at x=300 and x=306, predictions
at x=300 and x=301:

    IoU     g0       g1
      p0   1.000*   0.478
      p1   0.889*   0.545*      * clears the 0.5 threshold

p0 takes g0. p1's argmax is also g0, now used, so p1 is dropped -- while
IoU(p1, g1) = 0.545 and g1 is free.

    _match              tp=1  fp=1  fn=1
    greedy one-to-one   tp=2  fp=0  fn=0

The missed pair is charged twice: once as a false positive, once as a false
negative. Across lane gaps 4..17 px and prediction offsets in 0.5 px steps,
29 of 294 combinations under-count.

Over 400 random four-lane frames through the public CULaneF1 class, where a
quarter of the predictions are pulled toward a neighbouring lane:

    _match              tp=1218  fp=382  fn=382   F1=0.7612
    greedy one-to-one   tp=1236  fp=364  fn=364   F1=0.7725

Both matchers see the same IoU matrices, so the difference is the matching
rule alone. F1 is understated by 0.0113 on that sample, and the direction is
fixed: the previous rule can never count more.

The change walks the matrix best-pair-first and skips a pair only when the
prediction or the lane is already used -- the docstring's rule, applied to the
whole matrix rather than to one column per row. It stays pure numpy; the loop
breaks at the first sub-threshold pair because the order is sorted.

One thing I am not claiming: greedy is not always the maximum matching. It can
commit a high pair that blocks two lower ones. On the 40-frame sample in the
tests greedy takes 129 where an exhaustive search takes 130. Closing that gap
means Hungarian assignment, which would pull scipy into a module whose first
line advertises "pure numpy (no cv2, no external eval binary)", and would
depart from the rule the docstring states. If you would rather have the
optimum I am happy to send that instead -- it is a small change on top of this
one.

Adds ngperception/lane/tests/test_culane_metric.py -- 12 tests, no dataset, no
model, no GPU. Against this branch: 12 passed. Against the file as it stands on
main: 3 failed, 9 passed --

    FAILED test_a_prediction_is_not_dropped_while_a_lane_it_matches_is_free
    FAILED test_the_sweep_that_found_it_reports_no_under_counting
    FAILED test_the_class_agrees_with_an_independent_greedy_matcher (124 vs 129)

The 9 that pass either way are the properties this must not break: one lane per
prediction, one prediction per lane, counts adding up, the threshold respected,
the empty cases, and a perfect prediction scoring 1.0.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants