Skip to content

kitti_eval: name class 3 the class clean_data actually filters for - #10

Open
egeboy35 wants to merge 1 commit into
lkk688:mainfrom
egeboy35:fix/kitti-eval-class3-name
Open

kitti_eval: name class 3 the class clean_data actually filters for#10
egeboy35 wants to merge 1 commit into
lkk688:mainfrom
egeboy35:fix/kitti-eval-class3-name

Conversation

@egeboy35

Copy link
Copy Markdown

clean_data decides which annotations belong to a class index:

CLASS_NAMES = ['car', 'pedestrian', 'cyclist', 'van', 'person_sitting', 'truck']

get_official_eval_result names the same indices for the results table, and inverts that map so a caller can pass a class name:

3: 'Sign', #'Van',

Index by index, it is the only one of six that disagrees

0: filter=car             label=Car             same
1: filter=pedestrian      label=Pedestrian      same
2: filter=cyclist         label=Cyclist         same
3: filter=van             label=Sign            <-- differs
4: filter=person_sitting  label=Person_sitting  same
5: filter=truck           label=Truck           same

Two consequences:

  • name_to_class['Sign'] = 3, so a caller asking for signs is given the van numbers, under the heading "Sign".
  • name_to_class['Van'] is absent, so a caller using KITTI's own class name gets a KeyError.

Three things in the repo say which one was meant

  1. get_coco_eval_result, 120 lines further down the same file, has 3: 'Van'.
  2. clean_data line 50 treats Van as a KITTI class in its own right: current_cls_name == "car" and gt_name == "van" → neighbour, not a false positive.
  3. min_overlaps gives index 3 a 0.7 IoU threshold — the vehicle threshold, where pedestrian and cyclist get 0.5.

And Sign is a Waymo class, not a KITTI one. This repository already says so: detection3d/waymo2kitti_async.py:87 is 'SIGN': 'Sign' # not in kitti.

The commented-out #'Van' on the same line is what this restores.

Tests

Adds kitti_eval/tests/test_class_names_agree.py — 10 tests. They read the source with ast rather than importing eval.py, which compiles its kernels with @numba.jit at import time; nothing here makes a claim about numba. Standard library only, no dataset, no GPU.

Against this branch: 10 passed. Against the file as it stands on main: 4 failed, 6 passed:

FAILED test_every_index_names_the_class_clean_data_filters_for[get_official_eval_result]
FAILED test_the_two_maps_agree_with_each_other
FAILED test_a_caller_can_name_every_class_the_filter_accepts[get_official_eval_result]
FAILED test_no_index_map_carries_a_class_kitti_does_not_have

The same two parametrised tests pass for get_coco_eval_result either way — which is the internal contradiction showing up in the test report rather than in prose.

pytest DeepDataMiningLearning/ngperception/detection/kitti_eval/tests

@egeboy35
egeboy35 force-pushed the fix/kitti-eval-class3-name branch from 5954916 to a2f866d Compare September 1, 2026 12:28
`clean_data` decides which annotations belong to a class index:

    CLASS_NAMES = ['car', 'pedestrian', 'cyclist', 'van', 'person_sitting', 'truck']

`get_official_eval_result` names the same indices for the results table, and
inverts that map so a caller can pass a class name:

    3: 'Sign', #'Van',

Index by index, that is the only one of the six that disagrees:

    0: filter=car             label=Car             same
    1: filter=pedestrian      label=Pedestrian      same
    2: filter=cyclist         label=Cyclist         same
    3: filter=van             label=Sign            <-- differs
    4: filter=person_sitting  label=Person_sitting  same
    5: filter=truck           label=Truck           same

Two consequences:

  * `name_to_class['Sign'] = 3`, so a caller asking for signs is given the van
    numbers, under the heading "Sign".
  * `name_to_class['Van']` is absent, so a caller using KITTI's own class name
    gets a KeyError.

Three things in the repository say which one was meant. `get_coco_eval_result`,
120 lines further down the same file, has `3: 'Van'`. `clean_data` line 50
treats Van as a KITTI class in its own right (`current_cls_name == "car" and
gt_name == "van"` -> neighbour, not a false positive). And min_overlaps gives
index 3 a 0.7 IoU threshold -- the vehicle threshold, where pedestrian and
cyclist get 0.5.

`Sign` is a Waymo class, not a KITTI one, and this repository already says so:
detection3d/waymo2kitti_async.py:87 is `'SIGN': 'Sign'  # not in kitti`.

The commented-out `#'Van'` on the same line is what the fix restores.

Adds kitti_eval/tests/test_class_names_agree.py -- 10 tests. They read the
source with `ast` rather than importing eval.py, which compiles its kernels
with `@numba.jit` at import time; nothing here makes a claim about numba.
Standard library only, no dataset, no GPU.

Against this branch: 10 passed. Against the file as it stands on main:
4 failed, 6 passed --

    FAILED test_every_index_names_the_class_clean_data_filters_for[get_official_eval_result]
    FAILED test_the_two_maps_agree_with_each_other
    FAILED test_a_caller_can_name_every_class_the_filter_accepts[get_official_eval_result]
    FAILED test_no_index_map_carries_a_class_kitti_does_not_have

The same two parametrised tests pass for `get_coco_eval_result` either way,
which is the internal contradiction showing up in the report rather than in
prose.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@egeboy35
egeboy35 force-pushed the fix/kitti-eval-class3-name branch from a2f866d to 5ac9123 Compare September 1, 2026 12:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant