kitti_eval: name class 3 the class clean_data actually filters for - #10
Open
egeboy35 wants to merge 1 commit into
Open
kitti_eval: name class 3 the class clean_data actually filters for#10egeboy35 wants to merge 1 commit into
egeboy35 wants to merge 1 commit into
Conversation
egeboy35
force-pushed
the
fix/kitti-eval-class3-name
branch
from
September 1, 2026 12:28
5954916 to
a2f866d
Compare
`clean_data` decides which annotations belong to a class index:
CLASS_NAMES = ['car', 'pedestrian', 'cyclist', 'van', 'person_sitting', 'truck']
`get_official_eval_result` names the same indices for the results table, and
inverts that map so a caller can pass a class name:
3: 'Sign', #'Van',
Index by index, that is the only one of the six that disagrees:
0: filter=car label=Car same
1: filter=pedestrian label=Pedestrian same
2: filter=cyclist label=Cyclist same
3: filter=van label=Sign <-- differs
4: filter=person_sitting label=Person_sitting same
5: filter=truck label=Truck same
Two consequences:
* `name_to_class['Sign'] = 3`, so a caller asking for signs is given the van
numbers, under the heading "Sign".
* `name_to_class['Van']` is absent, so a caller using KITTI's own class name
gets a KeyError.
Three things in the repository say which one was meant. `get_coco_eval_result`,
120 lines further down the same file, has `3: 'Van'`. `clean_data` line 50
treats Van as a KITTI class in its own right (`current_cls_name == "car" and
gt_name == "van"` -> neighbour, not a false positive). And min_overlaps gives
index 3 a 0.7 IoU threshold -- the vehicle threshold, where pedestrian and
cyclist get 0.5.
`Sign` is a Waymo class, not a KITTI one, and this repository already says so:
detection3d/waymo2kitti_async.py:87 is `'SIGN': 'Sign' # not in kitti`.
The commented-out `#'Van'` on the same line is what the fix restores.
Adds kitti_eval/tests/test_class_names_agree.py -- 10 tests. They read the
source with `ast` rather than importing eval.py, which compiles its kernels
with `@numba.jit` at import time; nothing here makes a claim about numba.
Standard library only, no dataset, no GPU.
Against this branch: 10 passed. Against the file as it stands on main:
4 failed, 6 passed --
FAILED test_every_index_names_the_class_clean_data_filters_for[get_official_eval_result]
FAILED test_the_two_maps_agree_with_each_other
FAILED test_a_caller_can_name_every_class_the_filter_accepts[get_official_eval_result]
FAILED test_no_index_map_carries_a_class_kitti_does_not_have
The same two parametrised tests pass for `get_coco_eval_result` either way,
which is the internal contradiction showing up in the report rather than in
prose.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
egeboy35
force-pushed
the
fix/kitti-eval-class3-name
branch
from
September 1, 2026 12:43
a2f866d to
5ac9123
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
clean_datadecides which annotations belong to a class index:get_official_eval_resultnames the same indices for the results table, and inverts that map so a caller can pass a class name:Index by index, it is the only one of six that disagrees
Two consequences:
name_to_class['Sign'] = 3, so a caller asking for signs is given the van numbers, under the heading "Sign".name_to_class['Van']is absent, so a caller using KITTI's own class name gets aKeyError.Three things in the repo say which one was meant
get_coco_eval_result, 120 lines further down the same file, has3: 'Van'.clean_dataline 50 treats Van as a KITTI class in its own right:current_cls_name == "car" and gt_name == "van"→ neighbour, not a false positive.min_overlapsgives index 3 a 0.7 IoU threshold — the vehicle threshold, where pedestrian and cyclist get 0.5.And
Signis a Waymo class, not a KITTI one. This repository already says so:detection3d/waymo2kitti_async.py:87is'SIGN': 'Sign' # not in kitti.The commented-out
#'Van'on the same line is what this restores.Tests
Adds
kitti_eval/tests/test_class_names_agree.py— 10 tests. They read the source withastrather than importingeval.py, which compiles its kernels with@numba.jitat import time; nothing here makes a claim about numba. Standard library only, no dataset, no GPU.Against this branch: 10 passed. Against the file as it stands on
main: 4 failed, 6 passed:The same two parametrised tests pass for
get_coco_eval_resulteither way — which is the internal contradiction showing up in the test report rather than in prose.