mixed_dataset: let the shuffle reach training instead of being discarded

[egeboy35]

build_mixed shuffles the train image list, and the comment above it states why:

Shuffle the train order so that taking the first --max-images during training yields a balanced (cross-source) subset -- needed for clean data-size studies.

That order does not survive to the trainer. img_id is assigned inside the per-source loop, so each source owns one contiguous id block, and the consumer re-sorts. torchvision's CocoDetection.__init__ does:

self.ids = list(sorted(self.coco.imgs.keys()))

which discards the JSON list order, after which datasets.py takes the first --max-images in id order. Sorting therefore front-loads whichever source happened to be written first.

Measured

Ran the repository's own build_mixed with three sources x 400 train / 150 test images, then read the result back through CocoDetection:

shuffle achieved, in the JSON list : waymo 148, kitti 124, nuimages 128
ids as assigned                    : kitti 0..399, waymo 550..949, nuimages 1100..1499

train --max-images 200  -> kitti 200
train --max-images 400  -> kitti 400
train --max-images 800  -> kitti 400, waymo 400
val   --max-images 200  -> kitti 150, waymo 50

train.py:63 defaults to max_images = 400, so --dataset mixed with nothing else set trains on a single source.

The held-out block has the same shape. It is written as val.json and labelled "held-out mixed", and run_eval.py defaults to --mixed-split val --max-images 200 -- which read 150 KITTI + 50 Waymo and never reached nuImages.

Change

Both lists are shuffled (with separate seeds) and the image ids are renumbered to follow that order, with annotations[].image_id remapped to match. Train takes 0..T-1 and the held-out block the range above it, so the two stay disjoint by construction the way this function's docstring promises.

Same build, after:

train --max-images 200  -> waymo 81,  kitti 57,  nuimages 62
train --max-images 400  -> waymo 148, kitti 124, nuimages 128
val   --max-images 200  -> waymo 72,  kitti 65,  nuimages 63

Invariants re-checked on the produced files:

• train/val image-id intersection: 0
• every annotation still resolves inside its own split (train and val both)
• test_<source>.json unchanged at 150 images of one source each -- those filter on "source", not on id

One file, +23 lines.

What I could not run, and what I substituted

I have no KITTI/Waymo/nuImages data and no GPU here, so two things were stubbed. Neither is under test:

• _source_dataset -- replaced with a stub honouring the contract the loop uses (len(ds), ds[i] -> s.image (PIL), s.gt_boxes, s.gt_labels), so no download is needed.
• torchvision.datasets.CocoDetection -- torchvision is not installed in my environment, so the class was reconstructed keeping the one line that matters verbatim from torchvision 0.28.0's own datasets/coco.py, read out of the wheel: self.ids = list(sorted(self.coco.imgs.keys())).

pycocotools is the real package, and the id-selection logic under test is this repository's own code. If your torchvision pin orders ids differently, the premise changes and this PR should be closed -- worth a 10-second check on your side.

A question, not a claim

TUTORIAL.md 18.3 runs --dataset mixed at the 400 default and reports a fine-tune result whose shape (kitti up, waymo and nuimages down) is what training on KITTI alone would look like, while 18.4 passes --max-images 99999 and recovers the other two. Whether that section is measuring catastrophic forgetting or this bug is a question for you -- I have no way to tell from outside, and I am not asserting either reading.