Trained by Nuplan dataset / epoch 3

[riita10069]

@m-zain-khawaja
I share update of today's meeting.

Validation Score

Epoch 1
Selection Score 0.19672
ADE@6.4 1.415 m
FDE@6.4 4.226 m

Epoch 2
Selection Score 0.20092
ADE@6.4 1.305 m
FDE@6.4 4.021 m

⭐️Epoch 3
Selection Score 0.20262
ADE@6.4 ** 1.261 m **
FDE@6.4 ** 3.856 m **

checkpoint

https://auto-e2e-platform-checkpoints-381491877296.s3.us-west-2.amazonaws.com/ray-train/nuplan-p5-b4-resume-e3-7599a6d3-r3-nuplan_full-ray-8-full/checkpoint_2026-09-06_06-13-01.542568/checkpoint.pt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAVRUVQCWYELRKZ4WK%2F20260908%2Fus-west-2%2Fs3%2Faws4_request

Source Code

On experimental/reactive-bev-learning Branch

[m-zain-khawaja]

@riita10069 - the checkpoint link appears to be broken, could you please update it, thanks

[CodeName-Detective]

@riita10069 and @m-zain-khawaja, I will take this checkpoint and fine-tune on KIT Scenes.

Do you want me to freeze any modules in the model or fine-tune it completely?

[cgnarendiran]

Please share the checkpoint link again, it's broken, thanks!

[riita10069]

Fine tuning on KITScenes has been completed.
The situation has become complex with various factors such as the baseline, post-fine tuning, whether to freeze BEVFormer, and whether to use BEV Segmentation Auxiliary Loss.
Rather than passing around links, we should provide the models on Hugging Face.
The only way forward is for more people to conduct more experiments across more situations.
Zain, please coordinate with us on the Hugging Face matter.

[CodeName-Detective]

Makes sense....

[m-zain-khawaja]

@riita10069 - to publish on HuggingFace, I need:

1. ADE and FDE @3s of baseline model (NuPlan training) on NuPlan validation samples
2. ADE and FDE @6.4s of baseline model (NuPlan training) on NuPlan validation samples
3. Visualizations of baseline model performance on NuPlan
4. ADE and FDE @3s of finetuned model (NuPlan training and KIT Fine tuning) on KIT validation samples
5. ADE and FDE @6.4s of finetuned model (NuPlan training and KIT Fine tuning) on KIT validation samples
6. Visualizations of finetuned model performance on KIT

[riita10069]

I'm planning to talk to Mr. Mitsudome tomorrow morning.

I'll do my best to have it updated by the time of the meeting Tomorrow evening.

[riita10069]

KITScenes Val v3.5、11,035 samples、Camera + HD Map + oracle Route

   Version    Model                            1秒                2秒                3秒                5秒    vertiacal    holizontal
  ━━━━━━━━━  ━━━━━━━━━━━━━━━━━━━  ━━━━━━━━━━━━━━━━━  ━━━━━━━━━━━━━━━━━  ━━━━━━━━━━━━━━━━━  ━━━━━━━━━━━━━━━━━  ━━━━━━━━  ━━━━━━━━
        64    nuPlan Epoch 4       0.2030 / 0.4066    0.5018 / 1.1935    0.9503 / 2.4508    2.2919 / 6.2272    1.2082    1.6448
  ─────────  ───────────────────  ─────────────────  ─────────────────  ─────────────────  ─────────────────  ────────  ────────
        65    nuPlan Epoch 5       0.1951 / 0.3779    0.4723 / 1.1331    0.9147 / 2.4226    2.2951 / 6.3885    1.1911    1.6633
  ─────────  ───────────────────  ─────────────────  ─────────────────  ─────────────────  ─────────────────  ────────  ────────
        66    KITScenes Epoch 5    0.1472 / 0.2850    0.3729 / 0.9239    0.7449 / 2.0196    1.9405 / 5.5645    0.9844    1.4025
  ─────────  ───────────────────  ─────────────────  ─────────────────  ─────────────────  ─────────────────  ────────  ────────
        67    KITScenes Epoch 7    0.1347 / 0.2831    0.3886 / 1.0101    0.8035 / 2.2127    2.0952 / 5.9509    1.0220    1.5659

KITScenes Test v1.0、23,690 samples、Camera-only

   Version    Model                            1秒                2秒                3秒                5秒    vertical    holizontal
  ━━━━━━━━━  ━━━━━━━━━━━━━━━━━━━  ━━━━━━━━━━━━━━━━━  ━━━━━━━━━━━━━━━━━  ━━━━━━━━━━━━━━━━━  ━━━━━━━━━━━━━━━━━  ━━━━━━━━  ━━━━━━━━
        64    nuPlan Epoch 4       0.1916 / 0.3899    0.4817 / 1.1514    0.9181 / 2.3895    2.2840 / 6.4266    1.2167    1.6287
  ─────────  ───────────────────  ─────────────────  ─────────────────  ─────────────────  ─────────────────  ────────  ────────
        65    nuPlan Epoch 5       0.1602 / 0.3134    0.3979 / 0.9727    0.7924 / 2.1621    2.1120 / 6.1751    1.0584    1.5496
  ─────────  ───────────────────  ─────────────────  ─────────────────  ─────────────────  ─────────────────  ────────  ────────
        66    KITScenes Epoch 5    0.1463 / 0.3107    0.4121 / 1.0453    0.8263 / 2.2199    2.1042 / 5.9422    1.2044    1.4051
  ─────────  ───────────────────  ─────────────────  ─────────────────  ─────────────────  ─────────────────  ────────  ────────
        67    KITScenes Epoch 7    0.1680 / 0.3754    0.4923 / 1.2512    0.9831 / 2.6242    2.4699 / 6.8964    1.4730    1.6598

[riita10069]

@m-zain-khawaja The evaluation result is here, and all of trajectory is on my dashboard, you can check them

https://d2itskdqq39tx1.cloudfront.net/

And I have updated the occupancy on the dashboard

the new one is learned by nuplan only 4 epochs.
the classes are added, how do you feel, lets talk at the weekly meeting

[m-zain-khawaja]

@riita10069 - could you please share metrics in this format:

• ADE and FDE @3s of baseline model (NuPlan training) on NuPlan validation samples
• ADE and FDE @6.4s of baseline model (NuPlan training) on NuPlan validation samples
• ADE and FDE @3s of finetuned model (NuPlan training and KIT Fine tuning) on KIT validation samples
• ADE and FDE @6.4s of finetuned model (NuPlan training and KIT Fine tuning) on KIT validation samples

I also couldn't see the NuPlan dataset visualizations

[riita10069]

@m-zain-khawaja

• ADE and FDE @3s of baseline model on nuPlan validation samples: Not recorded

ADE and FDE @6.4s of baseline model on nuPlan validation samples: 1.2578 m / 3.8450 m

ADE and FDE @3s of fine-tuned model on KIT validation samples: 0.7124 m / 1.9034 m

ADE and FDE @6.4s of fine-tuned model on KIT validation samples: 2.6326 m / 7.5567 m

[riita10069]

The Internal Validation Sample is arbitrarily selected from multiple Epoch models for the purpose of comparing between Epochs.
To prevent leakage, I believe we should use Test samples instead. (Specifically, I prefer the Validation dataset from KITScenes)

[riita10069]

You can access to the model checkpoint from this huggingface
https://huggingface.co/AutowareFoundation/auto_e2e

[riita10069]

I'm strongly interested in METEOR, It looks much better than BEVFormer.
Because the dataset size and quolity are great.

https://github.com/tier4/METEOR

In my opinion, we should use Co-MLOps dataset, it's neat dataset.
and we should use more high quality BEV annotation.

In my experience, model architecture is a trivial matter, while data volume, annotation quality, and ODD diversity are far more important.
METEOR seems to be working well. The weights are also publicly available.
We need to leverage this while maintaining the AutoE2E philosophy.

[riita10069]

• evaluate METEOR using KITScenes ADE/FDE and Occupancy Dashboard