[Pi0.5][Jetson Thor] Unable to Reproduce the Documented 27.17 ms 2-View NVFP4 + FA4 Baseline

[JuShuai-gif]

Issue

I ran the Pi0.5 2-view NVFP4 + FA4 end-to-end benchmark on an NVIDIA Jetson AGX Thor.

The README reports the following 2-view performance:

• FP8 P50: 38.70 ms
• NVFP4 + FA4 P50: 27.17 ms
• Throughput: 37 Hz
• Speedup: 1.424×

The results measured in my current environment are:

• FP8 P50: 41.603 ms
• NVFP4 + FA4 P50: 29.135 ms
• NVFP4 + FA4 P95: 29.252 ms
• Throughput: approximately 34.3 Hz
• Speedup: 1.428×

The absolute FP4 latency is approximately 1.97 ms, or 7.2%, slower than the README result. The FP8 baseline is also approximately 2.90 ms, or 7.5%, slower. However, the relative FP4-over-FP8 speedup is nearly identical to the documented result.

One important limitation is that the official pi05_libero checkpoint and original libero_obs_2v_n8.npz fixture used for the documented baseline are not available on this system.

For this preliminary performance test, I used the existing pi05base weights, LIBERO demo normalization statistics, and a temporary fixture constructed from the locally cached LIBERO dataset. Therefore, this is a preliminary performance reproduction rather than a formal numerical-fidelity reproduction.

Environment

• Hardware: NVIDIA Jetson AGX Thor Developer Kit, SM110
• Power mode: MAXN
• CPU: 14 cores locked at 2601 MHz
• GPU GPC: Min/Max/Current all at 1575 MHz
• GPU NVD: Min/Max/Current all at 1692 MHz
• EMC: Min/Max/Current all at 4266 MHz
• Container: nvcr.io/nvidia/pytorch:26.05-py3
• PyTorch: 2.12.0a0+5aff3928d8.nv26.05
• Container CUDA: 13.2
• Host driver: 580.00
• FlashRT commit: 7fd75d20c0528f4c3ba70e23b9fbf579dbc68211
• Worktree: dirty
• Construction path: flash_rt.load_model()
• Attention backend: FA4 active
• Camera views: 2
• Denoising steps: 10
• Autotune level: 3
• Warmup: 20 iterations per precision mode
• Timed iterations: 100 per precision mode
• FP8 and FP4 were executed in separate child processes

FA4 runtime environment:

• nvidia-cutlass-dsl==4.4.2
• quack-kernels==0.4.1
• CUTE_DSL_ARCH=sm_110a
• FLASH_ATTENTION_ARCH=sm_100a

Latest Results

FP8 and FP4 were each warmed up for 20 iterations, followed by 100 timed full pipe.infer() calls.

Mode	Min	P50	P95	Max
FP8 + FA4	41.387 ms	41.603 ms	41.723 ms	41.866 ms
NVFP4 + FA4	28.981 ms	29.135 ms	29.252 ms	29.304 ms

Comparison with the main README result:

Metric	README	Current result	Difference
FP8 P50	38.70 ms	41.603 ms	+2.903 ms
FP4 P50	27.17 ms	29.135 ms	+1.965 ms
FP4 throughput	37 Hz	34.3 Hz	-2.7 Hz
FP4 speedup	1.424×	1.428×	Nearly identical

Comparison with the documented public API confirmation run:

Metric	Public API confirmation	Current result	Difference
FP8 P50	38.62 ms	41.603 ms	+2.983 ms
FP4 P50	27.29 ms	29.135 ms	+1.845 ms
FP4 P95	27.34 ms	29.252 ms	+1.912 ms

Relative FP4 improvement over FP8 in the current run:

• P50 latency reduction: 12.468 ms
• Speedup: 1.428×
• Relative latency reduction: approximately 30.0%

The relative FP4 speedup is consistent with the README, but the absolute latency of both FP8 and FP4 is approximately 7% slower.

Numerical Fidelity

All performance gates passed:

• fp4_p50_faster_than_fp8=true
• fp4_fa4_p50_beats_published_by_2ms=true
• fp4_fa4_2v_p95_at_most_40_ms=true

However, the numerical-fidelity gates did not pass:

Metric	Current result	Required gate
Raw cosine	0.992483	≥ 0.995
Worst raw sample cosine	0.968467	≥ 0.995
Final action cosine	0.997008	≥ 0.999
Worst action sample cosine	0.979684	≥ 0.995

These fidelity failures are likely related to the temporary checkpoint composition, normalization statistics, and constructed fixture. They should not be interpreted as a fidelity failure of the official pi05_libero checkpoint.

Investigation

1. No other GPU compute processes were running before the benchmark.

2. nvpmodel was set to MAXN, and jetson_clocks --show reported:

   gpu-gpc-0 MinFreq=1575000000 MaxFreq=1575000000 CurrentFreq=1575000000
   gpu-nvd-0 MinFreq=1692000000 MaxFreq=1692000000 CurrentFreq=1692000000
   EMC MinFreq=4266000000 MaxFreq=4266000000 CurrentFreq=4266000000
   NV Power Mode: MAXN
   

3. Both FP8 and FP4 are approximately 7% slower than the README results, while the relative speedup is 1.428× compared with the documented 1.424×.

   This does not appear to be a regression isolated to the FP4 path. It looks more like the entire run is operating in a slower system-performance regime.

4. The documentation mentions that Thor may drift between fast and slow sustained-load regimes, differing by approximately 3 ms even when the GPC and NVD clocks are locked. It also notes that the EMC cap acts as a ceiling rather than a guaranteed lock of the internal memory-performance state.

   The current FP8 result is approximately 3 ms slower than the documented FP8 baseline, which appears consistent with this behavior.

5. The benchmark script currently uses an older published latency target:

   published_sota_p50_ms = 36.3 ms
   required_margin_ms = 2.0 ms
   target_p50_ms = 34.3 ms
   

   Therefore, the measured 29.135 ms result passes the existing performance gate, but it does not reach the latest 27.17 ms result reported in the README.

Questions

1. Could you provide the complete benchmark command, commit, and container version used for the documented 27.17 ms 2-view baseline?

2. Could you provide, or explain how to obtain, the exact pi05_libero safetensors checkpoint, normalization statistics, and libero_obs_2v_n8.npz fixture used for the baseline?

3. The main README reports 27.17 ms using direct frontend construction, while the public API confirmation run reports 27.29 ms. Should 27.29 ms be considered the expected target for the load_model() path?

4. Can the documented flash_rt_fp4 and flash_rt_kernels binaries be reproduced directly from a specific commit? Are there specific CUTLASS DSL, CUDA, PyTorch, or container versions that must also be fixed?

5. Is there a recommended way to determine whether Thor is operating in the documented fast or slow sustained-load regime? In addition to MAXN, jetson_clocks, and checking the sysfs min_freq, max_freq, and cur_freq values, are there additional EMC, power, or thermal settings that should be fixed?

6. Since both FP8 and FP4 are approximately 7% slower while the relative speedup remains consistent, which stages or kernels would you recommend profiling first?

Thank you very much for your help!

[LiangSu8899]

Thanks for the detailed report. The environment information and, especially, your note about substituting the checkpoint made both observations much easier for me to reproduce.

The short version is:

• The latency difference does not appear to be a FlashRT regression or a hardware fault. On Thor, 20 warmup iterations are often not sufficient.
• The fidelity failures are consistent with the pi05_base substitution, as you suspected. I reproduced the same pattern by running pi05_base through the same benchmark.

Latency

I tested the same commit (7fd75d20) with the official pi05_libero checkpoint, MAXN, and jetson_clocks enabled.

2-view	FP8 p50	FP4 p50	Gates
20 warmup	41.38 ms	29.05 ms	pass
300 warmup	38.62 ms	27.25 ms	all pass
README	38.70 ms	27.17 ms	all pass
Your report	41.603 ms	29.135 ms	fidelity fail

3-view	FP4 p50
20 warmup	33.62 ms
300 warmup	31.84 ms
README	31.74 ms

Your 41.60 / 29.14 ms result is within approximately 0.2 ms of my 20-warmup run, so I do not see evidence of a problem with your Thor.

I could also observe warmup completing during the timed measurement. In one 3-view run, the first 76 samples remained around 33.6 ms, after which the remaining samples dropped to approximately 32.0 ms. This also explains why the results may look stable even when warmup is incomplete: three of my 20-warmup 2-view runs had a p50 spread of only 0.024 ms.

I suggest using approximately 300 warmup iterations, or dividing the timed samples into ten groups and checking their medians:

np.array_split(samples_ms, 10)

If the medians contain a clear step, the system was still warming up during the measurement.

One additional caveat from my own Thor testing is that warmup and tactic selection are not completely stable. Depending on the machine state, I also regularly observe approximately ±2–4 ms of variation. When the absolute latency looks unusual, I normally check two things first:

1. whether the actual memory bandwidth is at the expected level; and
2. whether warmup was sufficient and the timed samples were stable from the beginning.

On my machine, the reference checks were approximately:

• 256.5 GB/s DRAM read bandwidth, or 94% of the theoretical 273 GB/s;
• 85.1 TFLOPS for an 8192³ FP16 GEMM without performance droop;
• EMC actually running at 4266 MHz;
• Tj around 45–55°C.

If your measurements are in a similar range, the hardware is likely healthy.

Fidelity and calibration data

The fidelity gate compares NVFP4 against FP8 using the same weights in the same run. Changing the checkpoint therefore affects the result through calibration, rather than simply because the weights produce different actions.

The NVFP4 path is calibrated rather than statically cast:

• AWQ (awq_alpha=0.8) derives per-channel weight rescaling from the activation maxima measured on the eight calibration observations.
• The per-16-element block scale factors are based on the observed dynamic range.
• The encoder’s FP8 99.9-percentile scales come from the same observations.

These calculations assume that the calibration samples represent the inputs the model will encounter during actual inference.

This is particularly important at low precision: using real calibration data that matches the model’s actual task and deployment distribution is critical for preserving accuracy. A small or mismatched calibration set may still produce a reasonable average result while causing large errors on a few samples outside the calibrated range.

pi05_base is not fine-tuned for LIBERO. On LIBERO frames, its activations have wider tails and more per-channel outliers than the matched calibration expects. A 4-bit representation has limited headroom for this mismatch. If the checkpoint, normalization statistics, and fixture also come from different sources, all three calibration inputs are mismatched.

I measured the following on the same machine with the same fixtures, changing only the checkpoint:

Model	raw cos	worst raw	action cos	worst action
pi05_libero 2v	0.998491	0.997046	0.999566	0.998752
pi05_base 2v	0.981490	0.891982	0.972497	0.962654
Your pi05_base 2v	0.992483	0.968467	0.997008	0.979684
pi05_libero 3v	0.997396	0.994791	0.998472	0.996124
pi05_base 3v	0.958712	0.863549	0.938451	0.797931

The worst sample drops much more than the average: 0.892 / 0.798 compared with averages of 0.981 / 0.959. This is consistent with a calibration-data mismatch, where a few frames fall outside the calibrated range while most remain reasonable. Your 0.968 worst value against a 0.992 average shows a similar pattern.

As expected, the checkpoint did not materially affect latency:

3-view pi05_base:   31.889 ms
3-view pi05_libero: 31.809 ms

More calibration details are available in:

• docs/calibration.md
• docs/pi05_native_calibration.md
• docs/extension/calibration.md

Getting the matching weights and data

The checkpoint is:

gs://openpi-assets/checkpoints/pi05_libero

It is approximately 12.4 GB and can be converted with the pinned OpenPI commit:

git clone https://github.com/Physical-Intelligence/openpi.git && cd openpi
git checkout 15a9616a00943ada6c20a0f158e3adb39df2ccac

python -c "from openpi.shared import download; print(download.maybe_download('gs://openpi-assets/checkpoints/pi05_libero'))"

python examples/convert_jax_model_to_pytorch.py \
  --config-name pi05_libero \
  --checkpoint-dir ~/.cache/openpi/openpi-assets/checkpoints/pi05_libero \
  --output-path ~/.cache/openpi/openpi-assets/checkpoints/pi05_libero_pytorch

The resulting model.safetensors should be 7,233,650,408 bytes.

The matching normalization statistics are included at:

assets/physical-intelligence/libero/norm_stats.json

They do not require a separate download, but the copy shipped with this checkpoint should be used.

The LIBERO conversion used by OpenPI is available here:

https://huggingface.co/datasets/physical-intelligence/libero

The fixture contains eight real LIBERO observations:

Key	Shape	dtype
n	scalar	int64 (= 8)
img_<i>	(224, 224, 3)	uint8
wrist_<i>	(224, 224, 3)	uint8
wrist_right_<i>	(224, 224, 3)	uint8 (3-view only)
state_<i>	(8,)	float32

The caller should provide raw uint8 images and raw state, without normalization or discretization.

Answers to your questions

1. Command, commit, and container for the 27.17 ms baseline

PYTHONPATH=<repo-root> python tests/bench_pi05_decoder_fp4_e2e.py \
  --num-views 2 \
  --checkpoint <pi05_libero_pytorch> \
  --fixture <dir>/libero_obs_2v_n8.npz \
  --output-dir <empty-dir>

With sufficient warmup, I get 27.250 ms with all gates passing at your commit (7fd75d20), directly on the host with Torch 2.13.0+cu130.

The original documented session used Torch 2.10.0 / CUDA 13.0. However, Torch does not appear to be the relevant variable because Torch 2.13 reaches the same latency. Your 26.05 container should therefore be fine.

2. Checkpoint, normalization statistics, and fixture

Please see the matching inputs and fixture schema above.

3. Is 27.29 ms the expected load_model() target?

Yes. I would consider approximately 27.2–27.3 ms the target for either construction path.

I measured 27.250 ms through load_model(). The 0.12 ms difference between 27.17 and 27.29 ms is much smaller than the warmup and run-to-run variation discussed above.

4. Are the binaries reproducible from a commit?

They may not be reproducible byte-for-byte, so I would not use SHA-256 equality as the acceptance test.

I rebuilt both binaries from source and obtained different hashes, but the 3-view cosine results were bit-identical:

0.997396 / 0.994791 / 0.998472 / 0.996124

The latency was also in the same range.

The important versions and settings to pin are:

• CUTLASS v4.4.2 at third_party/cutlass;
• cmake -DGPU_ARCH=110;
• CUDA 13.0 nvcc;
• nvidia-cutlass-dsl==4.4.2;
• quack-kernels==0.4.1;
• the same Python interpreter for CMake and import flash_rt;
• pass -DPython3_EXECUTABLE=;
• build with -j 2, since the CUTLASS translation units use substantial RAM.

Using -DPython_EXECUTABLE= may silently fall back to /usr/bin/python3.

5. How can I tell whether Thor is in the slow state?

In my testing, this is usually warmup/tactic and machine-state variation rather than a distinct hardware fault.

MAXN, jetson_clocks, devfreq, and EMC may report the same values in both cases, so there is no single sysfs value that reliably identifies it.

I recommend:

• using sufficient warmup;
• checking the ten group medians for a step;
• verifying actual memory bandwidth if the absolute latency remains a few milliseconds away from the reference;
• repeating the benchmark when necessary, since I have also observed approximately ±2–4 ms variation between machine states.

6. Which stages or kernels should be profiled first?

For this particular result, I would not begin with individual kernels.

FP8 and FP4 changed by nearly the same percentage, while the relative speedup remained 1.428× compared with the documented 1.424×. This pattern is more consistent with a shared run-level effect, such as incomplete warmup or machine-state variation, than with a regression isolated to the FP4 path.

Your observation about:

PUBLISHED_SOTA_P50_MS = {1: 30.5, 2: 36.3, 3: 42.8}

is correct. This is a regression floor inherited from an earlier tier, rather than the current README target. I will clarify that distinction and the warmup recommendation in the documentation.

Thanks again for taking the time to prepare such a careful report. The details you included made this much easier for me to investigate.