SplatAD: render lidar through get_outputs_for_lidar (fixes #75, #79)

[YuvrajPuyam]

Rendering lidar from a trained SplatAD checkpoint fails with ValueError: Unknown sensor type (#75, #79). It's been open a while and there are a couple of partial snippets in the thread, so here's a proper fix.

What's going on

SplatADModel inherits get_outputs_for_lidar from ADModel. That implementation is for the ray-based models: it turns the scan into a RayBundle and calls forward(ray_bundle=...), and SplatADModel.get_outputs only accepts Cameras/Lidars, so it raises. Both render.py (dataset --render-point-clouds True) and the viewer's _render_lidar go through that path.

The snippet in the thread (a get_outputs_for_lidar that returns get_outputs(lidar)) gets you past the exception but not much further: render.py expects (outputs, batch) with a point cloud in outputs["points"], and since it reads raw dataset entries rather than the datamanager's cached ones, there is no raster_pts for get_lidar_outputs to rasterize.

The change

• SplatADModel.get_outputs_for_lidar(lidar, batch=None): builds raster_pts from batch["lidar"] when it isn't there (using the datamanager's own raster code), uses a precomputed one if present, or falls back to the sensor's azimuth/elevation grid if no scan is given. Returns (outputs, batch); outputs["points"] is [B, H, W, 3] in the lidar frame, same layout as ray_drop_prob, plus outputs["points_valid"] for cells that had a real return.
• full_images_lidar_datamanager.py: _lidar_to_raster_pts becomes a staticmethod and the body of _add_metadata moves into a static add_raster_metadata(lidar, data, device) so the model can call it. The per-lidar-type elevation/azimuth setup in _load_lidars is pulled out into get_lidar_raster_params. No behavior change for training.
• render.py: also mask by points_valid when it's present (NeuRAD unaffected).
• A small test that the override exists and the raster params are sane for every LidarType.

Checked

PandaSet 001, SplatAD trained 30k iterations, A30. Same checkpoint before and after: render.py dataset --render-point-clouds True crashed at 0/40 lidars on main, with this it renders (~0.3 s per scan). Against the real sweeps on the 10 eval scans, using the model's own eval formulas: median range error 1.3 cm (1.2-1.5 across scans), ray-drop accuracy 96.7%, intensity RMSE 0.083. Across all ten SplatAD PandaSet sequences (30k iterations each, 100 eval scans total): median range error 1.2 cm (0.2-2.2 per scene), ray-drop accuracy 96.7%, intensity RMSE 0.061; per-scene table in the comments. That's in line with what the paper reports for fully trained scenes (1-2 cm, 92.6-96.7%), so the rendering path is doing what training was doing. Real vs rendered, BEV:

Viewer

The viewer's _render_lidar had the same problem (it builds a RayBundle from the control panel). @xdtyjwj's snippet in #75 showed that handing the model a Lidars grid works there, so the second commit does that properly: for models that set renders_lidar_by_rasterization, _render_lidar builds a Lidars from the panel's actual beams / FoV / azimuth resolution / position and calls get_outputs_for_lidar; ray-based models keep the RayBundle path unchanged. The ray-drop and distance-cutoff filters stay in place. The grid branch lays cells out the way the datamanager does (azimuth ascending from -180, padded to whole tiles, padded cells masked out) and gives the virtual sensor zero velocities so the rolling-shutter code is happy. One small thing surfaced on the way: Lidars never registered azimuths/elevations as [*num_lidars, n, 1] fields in _field_custom_dimensions, so constructing one with a grid failed to broadcast (I suspect that's what pushed the thread's workaround into editing lidars.py); fixed by registering them. I checked it headlessly against a checkpoint with the viewer defaults (24 beams, 0.5 deg) and an odd size that needs padding (30 beams, 0.3 deg); I haven't clicked through the actual viser UI on the cluster, so a quick look from someone with a display would be welcome.

Unrelated but you'll hit it right after this: plot_lidar_points in render.py needs kaleido>=1 (and a Chrome install) with plotly 7. Left alone here.

Fixes #75, fixes #79.

[YuvrajPuyam]

Numbers from a 30k-iteration checkpoint are in (PandaSet 001, 10 eval scans, same script as before): median range error 1.3 cm (1.2-1.5 across scans), ray-drop accuracy 96.7%, intensity RMSE 0.083, about 0.3 s per scan on an A30. That lines up with the paper's fully-trained figures, so the rendering path matches what training sees. Updated the description and swapped the BEV plot to this checkpoint.

The viewer path on the same checkpoint, rendered headlessly with the panel defaults (24 beams, 0.5 deg):

[YuvrajPuyam]

Same check across all ten SplatAD PandaSet sequences now (30k iterations each, 10 eval scans per scene, 100 scans total): median range error 1.2 cm, ray-drop accuracy 96.7%, intensity RMSE 0.061. Per scene:

seq	median range err (cm)	ray-drop acc (%)	intensity RMSE
001	1.31	96.7	0.083
011	1.46	96.6	0.064
016	0.92	97.0	0.057
028	0.24	98.4	0.055
053	1.15	97.1	0.065
063	2.11	95.9	0.062
084	2.16	95.4	0.057
106	1.13	96.3	0.060
123	0.92	97.0	0.057
158	1.44	96.4	0.050

Updated the description with the aggregate. Script that produced these is the same one as before (renders through get_outputs_for_lidar, metrics via filter_lidar_pred_and_gt); happy to add it under scripts/ if useful.

[georghess]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2a564e6bb7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Bracket the final elevation tile

When the viewer uses its default 24 beams over -15° to 15°, boundaries contains elevations 0, 8, and 16, so this sets the maximum boundary to about 6.87° even though seven valid beams extend through 15°. get_lidar_outputs passes that value as max_elevation and as the final tile boundary, causing the upper portion of the requested vertical FoV to be omitted or assigned incorrectly. Construct boundaries around every eight-channel group, with the final boundary above elevations[-1], as get_lidar_raster_params does.

Useful? React with 👍 / 👎.

[YuvrajPuyam]

Good catch, that was a real bug in the grid branch: I built the tile boundaries as elevations[::8] plus one degree above the last tile start, so the viewer's default 24 beams over -15..15 deg ended at 6.87 deg and the top tiles were dropped. Pushed a fix that uses the same construction as get_lidar_raster_params (one degree below the first channel, midpoints between 8-channel groups, one degree above the last), factored into grid_tile_elevation_boundaries with a unit test that checks every channel lands in exactly one tile for that 24-beam case. Re-ran the headless viewer-grid check on a checkpoint as well (24 beams / 0.5 deg and 30 beams / 0.3 deg): both render, all cells valid.

[YuvrajPuyam]

@georghess Codex comment is addressed (the grid branch was not bracketing the last elevation tile, fixed with a helper and a unit test), branch is up to date with main, tests and ruff clean at the pinned 0.1.13.

The red check is not from this PR: Core Tests fails the same way on main, the second uv pip install is missing --system. I have a fix for that plus the ruff leftovers from the py123d merge on a branch, happy to open it separately if useful.