A robotics perception engineering project focused on building 3D spatial understanding, multi-object tracking, and temporal perception using synchronized RGB cameras and LiDAR.
Built using CARLA, ROS2, OpenCV, YOLO, and LiDAR-based 3D detection, the project progresses through camera–LiDAR integration, multi-camera perception, sensor fusion, unified spatial perception, cross-camera object association, and 2D/3D multi-object tracking, establishing a foundation for 360° environmental understanding and dynamic scene perception.
This project builds upon the 2D perception stack developed in the companion repository:
ROS2 Autonomous Perception Stack — 2D Perception
| Category | Technologies |
|---|---|
| Simulation | CARLA 0.9.15 |
| Robotics Middleware | ROS2 Humble |
| Computer Vision | OpenCV |
| Object Detection | YOLOv8m-seg INT8 / YOLO26m FP16 |
| 3D Detection | PointPillars / OpenPCDet |
| 3D Tracking | AB3DMOT |
| 3D Sensor | LiDAR |
| Programming Language | Python |
| Communication | CycloneDDS |
| Environment | Windows 11 + WSL2 Ubuntu 22.04 |
RGB Cameras + LiDAR
│
▼
Sensor Synchronization
│
▼
Multi-Camera Perception
│
▼
Camera–LiDAR Fusion
│
▼
360° Panoramic Perception
│
▼
Unified Spatial Perception
│
▼
Cross-Camera Object Association
│
▼
Unified World Representation
│
▼
2D Multi-Object Tracking
│
▼
3D Multi-Object Tracking
Established the foundation for spatial perception by integrating LiDAR with RGB cameras, calibrating multi-sensor geometry, associating 2D detections with 3D point clouds, and preparing the perception pipeline for object-level distance estimation.
-
✅ 7A — LiDAR Integration
- Integrated a 32-channel LiDAR sensor with the RGB perception pipeline.
- Published synchronized
PointCloud2data and validated real-time visualization. - 📁 View Phase 7A
-
✅ 7B — Camera–LiDAR Calibration
- Calibrated RGB camera and LiDAR sensors using intrinsic and extrinsic parameters.
- Implemented point cloud projection and verified exact timestamp synchronization.
- 📁 View Phase 7B
-
✅ 7C — 2D–3D Association
- Associated projected LiDAR points with YOLOv8 segmentation masks to generate object-specific point clouds.
- Developed deterministic recording and offline replay pipelines for repeatable perception experiments.
- 📁 View Phase 7C
-
✅ 7D — Object Distance Estimation
- LiDAR-based object distance estimation
- Monocular distance estimation
- Camera–LiDAR distance fusion
- 📁 View Phase 7D
Expanded the perception pipeline from a single forward-facing camera to a synchronized 360° multi-sensor perception system by integrating multiple RGB cameras and LiDAR. This phase establishes the foundation for surround perception, multi-camera sensor fusion, and unified environmental understanding.
-
✅ 8A — 360° Camera–LiDAR Perception
- Developed a synchronized 360° perception pipeline using four RGB cameras and a 32-channel LiDAR sensor.
- Implemented deterministic recording, offline replay, multi-camera perception, and LiDAR projection for repeatable perception experiments.
- 📁 View Phase 8A
-
✅ 8B — 360° Panoramic Distance Estimation
- Extended the 360° perception pipeline with panoramic image generation and LiDAR-based object distance estimation.
- Implemented multi-camera object detection, camera–LiDAR projection, object-level point association, cylindrical panoramic stitching, and distance-aware surround visualization.
- 📁 View Phase 8B
Developed a unified spatial perception framework by transforming synchronized multi-camera detections and LiDAR observations into a common ego-centric world representation. This phase establishes the foundation for spatial scene understanding through object localization, Bird's-Eye View generation, and cross-camera object reasoning.
-
✅ 9A — Unified Spatial Perception
- Localized detected objects from four synchronized RGB cameras into a unified ego coordinate frame using Camera–LiDAR fusion.
- Generated a unified Bird's-Eye View (BEV) and validated coordinate transformations through camera-specific yaw correction.
- 📁 View Phase 9A
-
✅ 9B — Cross-Camera Object Merging
- Associated duplicate object detections across overlapping camera views using bearing-based overlap filtering and Hungarian assignment.
- Merged duplicate observations into a unified object representation for consistent 360° spatial perception.
- 📁 View Phase 9B
Extending the perception pipeline with temporal multi-object tracking using YOLO26m TensorRT FP16 and ByteTrack.
-
✅ 10A — 2D Multi-Object Tracking
- Integrated YOLO26m TensorRT FP16 with the official ByteTrack implementation for persistent 2D object identities across consecutive frames.
- Implemented track lifecycle management and tracking-focused visualization using synchronized offline replay data.
- 📁 View Phase 10A
-
✅ 10B — 3D Multi-Object Tracking
- Integrated geometry-based and PointPillars-based 3D detection with AB3DMOT for persistent 3D object tracking.
- Enabled modular comparison of 3D detection approaches using a common tracking and visualization framework.
- 📁 View Phase 10B
-
✅ 10C — Multi-Camera 3D Multi-Object Tracking
-
Extended 3D tracking to four synchronized RGB cameras and LiDAR with geometric cross-camera association and duplicate object merging.
-
Integrated AB3DMOT to maintain persistent 3D Track IDs across the unified 360° ego-centric scene.
-
- Documentation will be published after project milestone release.
- ✅ 11A — Ego Motion Estimation
- ✅ 11B — Ego Surrounding Object Motion Estimation
Integrated a 32-channel LiDAR sensor into the ROS2 perception pipeline and validated synchronized PointCloud2 visualization in RViz2.
Projected LiDAR points onto synchronized RGB images through camera calibration, coordinate transformation, and perspective projection.
Associated projected LiDAR points with YOLOv8 segmentation masks to generate object-specific point clouds using a deterministic offline replay pipeline.
Estimated object distances using monocular camera geometry, LiDAR point clouds, and camera–LiDAR sensor fusion.
Established a synchronized 360° perception pipeline using four RGB cameras and LiDAR with deterministic recording, offline replay, and unified surround-view perception.
Extended the 360° Camera–LiDAR perception pipeline with panoramic image generation, LiDAR-based object distance estimation, and distance-aware surround visualization using synchronized multi-camera perception.
Unified spatial perception with ego-coordinate object localization and Bird's-Eye View generation from synchronized multi-camera and LiDAR observations.
Bird's-Eye View visualization used to validate ego-coordinate transformations and correct left and right camera yaw, resulting in consistent object localization across all synchronized camera views.
Associated duplicate object detections across overlapping camera views using bearing-based overlap filtering and Hungarian assignment.
Persistent 2D object identities maintained across consecutive frames using YOLO26m TensorRT FP16 and ByteTrack on synchronized offline replay data.
Persistent 3D object identities maintained across consecutive LiDAR frames using PointPillars 3D detection and AB3DMOT tracking with synchronized front-camera visualization.
Unified 360° 3D object tracking using four synchronized RGB cameras, LiDAR, geometric cross-camera association, and AB3DMOT with persistent 3D Track IDs.
Detailed planning, development notes, experiments, and engineering decisions are maintained separately:
roadmap/README.md