EgoExo is a ROS-based ego-to-exo view augmentation pipeline built on top of ORB-SLAM3. It takes a monocular (or stereo/RGB-D) camera feed, optionally IMU stream, runs visual(-inertial) SLAM, and augments the egocentric view with exocentric overlays. It also publishes a 2.5D ground skin reconstruction and augments a 3D robot mesh in the same spatial context.
Exo view (fixed trailing distance) |
Exo view (fixed POV) |
2.5D ground reconstruction |
Tested on Ubuntu 20.04 with ROS1 Noetic. A Docker container is provided and recommended for easy integration.
mkdir -p ~/catkin_ws/src
cd ~/catkin_ws/src
git clone https://github.com/uf-robopi/EgoExo.gitcd ~/catkin_ws/src/egoexo
docker build -t egoexo ../run_docker.shThis mounts your local egoexo folder into the container, so any changes you make on your host machine are immediately reflected inside the container.
catkin build egoexo
source devel/setup.bashAll parameters related to EgoExo pipeline are in a single config file:
config/egoexo_config.yaml
Key sections to adjust before running on your own data:
| Section | What to change |
|---|---|
camera_matrix |
Your camera's intrinsic parameters (fx, fy, cx, cy) |
distortion |
Your camera's distortion coefficients |
exo_view |
Robot point cloud file, projection scale (optional) |
exo_mesh |
Robot mesh file, scale, initial orientation (optional) |
ground |
Ground reconstruction parameters (optional) |
For ORB-SLAM3 specific parameters, edit the config file in: config/Monocular/MyConfig.yaml
A reference example for the EuRoC dataset is provided at: config/Monocular/EuRoC.yaml
Refer to config/Monocular-Inertial/ folder if you are running monocular-inertial SLAM.
Place your robot's 3D model (.stl) and point cloud (.npy) files in assets/. The BlueROV2 model is provided.
A sample recorded bag file is available for download here: Dropbox link
The bag file should contain at minimum a monocular camera topic. Stereo, RGB-D, and IMU topics are only needed to run respective type of SLAM.
# Terminal 1 - inside the container, launch EgoExo
roslaunch egoexo egoexo_mono_slam.launch
# Terminal 2 - play the bag file
rosbag play your_recording.bagSet mode: "video" and provide the path to your video in egoexo_config.yaml:
image_publisher:
mode: "video"
video_file: "/path/to/your/video.mp4"
publish_hz: 20.0Then launch:
roslaunch egoexo egoexo_mono_slam.launchSet mode: "images" in egoexo.yaml:
image_publisher:
mode: "images"
image_folder: "/path/to/your/images"
image_pattern: "frame_*.jpg"
publish_hz: 20.0Then launch:
roslaunch egoexo egoexo_mono_slam.launchThe Rviz window should look like the following:
# Save the current map
rosservice call /orb_slam3/save_map my_map
# Save the estimated trajectory
rosservice call /orb_slam3/save_traj my_trajectoryMap files are saved as .osa in ~/.ros/. To load a saved map, set System.LoadAtlasFromFile in your camera config yaml.
EgoExo subscribes to
| Topic | Type | Description |
|---|---|---|
/orb_slam3/camera_pose |
geometry_msgs/PoseStamped |
Camera pose from SLAM |
/orb_slam3/tracked_points |
sensor_msgs/PointCloud2 |
Tracked 3D map points |
/usb_cam/image_raw |
sensor_msgs/Image |
Input camera image |
EgoExo publishes
| Topic | Type | Description |
|---|---|---|
/egoexo/exo_image |
sensor_msgs/Image |
Exocentric image with augment ROV pose |
/egoexo/rov_marker |
visualization_msgs/Marker |
3D robot mesh in world frame |
/egoexo/ground_skin |
visualization_msgs/Marker |
2.5D textured ground reconstruction |
This work is published in the following research papers.
EgoExo++ (IJRR 2026)
Adnan Abdullah, Ruo Chen, Ioannis Rekleitis, and Md Jahidul Islam. EgoExo++: Integrating On-demand Exocentric Visuals with 2.5D Ground Surface Estimation for Interactive Teleoperation of Underwater ROVs. International Journal of Robotics Research (IJRR), accepted April 2026. [Paper] [Demo]
Ego-to-Exo (ISRR 2024)
Adnan Abdullah, Ruo Chen, Ioannis Rekleitis, and Md Jahidul Islam. Ego-to-Exo: Interfacing Third Person Visuals from Egocentric Views in Real-time for Improved ROV Teleoperation. International Symposium on Robotics Research (ISRR), 2024. [Paper] [Demo]
If you use this code in your research, please cite our work:
@article{abdullah2026ego,
title = {EgoExo++: Integrating On-demand Exocentric Visuals with 2.5D Ground Surface Estimation for Interactive Teleoperation of Underwater ROVs},
author = {Abdullah, Adnan and Chen, Ruo and Rekleitis, Ioannis and Islam, Md Jahidul},
journal = {International Journal of Robotics Research},
year = {2026},
note = {Accepted April 2026},
url = {https://arxiv.org/pdf/2407.00848}
}
@inproceedings{abdullah2024ego,
title = {Ego-to-Exo: Interfacing Third Person Visuals from Egocentric Views in Real-time for Improved ROV Teleoperation},
author = {Abdullah, Adnan and Chen, Ruo and Rekleitis, Ioannis and Islam, Md Jahidul},
booktitle = {International Symposium on Robotics Research (ISRR)},
year = {2024},
url = {https://par.nsf.gov/servlets/purl/10614050}
}This project builds on the following open-source works:
- ORB-SLAM3 - Campos et al., University of Zaragoza. The core SLAM system.
- orb_slam3_ros - ROS1 wrapper for ORB-SLAM3, created by Thien Nguyen.
- Pangolin - MIT license.
- DBoW2 - BSD license.
- g2o - BSD license.
- Sophus - MIT license.
This work was supported in part by NSF grants #2330416, #1943205, and #2024741.
The authors thank the Woodville Karst Plain Project (WKPP), CINDAQ, Global Underwater Explorers (GUE), Ricardo Constantino, and Project Baseline for providing access to challenging underwater cave environments used in this research. We also thank the participants of our user study for their time and feedback.
For questions or issues, please open a GitHub issue.