LoTIS: Learning to Localize Reference Trajectories in Image-Space for Visual Navigation

Finn Busch, Matti Vahs, Quantao Yang, Jesús Gerardo Ortega Peimbert, Yixi Cai, Jana Tumova, Olov Andersson

Division of Robotics, Perception, and Learning at KTH Royal Institute of Technology

LoTIS is a model for visual navigation that provides robot-agnostic image-space guidance by localizing a reference RGB trajectory in the robot's current view. Given a reference trajectory (a sequence of RGB images) and a query image from the robot's current viewpoint, LoTIS predicts the 2D image-space coordinates, visibility, and relative distance of each trajectory pose as it would appear in the query view.

🆕 New — Navigation evaluation. Reproduce the paper's Habitat benchmark and evaluate your own method against LoTIS. See Navigation Evaluation.

Setup

1. Clone this repository and install dependencies

git clone https://github.com/KTH-RPL/lotis.git
cd lotis

Dependencies are managed with uv. uv run will automatically install everything on first use. To install manually:

uv sync

2. Clone the DINOv3 repository

LoTIS uses DINOv3 as a frozen backbone. Clone Meta's repository into the project root:

git clone https://github.com/facebookresearch/dinov3.git

3. Download DINOv3 weights

Request access to DINOv3 pretrained weights at ai.meta.com/resources/models-and-libraries/dinov3-downloads. Once approved, you will receive an email with download URLs. Download the ViT-B/16 pretrain weights (dinov3_vitb16_pretrain.pth) and set the path:

export DINOV3_WEIGHTS=/path/to/dinov3_vitb16_pretrain.pth

Or pass it directly via --dinov3-weights when running scripts.

4. Download LoTIS model weights

huggingface-cli download fnnBsch/lotis final_model.pth final_config.yaml --local-dir .

Running inference

Localize a reference trajectory in a query (image, video, or directory), visualized live in Rerun:

uv run python infer.py \
    --trajectory examples/00_KTH_Campus/Courtyard/trajectory.mp4 \
    --query examples/00_KTH_Campus/Courtyard/queries/Forward/query.mp4

Live webcam — encode a trajectory once, then localize from your camera in real time:

uv run python infer.py \
    --trajectory path/to/trajectory.mp4 \
    --usb-cam

Use --cam-id 1 to select a different camera. All infer.py options:

--trajectory PATH       Reference trajectory (video, image dir, or single image)
--query PATH            Query input (video, image dir, or single image)
--usb-cam               Use webcam as live query  [mutually exclusive with --query]
--cam-id INT            Camera device ID (default: 0)
--checkpoint PATH       LoTIS checkpoint (default: final_model.pth)
--config PATH           LoTIS config (default: final_config.yaml)
--dinov3-weights PATH   DINOv3 weights file (or set DINOV3_WEIGHTS)
--dinov3-repo PATH      Path to cloned dinov3 repo (default: ./dinov3)
--device cuda|cpu       Inference device (auto-detected if not set)
--vis-threshold FLOAT   Visibility threshold for displaying points (default: 0.5)

Rerun also accepts additional flags (e.g. --serve to stream to a remote viewer). Run python infer.py --help for the full list.

Remote visualization

If running on a remote machine, start the Rerun viewer locally and forward the port:

# On your local machine
rerun

# Forward the Rerun port from remote to local
ssh -R 9876:localhost:9876 <remote-host>

# On the remote machine
uv run python infer.py --trajectory ... --query ... --connect

Gradio demo

A full interactive demo is available at the HuggingFace Space. To run it locally:

uv run python app.py

Set DINOV3_WEIGHTS (and optionally DINOV3_REPO) before running.

Python API

from lotis import TrajectoryEncoding, TrajectoryLocalizer

localizer = TrajectoryLocalizer.from_checkpoint(
    checkpoint_path="final_model.pth",
    config_path="final_config.yaml",
    dinov3_weights="/path/to/dinov3_vitb16_pretrain.pth",
)

# Encode a trajectory — do this once and reuse
encoding = localizer.encode_trajectory("path/to/trajectory.mp4")

# Localize a query image
result = localizer.localize("query.jpg", encoding)

print(f"Closest trajectory frame: {result.closest_frame()}")
print(f"Visible frames: {result.visible_indices()}")

# Save and reload the trajectory encoding
import torch
torch.save(encoding.to_dict(), "encoding.pt")
encoding = TrajectoryEncoding.from_dict(torch.load("encoding.pt"))

localize() accepts single images or lists, and single encodings or lists — see lotis/localizer.py for the full batching API.

Navigation Evaluation

The release includes a small, method-neutral evaluation runner for already-generated Habitat evaluation datasets. It does not include data generation code or the paper baseline implementations.

Install Habitat-Sim 0.3.3 for evaluation:

uv pip install pip
HEADLESS=True uv pip install --no-build-isolation \
    "habitat-sim @ git+https://github.com/facebookresearch/habitat-sim.git@v0.3.3"

Datasets and scenes

Two assets are required: the Habitat scene meshes (Gibson, HM3D) and the LoTIS evaluation dataset (the pre-generated trajectory-query episodes).

Habitat scenes. Point the evaluator at the directory containing your scene_datasets/ tree with --scene-data-dir (or the HABITAT_DATA_DIR environment variable) — this overrides the data_path in every dataset config, so you never edit YAML:

uv run python -m nav_eval.run --dataset gibson --method lotis --scene-data-dir /path/to/habitat_data

Expected layout under that directory:

<scene-data-dir>/
  scene_datasets/
    gibson/            # Gibson .glb scenes + gibson.scene_dataset_config.json
    hm3d/val/          # HM3D val scenes + hm3d_val_basis.scene_dataset_config.json

🚧 TODO — scene download instructions coming soon. Steps for obtaining the Gibson (license agreement) and HM3D (access request) Habitat scene datasets will be added here.

LoTIS evaluation dataset. Both datasets ship as tarballs in a single Hugging Face dataset repo (fnnBsch/lotis-eval). Download, extract, and point each config's dataset_path at the extracted folder:

hf download fnnBsch/lotis-eval --repo-type dataset --local-dir /path/to/lotis-eval
tar xzf /path/to/lotis-eval/gibson.tar.gz -C /path/to/lotis-eval
tar xzf /path/to/lotis-eval/hm3d.tar.gz   -C /path/to/lotis-eval
# gibson.yaml -> dataset_path: /path/to/lotis-eval/gibson
# hm3d.yaml   -> dataset_path: /path/to/lotis-eval/hm3d

Note — frames are rendered on the fly. Because the Habitat scene datasets are under licenses that do not permit redistributing rendered imagery, the evaluation dataset ships poses only, without RGB. The reference-trajectory and query frames are generated from your local scenes when you run the evaluation (or the explorer below). Reconstructing the frames this way can introduce small variations from the exact images used in the paper, so expect metrics to match closely rather than bit-for-bit.

Explore the dataset

Since the frames are not shipped, use the explorer to render an episode — its reference trajectory, query view, and goal frame — into a Rerun viewer:

uv run python -m nav_eval.explore \
    --dataset gibson --start on --direction forward --camera cross --index 0

Change --start / --direction / --camera to pick the condition and --index to page through episodes. Requires the Habitat scenes to be set up (same as running the eval).

Run LoTIS on all Gibson evaluation cases:

uv run python -m nav_eval.run \
    --dataset gibson \
    --method lotis

By default this expands to all valid combinations of start type (on, off), direction (forward, backward, any), and camera/reference condition (matched, cross). The any direction is only valid for off-trajectory starts, so on + any is skipped automatically.

Filter the matrix when needed:

uv run python -m nav_eval.run \
    --dataset gibson \
    --method lotis \
    --start off \
    --direction forward \
    --camera cross \
    --max-episodes 8

Use --max-episodes for quick smoke tests, then remove it for the full run. The terminal shows a live dashboard; the full per-episode log, results.json, and the resolved configs are written under eval_results/<dataset>/<case>/<method>/.

Live visualization

Add --visualize to stream the run to a Rerun viewer — the live camera feed, the goal image, and, for LoTIS, the predicted image-space trajectory points overlaid on the current view:

uv run python -m nav_eval.run --dataset gibson --method lotis --max-episodes 4 --visualize

This spawns a viewer locally. On a remote machine, run rerun locally, forward the port (ssh -R 9876:localhost:9876 <remote-host>), and pass --rerun-connect rerun+http://127.0.0.1:9876/proxy.

Reproducing the paper results

Run the full evaluation matrix on both datasets:

uv run python -m nav_eval.run --dataset gibson --method lotis
uv run python -m nav_eval.run --dataset hm3d --method lotis

Each command expands to all valid start / direction / camera conditions reported in the paper.

Evaluating your own method

The evaluator is method-neutral: it owns Habitat setup, episode iteration, and metrics, and only asks your method for velocity commands. See nav_eval/WRITING_A_METHOD.md for the interface, the observation and episode fields, and a runnable example.

Compare two result directories:

uv run python -m nav_eval.compare \
    --reference /path/to/reference/ours \
    --candidate eval_results/gibson/off_start_forward_cross_camera/lotis

Roadmap

• Inference code + Gradio Demo
• Evaluation runner for pre-generated datasets
• Training code

Citation

@misc{busch2026learninglocalizereferencetrajectories,
      title={Learning to Localize Reference Trajectories in Image-Space for Visual Navigation}, 
      author={Finn Lukas Busch and Matti Vahs and Quantao Yang and Jesús Gerardo Ortega Peimbert and Yixi Cai and Jana Tumova and Olov Andersson},
      year={2026},
      eprint={2602.18803},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2602.18803}, 
}