Add sequential matcher with bundled vocab tree to colmap.py - #70
Merged
Conversation
Add --use_sequential flag to run colmap sequential_matcher, which does sliding-window matching (--sequential_overlap, default 10) plus vocab-tree loop detection for spatially-near-but-sequentially-distant frames. Suits sequential captures where consecutive images sometimes lack direct overlap. Commit a default Flickr100K/32K-word vocab_tree.bin next to the script and resolve it via resolve_vocab_tree_path(): a per-session <source_dir>/vocab_tree.bin still overrides, else fall back to the bundled default. The existing --use_vocab_tree path now uses the same resolver. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Query every image for loop detection (period 1), retrieve 60 candidates with soft visual-word assignment (num_nearest_neighbors 5) for a well-connected graph. Add tight geometric verification (max_ratio 0.8, min_num_inliers 15) so the extra candidate edges that survive are reliable rather than spurious. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Indexing every extracted feature (up to max_num_features, 16384) into the vocab tree dominated matching runtime (~50s/image). Set loop_detection_max_num_features to 4096 so the index uses only the top-scale features (which dominate retrieval), cutting indexing ~4-8x. Full pairwise SIFT matching still uses all features, so map density is unaffected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Querying every image (period 1) against 60 candidates made the loop-detection matching pass take ~12h on a 765-image dataset. Consecutive frames retrieve nearly identical loop candidates, so period 1 is mostly redundant -- a revisit spans many frames and is still caught at period 5. Set period 5 and num_images 40 for ~5x faster matching (~2.5h) with negligible recall loss. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The script already writes ARKit position pose priors to the pose_priors table, but neither colmap mapper nor glomap mapper consumes them, so they were dormant during reconstruction -- a likely cause of disconnected components when visual matching between clusters is too sparse. Add run_colmap_pose_prior_mapping() and a --use_pose_prior flag that routes reconstruction through colmap pose_prior_mapper, which registers cameras using their ARKit positions (robust loss on prior position, DB covariance). Takes precedence over --use_glomap; mirrors the intrinsics-fixed BA settings. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ntion Vision-only SfM (incremental/GLOMAP) cannot cohere on wide-baseline, low-parallax capture (park foliage): despite rich matches (median 331 inliers/pair), two-view relative poses are inconsistent, so GLOMAP connects only 192/673 images and the incremental mapper dies at init. Add --use_arkit_poses: seed a COLMAP model with the trusted ARKit poses for every frame and run point_triangulator, giving a fully connected 673-image / ~52k-point map (poses never depend on the visual view graph). Fix the ARKit<->COLMAP coordinate convention, which was geometrically wrong: it applied a world-axis column flip instead of the standard camera-axis flip on an inverted matrix (ARKit camera-to-world -> COLMAP world-to-camera: R_w2c = diag(1,-1,-1) @ R_c2w^T, t = -R_w2c @ C). Correct convention triangulates ~34k+ points vs ~4.8k for the old one. Fix flows through: - colmap_pose.py: extract_rotation_translation_from_extrinsics + ColmapPose - arkit_integration.py / database_operations.py: derive camera center as -R^T t - colmap_database.cpp: colmapPoseToMatrix + drop the landmark/camera y/z flips, keeping the C++ refiner consistent with the corrected Python convention. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ARKit VIO drift splits multi-view observations into separate 2-view points that fail the >=3-sightings cull. A 12px merge/complete/filter threshold (was 6px) rejoins them into longer tracks, roughly doubling final usable landmarks after refinement (15.7k -> 31k) at essentially the same reprojection error (0.39 -> 0.42px). The refiner's BA + outlier removal prunes any bad merges. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Both were empirically validated as non-viable for this data (see docs/RECONSTRUCTION_CLEANUP_PLAN.md): - pose_prior_mapper stalls at incremental init even with correct, tight priors (init is pure vision); the low-parallax view graph can't bootstrap. - Injecting ARKit odometry into two_view_geometries has no effect: COLMAP/GLOMAP re-estimate relative poses from feature matches and filter edges with <30 inliers, so pose-only edges are discarded. Odometry belongs in the g2o BA (ColmapRefiner already adds it), not the SfM view graph. Removed: --use_pose_prior + run_colmap_pose_prior_mapping; insert_arkit_odometry + insert_two_view_geometries_from_arkit and its helpers (image_ids_to_pair_id, array_to_blob, compute_relative_pose_from_arkit); the pose_priors write in create_colmap_database. Restored the full pipeline (setup + extract + match) that was commented out for the resume workflow. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- docs/RECONSTRUCTION_CLEANUP_PLAN.md: findings, keep/remove decisions, future --use_hybrid design, and merge action items. - gitignore .claude/ (Claude Code metadata) and debug.txt (g2o Hessian dump). - Untrack committed __pycache__/*.pyc (already covered by *.pyc). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Point RECONSTRUCTION.md at --use_sequential --use_arkit_poses as the path for large / low-parallax capture, explain when vision-only SfM fragments, drop the removed ARKit-odometry pipeline step, and note alignment is skipped for the ARKit-pose path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a
--use_sequentialmatching mode toscript/colmap/colmap.py, tuned for sequential iPhone captures where consecutive frames sometimes lack direct overlap. It runscolmap sequential_matcher— a sliding window over frames (in filename/capture order) plus vocab-tree loop detection for spatially-near-but-sequentially-distant frames — giving a well-connected match graph without a full O(N²) exhaustive pass.Changes
--use_sequentialflag (+--sequential_overlap, default 10). Uses--SequentialMatching.quadratic_overlapso the window reaches frames at exponentially increasing distances.loop_detection_period 1(query every image),loop_detection_num_images 60,loop_detection_num_nearest_neighbors 5(soft visual-word assignment for higher retrieval recall).max_ratio 0.8,min_num_inliers 15) so the extra loop-closure candidate edges that survive are reliable, not spurious.script/colmap/vocab_tree.bin(15 MB, Flickr100K / 32K words). Newresolve_vocab_tree_path(): a per-session<source_dir>/vocab_tree.binoverrides, else the bundled default is used. The existing--use_vocab_treepath now shares this resolver.Usage
Works out of the box with loop detection — no per-session vocab tree setup needed.
🤖 Generated with Claude Code