Support for fisheye (instaX5). - #536
Draft
michalpelka wants to merge 22 commits into
Draft
michalpelka wants to merge 22 commits into
michalpelka wants to merge 22 commits into
Conversation
…ntensity windows, 2D overlay, CLI, error display - Fix broken intensity grayscale (was unnormalized glColor3f) and add a calibration window (min/max/gamma, auto-range) for it. - Add an extrinsics calibration window with live-preview sliders and precise +/- angle nudge buttons for the camera-to-LiDAR pose. - Add an optional 2D projection overlay (intensity or depth, jet colormap, alpha, decimation) clipped to the image viewport. - Add --photo/--laz CLI flags to load files on startup. - Show per-pair reprojection error and RMS in the point-pair lists, with a "remove pair" action that keeps the 2D/3D correspondence lists in sync. - Add Space as a shortcut to toggle RGB <-> intensity in the 3D view. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…viewer calib::Intrinsics gains a CameraModel tag (Pinhole | Equirectangular) plus the image dimensions, which are what an equirectangular camera projects with in place of fx/fy/cx/cy, and projectPoint branches on it. The change is purely additive: projectPoint's signature is unchanged and the model defaults to Pinhole, so camera_lidar_calibration keeps building and behaving identically. It does not yet read or write the "model" key -- the comment on CameraModel records that gap, the other pinhole-only spots in that app, and how the vendored equirectangular observation equations drop into the solver when it is picked up. camera_lidar_trajectory_viewer wires the model up end to end: - loadCalib reads a "model" key, at the top level or under "intrinsics". - The image scanner accepts equirectangular_<ts>.jpg and bare <ts>.jpg alongside cam0_<ts>.jpg, and infers the model from the prefix when the calibration doesn't name one. Both loaders share one parse now instead of two copies, and the model is resolved in a single place because the calibration and the images arrive in either order. - colorize() calls calib::projectPoint instead of duplicating the distortion math inline, over intrinsics scaled by a new image-scale control -- a chunk of 360 frames is ~2.2 GB at full size and multi-image coloring holds a whole chunk resident. - Image dimensions come from the first scanned frame rather than a hardcoded 4656x3496 that matched neither camera; they drive the ROI default, the frustums and COLMAP's cameras.txt. - Frustums become a position marker and axis triad for a camera with no frustum; COLMAP export refuses, since its text model has no equirectangular type; the ROS CameraInfo reports "equirectangular" with no K rather than a pinhole that would mislead consumers, and rectification is skipped. New calib_core/tests (doctest, following shared/tests) cover the seam wrap, the poles, a bearing round trip, behind-the-camera points and the extrinsics, and pin the pinhole path against the refactor with hand-computed values. Also fixes a pre-existing include in RosExport.cpp that fails to compile whenever CALIB_ENABLE_ROS_EXPORT is ON. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…r_calibration calib::CameraModel gains a Mei enumerator for the Insta360 rig's per-lens fisheye (distortion_model insta360_mei_v2). projectPoint does not re-derive the unified-sphere math -- it wraps the existing MeiCamera (MeiCamera.h/.cpp), which until now was an orphan: absent from every CMakeLists source list, and its own quoted "MeiCamera.h" include didn't match this repo's include/CalibCore layout, so it had never actually been compiled here. Intrinsics gains `xi`; k1/k2/k3 and p1/p2 are reused as Mei's own (non-rational) polynomial, k4/k5/k6 unused. calib_core therefore now compiles MeiCamera.cpp and picks up OpenCV's header-only Point2d/Point3d and yaml-cpp (camera_info.yaml loading) for it. camera_lidar_calibration is wired up end to end: - loadIntrinsics detects a Mei camera_info.yaml by its distortion_model key and loads it via LoadMeiCamera; the OpenCV-YAML path now resets model/xi explicitly so a Mei calibration can't linger behind a pinhole one. - load/saveCalibration round-trip "model" and "xi". - Intrinsics naming a different resolution than the loaded image are auto-scaled (calib::scaleIntrinsics) rather than only warned about, in whichever order the image and the calibration arrive; saveCalibration records the resolution they apply to (width/height) so the next load can do the same. - rebuildImageTexture skips initUndistortRectifyMap for non-pinhole models: it assumes OpenCV's rational pinhole model and would mis-warp a fisheye rather than rectify it, so a Mei image is always shown raw. - Both GLSL shaders gain a Mei branch mirroring MeiCamera::Project, so the projection overlay and Camera-RGB coloring work against that raw image. The clip weight is Xs.z+xi, the model's own "in front of the camera" test (it reduces exactly to the pinhole z_cam when xi==0); using the range there instead let points behind the camera through the hardware clip. - projectPoint applies that same Xs.z+xi>0 guard on the CPU side. MeiCamera:: Project has no domain guard of its own and the projection isn't injective past its valid dome, so a point behind the camera could otherwise be scored as visible at a plausible-looking pixel. - drawCameraFrustum becomes a position marker plus axis triad for non-pinhole models -- a rectangular pyramid misrepresents a fisheye's field of view -- matching what camera_lidar_trajectory_viewer already does for equirectangular. - The Intrinsics panel gains a model combo and an xi drag, and hides k4/k5/k6 for Mei rather than showing dead controls. New solveExtrinsicsMeiCeres: a Ceres-based extrinsics solver for Mei, since no vendored analytic Jacobian exists for this model as it does for Pinhole. It is gated behind -DCALIB_ENABLE_CERES, OFF by default because the README advertises depending on nothing but Eigen for optimization; without it the function compiles to a stub reporting why it is unavailable, so callers need no #ifdef of their own. The Pinhole solver is untouched and stays analytic. Tests check the Mei projection against MeiCamera::Project directly (so the wrapper is verified as delegation, not as a re-implementation), the behind-the-camera rejection for xi<1 alongside full-sphere coverage at xi>=1, scaleIntrinsics for Mei, and the no-Ceres stub. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit f78fe80 on fork/mp/work) Adapted during the cherry-pick: the source branch carries the fast-rotation image filter (#526), which this branch does not, and f78fe80 touches the same lines. The filter's machinery -- poseAngSpeedDeg, poseAngSpeedMax, filterFastImages, maxImageAngSpeedDeg, angFilteredImgs and the angularSpeedDegAt()/computePoseAngularSpeedDeg() helpers -- was dropped rather than carried along: the calls came across in the conflicted hunks but their definitions did not, so keeping them would not have compiled. Bringing the filter here is a matter for cherry-picking #526 on its own. What the commit actually contributes is kept whole: the intensity drawing, the Camera.cpp fold-back guard ported from #527, and the imageTimeOffsetMs camera/LiDAR clock offset with its imageOffsetNs() plumbing. Note that imageTimeOffsetMs has no writer -- no widget, CLI flag or calibration-file key sets it, here or on the source branch -- so it stays 0 and the offset plumbing is inert until something wires it up. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The calibration loader reads "mei"/"insta360_mei_v2" and xi, the 5 cm near clip now applies to Mei as well as pinhole (its depth is a range rather than a z, but 5 cm means the same thing physically), and every non-pinhole model draws an axis triad instead of a frustum -- a fisheye sees far more than the pyramid fx/fy/cx/cy imply. Rectification and the COLMAP export are gated on Pinhole: initUndistortRectifyMap would mis-warp a fisheye rather than rectify it, and no COLMAP camera type carries an xi. The ROS 2 export reports the rig's own insta360_mei_v2 distortion tag rather than claiming plumb_bob, with xi appended to d since CameraInfo has nowhere else to put it. Also: image filenames parse as "<any prefix>_<timestamp_ns>.jpg", so the 360 rig's per-lens frames work without the parser knowing the list of rigs, and the camera/LiDAR clock offset becomes timeOffsetSec (seconds, double), applied to the stamps the ROS 2 bag is written with as well. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The mask is the free-form counterpart of the ROI: a point projecting onto a masked-out pixel stays uncolored, which is what it takes to drop the operator and the rig itself out of a 360 frame -- no rectangle can cut those out without cutting out the scene with them. Loaded from any image OpenCV reads (File > Open Image Mask..., the Calibration panel, or drag & drop) and thresholded to a strict 0/255, so a hand-painted PNG and a jpeg with compression noise behave the same. It is resampled to whatever size the frames are actually read at, so its own resolution doesn't have to match: one drawn over a downscaled copy of a frame works as well as a full-res one. Rejected pixels are tinted red over the image preview, and the "In ROI" point color mode becomes "In ROI / mask", painting what either filter rejects. The ROI had the scale mismatch the mask sidesteps: it is specified in full-resolution pixels (calib::Roi) but was compared against pixels of the downscaled images the colorizer reads, stretching it by 1/imgScale at any Image scale below 1. calib::scaleRoi now scales it alongside the intrinsics. It rounds both edges and subtracts, rather than scaling the width on its own, so abutting rectangles cannot come back overlapping, and it never lets a non-empty rectangle collapse to w/h == 0 -- the sentinel every caller reads as "no ROI set", i.e. accept everything. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ominator
The Mei domain guard was "Xs.z + xi > 0", which is the right limit only for
xi <= 1. For xi > 1 -- both rigs in use are calibrated at xi = 2.0 -- that
expression never goes non-positive (Xs.z + 2 stays in [1, 3] for every
direction on the sphere), so the guard never fired and every direction,
including straight backwards, produced a finite pixel.
It is worse than an unfiltered projection: r(theta) = sin/(cos + xi) is only
injective up to its turning point at cos(theta) = -1/xi, and past that the
radius shrinks again, folding far-off-axis directions back onto real pixels
rather than pushing them out of frame. At theta = 180 deg the radius is
exactly 0, so a point directly behind the camera lands on (cx, cy) -- dead
centre. That is the "masked points still get colours" symptom: geometry
behind the rig painting into the middle of the image.
The limit both regimes share:
xi <= 1: Xs.z > -xi (blow-up; reduces to Pinhole's pc.z > 0 at xi = 0)
xi > 1: Xs.z > -1/xi (fold-back)
applied in calib::projectPoint and in both of camera_lidar_calibration's GLSL
shaders -- kProjVS's clip weight becomes the distance inside that dome rather
than the denominator, and kPointVS's fragCamDepth carries the same sign for
the Camera-RGB "seen" test.
For a real lens the calibrated image circle is tighter still (the 3840x3840
xi = 2.0 front lens reaches its edge at theta = 101.2 deg, i.e. ~202 deg FOV,
while the fold is at 120 deg), so the bounds check remains the binding
constraint in normal use; this only removes the directions that were bypassing
it entirely.
Tests replace a case that asserted the old behaviour outright ("xi >= 1 covers
the full sphere -- straight behind still succeeds") with the cutoff at
acos(-1/xi), acos(-xi) and the xi = 0 pinhole half-space. The helper building
those directions needs an explicit -> Eigen::Vector3f: with auto it deduces an
expression template holding a reference to the temporary, which dangles and
made the tests report nonsense.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ry viewer Upstream's fix for colorizing invalid points (#527) applies the rational model's fold-back cutoff inline in TrajectoryViewer.cpp, against local d_k1..d_k6 distortion variables. This branch had already moved that same logic into calib_core when coloring switched to calib::projectPoint, deleting those locals with the rest of the inline distortion math. Both sides merged without a textual conflict, leaving a call to maxValidRadiusSq() with arguments that no longer exist -- so the app did not compile after the rebase. Removes the reinstated copy of the function and its call site. The cutoff itself is unchanged in behaviour: calib::projectPoint runs it internally (Camera.cpp's cachedMaxValidRadiusSq), which is where the equirectangular commit deliberately put it so every caller is covered rather than this one call site. rMaxSq had no other reader. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
build-ceres/ and rosbags.zip sat untracked and unignored, so `git add -A` would sweep 424 MB of build output into a commit. /build-* follows the anchored style of the existing /build2 and /build3 entries and covers future variants; /rosbags.zip is anchored to the root so the tracked rosbags/ source directory is unaffected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
saveCalibrationJson writes camera_pose.inverse(), which is the R_wc/C the shared schema calls for, but loadCalibrationJson read those fields straight back into camera_pose without inverting. The two are not the same thing: the *.reg file stores camera_pose verbatim (the LiDAR-to-camera transform, p_cam = M*p), while the JSON's camera_rotation_matrix_in_world and camera_position_in_world_xyz are R_wc and C (p_cam = R_wc^T * (p - C)). Reading them the same way left the pose inverted, which mis-projects plausibly rather than failing, and meant loading a file this app had just saved did not reproduce the pose it saved. Load now converts, so the pair round-trips exactly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ribution calib_core had picked up find_package(yaml-cpp REQUIRED), but nothing else in the repo uses yaml-cpp, no CI workflow installs it, and Windows/macOS take their dependencies from pre-downloaded binaries with no equivalent path for it -- so the build configured only where it happened to be installed already. cv::FileStorage cannot substitute (it rejects plain YAML without a %YAML:1.0 header), so LoadMeiCamera now parses this rig's flat camera_info.yaml directly, covered by tests. MeiCamera::Project takes Eigen rather than cv::Point, which drops OpenCV from calib_core's public interface and restores its "nothing but Eigen/LASzip/std" rule. Also: - modelToString/modelFromString were declared static in Camera.h. At namespace scope that is internal linkage, so every includer warned about an unused function and the definitions in Camera.cpp were never exported -- App.cpp's call sites would not have linked. Their -Wswitch rationale moved across with them. - Removed MeiCamera::Unproject and AppState::imageTimeOffsetMs, neither of which had any caller; imageTimeOffsetMs also duplicated timeOffsetSec and documented behavior it did not have. - Camera.h's CameraModel comment claimed camera_lidar_calibration neither reads nor writes the "model" key, which all three of loadIntrinsics, loadCalibration and saveCalibration now do. - Fixed 55 clang-format 21 violations across App.cpp, TrajectoryViewer.cpp and manual_color.cpp; apps/ is clean again under the version CI runs. - Camera.h's documentation is now doxygen, matching the //! and @PARAM style the rest of the codebase uses. - Trimmed comments that restated the code or speculated about future work, and dropped a personal absolute path and a pointer to a file that does not exist. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Matches the //! and @param/@return/@note style Camera.h and the rest of the codebase use, so calib_core's public headers no longer mix two conventions. Struct members are documented individually, and the contracts that were buried in prose -- what each function returns on failure, which pixel frame each solver expects, that errorMessage is non-defaulted so a failure reason cannot be dropped -- are now tagged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Completes the conversion, so every documented declaration in calib_core's public headers now uses //! with @param/@return/@note. Code is untouched -- these two predate the .clang-format brace style and calib_core is outside the format check's scope, so only the comments changed. Two contracts that were previously implicit are now written down: Trajectory::nearest binary-searches, so it assumes poses are sorted and gives an arbitrary answer otherwise, and it clamps to the first or last pose rather than failing when the timestamp falls outside the trajectory. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
MeiCamera duplicated ten fields that Intrinsics already had, so the same numbers were copied twice: once from the yaml into a MeiCamera and then into an Intrinsics on load, and again from Intrinsics into a throwaway MeiCamera on every projectPoint call. The struct and its header are gone. loadMeiIntrinsics fills an Intrinsics directly, and the unified-sphere projection now lives in Camera.cpp's Mei branch beside the pinhole and equirectangular ones -- which is also where its domain guard already was, so the math and the guard that protects it are no longer split across two files. src/MeiCamera.cpp becomes src/MeiIntrinsics.cpp, now purely the yaml reader. frameId is dropped, nothing having read it. distortion_model is still checked and still warns when it is not insta360_mei_v2, but stays internal to the loader rather than being exposed as a field. The test that checked projectPoint against MeiCamera::Project had nothing left to compare against, so it is replaced by one pinning the projection to reference values captured from the previous implementation -- they pass unchanged, so this refactor is behavior-preserving. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Frames now go out exactly as captured and CameraInfo always carries the real distortion, so consumers that want rectified images undistort from it. Rectification only ever applied to Pinhole anyway -- OpenCV's initUndistortRectifyMap has nothing to say about a 360 panorama, and Mei's k1/k2/k3/p1/p2 are its own polynomial applied after a unit-sphere step a K/D pair cannot express -- so it was a per-model special case guarded by a checkbox that was disabled for two of the three models. Removing it also removes the re-encode it forced: with nothing to rectify, the compressed path copies the source jpeg verbatim in every case rather than decoding and re-encoding it, so exported images no longer lose a generation of jpeg quality. imageFiles is jpeg-only (see imageTsFromName), so the verbatim copy is safe. The undistortCamera option, the rectify maps, and the now-unused calib3d and imgproc includes are gone with it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Cuts design history, narration of what other files do, and a block of commented-out dynamic-subsampling code that git still has. Comment blocks of five lines or more drop from 171 lines to 106, with none over 10. File-scope declarations and AppState members now use //! (//!< for trailing member comments), matching Camera.h and the rest of calib_core; @param/@return are added where a function has an out-parameter or a non-obvious failure case. Section dividers and in-body comments stay plain //. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Cuts the rationale that narrated step2's internals and the camera-framing formula. Blocks of five lines or more are now 13 / 77 lines, down from 25 / 171 before this pass started. Also fixes a misattributed doc block: the mask commit inserted actionOpenMask directly beneath handleDroppedPath's comment, so actionOpenMask was documented as the drag & drop handler while handleDroppedPath carried a near-duplicate of the same text further down. actionOpenMask now describes itself, and the drag & drop description survives once, on the function it belongs to. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Michał Pełka <michalpelka@gmail.com>
PointCloud.cpp includes nlohmann/json.hpp but calib_core never added the bundled 3rdparty/json/include dir to its include paths, so the header wasn't found on macOS/AppleClang builds. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- App.cpp: drop a stray blank line before a closing brace that clang-format flags. - test_camera.cpp: widen the epsilon on the two "top edge" pole assertions. asinf(-1) isn't correctly rounded on every platform's libm (macOS/AppleClang's included) -- the derivative of asin is infinite at the pole, so even a 1-ULP wobble there gets amplified through the pixel-height scale. doctest::Approx's default epsilon is an absolute tolerance too tight for that when comparing against 0, which made these two checks flaky on macOS CI. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…b to stderr Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…, tf) New console tool that writes a processed lidar_odometry_step_1 session (session.json) to an MCAP file containing already-undistorted lidar points placed in the map frame, a /tf stream of map->lidar transforms taken from each chunk's local_trajectory, and optionally /imu re-read from the original mandeye recording directory (a session keeps no raw IMU samples). Extends rosbags::McapWriter with tf2_msgs/msg/TFMessage support (hand-rolled ros2msg/CDR encoding, same approach as the existing PointCloud2/Imu channels, no real ROS dependency) and a decoupled point-cloud frame_id so a cloud can be published in a different frame than the Imu/tf child frame. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Few rather large changes to work with camera.
hdmapping-camera_lidar_calibration(currently with Ceres)hdmapping-camera_lidar_trajectory_viewer.meta.jsoninhdmapping-camera_lidar_trajectory_viewerfor build-in camera.