A reconstruction can look almost right while being wrong in several different ways. A wall bends slightly. Two views disagree at the edge of a person. A scene seems to move when the camera rotates. The temptation is to tune the reconstruction algorithm. Sometimes the disagreement begins earlier, in what the system believes the sensors measured.
Kalibr makes that foundation explicit. Its calibration tools cover camera intrinsics, relationships between cameras, and spatial and temporal alignment between a camera system and an IMU. Those are distinct problems even when the final application needs all of them at once.
Intrinsics describe how a camera turns a ray into an image position. Focal parameters, the principal point and lens distortion affect that mapping. Extrinsics describe where one sensor sits relative to another. A point measured in the left camera's coordinates needs a rotation and translation before it can be compared with a point measured by the right camera.
Time is the less visible coordinate. Two correctly calibrated cameras can still disagree if their images describe different instants. A person moving sideways may appear in two positions. An optimizer might try to explain the discrepancy with geometry, but no fixed geometric correction can make both instants become one.
A camera and IMU make this particularly clear. The IMU reports motion at a high rate, while the camera observes changes in the image. If one stream is shifted in time, a sharp rotation produces measurements that seem inconsistent even though each sensor is behaving normally. Estimating the offset is part of making their observations comparable.
The calibration recording must also contain informative motion. If a rig only moves in one direction, some relationships remain difficult to distinguish. If a calibration target stays near the image center, the lens behavior at the edges is weakly observed. More frames of the same limited view may add less useful information than a shorter recording with varied coverage.
I would inspect residuals as pictures, not only as one average number. An error that grows toward the edges suggests a different problem from an error that changes with movement speed. A consistent directional offset has a different character from random noise. The pattern can point toward the assumption that needs attention.
This matters for systems such as LSD-SLAM, where image measurements are used to estimate motion and scene structure. The algorithm cannot recover a perfect world from an internally inconsistent description of its camera. Calibration errors become part of the evidence it is trying to explain.
I find this stage satisfying because it replaces vague visual discomfort with testable relationships. Which ray produced this pixel? In whose coordinate system is that point expressed? When was the measurement taken? Once those answers agree, the reconstruction algorithm has a much fairer problem to solve.