A fisheye image looks distorted because we are accustomed to photographs that approximate perspective projection over a smaller field of view. Straightening every line is not necessarily the right goal. For a wide-angle camera, the more useful question is which direction in space produced each pixel.
The older paper Fish-Eye-Stereo: Calibration and Epipolar Rectification, by Steffen Abraham and Wolfgang Forstner, is a helpful reference here. Published in the mid-2000s, it treats calibration and stereo rectification together, including projection models designed for very wide fields of view. The geometry remains relevant even when the surrounding capture hardware changes.
A simple comparison explains why the model matters. For a perspective camera, image radius grows with the tangent of the angle from the optical axis. Near a right angle, that relationship becomes unbounded. An equidistant fisheye model instead makes radius proportional to angle. These are different mappings, not merely stronger and weaker settings of one cosmetic distortion effect.
Calibration estimates the parameters of the chosen mapping and the deviations of the real lens from it. A model can fit the central portion of an image reasonably well and still fail near the edge. If calibration observations barely reach that edge, a low average fitting error can give a false sense of confidence.
Stereo adds the relationship between two cameras. A point visible in one view constrains where its match can appear in the other. Rectification transforms the images into a representation that makes that search easier, often by arranging corresponding candidates along matching rows.
With fisheye coverage, forcing everything into an ordinary perspective image can sacrifice much of the field of view or stretch the boundaries severely. The rectification representation must be chosen for the rays it needs to retain. A visually familiar rectangular preview may be a poor computational representation of the captured directions.
Resampling also spends information. Some output pixels stretch a small region of the source, while others compress many source samples together. Enlarging the output image doesn't create new optical detail in the stretched region. The mapping changes how the existing samples are distributed.
For a practical check, I would use features at different depths and positions across the overlap, not only a tidy target in the center. A pair of views can appear aligned on one distant surface while disagreeing strongly on a nearby object. The test should exercise the geometry the final application depends on.
What I like about calibration work is that it gives the pretty preview a precise foundation. Once a pixel has a trustworthy ray, stitching, depth estimation and headset presentation can each build on the same understanding. Without that agreement, every later correction risks becoming a special case for the current image.