I started a small stitching experiment called theta_stitch this month. The problem looks compact when reduced to two circular images. Map them somewhere sensible, find the overlap, and join them. Almost every word in that description hides a decision.
A fisheye image is not simply a photograph with unusually bent edges. Its pixels represent directions according to a lens projection. Before comparing pixels from two cameras, the pipeline needs a relationship between those pixels and rays in space. Otherwise a seam adjustment can accidentally become compensation for an incorrect lens model.
Choose a space in which to compare
An equirectangular panorama represents horizontal and vertical directions on a rectangle. That makes it convenient for delivery and inspection, but it introduces its own distortion. Near the poles, a large horizontal stretch of the image corresponds to a small neighborhood of directions.
For a conceptual stitching pipeline, I would separate the operations. First interpret source pixels through the lens model. Then orient those rays relative to the other camera. Finally sample them into the chosen output projection. Keeping the steps distinct makes it easier to ask which assumption is responsible for a visible error.
Nearby and distant objects shift differently between two lens positions. One global alignment cannot remove all of that parallax.
For example, a rotated horizon suggests a different problem from a vertical object that doubles only near the seam. A brightness band suggests another class again. If every adjustment is hidden in one large warp, correcting one symptom can make a second symptom harder to understand.
The near object refuses to cooperate
Two physical lenses do not occupy exactly the same point. A nearby object therefore moves against the background between their views. Align the background perfectly and the object may split. Align the object and the background may shift.
Blending spreads that disagreement over an area; it does not remove it. A soft transition can hide a mild exposure change while making a doubled edge more obvious. A hard transition can preserve sharpness but reveal a sudden jump in position.
The useful question becomes where the seam should pass and which view should supply each region. A featureless wall gives the seam somewhere quiet to go. A person's face or a thin railing gives it nowhere forgiving. That is why a visually simple test scene can be more revealing than an attractive landscape.
Facebook's stereo stitching discussion is a helpful larger reference for these geometric difficulties. A two-camera setup has fewer inputs, but it does not escape the fact that different viewpoints see different information.
For this experiment, the productive unit is a diagnosable error rather than a finished panorama. Lens mapping, orientation, color matching and seam placement should each have an output that can be inspected. A clean final image is welcome. Knowing why the unclean image failed is what makes the next one possible.