A video can decode correctly and still look completely wrong in a headset. The frame may contain two fisheye images, but a decoder only returns pixels. It doesn't know which pixels belong to each eye or which direction in space each sample represents.
This month, my VR180 broadcast work has been moving through that boundary between video and geometry. The experiments include camera mesh data, a parser and work on the mesh-processing path. What interests me is how much of the viewing experience depends on information that a normal flat video player can ignore.
The pixels need an interpretation
Suppose a frame contains a left-eye fisheye image beside a right-eye fisheye image. Drawing the whole frame onto a rectangle gives a useful diagnostic preview. Drawing it onto a sphere with arbitrary UV coordinates does not automatically produce the intended stereo view. The lens projection and the packing layout have to agree with the geometry used for playback.
Google's VR180 format description builds on Spherical Video Metadata V2. Mesh projection allows the file to describe how image samples map into viewing directions. Camera motion metadata is a separate component that can support stabilization. Encoding the picture and describing its spatial interpretation are related tasks, but they are not the same task.
The codec delivers pixels; the projection and stereo layout describe how a compatible player should present them.
A mesh representation is useful because it doesn't force every camera into one idealized fisheye equation. Vertices and texture coordinates can describe a mapping that the player renders. The image remains a normal video texture; the geometry tells the renderer how to present it.
Yasuhiro Taniuchi's VR180 Mesh Projection Box Parser provides a concrete example. It reads the relevant metadata and constructs a Unity mesh for visualization. The binary parsing and the Unity presentation are distinct pieces, which makes the project useful for understanding both sides of the boundary.
Parsing is part of playback
An MP4 file is structured into boxes with types and lengths. A parser has to navigate those containers and interpret the payload of the relevant projection box. A malformed length or an unexpected structure should produce a controlled failure, not a mesh built from arbitrary bytes.
Even valid parsed numbers can be interpreted incorrectly. Coordinate handedness, triangle winding, texture origin and eye assignment all matter. Swapping the stereo views can reverse depth. Flipping a texture coordinate can create an image that seems partly recognizable while placing every feature in the wrong direction.
A useful inspection sequence starts with simple evidence. Show each eye as a flat texture. Display the projection mesh without the video. Then combine one eye with its mapping before enabling stereo. Each stage checks the interpretation required by the next.
The spatial-media tools and specification expose another distinction. Adding metadata can tell a compatible player what a file means. It cannot make an incorrectly captured or incorrectly packed image geometrically correct. A label and the data it labels have to agree.
A stream has to preserve the contract
Broadcasting adds an operational question: does the spatial description survive the route from producer to player? A process that re-encodes, repackages or copies selected tracks may preserve the visible frame while losing information needed for immersive playback. A successful flat preview is therefore an incomplete end-to-end check.
I want the debugging view to make that contract visible: which projection was read, which eye uses which image region, and which geometry the player constructed. These are small details compared with the apparent complexity of streaming video, but a mistake in any one of them can make the final experience unusable.
VR180 is a good reminder that a media format includes meaning as well as compression. The codec gets the pixels to the viewer. The projection data tells the viewer where those pixels belong.