A photograph changes when I move past it because I see the paper from a different angle. The scene inside it doesn't respond. Nearby objects don't slide against the background, and moving my head cannot reveal the part of a wall hidden behind a chair.
View synthesis tries to supply those missing changes. Google's DeepView project learns to construct multiplane image representations from several photographs. These layers can then be rendered from nearby viewpoints. The project combines a geometric representation with a learned reconstruction process rather than asking a flat image to behave like an unrestricted 3D scene.
A multiplane image is easiest to imagine as a stack of transparent sheets at different depths. Each sheet carries color and opacity. From the original view they combine into one picture. Move the virtual camera, and the sheets shift relative to one another according to their depth before being composited again.
The layers matter at occlusion boundaries. If a chair sits in front of a wall, a single depth value and color at each source pixel describe only the visible surface. When the viewpoint moves, new wall pixels should appear beside the chair. A layered representation has somewhere to store background information that another input camera observed.
Not every layer needs to be an opaque surface. Transparency allows the representation to distribute appearance across depth hypotheses. That flexibility can help it reproduce observations, although it also means the layers shouldn't be interpreted as a perfectly measured mesh of the physical scene.
DeepView's learned-gradient-descent approach uses the relationship between rendered predictions and input images to guide reconstruction. The model learns how to refine the representation. Geometry supplies a structured way to render and compare views; learning helps determine what information belongs in the representation.
I find this middle ground appealing. A conventional mesh can be difficult to reconstruct around thin structures, reflections or weakly textured surfaces. A collection of unrelated photographs preserves appearance but makes movement difficult. Layered images provide enough spatial organization to support useful motion without requiring every surface to become a clean polygonal object.
The valid movement region still comes from the evidence. A collection of cameras facing one side of an object doesn't reveal its hidden back. As the requested viewpoint moves farther from the observations, missing content and incorrect depth assignments become more visible. A smooth camera move is a stronger test than a few selected still frames because discontinuities have nowhere to hide.
For VR, a small amount of convincing local parallax can already be valuable. Leaning around a foreground object helps the scene feel spatial. The capture doesn't have to promise that I can walk anywhere. It needs to make the permitted movement coherent, and to avoid inviting movement beyond the information it contains.
The result is a different kind of image: one that includes a limited set of answers to where the viewer might go next.