Detecting a person tells a program which pixels belong to them. Removing that person asks a much harder question: what should those pixels become?
Jason Mayes's Real-Time Person Removal experiment makes the distinction unusually clear. It runs in the browser with TensorFlow.js and builds up knowledge of the background over time. The effect is compelling because the visible room continues to move and change while a person disappears from it. Mayes describes it as an experiment, with situations where it will not work perfectly.
The background is the essential ingredient. Suppose somebody walks across a bookcase. Before they arrive, the camera can see the shelves. While they pass, a mask identifies their silhouette and the program can substitute previously observed shelf pixels. Once they leave, the live image becomes useful again. The system has borrowed information from another moment.
Now suppose the person has been standing in front of the bookcase since the camera started. There is no observation of the hidden books. A program can retain an older image if one exists, or invent a plausible replacement, but those are different promises. A convincing patch is not evidence of what was actually behind the person.
This is why camera motion is such a demanding case. A saved background pixel refers to a particular viewing direction. Move the camera and the bookcase shifts, while nearby objects shift by different amounts. Reusing old pixels now requires alignment, and a single flat transformation may not explain the whole room.
The boundary keeps moving
The silhouette introduces another set of compromises. Hair, fingers and motion blur contain mixtures of person and background. A binary cutout produces sharp edges where the original camera recorded soft ones. A softer mask can avoid that cutout look, but it also lets traces of the person leak through. The useful setting depends on how quickly the subject moves and how visible the edge is.
I am particularly interested in the effect over time. One excellent frame proves relatively little. A mask that changes its mind about a sleeve several times a second produces flicker, and a background estimate that updates too eagerly can absorb part of a stationary person. Updating too slowly leaves stale objects behind when the room changes.
A sensible evaluation would include a person entering an empty scene, stopping, crossing in front of moving objects, and leaving. Each case asks a different question. Can the background be learned? Can it be preserved? Can it be updated without swallowing the subject?
The browser is a useful place to expose these tradeoffs because the interaction is immediate. A webcam, a model and a canvas can turn a computer-vision idea into something a visitor understands in seconds. What I take from this experiment is a reminder to look beyond the detector. The replacement image, and its history, carry much of the illusion.