A row of body keypoints is an appealing input for an interactive scene. It suggests that a person's motion can become a cursor, a particle source or an animation control without requiring a wearable sensor. But coordinates are only the beginning of the interface.
The OpenPose Unity Plugin, by Tianyi Zhao, Gines Hidalgo and Yaser Sheikh, wraps OpenPose output for Unity and includes examples. That bridge removes some integration work. It doesn't decide what the measurements should mean inside a particular scene.
Start with a simple example: use a detected wrist to move a bright point over the camera image. The mapping needs to account for the image dimensions, its orientation and whether the preview is mirrored. A person expects their reflected hand to move one way in a selfie-style display; a world-space camera view may use a different convention. Mixing those conventions produces an interface that feels wrong even when the detector is accurate.
Moving a 3D character is a larger step. A two-dimensional joint location does not uniquely determine depth, bone orientation or body scale. Several poses can project to similar image positions. Retargeting therefore requires assumptions, constraints or additional measurements. A skeleton drawn convincingly over video is not automatically a complete motion-capture solution.
Confidence should be treated as part of the data. If a wrist is hidden behind the body, the application must choose whether to hold its last position, fade the effect, estimate a continuation or stop using that joint. Each choice creates a different experience. Snapping to an unreliable estimate is usually the most visible way to expose uncertainty.
Smoothing can reduce jitter, but it introduces delay. A heavily filtered hand may look calm while trailing noticeably behind the person. The correct balance depends on the effect. A slow background response can tolerate more filtering than a cursor that is supposed to touch a small target.
Identity across frames is another design concern when several people are present. The application shouldn't assume that the first detected skeleton always belongs to the same visitor. A change in ordering can make one person's effects jump to another body. Stable association is an application requirement separate from detecting plausible poses in one image.
For an installation, I would begin with one robust gesture or one body region and make the tracking state visible. A visitor should understand where to stand and whether the system can see them. Adding more joints is useful only if it supports a clearer interaction.
What makes the Unity bridge valuable is the speed at which these ideas can become spatial prototypes. The remaining work belongs to interaction design: deciding how uncertain measurements become legible, forgiving behavior. A detector sees keypoints. An interface has to help a person understand what to do with them.