When the camera moves in front of a real window, the view outside shifts: nearby objects slide across the frame faster than distant ones. An LED wall has to reproduce that shift. It does so by tracking the physical camera and moving a virtual camera in the game engine in step with it. If the two disagree, the background slides, floats or swims, and the illusion goes.
What tracking measures
Position and rotation of the camera, six values in all, many times per second. The engine uses them to render the part of the virtual world the lens would see from exactly that spot, and sends it to the wall.
Lens data matters as much as position
Focal length, focus distance and lens distortion change what the camera sees. The virtual camera needs the same values. A zoom that is not reported to the engine makes the background stay the same size while the foreground changes, which looks wrong at once. Lenses are calibrated before the shoot so the engine can match their behaviour.
Common ways to track
Optical. Infrared cameras around the stage watch markers or a tracking device on the camera. It is precise and works well in a fixed space, but needs line of sight and a calibrated volume.
Camera-mounted sensors. A small unit on the camera looks at the stage, using reference points to work out where it is. It travels well and sets up quickly, with accuracy that depends on the reference points staying visible.
Mechanical. Encoders on a crane, dolly or head report position directly. They are very stable for repeated moves, and limited to the equipment they are fitted to.
Latency is what the eye catches
Tracking data, rendering and the wall's refresh all take time. If the background arrives a few frames behind the camera, it lags during a pan. Teams keep that delay small and, where possible, compensate for what is left. Fast whip pans are the hardest test, so we try one in the camera test.
The inner frustum
Rendering an entire wall at full quality is expensive. Instead the engine renders a high-quality window, the frustum, for exactly what the camera sees, and a simpler version for the rest, which lights the set. Move the camera quickly and the edge of the frustum can briefly show. Keeping moves smooth and the frustum slightly larger than the frame keeps it out of shot.
What can go wrong
Reflective surfaces or stray infrared light confusing an optical system
Markers or sensors knocked out of position between takes
A lens swapped without recalibrating
Wall and set not aligned to the same real-world origin, so the floor seems to tilt
What to do before the shoot
Tell the stage which camera, lenses and rigs you plan to use, run the lens calibration for each, and shoot a test with the kind of movement the real scene needs. Tracking problems are cheap to find in a test and expensive to find in an edit.
Have a camera move in mind? Describe it at wlab@wlab.tech and we will tell you how we would track it.
Cover image: an illustration generated for this article, not a photograph of a WLAB shoot.
