CameraSmartPlate

Matchmoving, recovering where the camera was for every frame, is some of the most labor-intensive work in visual effects. Neural Radiance Fields (NeRF) have the same dependency: a NeRF learns a scene from 2D images, and it needs to know the pose of the camera that took each one. Usually that pose is solved for afterwards from the images themselves.

We built CameraSmartPlate to record the pose instead. A plate that rides with the camera carries a visual-inertial tracking sensor, and every frame it captures leaves with its position and orientation attached.

How it works

The tracking comes from VIO-SLAM. Visual-inertial odometry combines camera images with an inertial measurement unit (IMU) to estimate the device's pose over time, and SLAM uses that to map the environment while tracking the device within it.

• Pose source. The first implementation uses an Intel RealSense T265, which runs VIO-SLAM on board, the same sensor we chose as the base camera for spatial understanding in HoloFlect. A USB camera alongside it supplies the images.
• A device-agnostic manager. A base class handles zeroing the pose to a reference position, reading pose relative to it, taking a snapshot and repeating snapshots at a set interval for time-lapse capture. Each device subclass supplies only how to read a pose and an image, so the system can follow better SLAM sensors and cameras as they arrive.
• Pose in the file. Each snapshot is saved as a PNG with the relative position and rotation written into its metadata, so the image and its pose can't be separated.
• Calibration. A tool computes camera intrinsics and lens distortion from a directory of chessboard images and saves them as JSON. A second, experimental approach solves for intrinsics by least squares when each image's extrinsics are already known from the tracker.

Where it leads

• VFX and post-production: camera tracking without the manual solve.
• Virtual production: CG environments and characters visualized live on set.
• NeRF capture: posed images ready for training, with upgradable cameras.
• AR, VR, robotics and architecture: anywhere real-time mapping and localization help.

It's part of the same push toward sensors that understand space as our xR ecosystem research.