Big Blue Ceiling

North Star Stereoscopic Telepresence

A preview of a fully 3D, ultra-HD video call with realtime perspective shift.

Eddie OffermannHardware selection and prototyping, full software stack, networking, realtime robotics control

Role
Hardware selection and prototyping, full software stack, networking, realtime robotics control
Areas
Telepresence, Hardware, Computer Vision, Research
A stereo camera rig, two cinema cameras with zoom lenses side by side in a black printed cage, hanging from the end of a grey robotic arm in an empty room.

The brief: demonstrate, by any means necessary, the highest possible quality telepresence. That meant 3D, room scale, and an observer free to move, so people could experience what a fully 3D ultra-HD video call with realtime perspective shift might feel like twenty years out.

We proposed the solution, selected the hardware, built the rigs and wrote the software. A pair of cinema cameras sits in a 3D-printed cage at a fixed spacing, and the cage hangs from the wrist of a robotic arm. An additional high-fidelity RGB and depth sensor rode on the rig to collect extra data during development and testing. Realtime head tracking of the observer drives the arm, so the remote viewpoint follows the viewer's head. Fiber networking delivers stereo 4K video to a stereoscopic display in under 16 ms, less than a single frame.

Getting there took a series of smaller experiments: choosing cameras and lenses, tracking a head with a VR controller on a strap, and measuring how fast the tracking could report movement before the arm ever moved.

Part of Enhanced Presence.

  • Putting the motion-control rig through its paces.
  • Measuring tracking latency and movement in realtime.
  • A robotic arm mounted on a black pedestal reaching across a room, with the stereo camera rig at its end.
    The arm moves the cameras to match the remote viewer's head.
  • The full length of the robotic arm extended across the room toward the camera rig, with a wooden door behind.
    The full reach of the arm.
  • Close view of the stereo rig hanging from the arm's wrist: two cameras on a black printed plate with large lenses angled toward the room.
    The stereo pair on the arm's wrist plate.
  • Front view of the two camera lenses side by side under the arm's wrist, in an empty room with a concrete floor.
    The cameras from the subject's point of view.
  • Front view of the stereo rig against a white wall: two camera lenses side by side in a black printed cage, with a grey RGB and depth sensor mounted above them.
    The printed cage holds the stereo pair at a fixed spacing.
  • The stereo pair on a ball head on a desk in a warehouse, with a grey RGB and depth sensor mounted above the lenses.
    An early configuration with an additional RGB and depth sensor above the lenses.
  • Top-down view of the stereo cage on a white surface, with a grey perforated RGB and depth sensor between the two cameras and the lenses pointing down.
    From above, with the additional RGB and depth sensor seated between the cameras.
  • The two cameras side by side in the black cage on a white desk in a workshop.
    The assembled rig before it went on the arm.
  • A white foam mannequin head on a stand wearing a strap with a VR controller mounted on top of it, in front of a whiteboard.
    Head tracking, tested on a mannequin with a tracked controller on a head strap.
  • Side view of the mannequin head with the controller held upright on a printed mount on the head strap.
    The tracker mount from the side.
  • A monitor showing green text readouts of sync rate, sample latency, signal latency and movement figures beside a polar grid.
    Latency readouts from the tracking test.
  • A curved monitor showing two polar plots labeled XY and YZ with crosshairs and a running timer.
    Plotting head movement on two planes.

  • Enhanced Presence

    A run of prototypes chasing the feeling of being in the room with someone far away.

    • Telepresence
    • XR
    • Hardware
    • Computer Vision
    • Research

  • Glasses-Free 3D Telepresence

    Volumetric calls on a 45-inch autostereoscopic display, no glasses required.

    • Telepresence
    • Hardware
    • Computer Vision
    • Research

  • Projection AR Tabletop Gaming

    Augmented reality projected onto an ordinary table, with touch.

    • XR
    • Hardware
    • Computer Vision
    • Telepresence
    • Research

  • Transparent Volumetric Display

    A see-through, desktop-sized display that shows a remote person with realtime parallax.

    • Telepresence
    • Hardware
    • Computer Vision
    • Research