Applications

A spectrometer in every pocket changes who can do spectral imaging and where. The provisional patent application lists a broad set of uses, and our own background in film and mixed reality adds several more. Different applications use different parts of the system: some need full spectra, some a handful of tailored bands, some a single derived number.

Science and industry

Agriculture. Plants announce stress in the red and near-infrared before it shows to the eye. Vegetation indices such as NDVI compare those bands, and a phone that can estimate them can check plant health, water stress and nutrient status in the field, without a drone or a dedicated sensor.

Medicine and dermatology. Skin and tissue absorb and scatter light differently depending on blood, oxygenation and structure. Spectral estimates can support tissue characterization and wound monitoring, and the dataset's deliberate coverage of the full range of skin tones matters here more than anywhere.

Cultural heritage and art. Pigments that look alike can have different spectra, and some materials show up only outside the visible range. Spectral imaging helps identify pigments and reveal underdrawings, and a phone-based method makes it possible in a gallery or an archive rather than a conservation lab.

Industrial inspection. Sorting materials, checking coatings and catching contamination on a production line usually means a costly machine-vision camera. Commodity sensors with a spectral mapping can cover many of those jobs at a fraction of the cost.

Environmental monitoring. Water quality, algae and pollution all have spectral signatures. Distributed, cheap spectral sensing suits monitoring that needs many observations over a wide area.

Media production

This is where our own history comes in. Film and visual effects are, at bottom, about controlling and reproducing light, and RGB is a lossy description of light.

Matching under LED light. LED walls and LED lighting have spiky spectra, very different from daylight or tungsten. Two fabrics, two paints or two skin tones that match perfectly in the costume shop can separate on an LED volume, and the camera can see a mismatch the eye doesn't, or the reverse. Spectral captures of costumes, props and makeup on a phone could flag those metameric pairs before the shoot instead of in the grade.

Real material spectra for rendering. Spectral renderers handle dispersion and mixed lighting more accurately than RGB renderers, but they need spectral material data that's slow and expensive to measure. Phone-based spectral capture could feed reflectance spectra straight into a look-development library.

Better plates and better matches. Knowing the spectrum of what's in a plate helps match CG to it under any lighting, and helps match cameras to each other. The same characterization that underlies this method, measuring exactly how a camera's channels respond, is what precise camera matching needs anyway.

Capture that knows where it is. Paired with tracked capture like the CameraSmartPlate, spectral data could be attached to a position and a view, building material maps for virtual production sets. Combined with depth sensing, as the patent application describes, it extends to 3D spectral reconstruction of props and environments, alongside the geometry we already capture with the iOS LiDAR Scanner.

Mixed reality

AR content that looks pasted on often looks that way because its color and shading don't respond to the room's light the way real surfaces do. The provisional names "color-accurate virtual overlays" and "material-aware rendering" as uses. A headset or phone that can estimate both the illuminant and the materials in a room can light virtual objects with the room's actual spectrum and recognize what surfaces are made of, which also helps interaction and occlusion. That links this work directly to HoloFlect and to the sensor-agnostic interfaces of the xR Ecosystem, and to how the MRiot Vision Pro Module gives entertainment hardware an understanding of the people and things in front of it.

Choosing the output

Not every application needs a full cube. The method can output:

• a continuous spectrum per pixel, for research and look development;
• a fixed set of bands, tailored to a task;
• a derived index, such as a vegetation index or a pigment ratio;
• a classification score or anomaly map;
• material parameters for rendering in AR.

Smaller outputs mean smaller models, which is what makes on-device, real-time use practical.