Whisper.Sentis

Speech recognition in an app usually means sending audio to a cloud service or depending on whatever the platform provides. In Whisper.Sentis we run Whisper inside Unity itself, on Unity Sentis (the engine Unity later renamed Inference Engine), so transcription stays on the device and works the same on every platform Unity targets.

How it's built

Whisper is split into separate ONNX models that Sentis loads at runtime:

• a log-mel spectrogram model that turns 16 kHz audio into Whisper's input features;
• the audio encoder;
• a decoder, plus a decoder-with-past variant that reuses cached keys and values so each new token doesn't recompute the whole sequence.

A shared base class runs the encode and decode loop and turns tokens back into text with Whisper's vocabulary. The project has two scenes: one transcribes prerecorded clips, the other transcribes live.

Real-time transcription

Whisper works on fixed windows of audio, not a stream. The real-time component bridges that gap:

1. Microphone samples are appended to a ring buffer.
2. Every two seconds, the latest three seconds of audio are pulled out as a chunk, so consecutive chunks overlap by a second and words at a boundary aren't cut in half.
3. Each chunk is transcribed, and overlapping text is detected by matching the end of the previous result against the start of the new one, so it isn't repeated.
4. When listening stops, leftover samples are padded into one final chunk.

Sentis reference

We also published a reference guide to Unity Sentis: model loading, tensors, workers and backends, the functional API for editing a model graph, texture conversion, and performance and troubleshooting notes, with code examples throughout.

This was research into on-device speech for Frndo, whose app uses Apple's on-device recognizer instead. Related on-device ML work in Unity: Frndo's lip-sync system, which runs its viseme network on Sentis.