PiperTTS for Unity

Cloud text-to-speech sounds good but adds a network round-trip to every sentence and stops working without a connection. With PiperTTS for Unity we put a neural voice inside the app instead, so synthesis runs entirely on the user's device.

Architecture

The voices are Piper models: VITS networks exported to ONNX, compact enough to ship in an app. Rather than wire ONNX Runtime, a phonemizer and the model together separately on every platform, we use sherpa-onnx as a single C/C++ runtime that bundles ONNX Runtime, eSpeak-NG and piper-phonemize behind one C API. Unity talks to it through a thin P/Invoke layer.

• Windows and macOS load sherpa-onnx as a shared library.
• iOS links it statically, as the App Store requires.

Data flow

1. Text goes through a preprocessing step that strips expressions and emotes not meant to be spoken.
2. The manager component dispatches synthesis to a background thread, so the main thread never blocks.
3. sherpa-onnx phonemizes the text with eSpeak-NG and runs the VITS model.
4. The PCM samples come back to the main thread as a Unity AudioClip and play through a normal AudioSource.

Practical details

Adding a voice means dropping its folder into the project. Piper models published on Hugging Face lack the metadata sherpa-onnx expects, so the repo includes tools that generate the phoneme token map and inject that metadata. eSpeak-NG's compiled phoneme tables and dictionaries aren't produced by the sherpa-onnx build, and the docs cover where to get them. We also tracked licensing: eSpeak-NG is GPL-3.0, which matters when linking statically on iOS.

We built PiperTTS as part of our on-device voice research for Frndo, whose app uses a different, server-side voice. Its engine code is reused in OpenVoice for Unity.