Xiaomi open-sources OmniVoice: a model that covers voice clones of 646 languages, trained with purely open-source data to outperform commercial systems.

According to Beating, Xiaomi AI Lab's next-generation Kaldi team has open-sourced OmniVoice, a zero-shot speech cloning TTS (text-to-speech) model supporting 646 languages. It can clone timbres using only a few seconds of reference audio, and it works across languages: given a Chinese recording, the model can speak Japanese, Korean, or other languages using the same voice. The code, weights, and training data are all open-source and licensed under Apache 2.0. OmniVoice adopts a minimalist architecture. The entire model uses only a single bidirectional Transformer, directly mapping from text to multi-codebook acoustic tokens (discrete encodings of sound), eliminating the need for a two-stage pipeline of first converting to semantic tokens and then to acoustic tokens. Two key design features underpin this simple structure: a full-codebook random masking strategy improves training efficiency, and pre-trained parameters from a large language model are used for initialization to improve pronunciation accuracy. Inference speed is 40 times faster than real-time, and it runs directly in PyTorch without additional optimization. The training data comes from 50 open-source speech datasets, totaling 580,000 hours after noise reduction and quality screening. For languages with limited resources, dynamic upsampling ensures effective training. In tests with 24 languages, OmniVoice outperformed many commercial systems in both speech similarity and intelligibility. In tests with 102 languages, its intelligibility was close to or even better than real recordings. It can even synthesize speech in less common languages with less than 10 hours of training data. In addition to speech cloning, the model supports customized voice timbres for text descriptions (e.g., "male, middle-aged, very low tone" or "female, young, Sichuan dialect"), automatic noise reduction from noisy reference audio, insertion of intonation marks such as laughter and sighs, and pronunciation correction for polyphonic characters and proper nouns in Chinese and English.