Last updated 08-15-2026
Category:
Reviews:
Join thousands of AI enthusiasts in the World of AI!
VALL-E
VALL-E is a Microsoft Research text-to-speech model that clones a speaker's voice from a short audio clip and generates new speech from text. It belongs to the audio generation category because it synthesizes natural speech rather than editing existing recordings. The project page hosts sample audio for the original VALL-E model and later variants in the same research family.
Most TTS systems regress continuous waveforms or mel-spectrograms directly. VALL-E instead treats speech as a conditional language modeling task over discrete codes from a neural audio codec. Microsoft trained it on about 60,000 hours of English speech, far larger than typical TTS datasets. A 3-second enrolled recording of an unseen speaker is enough to drive zero-shot synthesis, and the model can keep the emotion and room tone present in that prompt.
The VALL-E family grew beyond the first paper. VALL-E X handles cross-lingual zero-shot TTS, VALL-E R adds phoneme monotonic alignment for more stable speech generation, and VALL-E 2 pairs repetition-aware sampling with grouped code modeling to reach human parity on LibriSpeech and VCTK benchmarks. Related lines like MELLE, FELLE, and PALLE explore continuous mel tokens and hybrid autoregressive plus parallel decoding.
Researchers, speech engineers, and curious listeners use VALL-E to hear what large-scale codec language models can do before building their own pipelines. The public samples are research demos, not a hosted API you can plug into a product without separate licensing and ethics review.
Clones an unseen speaker from a 3-second enrolled audio prompt
Pre-trained on about 60,000 hours of English speech data
VALL-E 2 reports human parity on LibriSpeech and VCTK zero-shot benchmarks
Preserves speaker emotion and acoustic environment from the prompt clip
VALL-E X extends zero-shot synthesis to cross-lingual scenarios
Sample pages cover seven model lines including MELLE, FELLE, and PALLE
Zero-shot cloning from a 3-second prompt removes the need for per-speaker fine-tuning in research demos
Codec language modeling scales to 60,000 training hours, which is unusually large for TTS
The public sample hub links VALL-E through VALL-E 2 plus MELLE, FELLE, and PALLE variants
Published ethics guidance covers misuse risks like voice spoofing and impersonation
The original valle-demo.github.io GitHub Pages site now returns a 404 error page
No hosted product API on the sample pages, only research audio demos
Voice cloning misuse risks mean real deployments need consent protocols and detection tooling
What is VALL-E?
VALL-E is a Microsoft Research neural codec language model for text-to-speech. It generates speech in a new speaker's voice using a short audio prompt plus text, treating synthesis as language modeling over discrete audio codes.
How long must a VALL-E voice prompt be?
VALL-E can synthesize speech from about a 3-second enrolled recording of an unseen speaker. Microsoft notes that similarity and naturalness still depend on prompt length, background noise, and recording quality.
Is VALL-E free to use?
VALL-E research sample pages are free to browse and listen to on Microsoft's site. There is no public self-serve API on the demo page, so product teams need separate research or licensing paths.
What datasets did VALL-E train on?
The original VALL-E paper describes pre-training on roughly 60,000 hours of English speech, far more than typical TTS corpora. VALL-E 2 evaluation references LibriSpeech and VCTK zero-shot test sets.
Does VALL-E support multiple languages?
The base VALL-E model targets English, but VALL-E X adds cross-lingual zero-shot TTS so a speaker prompt in one language can drive speech in another. Sample pages list separate audio for VALL-E X.
Can VALL-E copy a speaker's emotion?
Yes. Microsoft reports that VALL-E can preserve the emotion and acoustic environment captured in the prompt recording, and the ethics page warns that voice similarity depends on prompt quality and noise.