VibeVoice: Microsoft opens its voice AI with 60-minute transcription, real-time TTS and 50 languages

VibeVoice is Microsoft's commitment to cutting-edge voice AI in open source: an ecosystem that combines speech recognition (ASR), speech synthesis (TTS) and real-time generation in the same technical framework. The GitHub repository exceeds 49,000 stars, with documentation, demos in Colab and weights published in Hugging Face. For those who work with podcasts, meetings, accessibility or voice assistants, VibeVoice offers something unusual in open models: process up to an hour of audio without chopping it up and return who spoke, when and what they said.

What is VibeVoice and why it matters

Most conventional ASR systems break audio into short fragments—often 30 seconds—and lose overall context: speaker changes, cross-references, or technical terms that only make sense at the end of an hour of recording. VibeVoice-ASR attacks that problem with a 64K token context window that supports up to 60 continuous minutes in a single inference, according to the technical report and official documentation.

Instead of chaining ASR + diarization + temporal alignment into three distinct pipelines, the model unifies the three tasks into a structured generation: a rich transcription that intersperses speaker identity (“who”), timestamps (“when”) and content (“what”). It also supports custom hotwords—proper names, medical jargon, brands—that the user injects as context to improve accuracy in specific domains.

The three models of the family

Microsoft organizes VibeVoice into three lines, each with Hugging Face weights and different degrees of inference code availability:

• VibeVoice-ASR-7B — The most mature and adopted. ~9 billion parameters (according to Microsoft Foundry). It supports more than 50 languages without indicating explicit language, includes native code-switching and offers finetuning, inference with vLLM and direct integration with Transformers since March 2026.
• VibeVoice-Realtime-0.5B — Lightweight TTS for production deployment: ~300 ms to first audible audio, streaming text input, and robust generation of up to ~10 minutes. Includes experimental vocals in nine languages ​​(DE, FR, IT, JP, KR, NL, PL, PT, ES) and 11 styles in English.
• VibeVoice-TTS-1.5B — Long-lasting multi-voice synthesis: up to 90 minutes and 4 different speakers in a conversation. Accepted as Oral at ICLR 2026. The weights are still in Hugging Face, but Microsoft removed the TTS code from the repository in September 2025 after detecting misuse (deepfakes, misinformation).

VibeVoice-ASR in detail

The ASR is today the active pillar of the project. Microsoft opened it on January 21, 2026 and has since added finetuning code, vLLM support for faster inference, and the public playground on aka.ms/vibevoice-asr.

Features that distinguish it from Whisper, Deepgram or classic pipelines:

• One step — No sliding window or chunk reassembly; speaker coherence from beginning to end.
• Structured output — Speaker ID, timestamps and text in a single stream, not three tools chained together.
• Hotwords and context — Lists of terms or background descriptions that guide recognition.
• Native multilingual — More than 50 languages; handling language mixing within and between sentences.
• Open ecosystem — Local demo grading, per-file inference scripts, finetuning and catalog in Azure Foundry.

To test it locally, the documentation suggests:

python demo/vibevoice_asr_gradio_demo.py --model_path microsoft/VibeVoice-ASR --share

Or direct inference on audio files with vibevoice_asr_inference_from_file.py. The Transformers compatible variant is at VibeVoice-ASR-HF.

TTS in real time: VibeVoice-Realtime

Released in December 2025, VibeVoice-Realtime-0.5B targets cases where latency rules: voice assistants, live reading of news or narration while the user writes. With only 500 million parameters, it is deployable on modest hardware and supports streaming text: the model starts speaking before receiving the complete sentence.

Microsoft offers a Colab notebook to experiment without installing local dependencies. It is the most accessible way to evaluate voice quality compared to ElevenLabs, OpenAI TTS or other closed services — with the caveat that it is still a research model, not a polished commercial product.

The long TTS and the withdrawal of the code

VibeVoice-TTS-1.5B was one of the first open models capable of generating conversations of up to 90 minutes with up to four different voices, with expressive intonation and character coherence throughout the dialogue. The evaluation results published by Microsoft show preference of listeners over baselines in several scenarios:

However, in September 2025, Microsoft removed the TTS inference code from the repository. The official statement cites "instances where the tool was used in ways inconsistent with the stated intent"—in practice, deepfakes and disinformation with convincing synthetic voices. The weights in Hugging Face remain, but access to the full pipeline is restricted. It is a clear sign of the tension between open research and risk of abuse in high-fidelity speech synthesis.

Architecture: 7.5 Hz tokenizers and diffusion

The technical core of VibeVoice is continuous speech tokenizers (acoustic and semantic) that operate at an ultra-low frequency of 7.5 Hz. That compresses the audio into a much shorter sequence than traditional frame-to-frame approaches, making it feasible to process hours of content within the context of the LLM without exploding memory.

On top of that representation, VibeVoice uses a next-token diffusion framework: a large language model (based on Qwen2.5 1.5B in early versions) understands textual context and dialogue flow, while a diffusion head generates high-fidelity acoustic details. It's the same principle that allows you to go from "understanding what to say" to "how to sound when you say it" in a single end-to-end trained system.

How to get started

Depending on your profile, these are the most direct routes:

1. Just try — Playground ASR or Microsoft Foundry Labs (upload audio and get structured transcription).
2. Embed into code — Clone the repo, install dependencies, and use Transformers with microsoft/VibeVoice-ASR-HF.
3. Real-time TTS — Open the Realtime Colab or download the weights from VibeVoice-Realtime-0.5B.
4. Tune for your domain — Follow the guide in finetuning-asr/ in the repository to adapt ASR to specific vocabulary or accents.

Risks and limitations

Microsoft explicitly warns in the README: VibeVoice is for research and development, not for commercial production without additional testing. It inherits biases from the base model, can produce incorrect transcriptions, and—in the case of TTS—the risk of deepfakes is real. The MIT license makes it easy to use, but the responsibility for deploying it legally and ethically lies with the implementer. Microsoft recommends disclosing the use of AI when sharing generated content.

In summary

What is it? An open source family of Microsoft voice models (ASR, TTS, streaming). What stands out? Transcription of 60 minutes in one step with diarization and timestamps, 50+ languages and TTS in real time with 300 ms latency. Where to try it? aka.ms/vibevoice-asr, GitHub and Hugging Face. Caution? Long TTS lost its code due to abuse; the entire ecosystem remains experimental. For teams that process podcasts, minutes or multilingual content, VibeVoice-ASR is today one of the most ambitious open options on the market.