Audio-Visual Translation at Once, Not Cascaded

01. What was the cascade era of machine dubbing?

Machine dubbing began as a cascade: four tools in a row, each built and tuned on its own, each handing its result to the next.

1. 01

   Transcription.

   Speech-to-text writes down what was said.
2. 02

   Translation.

   A text model renders the transcript in the new language, without hearing the voice or seeing the face.
3. 03

   A voice.

   Text-to-speech reads the translation aloud, often in a stranger's voice.
4. 04

   Lip-sync.

   A separate tool moves the mouth to the new audio, after every word is fixed.

That chain made machine dubbing possible. Its limit sits in its shape: each stage guesses without the others. The translation cannot know the line has to fit a closed mouth. The voice cannot know the room it sits in. A misheard word or a line that runs long is passed down, and the error compounds at every stage after it.

Each tool got better over the years: a cleaner transcript, a sharper translation, a more natural voice. The finished dub improved more slowly, because the seams between the tools stayed where they were.

02. What does a buyer meet today?

In practice the cascade arrives as a stack of tools: a transcript from one, the translation from another, a voice from a third, lip-sync and subtitles from a fourth. A fix to one line means running every stage after it again, and someone on your team holds the files together. The tools that skip the stack trade quality against time. The assistive tools reach the quality with a team of VFX artists on the hard shots: weeks per project, and every language is a new project. The one-shot tools are fast and give up the rest: the voice drifts, the room drops out, and the lips fail on a hand over the face, a microphone, blur, bokeh, a side view or several people in frame, with no spatial reasoning and no understanding of the world. Familiar is one shot and the quality.

The other road is full human dubbing, translation and voice, at $70–150 per finished minute. The cost, the manpower and the time put it out of reach for most of the world: even top productions dub only a few languages, and smaller productions, shows and vertical dramas mostly don't dub at all.

HeyGenElevenLabsHeyGen + ElevenLabsONE-SHOT TOOLS · NO CONTROL

One shot today means giving up the rest.

The one-shot tools today give up the rest: the translation cannot be edited, the music and the room drop out, the voice drifts, and the lips fail on a hand over the face, a microphone, blur, bokeh, a side view or several people in frame. No spatial reasoning, no understanding of the world, no control. ElevenLabs' current Dubbing v2 edits only through an Enterprise API, and its full editor is the older model, in maintenance mode; HeyGen stitches a lip-sync over a dub like that, and the lip-sync breaks on the ordinary shots of any episode. Familiar is one shot and the quality: translation first, reviewed; then the voice, the lips and the scene together, through the hard shots.

54.7%

FEWER TRANSLATION ERRORS · FAMILIAR VS ELEVENLABS DUBBING V2

270%

ELEVENLABS MORE BACKGROUND-SOUND ERROR · THE LAUGHTER, THE MUSIC, THE AMBIENCE

+27.8%

FAMILIAR CLOSER TO THE REAL SPEAKER

FAMILIAR · ONE SHOT AND THE QUALITYFamiliar vs Flawless TrueSync →The ElevenLabs study →The demos →

If you are weighing ElevenLabs Dubbing, here is their documentation, quoted, beside our study:

ElevenLabsElevenLabsTHEIR DOCS, READ OCTOBER 30, 2026

Editing the translation means their old model.

On ElevenLabs' current Dubbing v2, in alpha, editing the translation and regenerating the audio through the API is for their Enterprise plans only. Their full editor, Dubbing Studio, is "only available via the v1 model" and "in maintenance mode", in their words. Dubbing v1 is the model our 418-pair study measured, and its output is mono.

+27.8%

FAMILIAR CLOSER TO THE REAL SPEAKER · ALL 11 LANGUAGES

261%

ELEVENLABS MORE BACKGROUND ERROR · THE MUSIC, THE AMBIENCE

+243.7%

FAMILIAR ON LAUGHTER AND REACTION SHAPE

ELEVENLABS DUBBING V1 · 418 PAIRED OUTPUTS · 11 LANGUAGES · AUGUST 5, 2026Read the study →Their docs ↗

03. What changes when one model makes every decision?

Familiar dubs with a Multimodal Localization Decision Model (MLDM). MLDM decides what to localize and how, across text, audio and visual, and the decisions agree because one model makes them. The definition is in The First Foundation Model for Multimodal Translation.

What you get back is the actor's own performance in another language: their voice, their lips, their expressions and the scene audio, carried across together. One upload goes out in every language at once: dubs in 30 languages, from footage in any of 99 source languages.

Deciding together also changes what can go wrong. A line that would run long is rendered to fit the time and the mouth before any voice is made. A laugh in the middle of a sentence stays a laugh. When two people talk over each other, the model knows who is saying which line, so the right lips move.

04. What does that change in the finished dub?

We run the same clips through both systems and score them the same way. Each figure comes from its own study, published on the benchmark.

The voice is measured too: Familiar sounds 27.8% more like the real speaker than ElevenLabs Dubbing v1, closer in all 11 languages, in the study quoted above.

YouTube's auto-dubbing showed 232.6% more background-sound error in its own study. The scene survives because the MLDM hears everything in the track, the laughter, the crowd, the music, finds the main voice, carries it into the new language in the speaker's own voice, and keeps the rest as it was.

05. What does "translation first, then the dub in one shot" mean?

++

Translation first, then the dub in one shot: the voice, the lips and the scene together.

The order matters. The translation comes first, and your creative directors review it: where a line can land more than one way, they see the renderings with the reasoning behind each and pick one. Only then is the dub made, once. Nothing new is made up. Not 'generative AI', but translation to enable creatives. MLDM for the tedious part.

Subtitles are cut to the conventions of Netflix's Timed Text Style Guide, the subtitle template standard the industry works to. Every dub is processed on Familiar's own eight-figure cluster of NVIDIA B300 GPUs in Toronto, Canada, never on a public cloud like Google Cloud or AWS, and never by a third-party inference provider. The files are in the outputs docs, the processing in private inference, and what it means for a catalog in What a Foundation Model for Multimodal Translation Means for Studios and Platforms.

06. Who makes the final call?

1. TODAY

   Your creative directors review the translation decisions.

   The renderings with the reasoning behind each; they pick one before anything goes out.
2. TODAY

   A Do Not Translate list and a one-sentence context travel with the job.

3. IN THE WORKS

   The audio and visual decisions, on the same review.

   What stays in the original language, a different face when the model chose the wrong speaker, and the text in the frame.

What changes for a catalog is reach. A show that would have stayed in one language can go out in 30 at once, carrying the actors' own performances, while your team still makes every call that matters.

That is the decision record; the full story is in The Decision Record, and what the model decides in What Is MLDM. The research behind it is in Pioneering Multimodal Translation. Human to human, expanding the reach.

QUESTIONS