What a Foundation Model for Multimodal Translation Means for Studios and Platforms

01. What does a foundation model change about dubbing?

Machine dubbing used to be a cascade of separate tools: transcription, then translation, then a voice, then lip-sync. Each stage guessed without the others, and the errors carried from one stage to the next. What a buyer meets today trades quality against time. The assistive tools reach the quality with a team of VFX artists: weeks per project, and every language is a new project. The one-shot tools are fast and give up the rest: the translation cannot be edited, the voice drifts, the music and the room drop out, and the lips fail on a hand over the face, a microphone, blur, bokeh, a side view or several people in frame. No spatial reasoning, no understanding of the world, no control. Familiar replaces the chain with one Multimodal Localization Decision Model (MLDM): one shot and the quality.

++

Translation first, then the dub in one shot: the voice, the lips and the scene together.

Audio-visual translation at once, not cascaded: the first of its kind. The definition is in The First Foundation Model for Multimodal Translation; the contrast with the cascade, step by step, is in Audio-Visual Translation at Once, Not Cascaded.

HeyGenElevenLabsHeyGen + ElevenLabsONE-SHOT TOOLS · NO CONTROL

One shot today means giving up the rest.

The one-shot tools today give up the rest: the translation cannot be edited, the music and the room drop out, the voice drifts, and the lips fail on a hand over the face, a microphone, blur, bokeh, a side view or several people in frame. No spatial reasoning, no understanding of the world, no control. ElevenLabs' current Dubbing v2 edits only through an Enterprise API, and its full editor is the older model, in maintenance mode; HeyGen stitches a lip-sync over a dub like that, and the lip-sync breaks on the ordinary shots of any episode. Familiar is one shot and the quality: translation first, reviewed; then the voice, the lips and the scene together, through the hard shots.

54.7%

FEWER TRANSLATION ERRORS · FAMILIAR VS ELEVENLABS DUBBING V2

270%

ELEVENLABS MORE BACKGROUND-SOUND ERROR · THE LAUGHTER, THE MUSIC, THE AMBIENCE

+27.8%

FAMILIAR CLOSER TO THE REAL SPEAKER

FAMILIAR · ONE SHOT AND THE QUALITYFamiliar vs Flawless TrueSync →The ElevenLabs study →The demos →

02. Can one upload cover every market we sell into?

Yes. Footage in any of 99 source languages goes in once, and the dubs come out in any of the 30 output languages you order: every language from one upload, each carrying the same performance. A season, a library or a feed of vertical episodes follows the same path, and the terms you set once follow the next episode.

Full human dubbing, translation plus voice, runs $70 to $150 per finished minute. At that rate it is out of reach for most of the world. Even top productions dub only a few languages, because of the cost, the people and the time it takes.

That is the gap Familiar closes. Not 'generative AI', but translation to enable creatives. MLDM for the tedious part. Human to human, expanding the reach.

03. Who decides how a line lands in another language?

The MLDM decides what to localize and how, across text, audio and visual. Your team stays in charge of the choices that matter. The translation comes first, and where a line can land more than one way, your creative directors see the options and the reasoning behind each before anything is dubbed.

Every decision is recorded with the job, and you can override any of it. The full story is in The Decision Record. On the words alone, the translation matches expert human translators: 100.3% of their quality, measured in the benchmark.

04. What comes back with a finished dub?

A delivery your post team can drop into the release, at the spec a streamer asks for. The scene audio stays as it was: the music, the ambience and the laughter are kept, and only the speech changes language.

The subtitles are built from the reviewed translation, cut to the conventions of Netflix's Timed Text Style Guide, the subtitle template standard the industry works to. Every file and format is listed in the outputs docs.

05. Where does our footage run?

Unreleased footage is the asset a studio protects most. Every dub is processed on Familiar's own eight-figure cluster of NVIDIA B300 GPUs in Toronto, Canada, on compute Familiar owns.

The path of a dub and the answers for your security review are in the private inference docs.

06. What is available today, and what is in the works?

1. TODAY

   Every language from one upload, with the actor's own voice and lips.

   30 output languages, 99 source languages in; up to 4K, a true 5.1 or stereo mix, subtitles.
2. TODAY

   Your creative directors review the translation decisions.

   The options with the reasoning behind each, recorded with the job.
3. IN THE WORKS

   The audio and visual decisions, on the same review.

   What stays in the original language, a different face, which text in the frame to translate.
4. IN THE WORKS

   Live dubbing.

Access is by Studio contract, sized to your catalog: talk with the team about your titles, your languages and your delivery spec. The team is PhDs who started the field of joint audio-visual human models: 22 papers, over 10,000 citations, with Prof. Xu Zheng and Prof. Youngjung Uh as advisors, backed by Samsung, RedBird and Gold House. The research is in our papers and Pioneering Multimodal Translation; the founder's story is on our mission page.

QUESTIONS