The First Foundation Model for Multimodal Translation

01. What is a foundation model for multimodal translation?

A video says more than its words. The speaker's voice, the way their mouth moves, the laughter in the room and the music under a scene all carry meaning. Translating a video well means carrying all of it across, and keeping it in agreement: the line has to fit the mouth, the voice has to sit in the scene, and the person on screen has to stay exactly who they are.

A foundation model for multimodal translation is one model that does that whole job. It takes the footage as one thing and decides how each part should land in the new language. Familiar built the first one, and calls it MLDM.

02. Why did dubbing go through separate tools until now?

Human dubbing has always been out of reach for most of the world. Even top productions dub only a few languages because of the cost, the people and the time; smaller productions, shows and vertical dramas don't dub at all. A full human dub, translation plus voice, runs $70 to $150 per finished minute.

03. What changes when one model carries it all at once?

++

Audio-visual translation at once, not cascaded: the first of its kind.

The same actor's performance comes through: their own voice, their lips moving to the new words, their expressions, and the scene audio around them. The laughter, the crowd and the music stay as they were. On the same clips, YouTube had 232.6% more background-sound error than Familiar. On the words alone, the translation matches the first-pass expert professional: 100.3% of their quality. The full numbers are on the benchmark.

It also changes the work around a dub. The high-quality route today is assistive: a team of VFX artists works shot by shot for weeks, and every language is a new project. The one-shot tools today are fast and give up the rest: the translation cannot be edited, the music and the room drop out, and the lips fail on the ordinary shots, a hand over the face, a microphone, blur, bokeh, a side view, several people in frame. No spatial reasoning, no understanding of the world. Familiar is one shot and the quality: one upload carries a video into every language you pick, at once, and the decisions agree because one model makes them, so nothing has to be stitched back together by hand.

HeyGenElevenLabsHeyGen + ElevenLabsONE-SHOT TOOLS · NO CONTROL

One shot today means giving up the rest.

The one-shot tools today give up the rest: the translation cannot be edited, the music and the room drop out, the voice drifts, and the lips fail on a hand over the face, a microphone, blur, bokeh, a side view or several people in frame. No spatial reasoning, no understanding of the world, no control. ElevenLabs' current Dubbing v2 edits only through an Enterprise API, and its full editor is the older model, in maintenance mode; HeyGen stitches a lip-sync over a dub like that, and the lip-sync breaks on the ordinary shots of any episode. Familiar is one shot and the quality: translation first, reviewed; then the voice, the lips and the scene together, through the hard shots.

54.7%

FEWER TRANSLATION ERRORS · FAMILIAR VS ELEVENLABS DUBBING V2

270%

ELEVENLABS MORE BACKGROUND-SOUND ERROR · THE LAUGHTER, THE MUSIC, THE AMBIENCE

+27.8%

FAMILIAR CLOSER TO THE REAL SPEAKER

FAMILIAR · ONE SHOT AND THE QUALITYFamiliar vs Flawless TrueSync →The ElevenLabs study →The demos →

04. What does MLDM decide, and what do you choose?

The Multimodal Localization Decision Model is the model itself. MLDM decides what to localize and how, across text, audio and visual. The full definition is in What Is a Multimodal Localization Decision Model (MLDM)?

1. TODAY

   The translation decisions, reviewed by your creative directors.

   Every decision is recorded with the job. You can override any of it.
2. IN THE WORKS

   The audio and visual decisions, on the same review.

3. IN THE WORKS

   The text in the frame, and live dubbing.

The record of those choices is in The Decision Record.

05. What does a studio get from a foundation model?

MLDM for the tedious part. The creative calls stay with the people who made the work; the model carries the rest into every language at once.

Every dub is processed on that cluster, never by a third-party inference provider, so your media is never leaked. Details are in the private inference docs and the outputs docs. Behind it are PhDs who started the field of joint audio-visual human models: 22 papers, over 10,000 citations, with Prof. Xu Zheng and Prof. Youngjung Uh as advisors, backed by Samsung, RedBird and Gold House. The papers are in our research.

06. Is a foundation model for translation generative AI?

Not in the traditional sense. IBM defines generative AI this way:

That is a strong tool for making new things. MLDM has no prompt, and nothing new is made up: it carries what is already there into another language. Not 'generative AI', but translation to enable creatives. Human to human, expanding the reach.

The other papers in this series: Audio-Visual Translation at Once, Not Cascaded, What a Foundation Model for Multimodal Translation Means for Studios and Platforms and Pioneering Multimodal Translation.

QUESTIONS