Pioneering Multimodal Translation: From Joint Audio-Visual Research to the First Foundation Model

01. Who started joint audio-visual human models?

A dub holds up only if the voice and the face stay one person. Familiar's researchers are PhDs who started the field of joint audio-visual human models: their paper JAM-Flow, at ECCV 2026, is the first joint audio-visual human model, the voice and the face of a speaking person treated as one problem.

The work covers the pieces multimodal translation needs: joint audio-visual human models, fast video models, editing and multimodal perception. Every paper and venue is in our research overview. The reason is personal: An Zhu Liu, the CEO, could not speak English to his classmates as a child and learned it from television, a story told on our mission page.

02. Why did dubbing go through separate tools?

Because each piece was solved on its own first. Machine dubbing arrived as a chain, and that chain is what made it possible at all. It also set its limit, and the tools built since still trade quality against time. The assistive tools reach the quality with a team of VFX artists and weeks per project, a new project per language. The one-shot tools are fast and give up the rest: their lips fail on a hand over the face, a microphone, blur, bokeh, a side view or several people in frame, with no spatial reasoning and no understanding of the world. Familiar is one shot and the quality.

++

Familiar is the first foundation model for multimodal translation: one model that carries the words, the voice, the lips and the scene into a new language, all at once.

The contrast step by step is in Audio-Visual Translation at Once, Not Cascaded; the definition is in The First Foundation Model for Multimodal Translation.

HeyGenElevenLabsHeyGen + ElevenLabsONE-SHOT TOOLS · NO CONTROL

One shot today means giving up the rest.

The one-shot tools today give up the rest: the translation cannot be edited, the music and the room drop out, the voice drifts, and the lips fail on a hand over the face, a microphone, blur, bokeh, a side view or several people in frame. No spatial reasoning, no understanding of the world, no control. ElevenLabs' current Dubbing v2 edits only through an Enterprise API, and its full editor is the older model, in maintenance mode; HeyGen stitches a lip-sync over a dub like that, and the lip-sync breaks on the ordinary shots of any episode. Familiar is one shot and the quality: translation first, reviewed; then the voice, the lips and the scene together, through the hard shots.

54.7%

FEWER TRANSLATION ERRORS · FAMILIAR VS ELEVENLABS DUBBING V2

270%

ELEVENLABS MORE BACKGROUND-SOUND ERROR · THE LAUGHTER, THE MUSIC, THE AMBIENCE

+27.8%

FAMILIAR CLOSER TO THE REAL SPEAKER

FAMILIAR · ONE SHOT AND THE QUALITYFamiliar vs Flawless TrueSync →The ElevenLabs study →The demos →

03. What does pioneering mean here?

Three things, together for the first time. The first is the audio and the visual carried into the new language at once, by one model, so the decisions agree: the voice is the actor's own, the lips follow the new words, and the laughter, the crowd and the music stay where they were.

The second is the order. Translation first, then the dub in one shot: the voice, the lips and the scene together. The words are settled, and reviewed by your creative directors, before anything is voiced.

The third is the record. Every decision the MLDM makes is recorded with the job, and you can override any of it. The full account is in The Decision Record; what MLDM is, in the definitional paper.

04. What does the model decide, and what do you choose?

A research result becomes a product when the people using it stay in charge. MLDM for the tedious part. The creative calls stay yours.

05. What is still in the works?

Pioneering means the first version, not the last. The translation choices are open today; the rest of the record opens in steps.

1. TODAY

   Your creative directors review the translation decisions.

   The renderings with the reasoning behind each; a Do Not Translate list and a one-sentence context travel with the job.
2. IN THE WORKS

   The audio and visual decisions, on the same review.

3. IN THE WORKS

   The text in the frame.

   A phone screen, a book cover, a street sign, written in the terms you already chose.
4. IN THE WORKS

   Live dubbing.

06. What does it change for the people on screen?

Human dubbing is out of reach for most of the world. Even top productions dub only a few languages, for the cost, the people and the time it takes; smaller productions, shows and vertical dramas mostly don't dub at all. A foundation model for multimodal translation gives every one of them the same performance in another language: the actor's voice, lips and expressions in 30 languages, delivered up to 4K with a true 5.1 surround or stereo mix in 24-bit PCM (the outputs).

The translation measures at 100.3% of expert human translation quality, a match; the studies are at /benchmark. Not 'generative AI', but translation to enable creatives. Human to human, expanding the reach. What it means for a catalog is in the paper for studios and platforms.

QUESTIONS