CONTROL

The Decision Record: How Creative Directors Direct the MLFM

FAMILIAR LABS RESEARCHUPDATED ALL ARTICLES
READ THIS INEnglishEspañolPortuguês中文日本語한국어FrançaisDeutschالعربيةBahasa IndonesiaРусскийItalianoTiếng ViệtไทยNederlandsΕλληνικάУкраїнськаHrvatskiLietuviųSlovenčinaSvenskaБългарскиDanskLatviešu

ABSTRACT

The MLFM is the first foundation model for multimodal translation, a new paradigm, the first of its kind. Every decision it makes is recorded with the dub: you read it, override it, and keep a choice for the next episode. The translation decisions are reviewed today; the audio and visual ones are in the works.
THE FIRST FOUNDATION MODEL FOR MULTIMODAL TRANSLATION

The decisions you direct today

  1. 01

    Translation comes back first.

  2. 02

    Where a line can land more than one way, the review step returns three renderings, each for a different intent and emotion, with the reasoning behind each.

  3. 03

    Your creative directors pick one or write their own, then render.

The same review covers the localization calls, what not to translate first: the slang kept as said, the name and the brand kept, a reference swapped for one the audience knows, what that culture avoids, and the profanity lowered for a younger cut.

A pun, Korean to English

"사과하려고 사과 사 왔어." rests on 사과 meaning both apology and apple. Three renderings come back:

  • "Apples. I'm sorry, right down to the core." The pun kept.
  • "I came to say sorry. I brought apples." The meaning only.
  • "An apple a day keeps the silent treatment away." A new joke.

The creative director picked the first and edited it.

A jab, Chinese to English

"你可真行啊。" reads as praise or as a jab, depending on the scene:

  • "Wow. You're incredible." Sincere.
  • "Well, aren't you something." Sarcastic, the pick.
  • "Look at you, hotshot." Teasing.

A podcast, an episode or a movie carries many such segments: the jokes, the conversations, the stories.

The audio decisions

The MLFM decides what to translate at all. It finds the main voice, carries it into the new language in the speaker's own voice, and keeps the rest as it was. Showing your team every close call in the audio is in the works.

What stays in the original language: a stadium chant, kept in Korean; a yell across a kitchen; a catchphrase the fans know. The background speaker no one can make out stays as it was, or comes back as a choice to translate after all.

The visual decisions

IN THE WORKS

WHO IS ACTUALLY SAYING IT
WHAT IS RECORDEDThe face a line is heard from, the narrator, the voice from off screen.
WHAT YOU CAN DOWhen the model picks the wrong speaker, you pick the face from the candidates, and the lips move on the right person.
THE TEXT IN THE FRAME
WHAT IS RECORDEDThe phone screen reading 내일 봐, a street sign, a store front, and what it kept as shot on the fly: the brand on a bag of Tostitos, a name.
WHAT YOU CAN DOEach item is written in the terms you already chose, as "See you tomorrow", or kept as shot.
EXPRESSIONS AND GESTURES
WHAT IS RECORDEDThey follow the new line's emphasis or the culture.
WHAT YOU CAN DOAn eyebrow raise the new audience reads differently can be held as shot.

Recorded with the dub

The record is what makes MLFM a new paradigm, the first of its kind: a model that shows its decisions.

++
The decision record, in the works, holds every decision with its options, the reasoning, what was chosen and by whom: the model, you, or memory.

It is open at review and locked at render. Change a decision on a delivered dub and render again; only the languages it touches render again. A dub rendered with review=auto still writes the record, with the model's choices. The dub itself is made in one shot from those decisions: the voice, the lips and the scene together. The one-shot tools today work the other way: upload once; you get what comes out. The high-quality route keeps the control by hand instead, with an assistive tool, a team of VFX artists and weeks per project. Familiar Labs is one shot and the quality.

HeyGenHeyGenElevenLabsElevenLabsHeyGen + ElevenLabsONE-SHOT TOOLS · NO CONTROL

One shot today means giving up the rest.

The one-shot tools today give up the rest: the translation cannot be edited, the music and the room drop out, the voice drifts, and the lips fail on a hand over the face, a microphone, blur, bokeh, a side view or several people in frame. No spatial reasoning, no understanding of the world, no control. ElevenLabs' current Dubbing v2 edits only through an Enterprise API, and its full editor is the older model, in maintenance mode; HeyGen stitches a lip-sync over a dub like that, and the lip-sync breaks on the ordinary shots of any episode. Familiar Labs is one shot and the quality: translation first, reviewed; then the voice, the lips and the scene together, through the hard shots.

54.7%

FEWER TRANSLATION ERRORS · FAMILIAR LABS VS ELEVENLABS DUBBING V2

270%

ELEVENLABS MORE BACKGROUND-SOUND ERROR · THE LAUGHTER, THE MUSIC, THE AMBIENCE

+27.8%

FAMILIAR LABS CLOSER TO THE REAL SPEAKER

FAMILIAR LABS · ONE SHOT AND THE QUALITYFamiliar Labs vs Flawless TrueSync The ElevenLabs study The demos

A choice made once follows the next episode

  1. TODAY

    Two standing decisions work today.

    Do Not Translate keeps a phrase as said, and Context gives the one-sentence briefing; you set both before the dub. The record holds the decisions the model made during it.
  2. IN THE WORKS

    Decision memory.

    Remember an override for a creator, one show, channel or person, and the next dub applies it first when the same line, term, name, chant, sign or face comes back. The catchphrase kept in the original follows episode 2 without a second review.

On the review screen and in the API

In the works: the review screen grows a decisions rail. A timeline of the episode carries a marker per decision, filtered by translation, audio or visual; each card shows the moment, the options with the reasoning and the model's pick lit. One tap picks another option, types your own, keeps the original, or remembers the choice for this creator.

The API returns the same record, so a choice made on the screen shows in the API and the other way round. The planned surface, all in the works:

GET /v1/dubs/{dub_id}/decisions
WHAT IT DOESReads the record.
PATCH /v1/dubs/{dub_id}/decisions/{decision_id}
WHAT IT DOESOverrides one decision.
decisions.*
WHAT IT DOESThe settings on a new dub set how many renderings come back, the profanity level, and which calls come back to you.
dub.decisions_ready
WHAT IT DOESThe webhook fires when the record is open.
decisions.json
WHAT IT DOESCarries the full record with every delivered dub.

The reference is at Decisions; today's review step is at Transcript & review.

QUESTIONS

Which decisions can I change today?

The translation decisions. A line that can land more than one way comes back as three renderings, each with its reasoning; your creative directors pick one or write their own, then render. The audio and visual decisions are in the works.

Can I keep a chant in the original language?

That choice is in the works. A stadium chant, a yell or a catchphrase will come back as an audio decision you can keep in the source language, and the choice can follow the next episode.

REFERENCES

  1. [1]What Is a Multimodal Localization Foundation Model (MLFM)?
  2. [2]API: Decisions
  3. [3]API: Transcript & review

Multimodal Translation

Familiar Labs builds generationally better dubbing with a Multimodal Localization Foundation Model (MLFM), the first foundation model for multimodal translation: one model decides what to localize across text, audio and visual, and shows you the choices. Your voice and lips, carried into new languages. They're a team of 7 with PhDs, backed by Samsung, RedBird, Gold House. Our mission

See the full benchmark →Talk with the team