ABSTRACT
01.What does an MLDM decide?
Text
Which rendering of a line carries it into the new language: a joke that has to land, a slang word, a name to keep as said, what a culture avoids, the profanity level that fits the audience. The MLDM weighs the options itself. When a line can land more than one way,your creative directors see three renderings and pick one.
Audio
Which voice is the one to translate. The laughter, the crowd and a background speaker no one can make out stay as they were, under the new dialogue. The voice in the new language is the speaker's own.
Visual
Who on screen is saying it, including who narrates and who speaks from off screen. The lips follow the new words, the expressions follow when the emphasis calls for it, and gestures follow when the meaning needs new emphasis. The text and objects in the frame, a phone screen, a book cover, a street sign, a store front, are in the works.
02.Why one model?
Most dubbing tools run a chain: speech-to-text, then translation, then a voice, then lip-sync, each stage tuned on its own. That chain made machine dubbing possible. Each stage still guesses without the others, so the errors compound from one stage to the next.
An MLDM makes the decisions together. The translation knows the mouth it has to fit, the voice knows the scene it sits in, and the person never changes.
03.What is the difference between localization and translation?
Translation is the words. Localization is what a culture needs from them: the right terms, the slang, the names kept as said, a reference swapped for one the audience knows, what that culture avoids, the profanity level that fits. An MLDM decides all of it, and in time the text written in the frame too.
04.Is an MLDM generative AI?
No. IBM defines it this way: "Generative AI is the branch of artificial intelligence that creates original content, such as text, images, video, audio and software code, in response to user prompts." That is a strong tool for making new things.
An MLDM has no prompt, and nothing new is made up: it decides how to localize what is already there. The full position is in Why Familiar Is Not Traditional Generative AI and Translation Is Not Generation.
05.Whose decisions are they?
Yours. When a line can land more than one way, your creative directors review three renderings and pick one before anything goes out. A Do Not Translate list and a one-sentence context travel with the job, and a choice made once follows the next episode. The controls are in the API docs.
QUESTIONS
What does MLDM stand for?
MLDM stands for Multimodal Localization Decision Model: one model that decides what to localize and how, across text, audio and visual. It picks the rendering of a line, the voice to translate, who on screen is speaking, and how the lips and the expressions follow.
Is an MLDM a video model?
Not in the usual sense. An MLDM works on footage that already exists and decides how to localize it: the words, the voice, the lips and the expressions. The person on screen stays exactly who they are, in every language.
Does an MLDM generate new footage?
No. There is no prompt, and nothing new is made up. An MLDM decides how to localize what is already there: the actor, the timing and the scene stay, while the words, the voice track and the lips change to fit the new language.
What does an MLDM decide in audio?
Which voice is the one to translate. The laughter, the crowd and a background speaker no one can make out stay as they were, under the new dialogue. The voice in the new language is the speaker's own.
Does an MLDM change expressions?
When the emphasis calls for it. The lips follow the new words in every line; the expressions follow when a line lands differently in the new language, and gestures follow when the meaning needs new emphasis. The person stays exactly who they are.
Does an MLDM translate text on screen?
Not yet: the text and objects in the frame, such as a phone screen, a book cover, a street sign or a store front, are in the works. Today the spoken words, the voice, the lips and the expressions are localized.
REFERENCES
Multimodal Translation
Familiar Labs builds generationally better dubbing with a Multimodal Localization Decision Model (MLDM). Your voice and lips, carried into new languages. They're a team of 8 with PhDs, backed by Samsung, RedBird, Gold House. Our mission