What Spider-Verse Got Right About Machine Learning

01.The line work

Hand-drawn ink lines do not sit still on a 3D face. In fxguide's account, written with Imageworks effects supervisor Pav Grochola, rule-based toon shaders missed the hand-drawn look, and lines mapped onto the surface stopped matching up when a character turned his head, particularly along the jawline. An artist could draw and nudge the lines by hand, frame by frame, across a feature film. So the team trained a model to predict where those lines belong as the camera angle and the expression change.

02.The artists drew the data

The training set was the artists' own work. They drew lines over turntable animation of each character from different angles and through expression changes; the first generation came to about 600 frames per character. The model gave animators a first pass, the animators nudged it where it was wrong, and those corrected frames became the second generation of training data. Grochola reports that the final model showed definite improvement over the first. The examples, the corrections and the final call all came from the artists.

03.A tool, kept for the sequel

For Across the Spider-Verse, Grochola told befores & afters that the tool trained on examples of the face linework was still in play: “we still used this tool extensively on this film.”

The filmmakers draw a clear line around it. On June 1, 2024, co-writer and producer Chris Miller posted on X, as reported by TheWrap: “There is no generative AI in ‘Beyond the Spider-Verse’ and there never will be.” He gave the reason too: the films set out to create new visual styles never seen in a studio CG film. A model trained on the team's own drawings fits that goal. The style is theirs; the machine carries it from frame to frame.

04.Machine learning for the tedious part.

Viewers saw it the same way. Two comments on SWAYNAMI's short, “Did Spider-Verse use AI the "RIGHT" way?”:

• “This is a tool. Cause its used to do the tedious annoying thing itd be a waste to use creatives for.” @lonewanderer-u5d
• “basically, they use it in the right purpose: doing the repetitive boring stuff so they have time to do creative stuff” @khanhkhanh7663

05.Not generative AI, but translation to enable creatives.

Dubbing has its own line work. The performance already exists: the actor, the timing, the face, the scene. Taking it into another language means the translation, the voice and the lip-sync, repeated for every line in every language. That is the tedious part, and it is what Familiar does, as a multimodal translation tool.

Generative AI in the traditional sense makes new content from a prompt. Familiar has no prompt and invents nothing: your video goes in, and the same performance comes out in another language, in the speaker's own voice, into any of 30 languages. Every line waits at a review screen for your edits before it renders. The performance stays the artist's.

All of the data we trained with ourselves are actually generated by the Familiar team.

We're another tool for the creative process.

QUESTIONS, ANSWERED

REFERENCES

1. [1]fxguide, Ink Lines and Machine Learning, with Pav Grochola of Sony Pictures Imageworks (May 20, 2019)
2. [2]befores & afters, Giving Spider-Man: Across the Spider-Verse that crucial hand-made quality (January 10, 2024)
3. [3]TheWrap, Beyond the Spider-Verse writer-producer Chris Miller shuts down rumors of generative AI use (June 2, 2024)
4. [4]SWAYNAMI, Did Spider-Verse use AI the "RIGHT" way? (YouTube short)
5. [5]Why Familiar is not traditional generative AI
6. [6]Familiar's mission