What is AI lip sync?

Every spoken sound has a mouth shape. Lips press shut for p, b and m; the top teeth touch the lower lip for f and v; the lips round for oo. Lip sync works by getting those shapes on screen at the moment each sound is heard, and blending one into the next the way a real mouth does.

There are two ways to get there. A video model with sound can make the face and the voice at once, from a prompt that holds the line: nothing exists before the generation, so the mouth and the words are invented together. Or a lip-sync model can take a video that already exists and an audio track, and redraw only the mouth area so it says the new words.

Phonemes and visemes

A phoneme is a sound of speech; a viseme is what it looks like on the face. In Microsoft’s speech documentation, a viseme is “the visual description of a phoneme”, and there is no one-to-one match: several sounds, such as s and z, look the same on a speaker’s face. Microsoft uses 22 visemes for its speech service and notes that the set varies by language, which is one reason lip sync that looks right in one language can look loose in another.

Avatar and game pipelines drive a 2D or 3D face from visemes directly: the voice engine names each mouth shape and when it starts. Video models don’t list visemes; they learn the link between sound and mouth from footage of people talking.

Audio-driven lip-sync models

Audio-driven lip sync starts from a face you already have. A well-known research example is Wav2Lip (2020): it lip-syncs “a talking face video of an arbitrary identity” to a target speech clip, and it learns from a strong lip-sync discriminator, a network that judges whether mouth and sound agree. An earlier Oxford paper, Out of time by Chung and Zisserman (2016), trained a network to measure the synchronization between mouth motion and speech in a video: the kind of judge such models learn from.

Model makers now sell the same route as a feature. Kling’s Lip Sync takes a video with a complete face and either an uploaded audio file or a voice made from text, and crops the audio if it runs longer than the video. Alibaba’s VideoRetalk takes a video and an audio file and asks for a clear, front-facing face. In each, the rest of the frame is kept and the lower face is redrawn.

Joint audio-video models

The newer route makes sound and picture in the same generation. Google introduced Veo 3 as generating sound effects, ambience and dialogue natively, and Kling 3.0 is described with character-level lip-sync and dialogue given to several characters in one scene. Research models such as Ovi (2025) show the idea in the open: one generative process for both, with two linked networks trading timing and meaning so the sound needs no alignment afterward. See native audio for the sound side in depth.

Which route you need depends on what you start with. If nothing exists yet, generate the face and the voice together; if footage or a voice already exists and must stay, you need an audio-driven tool.

These names overlap in search results and in tools, but they describe different jobs. Lip sync is the part they share: a mouth that matches the speech.

This 30-second take was made on DreamBench from the brief under it, on Seedance 2.5 with sound. The chef’s line is written in the brief in Japanese, in quotation marks, with who says it and how: “says quietly in Japanese”. Nothing was recorded; the voice, the words and the mouth were generated together.

Watch how the line is placed: after a pause, while the chef is still and facing the light. A spoken line lands best on a face that isn’t also doing something fast.

To make an AI character talk, write the line in the prompt, in quotation marks, and say who speaks it. A model with sound reads a quoted line as speech; a line without quotes is often treated as a description. A useful order:

• Keep it short: about two words a second of clip is a comfortable pace, so an 8-second shot holds one or two sentences.
• One speaker at a time, each line given to a named person; say that the others listen in silence.
• Put the speaker facing the camera or at three-quarters, with light on the face.
• Keep hands, food, cups and microphones away from the mouth while the line is spoken.
• If you need an accent, ask for it in words, such as “a soft Scottish accent”.

The AI video prompt generator has a spoken-line option for any model, and the Kling 3.0 prompt generator writes a line to be lip-synced in English, Chinese, Japanese, Korean or Spanish. Both write the prompt only; the clip is still made by the video model.

• Talking-to-camera ads and social clips, where a presenter says one line about a product.
• Short dramas and sketches, with two characters trading lines.
• Animated and stylized characters that speak, from a mascot to a cartoon animal.
• The same scene in several languages, each version generated with its own line.

• Fast speech: when sounds come faster than the frames can show them, the mouth smears into a blur or skips shapes. Slow the line down or cut words.
• Side views: both routes do best on a face seen from the front or at three-quarters; VideoRetalk asks for a front-facing face outright. In profile, half the mouth is hidden and errors show.
• Beards, mustaches and hands near the mouth hide the lip line, so the model has less to follow and more to invent.
• Teeth and tongue: they come out blurred, merged or change from frame to frame, most of all on open vowels in a close-up.
• Several speakers: the voice can come from the wrong face, or a listener’s lips move too. Name each speaker and give each line to one person.
• Languages: a model lip-syncs best in the languages it learned from most. Some makers list theirs, and a line in another language can come out less clear or in a different language.

Why does the voice in my AI video not match the mouth? Usually the line is too long or too fast for the clip, the speaker is turned away, or the prompt doesn’t say who speaks: fix that and generate again. Google itself names speech in short lines and audio synchronization as work in progress for Veo.

An audio-driven tool can put any words into any filmed face. That is the same technique behind many deepfakes, and it is why consent comes first: make a real person appear to say something only with their informed permission, and never to deceive, to endorse a product they don’t, or to put words in the mouth of a public figure.

DreamBench’s Terms of Use ask for informed permission before a real person’s likeness or voice is processed or depicted with AI, and rule out unauthorized real-person depictions, deceptive impersonation and false endorsements. Seedance on DreamBench refuses reference pictures of real human faces outright.

If a clip shows a real person saying something they didn’t say, label it as AI-made wherever you publish it.

On DreamBench, lip sync is generated together with the video. You describe the film, and when the brief has people speaking, DreamBench writes each line into the prompt of its scene in quotation marks, names who says it, keeps everyone else’s lips closed and describes each speaker’s voice the same way in every clip. A model with sound then makes the face and the voice in one pass. Each clip is generated on its own, so a voice can still vary a little from scene to scene; see character consistency.

Some models are made for it: HappyHorse 1.1 is on the menu for people speaking to camera, Kling 3.0 gives lines to several characters in one scene, and Wan 3.0 builds scenes with dialogue. Sound is not always speech, though: on DreamBench Wan 2.7 adds music or effects, and Vidu Q2 adds background music only.

What DreamBench doesn’t do: there is no upload for an audio file, so it can’t lip-sync a clip to your own recording, and it doesn’t dub an existing video into another language. Morph edits what is in a video you upload and keeps its original sound, unless you turn it off; it doesn’t take a new voice track. If a line comes out wrong, redo that scene alone. Each model’s page on the models list says what its sound includes.