Skip to content

Learn

What Is AI Lip Sync? How It Works

AI lip sync is a technique that changes the mouth movements of a person in a video so they match a new audio track. A model listens to the speech, works out which mouth shapes each sound needs and redraws the lower face frame by frame, so dubbed or re-recorded dialogue looks naturally spoken.

Start on the free Starter plan.

AI Lip Sync - FunAI
AI Lip Sync - FunAI

Sounds In, Mouth Shapes Out

A lip sync model hears the phonemes in your audio, turns them into the matching visemes and repaints only the mouth region, leaving the rest of the video as it was.

Open the Lipsync Tool

Phonemes to visemes

Each speech sound has a visible mouth shape. The model learns that mapping from real footage instead of fixed rules.

Only the mouth changes

Head pose, eyes, lighting and background come from your original video, so the edit stays in context.

Consistent across frames

The model keeps teeth, lips and identity stable over time, which is what separates believable sync from flicker.

Lip sync models on FunAI

Several makers offer lip sync, plus avatar models for when you only have a photo. Each model page lists its inputs and its price per run.

  • •
    Sync Lipsync 2.0:Takes a video and an audio clip and generates lip movement that matches the audio.
  • •
    LatentSync:A video-to-video lip sync model with guidance and loop-mode settings for when audio and video lengths differ.
  • •
    Kling Video Lipsync:Matches a person's lip movements to your audio from a 2 to 10 second video.
  • •
    HeyGen Lipsync Precision:Replaces or dubs the audio on an existing video with high-accuracy lip sync.
  • •
    OmniHuman:Not lip sync but its photo-based cousin: generates a talking video from one image of a person and an audio file.
Open the Lipsync Tool
AI Lip Sync Models

AI Lip Sync explained

Lip sync takes two inputs: a video of a face and an audio clip of speech or singing. The output is the same video, with the same head, eyes and background, but with a mouth that now forms the words in the new audio. It is the visual half of dubbing. The audio half is text-to-speech, voice cloning or translation, and lip sync makes the result look as if the person really said it.

Speech is made of phonemes, the distinct sounds of a language, and each one has a visible mouth shape called a viseme: P, B and M all press the lips together, 'oo' rounds them, 'f' and 'v' touch the lower lip to the teeth. Early systems mapped phonemes to visemes with hand-written rules. Modern lip sync models learn the mapping from video. An audio encoder turns the waveform into features, and a generator repaints the mouth region of each frame conditioned on those features and on the original face. Some, such as LatentSync, use a diffusion model working in latent space; all of them have to keep identity, teeth and timing consistent from frame to frame.

Lip sync is not the same as a talking-avatar model. Lip sync edits an existing video. An avatar model such as OmniHuman starts from a single photo plus audio and generates the whole performance, including head movement and expression. Use lip sync to dub a presenter into another language, fix a flubbed line, replace a voice-over or match a music video. Use an avatar model when all you have is a picture.

Tips for natural-looking lip sync

  • Clean audio syncs better: remove music and background noise first with the Audio Denoiser Tool.
  • Keep hands, microphones and hair away from the mouth; the model needs to see the lips to repaint them.
  • Match the pace: a translated line that runs much longer than the original forces fast, unnatural mouth movement, so tighten the script first.
  • Decide how length mismatches are handled before you run, for example with LatentSync loop mode, rather than trimming afterwards.
  • Only lip sync people who have agreed to it, and label synthetic speech wherever your platform or audience expects it.

How to Lip Sync a Video on FunAI

Feature visual preview 1
Feature visual preview 2
Feature visual preview 3
Feature visual preview 4
01

Prepare the video

Use a clip where the face is visible, reasonably front-on and well lit. Some models set limits; Kling Video Lipsync, for example, takes a 2 to 10 second video.

02

Prepare the audio

Record the new speech or create it on FunAI with the Text to Speech Tool, the AI Voice Cloning Tool, or the AI Dubbing Tool to translate the original speech.

03

Run the Lipsync Tool or a model

Open the Lipsync Tool, or a model page such as Sync Lipsync 2.0, LatentSync or HeyGen Lipsync Precision, upload the video and the audio, and run it.

04

Review the mouth closely

Play the result at full size and watch the closed-lip sounds (p, b, m) and the teeth. If they look soft or late, run another lip sync model on the same pair of files.

No public creations to show yet.

Everything you need to know