Lip Sync
Definition
AI lip sync is a technology that automatically adjusts a speaker's mouth movements in video to match new audio, typically in a different language. It enables seamless video dubbing where the speaker appears to naturally speak the target language. The technology combines facial landmark detection, generative video models, and audio-visual alignment.
How It Works
The system first detects facial landmarks and segments the mouth region in each video frame. A neural network then generates new mouth movements that match the phonemes of the target audio, blending them back into the original face. Advanced systems like Dubly and HeyGen also adjust jaw movement and facial expressions for more natural results.