Audio-Driven Face Video

img-and-audio2video turns a still portrait and a voice clip into a talking-head video. Give it one image and one audio file. It animates the face so the lips track the speech, then returns a single video. It runs on Socaity's cloud, so you need no GPU and no local copy of the model.

by lucataco · socaity.sdk.replicate.lucataco.img_and_audio2video

Capabilities

Animate a single portrait so its mouth and expression follow a voice track
Two inputs, both required: a portrait image and an audio clip
Returns one MP4 video, ready to drop into an edit
Pairs naturally with a hosted text-to-speech model to synthesise the voice line first

Install

Install the SDK once, then install the model so you can import its client.

pip install socaity
socaity install lucataco/img-and-audio2video

Authentication

Socaity proxies the Replicate call for you, so one API key covers everything. Set SOCAITY_API_KEY in your environment. You do not need a Replicate key. Get a Socaity key at socaity.ai/account/secrets.

export SOCAITY_API_KEY="sk_..."

Quick start

Generate a talking-head video from a portrait and a voice clip. .get_result() blocks until the job finishes and returns a single VideoFile.

import os
from socaity import ImageFile, AudioFile
from socaity.sdk.replicate.lucataco import img_and_audio2video

talk = img_and_audio2video(api_key=os.getenv("SOCAITY_API_KEY"))

# image and audio are both required.
video = talk(
    image=ImageFile().from_file("face.png"),
    audio=AudioFile().from_file("vocals.wav"),
).get_result()                  # single VideoFile

video.save("talking.mp4")

Methods

img_and_audio2video(api_key=None) -> client; client(image, audio) -> Job

Submit an image + audio job and return a Job object. talk(image=..., audio=...) and talk.run(image=..., audio=...) are aliases for the same call. Call .get_result() to block until the video is ready.

Parameter	Type	Default	Description
`image`	`ImageFile \| str`	`required`	Portrait to animate. Wrap a local path with ImageFile().from_file(...).
`audio`	`AudioFile \| str`	`required`	Speech audio that drives the lip movement. Wrap a local path with AudioFile().from_file(...).

Pricing & live playground

Current pricing and the interactive playground live on socaity.ai.

Open img-and-audio2video on socaity.ai

Next steps

Lip sync a portrait: step-by-step walkthrough that pairs this model with a hosted text-to-speech voice line.
Text to speech: synthesise the voice clip you feed in as the audio input.
Face restoration: clean up or upscale a portrait with hosted GFPGAN before animating it.

Face2Face

flux-schnell