Getting Audio2Face running isn't hard; the pipeline after it is

This thing is worth spending a weekend to go through. It runs, but it can't directly produce output—you have to connect the pipeline in the middle yourself.

The first day was mostly setting up the environment. Audio2Face traces back to NVIDIA's open-source line, and now some domestic companies are doing integration and packaging on top of it. I configured it on a B200 according to the docs, but the driver version didn't match, and I got a round of errors. What this thing does is listen to sound and recognize faces: you throw in an audio clip, and it outputs facial animation data to drive the mouth shapes and expressions of a 3D model. Sounds simple, but there are a ton of prerequisites. The model has to be in FBX or USD format and must be rigged first; the audio is best as clean WAV, and there are specifics about sample rate—the material mentioned the impact of 48kHz. I got stuck on model rigging the first day because that model's bone naming didn't match its spec.

On the third day, I got basic lip sync working. I used offline mode, one Chinese clip and one English clip, and the mouth shapes did keep up—it's phoneme-level alignment, where phonemes are the smallest units of speech. What surprised me was the upper face: eyebrows and around the eyes also moved, not just the mouth. You have to adjust Sensitivity and Emotion Intensity in the parameters; the defaults come out flat, and only after raising emotion intensity do expressions come alive. But there's a pitfall: if the audio isn't clean, like with background noise or music, it treats those as speech features to drive, and the face will twitch inexplicably.

A week later, I went through the whole pipeline. Real-time mode with Unreal's Live Link works, and the latency I measured is acceptable, but what really saves effort is the Python SDK for batch automation—dozens of audio clips queued up to run without clicking one by one.

But when exporting, pay attention to coordinate systems and track naming; this part isn't standardized, and it gets misaligned when imported into the engine. Also, it outputs blendshape weights, which are a set of values controlling the deformation amplitude of various facial parts. Whether this set of weights can be integrated into your own model is the watershed for the whole thing.

There's some background to clarify. There are similar solutions domestically. FACEGOOD's set has been criticized—the open-source part is incomplete, it can't generate ARKit blendshape weights, and can't be integrated into your own model. NVIDIA's line is relatively complete, but the cost is that it's tightly bound to its own ecosystem.

NVIDIA says that open-sourcing Audio2Face is to accelerate the deployment of AI-driven virtual avatars in gaming and customer service.

Back to my own judgment. This model is suitable for integrators with a 3D pipeline, an engine team, and a clear scenario for solution pre-research. It's not suitable for individual developers who want one-click digital human generation, nor for end customers who want to directly buy a talking avatar. It only saves the part of writing algorithms from scratch; you still have to get things running yourself. This logic is the same as my previous post on RDK NodeHub—assembly has been standardized, but the delivery part still has to be shouldered by yourself.

Listening to sound and recognizing faces really works, but whether it can be directly delivered depends on how well your own pipeline is connected.