VRM Lip Sync MVP #

Minimal TypeScript web app that loads a VRM avatar and drives its mouth from uploaded audio using the Web Audio API.

The project now includes a default sample avatar at public/default-avatar.vrm, so you can run it immediately without finding your own model first.

It also includes a Gemini Live text-to-speech loop: enter a prompt, send it to Gemini, receive native audio, and feed that audio back into the avatar player for lip sync.

Run #

1. Install dependencies: bun install
2. Start dev server: bun run dev
3. Open the local Vite URL.

Gemini setup #

Set your Gemini API key before starting Vite:

export VITE_GEMINI_API_KEY=your_key_here
bun run dev

Usage #

1. Load a .vrm file.
2. Load an audio file.
3. Press Play.

The avatar uses amplitude-based mouth opening, not phoneme-level viseme detection.

Next upgrades #

• Add phoneme/viseme inference instead of pure amplitude
• Add microphone input mode
• Add drag-and-drop and avatar presets
• Add eye blink / idle animation