Animation System Design #
Component: Animation System (Rust/Fyrox) Status: MVP Implemented (placeholder cube), Full Design Pending Related: Frontend Architecture, IPC Protocol
Implementation Status #
Current MVP (game/src/animation_state.rs, game/src/animation_events.rs, game/src/cube_animator.rs):
- ✅
BodyStateenum with all states (Idle, Working, TurningToCamera, Talking, TurningFromCamera) - ✅
IdleVariantenum (Sitting, Thinking, Stretching, LookingAround, Wandering) - ✅
WorkAnimationenum (Terminal, Reading, Writing, WebBrowsing, DeepThinking) - ✅
AnimationControllerwith state transitions - ✅ Idle variation timer (10-30 second random changes)
- ✅ Tool categorization for 15+ tool types
- ✅ Backend message integration (ToolStart, ToolEnd, AssistantTextDelta, AssistantMessageComplete)
- ✅ Color-coded placeholder cube for visual feedback
- ✅ Event-driven turn animations via convergence detection (replaces fixed timers!)
- ✅ Animation event system (
AnimationEvent,AnimationEventSourcetrait) - Fyrox-idiomatic API - ✅ Camera focus controller (
game/src/camera_animator.rs) - ✅ Sound effects system (
game/src/audio/mod.rs) - Mumble beeps during speaking, paper rustle for notes
Not Yet Implemented:
- Fyrox animation machine with skeletal animations
- Blend shape system for facial expressions
- 3D character model
Event-Driven Animation Architecture (MVP) #
The MVP uses convergence-based event detection instead of fixed timers for turn animations. This is more accurate and prepares for future Fyrox Animation Machine integration.
How It Works #
┌─────────────────────┐
│ AnimationController │ ← Decides state (TurningToCamera, Talking, etc.)
└──────────┬──────────┘
│ state query
▼
┌─────────────────────┐
│ CubeAnimator │ ← Interpolates position/rotation, detects convergence
└──────────┬──────────┘
│ AnimationEvent (when rotation converges)
▼
┌─────────────────────┐
│ AnimationController │ ← Processes events, transitions state
└─────────────────────┘
Event Types #
| Event | Trigger | Action |
|---|---|---|
turn_to_camera_complete |
Rotation converges to camera-facing | TurningToCamera → Talking |
turn_from_camera_complete |
Rotation converges to forward-facing | TurningFromCamera → Idle/Working |
Why Event-Driven? #
Old approach (fixed timers):
if self.turn_timer >= 1.0 { // Assumes 1 second turn
self.body_state = BodyState::Talking;
}
New approach (convergence detection):
// CubeAnimator detects when rotation reaches target
if rotation_angle < 0.01 { // Actually aligned
events.push(AnimationEvent::new("turn_to_camera_complete"));
}
// AnimationController responds to event
fn process_events(&mut self, events: impl Iterator<Item = AnimationEvent>) {
for event in events {
if event.name == "turn_to_camera_complete" {
self.body_state = BodyState::Talking;
}
}
}
Benefits:
- Responds to actual animation completion, not assumed duration
- Works correctly even if frame rate varies
- Seamlessly migrates to Fyrox Animation Machine (just swap event source)
Speaking Duration (Intentionally Timer-Based) #
Speaking duration remains timer-based because it depends on message length, not animation completion:
- Calculated as
chars / 50.0(50 characters/second reading pace) - Clamped to 1-30 seconds
- When real lip-sync arrives, the talking animation will LOOP until this timer expires
Asset Authoring Roadmap #
The placeholder cube is intentional for MVP development. This allows the animation state machine and backend integration to be tested without artist dependency.
Why Placeholder First? #
- Decouple development: Engineers can build animation logic while artists work on models
- Test state transitions: Color-coded cube makes state changes visible and debuggable
- Iterate on timing: Tune transition durations without reworking animation clips
- Reduce risk: Prove the system works before investing in art assets
Production Character Timeline #
| Phase | Milestone | Description |
|---|---|---|
| Phase 1 (Current) | Placeholder Cube | Color-coded states, all animation logic implemented |
| Phase 2 | Character Blockout | Basic mesh (~1K triangles) with skeleton, test rigging |
| Phase 3 | Animation Clips | Author 13 animation clips in Blender |
| Phase 4 | Character Final | Full detail mesh (~5K triangles), textures, blend shapes |
| Phase 5 | Polish | Sound effects, particle effects, final tuning |
Asset Specifications #
Character Model:
- Low-poly humanoid mesh (~5K triangles)
- Victorian-era clothing aesthetic
- Neutral pose for rigging
- PBR materials (albedo, normal, roughness, metallic)
Skeleton:
- ~50 bones humanoid rig
- Standard naming convention for Fyrox import
- IK targets for hands and feet
Facial Blend Shapes:
jaw_open- Talkingbrow_down- Thinking/concentrationsmile- Pleased expressionblink- Eye blink
Animation Clips (13 total):
- See "Animation Clips" table below for full list
- Export as separate files or NLA strips in Blender
- Target ~200KB per clip (~2.6MB total)
Asset Source Options #
- Custom Commission: Hire artist for bespoke Victorian character
- Mixamo Library: Start with Mixamo base, customize
- In-House: Create in Blender (requires 3D art skills)
- Asset Store: Purchase and modify existing character
Recommendation: Start with Mixamo base character to validate pipeline, then commission custom character for final release.
Integration Steps #
When ready to replace placeholder cube:
- Export character from Blender as GLTF
- Import into Fyrox editor
- Set up animation machine with states matching
AnimationController - Update
animation_state.rsto target real skeleton/mesh handles - Add blend shape support to
FaceController - Test all state transitions
- Tune blend times and timing
Overview #
The animation system brings the AI agent character to life through context-aware animations that correspond to tool usage, conversation state, and subagent spawning. It uses Fyrox's built-in animation state machine to blend between animations and synchronize with backend events.
Design Goals #
- Responsive: Animations triggered immediately on backend events (<100ms latency)
- Contextual: Different animations for different tools (Bash vs Read vs WebSearch)
- Smooth: Seamless blending between states (no jarring transitions)
- Performant: 60fps with real-time skeletal animation
- Extensible: Easy to add new animations for new tools
Animation Assets #
Character Rig #
- Skeleton: Humanoid rig (~50 bones)
- Skinned mesh: Low-poly character model (~5K triangles)
- Authored in: Blender, exported as FBX or GLTF
Animation Clips #
| Clip | Duration | Loop | Trigger |
|---|---|---|---|
| idle_sitting | 3s | Yes | Default state |
| idle_thinking | 4s | Yes | Random idle variation |
| idle_stretching | 5s | No | Random idle variation |
| idle_looking_around | 6s | Yes | Random idle variation |
| idle_wandering | 3s | Yes | Random idle variation (agent strolls around room) |
| turn_to_camera | 1s | No | Assistant message starts |
| turn_from_camera | 1s | No | Assistant message ends |
| talking | 2s | Yes | Talking (blended with face animation) |
| typing_terminal | 3s | Yes | Bash, Execute tools |
| reading_document | 3s | Yes | Read, Glob, Grep tools |
| writing_document | 3s | Yes | Write, Edit tools |
| web_browsing | 4s | Yes | WebSearch, WebFetch tools |
| deep_thinking | 5s | Yes | Task, Explore tools (subagent spawn) |
| gesture_explaining | 2s | No | Occasionally during talking |
Total storage: ~30 clips × ~200KB/clip = ~6MB
State Machine #
State Diagram #
┌─────────────────────────────────────────────────────────────┐
│ Idle Layer │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Sitting │ ───► │ Thinking │ ───► │Stretching│ │
│ │ │ ◄─── │ │ ◄─── │ │ │
│ └──────────┘ └──────────┘ └──────────┘ │
│ │ │ │ │
│ └──────────────────┴──────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────┐ │
│ │ Looking Around │ │
│ └─────────────────┘ │
│ │
└─────────────────────────┬───────────────────────────────────┘
│
tool_start event
│
▼
┌─────────────────────────────────────────────────────────────┐
│ Work Layer │
│ │
│ ┌───────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Terminal │ │ Reading │ │ Writing │ │
│ │ │ │ │ │ │ │
│ └───────────┘ └──────────┘ └──────────┘ │
│ │
│ ┌───────────┐ ┌──────────┐ │
│ │ Web │ │ Thinking │ │
│ │ │ │ │ │
│ └───────────┘ └──────────┘ │
│ │
└─────────────────────────┬───────────────────────────────────┘
│
tool_end event
│
▼
┌─────────────────────────────────────────────────────────────┐
│ Speaking Layer │
│ │
│ ┌────────────────┐ │
│ │ Turn to Camera │ ──► Talking ──► Turn from Camera │
│ └────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────┘
State Priorities #
When multiple animation triggers occur:
| Priority | Layer | Condition |
|---|---|---|
| 1 (highest) | Speaking | Assistant is sending text |
| 2 | Work | Tool is executing |
| 3 (lowest) | Idle | No other activity |
Blending: Higher-priority layers override lower layers with smooth transitions (500ms blend time).
Animation Controller #
pub struct AnimationController {
// Current state
body_state: BodyState,
face_state: FaceState,
// Animation machine
animation_machine: Handle<Machine>,
// Timing
idle_variation_timer: f32,
idle_variation_interval: f32, // 10-30 seconds
// Tool tracking
active_tool: Option<String>,
speaking: bool,
}
#[derive(Debug, Clone, PartialEq)]
pub enum BodyState {
Idle(IdleVariant),
Working(WorkAnimation),
TurningToCamera,
TurningFromCamera,
Talking,
}
#[derive(Debug, Clone, PartialEq)]
pub enum IdleVariant {
Sitting,
Thinking,
Stretching,
LookingAround,
Wandering, // Agent strolls to random positions in the room
}
#[derive(Debug, Clone, PartialEq)]
pub enum WorkAnimation {
Terminal, // Typing on keyboard
Reading, // Reading document
Writing, // Writing on paper
WebBrowsing, // Looking at screen, mouse movements
DeepThinking, // Leaning back, hand on chin
}
#[derive(Debug, Clone, PartialEq)]
pub enum FaceState {
Neutral,
Talking { intensity: f32 }, // 0.0 to 1.0
Thinking, // Furrowed brow
Pleased, // Slight smile
}
impl AnimationController {
pub fn update(&mut self, dt: f32, scene: &mut Scene) {
// Update idle variation timer
if matches!(self.body_state, BodyState::Idle(_)) {
self.idle_variation_timer += dt;
if self.idle_variation_timer >= self.idle_variation_interval {
self.trigger_idle_variation();
self.idle_variation_timer = 0.0;
self.idle_variation_interval = rand::thread_rng().gen_range(10.0..30.0);
}
}
// Update animation machine
scene.graph[self.animation_machine]
.as_animation_machine_mut()
.update(dt);
}
pub fn on_tool_start(&mut self, tool: &str, scene: &mut Scene) {
self.active_tool = Some(tool.to_string());
let work_anim = Self::categorize_tool(tool);
self.body_state = BodyState::Working(work_anim);
// Send transition event to animation machine
self.send_event("tool_start", work_anim.as_str(), scene);
}
pub fn on_tool_end(&mut self, scene: &mut Scene) {
self.active_tool = None;
// Return to idle if not speaking
if !self.speaking {
self.body_state = BodyState::Idle(IdleVariant::Sitting);
self.send_event("tool_end", "idle_sitting", scene);
}
}
pub fn on_speaking_start(&mut self, scene: &mut Scene) {
self.speaking = true;
self.body_state = BodyState::TurningToCamera;
self.send_event("speaking_start", "turn_to_camera", scene);
// After turn animation (1s), start talking
// TODO: Use animation event callback when turn_to_camera finishes
}
pub fn on_speaking_end(&mut self, scene: &mut Scene) {
self.speaking = false;
self.body_state = BodyState::TurningFromCamera;
self.send_event("speaking_end", "turn_from_camera", scene);
// After turn animation (1s), return to idle or tool animation
// TODO: Use animation event callback
}
pub fn on_text_delta(&mut self, _content: &str, scene: &mut Scene) {
// Update talking animation intensity based on text rate
// For now, just ensure we're in talking state
if matches!(self.body_state, BodyState::TurningToCamera) {
// Wait for turn to finish
} else if self.speaking {
self.body_state = BodyState::Talking;
self.send_event("talking", "talking", scene);
}
}
fn trigger_idle_variation(&mut self) {
// Randomly pick idle variant (20% chance for each)
use rand::Rng;
let mut rng = rand::thread_rng();
let variant = match rng.gen_range(0..5) {
0 => IdleVariant::Sitting,
1 => IdleVariant::Thinking,
2 => IdleVariant::Stretching,
3 => IdleVariant::LookingAround,
_ => IdleVariant::Wandering,
};
self.body_state = BodyState::Idle(variant);
// TODO: Send event to animation machine
}
fn categorize_tool(name: &str) -> WorkAnimation {
match name {
"Bash" | "BashOutput" | "Execute" => WorkAnimation::Terminal,
"Read" | "Glob" | "Grep" | "View" => WorkAnimation::Reading,
"Write" | "Edit" | "MultiEdit" => WorkAnimation::Writing,
"WebSearch" | "WebFetch" | "web_search" | "web_fetch" => WorkAnimation::WebBrowsing,
"Task" | "Explore" => WorkAnimation::DeepThinking,
"mcp__room-tools__create_note" => WorkAnimation::Writing,
"mcp__room-tools__look_around" | "mcp__room-tools__examine" => WorkAnimation::Reading,
_ => WorkAnimation::DeepThinking, // Fallback
}
}
fn send_event(&self, event_name: &str, target_animation: &str, scene: &mut Scene) {
// TODO: Fyrox animation machine API for events
// This would trigger a transition in the animation state machine
}
}
Fyrox Animation Machine Setup #
Fyrox provides a visual animation editor for creating state machines. Our machine would have:
States #
Each state plays an animation clip (looping or one-shot):
State::new("idle_sitting")
.with_animation("idle_sitting")
.with_loop(true)
State::new("turn_to_camera")
.with_animation("turn_to_camera")
.with_loop(false)
.with_exit_event("turn_complete") // Trigger when animation finishes
State::new("talking")
.with_animation("talking")
.with_loop(true)
Transitions #
Transitions define when to move between states:
Transition::new("idle_sitting", "turn_to_camera")
.with_condition(Condition::Event("speaking_start"))
.with_blend_time(0.3)
Transition::new("turn_to_camera", "talking")
.with_condition(Condition::Event("turn_complete"))
.with_blend_time(0.2)
Transition::new("talking", "turn_from_camera")
.with_condition(Condition::Event("speaking_end"))
.with_blend_time(0.3)
Transition::new("turn_from_camera", "idle_sitting")
.with_condition(Condition::Event("turn_complete"))
.with_blend_time(0.5)
Note: This is pseudocode. Actual Fyrox API uses MachineBuilder or visual editor.
Face Animation #
Blend Shapes (Morph Targets) #
Face expressions use blend shapes (vertex deformation):
| Blend Shape | Purpose |
|---|---|
jaw_open |
Talking (opens mouth) |
brow_down |
Thinking (furrowed brow) |
smile |
Pleased expression |
blink |
Eye blink (automatic, random) |
Face Controller #
pub struct FaceController {
state: FaceState,
mesh: Handle<Mesh>,
// Blend shape weights (0.0 to 1.0)
jaw_open: f32,
brow_down: f32,
smile: f32,
// Blink timer
blink_timer: f32,
next_blink: f32,
}
impl FaceController {
pub fn update(&mut self, dt: f32, scene: &mut Scene) {
// Smooth blend shape interpolation
match self.state {
FaceState::Neutral => {
self.jaw_open = Self::lerp(self.jaw_open, 0.0, dt * 5.0);
self.brow_down = Self::lerp(self.brow_down, 0.0, dt * 5.0);
self.smile = Self::lerp(self.smile, 0.0, dt * 5.0);
}
FaceState::Talking { intensity } => {
// Animate jaw opening (simple noise function for now)
let target = (self.jaw_open + dt * 10.0).sin().abs() * intensity;
self.jaw_open = Self::lerp(self.jaw_open, target, dt * 20.0);
}
FaceState::Thinking => {
self.brow_down = Self::lerp(self.brow_down, 0.6, dt * 3.0);
}
FaceState::Pleased => {
self.smile = Self::lerp(self.smile, 0.5, dt * 3.0);
}
}
// Automatic blinking
self.blink_timer += dt;
if self.blink_timer >= self.next_blink {
self.trigger_blink();
self.blink_timer = 0.0;
self.next_blink = rand::thread_rng().gen_range(2.0..6.0);
}
// Apply blend shapes to mesh
if let Some(mesh) = scene.graph.try_get_mut(self.mesh) {
if let Some(skinned) = mesh.as_mesh_mut() {
skinned.set_blend_shape_weight("jaw_open", self.jaw_open);
skinned.set_blend_shape_weight("brow_down", self.brow_down);
skinned.set_blend_shape_weight("smile", self.smile);
}
}
}
fn lerp(current: f32, target: f32, alpha: f32) -> f32 {
current + (target - current) * alpha.min(1.0)
}
fn trigger_blink(&self) {
// TODO: Play blink animation (quick close-open)
}
}
Rationale:
- Blend shapes over skeletal: Faces need subtle deformations that bones can't provide
- Simple talking: Jaw movement based on sine wave (cheap approximation of speech)
- Automatic blink: Adds life without backend coordination
Camera Control #
Current Implementation (MVP) #
The camera uses a fixed isometric view from the "open corner" of the room:
| Property | Value | Notes |
|---|---|---|
| Position | (7.0, 7.0, 7.0) |
Elevated, equal X/Z for true isometric |
| Look Target | (-0.5, 0.0, -0.5) |
Toward room center, offset to wall corner |
| Projection | Orthographic | No perspective distortion |
| Vertical Size | 6.0 |
Controls zoom (smaller = more zoomed in) |
This creates a classic isometric RPG-style view looking diagonally into the Victorian study from the front-right corner. The room is 6×6 units with walls at X=-3 (left) and Z=-3 (back), forming a cozy corner.
Implementation (in game/src/lib.rs, on_scene_loaded):
- Camera position and rotation set programmatically
- Orthographic projection applied via
camera.set_projection() - Fallback warnings logged if camera node not found or not a Camera component
Future Enhancement: Dynamic Camera Focus #
The following describes planned dynamic camera behavior (not yet implemented):
pub struct CameraController {
camera: Handle<Node>,
// Fixed isometric position (current MVP default)
default_position: Vec3, // (7.0, 7.0, 7.0)
default_target: Vec3, // (-0.5, 0.0, -0.5)
// Focus state (future enhancement)
focusing_on: Option<FocusTarget>,
focus_progress: f32, // 0.0 to 1.0
}
#[derive(Debug, Clone, PartialEq)]
pub enum FocusTarget {
Character, // During speaking (future)
Subagent(String), // When subagent spawns (future)
}
impl CameraController {
// FUTURE: Dynamic camera control - not implemented in MVP
pub fn focus_on_character(&mut self) {
self.focusing_on = Some(FocusTarget::Character);
self.focus_progress = 0.0;
}
pub fn update(&mut self, dt: f32, scene: &mut Scene) {
// FUTURE: Implement smooth camera transitions
// For MVP, camera remains fixed at isometric position
let target_pos = match &self.focusing_on {
Some(FocusTarget::Character) => {
// Move camera closer to character's face
Vec3::new(0.0, 1.5, 2.0)
}
Some(FocusTarget::Subagent(_id)) => {
// Look at subagent position
Vec3::new(2.0, 1.5, 2.0)
}
None => self.default_position,
};
// Smooth interpolation (2-second transition)
self.focus_progress = (self.focus_progress + dt * 0.5).min(1.0);
// ... lerp camera position ...
}
}
Rationale (Current MVP):
- Fixed isometric view: Consistent environmental visibility, classic aesthetic
- Orthographic projection: True isometric proportions without perspective distortion
- Simple implementation: No complex camera state management needed yet
Future (Phase 2):
- Focus on character: When speaking, camera smoothly pans closer (better view of expressions)
- Focus on subagent: When subagent spawns, camera shows spatial relationship
- Smooth transitions: 2-second interpolation avoids jarring jumps
Subagent Visualization #
Open design question: How do we represent subagents in the 3D scene?
Option 1: Separate Character #
- Spawn a second character model at the "lectern" surface
- Same rigging, different materials/colors (e.g., blue tint for subagent)
- When subagent is working, animate it independently
Pros: Clear visual distinction, can show parallel work Cons: Complexity (managing two animation controllers), memory overhead
Option 2: Particle Effect #
- Spawn glowing orb or particle effect at lectern
- Pulsates when subagent is active
- No detailed animation needed
Pros: Simple, low overhead Cons: Less immersive, no character presence
Option 3: UI Indicator Only #
- No 3D representation
- Show subagent status in UI overlay (e.g., "Subagent: Searching codebase...")
Pros: Simplest Cons: Breaks immersion, not spatial
Recommendation: Start with Option 3 (UI only) for MVP, consider Option 1 for future polish.
Performance #
Animation Playback #
- Skeletal animation: ~0.5ms per character at 50 bones
- Blend shape updates: ~0.1ms per frame (4 blend shapes)
- State machine updates: <0.1ms
Total: <1ms per frame (well under 16ms budget for 60fps)
Memory #
- Animation clips: ~6MB (all clips loaded)
- Character mesh: ~500KB (skinned mesh + textures)
- State machine: ~10KB
Total: ~7MB
Sound System (MVP) #
The sound system provides audio feedback for agent interactions, implemented in game/src/audio/mod.rs.
Sound Style Philosophy #
Hybrid Aesthetic: The agent "speaks" in chiptune (digital entity) but interacts with the Victorian study's physical objects using realistic analog sounds. Digital being in an analog world!
Implemented Sounds #
| Event | Sound | Implementation |
|---|---|---|
| Assistant speaking | Chiptune beep (looped) | beep_mumble.wav loops at ~150ms interval during AssistantTextDelta |
| Open note/image | Paper rustle | paper_rustle.wav plays once when viewer opens |
SoundManager #
pub struct SoundManager {
mumble_buffer: Option<SoundBufferResource>,
paper_rustle_buffer: Option<SoundBufferResource>,
is_speaking: bool,
last_beep_time: Option<Instant>,
master_volume: f32,
audio_enabled: bool,
active_sounds: Vec<Handle<Node>>,
}
Key methods:
start_speaking()/stop_speaking()- Control mumble loop stateupdate(scene)- Called each frame, plays beeps at interval during speakingplay_paper_rustle(scene)- One-shot paper soundset_volume(volume, scene)/set_enabled(enabled, scene)- Settings integration
Usage Flow #
- Initialization:
SoundManager::new(volume, enabled)inGame::init() - Asset loading:
load_sounds(resource_manager)inon_scene_loaded() - Speaking:
start_speaking()on firstAssistantTextDelta,stop_speaking()onAssistantMessageComplete - Frame update:
update(scene)handles timed beep playback - Settings:
apply_audio_settings()syncs Settings dialog changes
Future Expansion (Deferred) #
Tool category sounds (terminal typing, page flips) are intentionally omitted for MVP to avoid audio fatigue. The infrastructure supports adding more sounds later by:
- Adding new
SoundBufferResourcefields - Creating play methods following
play_paper_rustlepattern - Wiring to appropriate events in
lib.rs
Open Questions #
-
Audio sync: Should we play audio (keyboard typing, paper rustling) with animations?- RESOLVED: Implemented in
game/src/audio/mod.rs - Mumble beeps loop during assistant speaking (Animal Crossing style)
- Paper rustle plays when opening notes/images
- Volume and enable/disable controllable via Settings dialog
- Kept minimal to avoid audio fatigue (no tool sounds for MVP)
- RESOLVED: Implemented in
-
Animation events: How do we know when "turn_to_camera" finishes?
- Fyrox supports animation event markers
- Need to author events in Blender and export
- Alternative: Fixed timer (less accurate, simpler)
-
Subagent animation: See "Subagent Visualization" above
-
Gesture timing: When should character gesture while talking?
- Random (every 5-10 seconds)?
- Based on text content (punctuation, emphasis)?
- Manual trigger from backend (agent sends "gesture" hint)?
-
Tool animation feedback: Should tools send progress updates?
- E.g., "Bash: still running... (5s)"
- Would allow different animations for quick vs slow tools
- Current: Single animation regardless of duration
Related Documents #
- Frontend Architecture - Animation controller integration
- IPC Protocol - Tool events that trigger animations
- Backend Adapter - Subagent message handling