Project AURA
A delusion made real.
A real-time AI companion with a sub-second voice loop, an emotion-aware Live2D rig, and a control center that hot-swaps the entire AI brain at runtime.
Five distinct emotion states drive a Live2D rig in real time.
Each LLM reply is classified into an emotion vector— happy, playful, surprised, companion, or idle — and the avatar's Live2D parameters (eyes, mouth, head tilt) blend toward that state while the TTS audio plays. The ghost helper is summoned for companion mode.
detected from positive sentiment + exclamation tokens
The Pipeline
four hops, under 800msDemo
v0.1.1 · live voice loopThe Emotion System
five poses, blended over Live2D parametersAvatar Rendering
browser-native, zero external softwareThe avatar runs entirely in the browser — no VTube Studio, no external process. The animation loop is monkey-patched into coreModel.update() so every frame gets injected parameters right before GPU commit.
Organic Blink
A finite state machine fires random timers to trigger eye blinks. Probability and cooldown vary so the blink pattern never repeats identically.
Eye Saccades
Micro-randomization is added to gaze direction every frame via Math.random(). The eyes drift like real eyes — never perfectly still.
RMS Lip Sync
AnalyserNode computes Root Mean Square amplitude from the LiveKit audio track in a requestAnimationFrame loop and maps it directly to mouth open.
AI Memory & RAG
reads documents, recalls on demandTeaching AURA
Upload any document. AURA ingests, chunks, and embeds it — the knowledge becomes instantly searchable during live conversation.
AURA Remembers
Every utterance triggers a semantic search. Matching chunks are silently injected into the LLM context — AURA appears to just know.
The Interface
not just an avatar — a full control surfaceWhat I built · what's next
What's in v0.1.1
- ✦End-to-end voice loop under 800ms from speech end to TTS start.
- ✦Multilingual input/output — Deepgram + Qwen 3 handle EN ↔ ID switching mid-sentence.
- ✦Five-state emotion system with prompt-based classification per turn.
- ✦Live2D rig driven by emotion vectors, lip-sync from TTS phonemes.
- ✦Hot-reload model swap — change LLM provider/model live from the control center.
- ✦Per-context memory — every chat keeps its own personality, history, and creativity dial.
What's next
- →Tool use — function-calling for calendar, search, code execution.
- →Long-term memory with vector recall across contexts.
- →Vision input — webcam frame analysis so AURA can see what you're working on.
- →Avatar marketplace — pluggable Live2D rigs with rig-aware emotion mapping.
- →Mobile client for the conversation loop.
- →v1.0 when emotion classification moves on-device.
Building real-time AI? Let's compare notes.
I'm always happy to swap notes on Live2D rigging, sub-second voice pipelines, or emotion classification prompts. Or if you want AURA's roadmap to head somewhere specific — let me know.