Assistant Voice and Audio

The assistant module features a sophisticated voice interface, primarily centered around the "Nikki Aura" persona, supporting speech-to-text (STT) and text-to-speech (TTS) capabilities.

Voice UI Pages

The system provides two distinct voice interface pages:

voice.html - Always Listening Mode

Purpose: Continuous monitoring where agent name detection triggers processing.

Best For:

• Desktop/web use where power consumption is not a concern
• Scenarios where agent name is mentioned anywhere in speech
• Development and testing

How It Works:

1. Microphone continuously streams audio
2. VAD (Voice Activity Detection) determines speech boundaries
3. Audio is sent to server for STT
4. If agent name detected in transcription, query is processed
5. Response is streamed back via SSE

wake.html - Wake Word Mode

Purpose: Energy-efficient wake-word triggered interaction using Picovoice Porcupine.

Best For:

• Tablet kiosk devices (always-on home assistant)
• Smart speaker mode
• Low-power scenarios
• Privacy-conscious deployments (audio only sent after wake)

How It Works:

1. Local Porcupine engine listens for "Hey Nikki"
2. On wake-word detection, starts recording command
3. VAD determines end of speech
4. Audio sent to server for STT
5. Response streamed back, then returns to wake-word listening

Kiosk Mode

Both voice pages support kiosk mode for tablet deployment.

Activation

Add URL parameters to enable kiosk mode:

https://your-server/voice?kiosk=true&agent=nikki
https://your-server/wake?kiosk=true&agent=lenore

Kiosk Mode Features

The Nikki Aura Interface

Nikki Aura is a highly interactive, voice-driven UI designed for mobile and web.

Key Features

• Barge-in: Users can interrupt Nikki mid-sentence to provide new input.
• Hollow Ring Visuals: A multi-layered, glowing neon ring that oscillates based on audio frequency.
• Organic Streaming: Text is streamed word-by-word, perfectly synchronized with audio chunks for a natural feel.
• Extended Follow-up: The microphone stays active for a window after Nikki finishes speaking to encourage continuous conversation.
• Agent Theming: Each agent has unique colors (Nikki: Violet/Orange, Lenore: Indigo/Purple, etc.)
• Haptic Feedback: Physical vibration on supported mobile devices upon wake-word detection for tactile confirmation.
• Responsive Visuals: The Aura visualizer automatically scales for mobile, increasing the ring size while reducing complexity (fewer canvas points) to optimize performance.

Browser & Mobile Compatibility

• Permission Overlay: A dedicated UI layer appears if microphone access is blocked or needs manual initialization, ensuring a smooth experience in self-signed HTTPS or restrictive browser environments.
• AudioContext Resumption: Automatically handles browser "autoplay" restrictions by resuming the AudioContext upon the first user interaction (e.g., clicking the Mic button).
• Mobile History Drawer: Conversation history transforms into a full-screen drawer on mobile devices with a dedicated close button for better ergonomics.

Agent Themes

Colors are defined in nikki_voice.go:AgentThemes:

TTS Configuration

Per-Agent Defaults

Each agent has a preferred TTS provider configured in tts.go:GetAgentTTSConfig():

User Override

Users can override TTS provider via the dropdown:

• Agent Default: Uses agent's configured provider
• Inworld: High-quality streaming TTS
• ElevenLabs: Premium voice synthesis
• OpenRouter: OpenAI-compatible TTS

Accessibility

Both voice pages include screen reader support:

• ARIA Live Region: Announces state changes ("AI is thinking", "Listening for follow-up")
• Semantic HTML: Proper heading hierarchy and landmarks
• Keyboard Navigation: All controls keyboard-accessible
• Focus Management: Clear focus indicators on interactive elements

Audio Processing Stack

1. Speech-to-Text (STT)

The system uses assistant/stt.go to transcribe user audio. It supports multiple backends and can detect if a specific agent is being mentioned in the audio.

2. Text-to-Speech (TTS)

The system uses assistant/tts.go to generate high-quality audio responses. It is specifically tuned for the Nikki persona to ensure consistent tone and inflection.

3. Voice Handler

The VoiceHandler (in assistant/voice_handler.go) coordinates the flow: Audio In → STT → Agent Query → TTS → Audio Out.

4. TTS Text Sanitization

Before TTS generation, text is sanitized to remove voice-unfriendly elements:

• Emojis stripped (fox, sparkle, etc.)
• Markdown formatting removed
• Headers normalized
• Links converted to text

See tts.go:SanitizeForTTS() for implementation.

State Machine

Voice sessions follow a three-state model defined in nikki_voice.go:

┌───────┐  Wake/Agent   ┌────────┐  Response    ┌────────────┐
│ Idle  │ ────────────> │ Active │ ───────────> │ Follow-up  │
│       │ <──────────── │        │ <──────────── │            │
└───────┘  10s timeout  └────────┘  Speech done └────────────┘

• Idle: Waiting for trigger (wake word or agent name)
• Active: Processing query, generating response
• Follow-up: 10-second window for continuous conversation

Sequence Diagram: Voice Interaction

sequenceDiagram
    participant U as User (Web/Mobile)
    participant STT as stt.go
    participant A as Assistant Session
    participant TTS as tts.go
    participant UI as Nikki Aura UI

    U->>UI: Speaks Command
    UI->>STT: Transmit Audio Stream
    STT->>A: Transcribed Text
    A->>A: Generate Response
    A->>TTS: Text Response (Sanitized)
    TTS-->>UI: Audio Stream Chunks
    UI-->>U: Play Audio + Glow Ring

Key Files & Functions

Guidance for AI Agents

• Conciseness in Voice: When responding via voice, keep your answers concise and conversational. Avoid long lists or complex markdown tables.
• Prosody Cues: Use punctuation effectively to guide the TTS engine's inflection and pausing.
• No Visual Elements: Voice responses should not include emojis or markdown formatting as they won't translate to speech.

Cross-References

• Key-Packages/assistant/overview.md
• Features/assistant/session-management.md
• Features/assistant/webserver-api.md
• Key-Packages/dwhatsapp/overview.md - WhatsApp voice note handling