
Quest
Build a secure local voice interface that allows Brent to speak naturally to Ari and request computer actions, including unlocking the Brentputer, without requiring one exact memorized command phrase.
Core interaction
A lab microphone provides an always-available voice channel. Local speech recognition transcribes Brent’s request, and Ari interprets the semantic intent. Many ordinary phrasings can resolve to the same constrained action request, such as REQUEST_UNLOCK. Natural-language interpretation does not itself grant access.
Security architecture
Keep intelligence and authority separate. Ari may interpret what Brent means, but a deterministic security service independently decides whether a privileged action is authorized.
For computer unlocking, combine multiple signals:
- Possession: a physical keychain token must be nearby and cryptographically authenticate itself.
- Identity: local speaker verification checks Brent’s enrolled voiceprint.
- Intent: the spoken request must clearly express an authorized action such as unlocking the computer.
- Fallback: retain normal Windows PIN/password recovery so experimental authentication cannot lock Brent out.
Cryptographic proximity token
Do not rely on a permanent Bluetooth identifier or simple proximity beacon. The computer should issue a fresh random challenge and the keychain device should answer using a protected secret or private key. The computer verifies the response so captured Bluetooth traffic cannot simply be replayed later.
Voice pipeline
Use a lightweight local listening/wake layer, record the relevant utterance, perform local speech-to-text, perform speaker verification independently of transcription, and pass the resulting text to a constrained intent interpreter. The interpreter should emit only approved action types rather than arbitrary operating-system commands.
Windows integration
Investigate a Windows Credential Provider or equivalent bounded unlock bridge. Hodor is a candidate reference implementation because it exposes a Windows 11 Credential Provider intended to receive an unlock request from an external application. Test Credential Provider work cautiously with a VM and a reliable recovery path before deploying it to the primary machine.
Candidate open-source components
- Hodor: Windows Credential Provider / external unlock bridge.
- openWakeWord: local wake-word detection and custom wake models.
- Whisper-based local transcription: offline speech-to-text.
- SpeechBrain ECAPA-TDNN or comparable speaker-verification model: multi-sample voice enrollment and text-independent speaker verification.
Anti-replay and confidence
A recording of Brent’s voice must not be treated as equivalent to Brent being present. Explore anti-replay/liveness measures and combine voice evidence with the cryptographically authenticated physical token. Camera recognition can later become an additional confidence signal, but visual recognition alone should not silently unlock a computer. Presence establishes who appears to be there; an intentional request establishes consent to unlock.
Future multimodal interface
Integrate this system with Ari’s existing JARVIS-style spatial gesture interface. Cameras can eventually contribute face, body, gait, and room-context signals while microphones provide natural conversation and gesture/gaze provide additional control. The long-term goal is a continuous multimodal lab interface rather than isolated voice commands.
MVP
The first useful version should:
- Capture speech from the lab microphone locally.
- Transcribe flexible natural-language requests.
- Verify Brent’s speaker identity.
- Cryptographically verify the nearby keychain token.
- Map natural language into a small allowlist of explicit intents.
- Require the security daemon to authorize privileged intents independently.
- Unlock the Brentputer through a bounded Windows integration.
- Preserve PIN/password fallback and produce an auditable authentication log.
Design principle
Ari interprets intent. The security layer grants authority.
The system should feel conversational and flexible at the human-facing layer while remaining narrow, deterministic, recoverable, and auditable at the security boundary.
Definition of success
Brent can enter the lab with his authenticated physical token, naturally ask Ari to unlock the Brentputer using wording he has not pre-programmed, and have the system correctly identify Brent, understand the requested action, independently authorize it, and unlock the machine without exposing the Windows credential or relying on a reusable spoken password.
