Privacy-first emotion detection Chrome extension with LLM integration and emotional competence scoring
AffectFace has been tested on Chrome browsers and iPhone (Safari) only. Other browsers may experience compatibility issues.
The live app uses AI APIs (OpenAI, Anthropic, Hume AI) which incur costs per request. Please use the app thoughtfully and consider supporting this research.
Support This ResearchAffectFace is a production-ready Chrome extension (Manifest V3) that performs real-time emotion detection using audio and video analysis. It integrates with multiple LLM providers (OpenAI, Anthropic, local endpoints) and provides a comprehensive rubrics-based scoring system for evaluating emotional competence in AI responses.
Data access layer with CRUD operations for storage management
Business logic for emotion fusion, classification, and analysis
Workflow orchestration coordinating multiple engines
Track emotional patterns in therapy sessions for better outcomes
Study emotional competence in AI systems and LLM responses
Evaluate communication skills and empathy in professional settings
Provide emotion awareness for individuals seeking better self-understanding
TypeScript 5.3
Language
Vite 5.0
Build Tool
MediaPipe
ML Framework
Web Audio API
Audio Analysis
AffectFace analyzes emotions through three parallel streams that converge into a unified emotion vector, providing comprehensive emotional intelligence in real-time.
Real-time voice analysis capturing emotional truth that words can hide.
Advanced facial analysis using MediaPipe Face Landmarker with FACS-based emotion mapping.
Linguistic analysis providing explicit emotional content that audio and video might miss.
Intelligent weighted combination of audio, video, and text signals with dynamic adjustment.
Identifies when facial expressions and voice tone disagree - revealing emotional masking or suppression.
HIPAA-ready design with on-device processing and zero data transmission.
Comprehensive data export and analysis tools for researchers and professionals.
Pure data access layer - database-agnostic CRUD operations
Business logic and domain rules - the intelligence core
Workflow orchestration - coordinating multiple engines
AffectFace is fully open source (MIT License) to promote transparency, security, and innovation in emotion AI. The codebase includes comprehensive documentation, research findings, and real-world use cases from legal, therapy, and counseling professionals.
Therapists track emotional patterns over multiple sessions, identifying worsening depression before crisis and helping clients with alexithymia develop emotional awareness.
“This isn't replacing my clinical judgment—it's enhancing it with objective data.”
University counselors identify at-risk students during routine advising, enabling early intervention and preventing dropout through timely mental health support.
“Students are really good at hiding distress. AffectFace helps me identify those who need support.”
Primary care physicians improve pain assessment accuracy by detecting when patients under-report due to cultural factors or fear, leading to better pain management.
“When face, voice, and words don't align, I know to probe deeper.”
Researchers study the evolution of emotional intelligence in AI systems, comparing models from GPT-3 to GPT-4o to understand how affective computing capabilities have advanced.
“AffectFace enables longitudinal studies of AI emotional competence that weren't possible before.”