AffectFace

Privacy-first emotion detection Chrome extension with LLM integration and emotional competence scoring

Multi-modal AIReal-time ProcessingPrivacy-firstChrome ExtensionTypeScriptMediaPipe

Overview

AffectFace is a production-ready Chrome extension (Manifest V3) that performs real-time emotion detection using audio and video analysis. It integrates with multiple LLM providers (OpenAI, Anthropic, local endpoints) and provides a comprehensive rubrics-based scoring system for evaluating emotional competence in AI responses.

Multi-modal Detection
  • • Audio prosody analysis (pitch, loudness, tempo)
  • • Facial expression detection via MediaPipe
  • • Weighted fusion of audio, video, and text
  • • Real-time processing at 10 Hz
LLM Integration
  • • OpenAI (GPT-4, GPT-3.5)
  • • Anthropic (Claude 3)
  • • Local HTTP endpoints
  • • Real-time emotion context for queries
Rubrics Scoring
  • • 5 criteria evaluation system
  • • Weighted scoring with customization
  • • CSV export for analysis
  • • Dashboard with statistics
Privacy-First Design
  • • All analysis performed on-device
  • • No raw audio/video sent externally
  • • Explicit consent required
  • • Visible watermark during analysis

Architecture

VBD (Value-Based Development)
Built with strict separation of concerns and modular design

Accessors Layer

Data access layer with CRUD operations for storage management

Engines Layer

Business logic for emotion fusion, classification, and analysis

Managers Layer

Workflow orchestration coordinating multiple engines

Use Cases

Legal Professionals

Analyze client emotional states during consultations and depositions

Therapists & Counselors

Track emotional patterns in therapy sessions for better outcomes

AI Researchers

Study emotional competence in AI systems and LLM responses

Communication Training

Evaluate communication skills and empathy in professional settings

Technology Stack

TypeScript 5.3

Language

Vite 5.0

Build Tool

MediaPipe

ML Framework

Web Audio API

Audio Analysis