VibeSense: Hearing Emotions Through AI-Powered Facial Analysis

What if you could hear someone's emotions instead of seeing them? Meet VibeSense, a nwHacks 2026 (@nwPlusUBC) project built by Talia Feng, Anica Mok, Vi Do, and Gordon Chieng that helps blind and visually impaired people sense facial expressions in real-time. Here's how it works: 1️⃣ Real-Time Detection: A webcam captures facial expressions using DeepFace and TensorFlow, identifying seven core emotions as they happen. 2️⃣ Biometric Context: An iOS companion app built with Presage's SmartSpectra technology adds heartbeat and breathing data for a fuller emotional picture. 3️⃣ Smart Feedback: With a simple long press gesture, the system sends recent emotion data to Google's Gemini AI for a context aware summary, then plays distinct beep patterns through Arduino hardware so users can instantly interpret the mood of a conversation. By combining computer vision, biometric sensing, and AI, VibeSense opens the door to more confident, connected social interactions for the blind and visually impaired community. 🔗 Check out how they built it: https://devpost.com/software/vibesense