Sensitive Virtual Humans

Project description

This project aims to design a system that allows humans to start natural conversations with virtual humans. It focuses on processing data like body language, facial expressions, and voice to detect when and how to start a conversation. The project uses tools like Mediapipe and DeepFace to create an interactive and responsive system.

Context

Virtual humans are becoming important tools in many areas like teaching, healthcare, and online support. However, current systems often fail to understand body language or voice properly, making conversations feel unnatural. This project looks at how to improve these systems by studying how people use non-verbal signals, like gestures and facial expressions, to start and continue conversations. By creating a system that processes these signals, the project aims to improve how virtual humans interact with people, making them more helpful and realistic. This can be useful in teaching students, assisting patients, or even in customer support.

Results

The project will create a system that allows virtual humans to detect and respond to cues like gestures, gaze, and voice to start conversations. It will deliver:

1. A Proof of Concept that uses tools like Mediapipe, DeepFace and Voss to process body language and voice. The second part of the proof of concept is a 3D avatar that listens for the verbal and non-verbal conversational cues and responds by the use of an LLM.
2. A better understanding of how non-verbal cues like head movements and voice tone help start conversations and how to use them in virtual humans to improve interactions.
3. The design that is adaptable to future improvements in AI and sensor technology.

The system was tested in a controlled environment first and later in real-life situations. This project will help improve virtual humans for teaching, patient care, and customer support by making them more responsive and realistic in conversations.