{"id":4341,"date":"2026-03-27T13:08:44","date_gmt":"2026-03-27T13:08:44","guid":{"rendered":"https:\/\/www.justinecassell.com\/articulab\/?page_id=4341"},"modified":"2026-06-23T08:34:32","modified_gmt":"2026-06-23T08:34:32","slug":"son-of-sara","status":"publish","type":"page","link":"https:\/\/www.justinecassell.com\/articulab\/projects\/son-of-sara\/","title":{"rendered":"Son of Sara"},"content":{"rendered":"\n
\n

<\/p>\n\n\n\n

<\/p>\n\n\n\n

Son of Sara : Developing a new LLM-based<\/h1>\n\n\n\n

<\/p>\n\n\n\n

Embodied Conversational Agent.<\/h1>\n\n\n\n

<\/p>\n\n\n\n

<\/p>\n<\/div>\n\n\n\n

<\/div>\n\n\n\n

Overview<\/h2>\n\n\n\n

As part of our ongoing work on socially capable conversational agents, the team launched what we are calling Son of SARA (see the SARA project on this page<\/a>), an embodied conversational agent designed to support natural and effective interaction with human users, relying on both rapport and task effectiveness to assure good collaboration between human and agent. In this context, and as always, the project aims to equip dialogue agents with both verbal and non-verbal interactional skills that are essential for collaborative communication. Building on our lab\u2019s tradition to build socially aware ECAs, we aim to elevate their naturalness and adaptability, alongside other key attributes, using the robust generalization capabilities of LLMs and cutting-edge deep learning models.<\/p>\n\n\n\n

Our current system follows the team’s traditional approach, utilizing a modular\/cascaded pipeline with specialized modules for each conversational phenomenon. The processing and generative pipeline will allow the system, from the voice of the user, to generate a suitable agent response. Each module of the pipeline can process or generate a different combination of multimodalities (speech, gesture, head and face movements, (text, audio). To ensure that the complexity of the modular architecture and its deep learning models won\u2019t prevent the agent from interacting with the user in real time, we decided to go with an incremental system, following the approach of several dialogue systems and conversational agent from colleagues, such as (Schlangen & Skantze, 2011)<\/a>. <\/p>\n\n\n\n

<\/div>\n\n\n\n

First version : Focusing on speech<\/h2>\n\n\n\n

We initially focused on building a conversational agent, without the embodied part of the agent. We successively created an agent that people can interact with, in real time, with a fairly flexible approach towards the scenario of interaction (free discussion, Q&A, collaborative task, etc). That first version of the agent, that we might call the \u201cskeleton\u201d of the pipeline, contains all required models to allow a speech conversation with : It captures and transcribes the user\u2019s voice with the Microphone and Automatic speech recognition (ASR), generates a suitable text then spoken response with the LLM and Text to speech (TTS). The dialogue dynamics are handled with the Voice Activity Detection (VAD) and Dialogue Manager modules. To qualitatively test that first version of the agent, we mimicked the collaborative task used in our Collab  team\u2019s longitudinal study which consist in discussing an image and coming up with hypothesis regarding the different image\u2019s elements (you’ll soon be able to find more information in the corresponding Collab study page).<\/p>\n\n\n\n

<\/div>\n\n\n\n

Non Verbal Behavior<\/h2>\n\n\n\n

Continuing the first agent version, we focused on extending it beyond spoken interaction by adding a visual and embodied dimension to the conversation. Building on an existing conversational framework enabling spoken dialogue, the team developed the agent\u2019s virtual body and implemented the software infrastructure required to synchronize non-verbal behaviors with speech. This included the design and implementation of preliminary rule-based models for the generation of facial expressions and gestures, allowing the agent to produce visible communicative signals aligned with its speech turns. <\/p>\n\n\n\n

<\/div>\n\n\n\n
\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Figure 1 : Son of Sara\u2019s 20 years old virtual body during a conversation<\/h5>\n\n\n\n
<\/div>\n\n\n\n

The gesture generation system aims to produce plausible, semantically relevant co-speech gestures in real-time during agent-user interaction. Initially, the team explored a retrieval-based approach, selecting gestures from a pre-existing gesture library based on verbal and syntactic cues extracted from the agent’s speech. This system used transformer-based encoders (BGE\/MiniLM) to embed utterances, then ranked candidate gestures via consensus re-ranking across multiple semantic views. While functional and fast, this method lacks flexibility, often producing generic scripted gestures that are vaguely appropriate rather than movements finely tailored to context.<\/p>\n\n\n\n

<\/div>\n\n\n\n