Overview

At Articulab, we work toward conversational agents that adapt to humans, grounded in a deeper understanding of how people naturally communicate. This project contributes to that vision by exploring how virtual agents can establish rapport with their users. Rapport can be defined as the feeling of connection, ease of communication, and closeness that people experience during conversation. It is a fundamental aspect of human communication, crucial for effective and satisfying interactions, and has been shown to have positive outcomes in various contexts, including education, healthcare, and negotiation. Our aim in this project is to explore how we can enhance the social capabilities of current conversational models, making them not only capable of achieving task-specific goals (such as answering user questions) but also capable of establishing rapport with users over time so that the conversation and collaboration are more effective. To achieve our objective of building socially aware conversational agents, we combine findings from the fields of communication and psychology, among other social sciences, and our own research on human-human conversation. The results allow us to innovate on recent developments in large language models, including deploying paradigms like reinforcement learning.

Motivation

We are social beings, and if we want to build conversational agents that interact in a natural and effective way, it is important that, in designing such agents, we consider not only the task dimension (making models more effective and robust in addressing users’ needs) but also the social dimension of the interaction. Among the social phenomena that have been shown to play an important role in the outcome of interactions is rapport. Previous studies have demonstrated that rapport can lead to higher learning outcomes and better negotiation results, and research has shown that these findings from human–human conversation can be transferred to human–agent interactions. However, despite the importance of rapport in human communication, current work on conversational agents often overlooks this dimension in the design of their systems and tends to rely mainly on pretrained Large Language Models for generating responses. These models, which were not specifically trained for social purposes, suffer from several limitations, particularly in the social dimension, including a lack of socially adaptive behavior, a tendency to over-agree with users (sycophantic behavior), and a focus on satisfying short-term outcomes rather than steering conversations toward long-term goals, such as building rapport with users over time.  As these models become increasingly common and integrated into everyday human activities, addressing their social limitations becomes crucial, not only to make interactions more natural and appropriate, but also to avoid the drawbacks that arise from the absence of such social intelligence.

Methodology

Our methodology follows an interdisciplinary approach structured around two complementary dimensions. First, the project draws on findings from psychological and social studies of human–human conversation, as well as our own data and results on language and nonverbal behavior. Different theoretical models of rapport between humans have been proposed, including the dyadic rapport management model previously developed in our lab. These models provide a structured understanding of how rapport is built, maintained, and broken over the course of an interaction, and serve as a theoretical foundation for our computational approach. Second, the project integrates recent advances in Natural Language Processing, relying on state-of-the-art models such as speech and large language models. These models provide powerful capabilities for natural language understanding and generation, enabling end-to-end conversational systems. A major effort of the project is dedicated to combining these recent approaches in training conversational models with theory-driven insights on rapport management.

The question of building rapport is addressed at three levels:

• Perceptual level – enabling models to perceive and interpret users’ verbal and non-verbal behaviors in order to understand the state of the interaction.
• Reasoning level – allowing the model to adapt its conversational strategies based on the rapport established with users. The objective at this level is to steer the conversation in a way that balances rapport management with the achievement of task objectives.
• Behavioral level – enabling the model to generate appropriate verbal and non-verbal behaviors that reflect its conversational strategy, while maintaining rapport and achieving task objectives.

References

R. Zhao, A. Papangelis, and J. Cassell, “Towards a Dyadic Computational Model of Rapport Management for Human-Virtual Agent Interaction,” in Intelligent Virtual Agents, vol. 8637, T. Bickmore, S. Marsella, and C. Sidner, Eds., in Lecture Notes in Computer Science, vol. 8637. , Cham: Springer International Publishing, 2014, pp. 514–527. doi: 10.1007/978-3-319-09767-1_62.

T. Sinha and J. Cassell, “We Click, We Align, We Learn: Impact of Influence and Convergence Processes on Student Learning and Rapport Building,” in Proceedings of the 1st Workshop on Modeling INTERPERsonal SynchrONy And infLuence, Seattle Washington USA: ACM, Nov. 2015, pp. 13–20. Doi: 10.1145/2823513.2823516.

T. Sinha and J. Cassell, “We Click, We Align, We Learn: Impact of Influence and Convergence Processes on Student Learning and Rapport Building,” in Proceedings of the 1st Workshop on Modeling INTERPERsonal SynchrONy And infLuence, Seattle Washington USA: ACM, Nov. 2015, pp. 13–20. doi: 10.1145/2823513.2823516.

C. Clavel, M. Labeau, and J. Cassell, “Socio-conversational systems: Three challenges at the crossroads of fields,” Front. Robot. AI, vol. 9, Dec. 2022, doi: 10.3389/frobt.2022.937825.

See Other Projects