Overview

As part of the Articulab’s goals, to better understand the natural human communication and create conversational agents that adapt to us rather than forcing us to adapt to them, in this project, we work on the phenomenon of conversational grounding. Conversational grounding is the essential, yet often invisible, “glue” of human interaction. It is the iterative process by which we ensure that what I said is actually what you understood. While modern Large Language Models (LLMs) have mastered the art of speaking fluently, they frequently suffer from an “illusion of grounding”, where they mimic the surface forms of understanding (like saying “Okay, I got it”) without actually updating their internal mental models. This project bridges the gap between the theoretical foundations of cognitive science and the probabilistic nature of deep learning to move beyond simple intent classification.

By focusing on two fundamental pillars – the Process of Grounding (immediate negotiation of meaning) and the Representation of Common Ground (long-term memory and reasoning), this research introduces new annotated corpora, scalable evaluation benchmarks like IndiRef, and novel training methodologies. The ultimate goal is to evolve AI from a simple query-response tool into a truly collaborative partner capable of navigating the messy, non-linear, and multi-threaded nature of natural spoken dialogue. Work of this kind can be integrated into conversational agents to make them better conversational partners.

Motivation

The vision of a future integrated with intelligent agents acting as empathetic companions for the elderly or efficient industrial collaborators depends entirely on communicative competence. Currently, we are stuck between two extreme failure modes: the “confident” agent that hallucinates shared history (ordering pizza from a place you never mentioned), and the “overly cautious” agent that exhausts the user with twenty clarification questions for a simple task. These aren’t just interface quirks; they are fundamental bottlenecks in Human-Computer Interaction. If we must explain every coordinate and detail to a robot, it ceases to be an aid and becomes an obstacle. We need agents that follow the Principle of Least Collaborative Effort [1], knowing exactly when to be confident and when to be humble enough to ask for a repair. As the stakes rise, especially in caregiving or tutoring, the ability to dissolve the barrier between human intent and machine execution becomes a safety requirement, not just a luxury.

Methodology

Discretizing Grounding: Annotation and Analysis

The first phase of the research focused on transforming the abstract concept of grounding into a computable metric. We conducted a granular annotation of two major datasets: Meetup[2] (text-based) and Spot the Difference[3] (spoken). By extending the framework established by Traum and Allen[4], we discretized dialogue into Grounding Acts (GAs) such as Initiate, Repair, and Acknowledge, and Common Grounding Units (CGUs). This process revealed a fundamental adherence to the Principle of Least Collaborative Effort: while text-based users prefer implicit grounding (like “moving on” to the next task), spoken dialogue relies heavily on explicit backchannels (like “mhm” or “yeah”) to maintain the acoustic channel.

Diagnostic Benchmarking: The Perplexity Framework

To evaluate LLMs without the constraints of traditional metrics like BLEU or ROUGE (which fail to capture pragmatic validity), we developed a perplexity-based testing framework. We generated pairs of hand-crafted responses: one “Correct” (accounting for a repair or cancellation) and one “Incorrect” (fluent but ignoring the grounded update). We hypothesized that a model with true grounding capability should assign a lower perplexity to the correct response.

Our results indicated a scaling law with respect to pragmatic capabilities such as Grounding. While massive models like GPT-4 excel, smaller models often fall into the “Affirmation Bias,” reflexively agreeing with users even when the conversational state has been invalidated. To understand why smaller models fail despite fine-tuning, we designed an embedding-based diagnostic. We compared the internal vector representations of dialogues containing one of the grounding acts to paraphrased versions that lacked the repair but shared the same semantic meaning. By calculating a V-score, we measured whether the model’s internal state was closer to the correct grounded meaning or to the surface-level incorrect information. We discovered that smaller models rely on lexical overlap; they struggle to “unlearn” or suppress tokens that have been repaired or cancelled (e.g., still thinking about a “chair” after the speaker corrected it to “table”), whereas larger models successfully update their internal representations.

Recognizing that standard Causal Language Modeling (CLM) does not teach models what to avoid, we implemented a Positive-Negative Reward training strategy. This approach provides an explicit negative signal for pragmatically invalid responses, forcing the model to attend to the specific tokens that differentiate a grounded state from a hallucination. This methodology successfully unlocked robust grounding capabilities in medium-sized models (approx. 7B parameters), proving that pragmatic reasoning can be taught to models having a minimum number of parameters, through targeted feedback rather than just massive parameter scaling.

Measuring Persistence: The IndiRef Benchmark

Moving from local turn-taking to long-term memory, we introduced the IndiRef benchmark. This assesses a system’s ability to resolve relational references – identifying entities based on their temporal, spatial, or attributive relationship to the established common ground (e.g., “the first movie we watched after holidays”). By using adversarial test cases where multiple similar entities exist along with different perspectives of the speakers, we ensured that the model could only succeed if it maintained a persistent and operational common ground, rather than relying on shallow retrieval. On testing various representation techniques like Summarisation, Utterance Chunking and Agentic systems with Ontologies, we noticed that the transformation from raw dialogue to a constrained representation often misses the nuanced relational cues necessary for grounding.

Scaling Reasoning: Synthetic Data and GRPO

To overcome the scarcity of high-quality spoken interactive data, we developed a multi-phase synthetic data generation pipeline. This “environment-first” approach programmatically builds a 2D world, simulates navigator experiences, and utilizes LLMs to generate constrained, grounded dialogues based on those event logs. We applied Group Relative Policy Optimization (GRPO)[5] to train models on these synthetic Q/A pairs. This reinforcement learning framework rewarded the model for accurate multi-hop reasoning over a structured Agentic Ontology, significantly improving the system’s ability to maintain a persistent common ground over extended, complex interactions. Despite these gains, performance on the IndiRef benchmark reveals that situated dialogue[6] is still fundamentally underspecified, suggesting a need for more sophisticated representation techniques to resolve spatial and linguistic uncertainty.

Multimodal Common Ground Representation

To address the persistent ambiguity found in situated dialogues, our latest research integrates Visual Scaffolding into the Common Ground Representation. This framework shifts the paradigm from storing conversational state in texts or opaque latent vectors to an externalized state maintenance via a dynamic image canvas, motivated by the cognitive requirement for “mental imagery” to resolve linguistic underspecification. By forcing the agent to translate a text-based description into a concrete visual artifact, the system is compelled to resolve underspecified details (such as the exact position or color of an object) that a text-only model might ignore.

We implement this through a modular agentic pipeline consisting of an Observer (which filters and segments the dialogue stream), a Drawer (which generates or edits human-verifiable images), and a Reasoner (which performs Visual-RAG to answer complex queries). Our results on the IndiRef benchmark demonstrate that this multimodal scaffolding further improves performance in reasoning tasks, outperforming both standard long-context LLMs and state-of-the-art reasoning models. This suggests that the act of “visual thinking” or creating a “mental map” can allow agents to maintain a more robust, persistent, and verifiable common ground over extended interactions.

Related Publications

1) Biswesh Mohapatra, Seemab Hassan, Laurent Romary, and Justine Cassell. 2024. Conversational Grounding: Annotation and Analysis of Grounding Acts and Grounding Units. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 3967–3977, Torino, Italia. ELRA and ICCL.

2) Biswesh Mohapatra, Manav Nitin Kapadnis, Laurent Romary, and Justine Cassell. 2024. Evaluating the Effectiveness of Large Language Models in Establishing Conversational Grounding. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9767–9781, Miami, Florida, USA. Association for Computational Linguistics.

3) Biswesh Mohapatra, Théo Charlot, Giovanni Duca, Mayank Palan, Laurent Romary, and Justine Cassell. Frame of Reference: Addressing the Challenges of Common Ground Representation in Situational Dialogs. Findings of ACL 2026.

4) Mohapatra, Biswesh, Giovanni Duca, Laurent Romary and Justine Cassell. “Using Machine Mental Imagery for Representing Common Ground in Situated Dialogue.” (2026). Under review.

References

[1] Herbert Clark and Susan E. Brennan (1991). Grounding in Communication. In Perspectives on Socially Shared Cognition. pp. 13–1991. American Psychological Association.

[2] Ilinykh, Nikolai, Sina Zarrieß, and David Schlangen (Sept. 2019). Meet Up! A Corpus of Joint Activity Dialogues in a Visual Environment. In: Proceedings of the 23rd Workshop on the Semantics and Pragmatics of Dialogue – Full Papers. London, United Kingdom: SEMDIAL. url: http://semdial.org/anthology/Z19Ilinykh_semdial_0006.pdf 

[3] Lopes, José, Nils Hemmingsson, and Oliver Astrand (May 2018). The Spot the Difference corpus: a multi-modal corpus of spontaneous task oriented spoken interactions. In: Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). Miyazaki, Japan: European Language Resources Association (ELRA). url: https://aclanthology.org/L18-1305. 

[4] David Traum and James Allen (Aug. 1994). “A ”Speech Acts” Approach To Ground- ing In Conversation”. In: Proc. 2nd International Conference on Spoken Language Processing (ICSLP 1992). 

[5] Shao, Zhihong, Peiyi Wang, Qihao Zhu, Runxin Xu, Jun-Mei Song, Mingchuan Zhang, Y. K. Li, Yu Wu and Daya Guo. “DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models.” ArXiv abs/2402.03300 (2024) 

[6] Robert J. Ross. “Situated dialogue systems: agency & spatial meaning in task-oriented dialogue.” (2012).

See Other Projects