Grounding Language in Multi-Perspective Referential Communication

detalle del documento

IDENTIFICACIÓN

oai:arXiv.org:2410.03959

Tema

Computer Science - Computation and... Computer Science - Artificial Inte... Computer Science - Computer Vision... Computer Science - Graphics

Autor

Tang, Zineng Mao, Lingjun Suhr, Alane

Categoría

Computer Science

Año

2024

fecha de cotización

9/10/2024

Palabras clave

model paired comprehension science computer

Métrico

Resumen

We introduce a task and dataset for referring expression generation and comprehension in multi-agent embodied environments.

In this task, two agents in a shared scene must take into account one another's visual perspective, which may be different from their own, to both produce and understand references to objects in a scene and the spatial relations between them.

We collect a dataset of 2,970 human-written referring expressions, each paired with human comprehension judgments, and evaluate the performance of automated models as speakers and listeners paired with human partners, finding that model performance in both reference generation and comprehension lags behind that of pairs of human agents.

Finally, we experiment training an open-weight speaker model with evidence of communicative success when paired with a listener, resulting in an improvement from 58.9 to 69.3% in communicative success and even outperforming the strongest proprietary model.

;Comment: Accepted to EMNLP2024 Main

Tang, Zineng,Mao, Lingjun,Suhr, Alane, 2024, Grounding Language in Multi-Perspective Referential Communication