Document detail
ID

oai:arXiv.org:2405.10121

Topic
Computer Science - Computation and... Computer Science - Multimedia
Author
Zhang, Bo Ma, Hui Ding, Jian Wang, Jian Xu, Bo Lin, Hongfei
Category

Computer Science

Year

2024

listing date

5/22/2024

Keywords
generation knowledge zero-resource dialogue
Metrics

Abstract

Integrating multimodal knowledge into large language models (LLMs) represents a significant advancement in dialogue generation capabilities.

However, the effective incorporation of such knowledge in zero-resource scenarios remains a substantial challenge due to the scarcity of diverse, high-quality dialogue datasets.

To address this, we propose the Visual Implicit Knowledge Distillation Framework (VIKDF), an innovative approach aimed at enhancing LLMs for enriched dialogue generation in zero-resource contexts by leveraging implicit multimodal knowledge.

VIKDF comprises two main stages: knowledge distillation, using an Implicit Query Transformer to extract and encode visual implicit knowledge from image-text pairs into knowledge vectors; and knowledge integration, employing a novel Bidirectional Variational Information Fusion technique to seamlessly integrate these distilled vectors into LLMs.

This enables the LLMs to generate dialogues that are not only coherent and engaging but also exhibit a deep understanding of the context through implicit multimodal cues, effectively overcoming the limitations of zero-resource scenarios.

Our extensive experimentation across two dialogue datasets shows that VIKDF outperforms existing state-of-the-art models in generating high-quality dialogues.

The code will be publicly available following acceptance.

;Comment: Under Review

Zhang, Bo,Ma, Hui,Ding, Jian,Wang, Jian,Xu, Bo,Lin, Hongfei, 2024, Distilling Implicit Multimodal Knowledge into LLMs for Zero-Resource Dialogue Generation

Document

Open

Share

Source

Articles recommended by ES/IODE AI

Elliptical Attention
language weights elliptical computer
Benefit finding among family caregivers of patients with advanced cancer in a palliative treatment: a qualitative study
benefit finding advanced cancer palliative caregivers care positive qualitative study cancer positive patients caregivers