Dokumentdetails
ID
oai:arXiv.org:2403.09919
Thema
Computer Science - Computation and... Computer Science - Machine Learnin...Autor
Zhang, Aonan Wang, Chong Wang, Yi Zhang, Xuanyu Cheng, YunfeiKategorie
Computer Science
Jahr
2024
Auflistungsdatum
05.06.2024
Schlüsselwörter
draft models decoding approach speculative languageZusammenfassung
In this paper, we introduce an improved approach of speculative decoding aimed at enhancing the efficiency of serving large language models.
Our method capitalizes on the strengths of two established techniques: the classic two-model speculative decoding approach, and the more recent single-model approach, Medusa.
Drawing inspiration from Medusa, our approach adopts a single-model strategy for speculative decoding.
However, our method distinguishes itself by employing a single, lightweight draft head with a recurrent dependency design, akin in essence to the small, draft model uses in classic speculative decoding, but without the complexities of the full transformer architecture.
And because of the recurrent dependency, we can use beam search to swiftly filter out undesired candidates with the draft head.
The outcome is a method that combines the simplicity of single-model design and avoids the need to create a data-dependent tree attention structure only for inference in Medusa.
We empirically demonstrate the effectiveness of the proposed method on several popular open source language models, along with a comprehensive analysis of the trade-offs involved in adopting this approach.
;Comment: 11 pages, 6 figures
Zhang, Aonan,Wang, Chong,Wang, Yi,Zhang, Xuanyu,Cheng, Yunfei, 2024, Recurrent Drafter for Fast Speculative Decoding in Large Language Models
Configurable Safety Tuning of Language Models with Synthetic Preference Data
data preference cst dpo configurable language safety
Affective flexibility as a developmental building block of cognitive reappraisal: An fMRI study
cognitive success affective flexibility
