detalle del documento
IDENTIFICACIÓN
oai:arXiv.org:2411.04424
Tema
Computer Science - Computation and... Computer Science - Artificial Inte...Autor
Gao, Yicheng Xu, Gonghan Wang, Zhe Cohan, ArmanCategoría
Computer Science
Año
2024
fecha de cotización
1/1/2025
Palabras clave
estimation methods llm evaluatorsResumen
Recent advances in large language models (LLMs) show the potential of using LLMs as evaluators for assessing the quality of text generations from LLMs.
However, applying LLM evaluators naively to compare or judge between different systems can lead to unreliable results due to the intrinsic win rate estimation bias of LLM evaluators.
In order to mitigate this problem, we propose two calibration methods, Bayesian Win Rate Sampling (BWRS) and Bayesian Dawid-Skene, both of which leverage Bayesian inference to more accurately infer the true win rate of generative language models.
We empirically validate our methods on six datasets covering story generation, summarization, and instruction following tasks.
We show that both our methods are effective in improving the accuracy of win rate estimation using LLMs as evaluators, offering a promising direction for reliable automatic text quality evaluation.
;Comment: Accepted by EMNLP 2024
Gao, Yicheng,Xu, Gonghan,Wang, Zhe,Cohan, Arman, 2024, Bayesian Calibration of Win Rate Estimation with LLM Evaluators
Optimal Sensing of Momentum Kicks with a Feedback-Controlled Nanomechanical Resonator
sensing momentum kicks resonator
Shared genetic aetiology of Alzheimer’s disease and age-related macular degeneration by APOC1 and APOE genes
gene diagnostic based analysis amd ad apoe apoc1 identified genes pleiotropy
