Détail du document
Identifiant
oai:arXiv.org:2408.13430
Sujet
Statistics - Applications Computer Science - Digital Librari... Computer Science - Computer Scienc... Computer Science - Machine Learnin... Statistics - Machine LearningAuteur
Su, Buxin Zhang, Jiayao Collina, Natalie Yan, Yuling Li, Didong Cho, Kyunghyun Fan, Jianqing Roth, Aaron Su, Weijie J.Catégorie
Computer Science
Année
2024
Date de référencement
28/08/2024
Mots clés
peer author-provided analysis rankings scores review learning machine science computerRésumé
We conducted an experiment during the review process of the 2023 International Conference on Machine Learning (ICML) that requested authors with multiple submissions to rank their own papers based on perceived quality.
We received 1,342 rankings, each from a distinct author, pertaining to 2,592 submissions.
In this paper, we present an empirical analysis of how author-provided rankings could be leveraged to improve peer review processes at machine learning conferences.
We focus on the Isotonic Mechanism, which calibrates raw review scores using author-provided rankings.
Our analysis demonstrates that the ranking-calibrated scores outperform raw scores in estimating the ground truth ``expected review scores'' in both squared and absolute error metrics.
Moreover, we propose several cautious, low-risk approaches to using the Isotonic Mechanism and author-provided rankings in peer review processes, including assisting senior area chairs' oversight of area chairs' recommendations, supporting the selection of paper awards, and guiding the recruitment of emergency reviewers.
We conclude the paper by addressing the study's limitations and proposing future research directions.
;Comment: See more details about the experiment at https://openrank.cc/
Su, Buxin,Zhang, Jiayao,Collina, Natalie,Yan, Yuling,Li, Didong,Cho, Kyunghyun,Fan, Jianqing,Roth, Aaron,Su, Weijie J., 2024, Analysis of the ICML 2023 Ranking Data: Can Authors' Opinions of Their Own Papers Assist Peer Review in Machine Learning?
Analysis of the ICML 2023 Ranking Data: Can Authors' Opinions of Their Own Papers Assist Peer Review in Machine Learning?
peer author-provided analysis rankings scores review learning machine science computer
