Document detail
ID
oai:arXiv.org:2405.18194
Topic
Computer Science - Machine Learnin... Computer Science - Cryptography an...Author
Ding, Youlong Wu, Xueyang Meng, Yining Luo, Yonggang Wang, Hao Pan, WeikeCategory
Computer Science
Year
2024
listing date
8/28/2024
Keywords
learning transformer trainingAbstract
Deep learning with differential privacy (DP) has garnered significant attention over the past years, leading to the development of numerous methods aimed at enhancing model accuracy and training efficiency.
This paper delves into the problem of training Transformer models with differential privacy.
Our treatment is modular: the logic is to `reduce' the problem of training DP Transformer to the more basic problem of training DP vanilla neural nets.
The latter is better understood and amenable to many model-agnostic methods.
Such `reduction' is done by first identifying the hardness unique to DP Transformer training: the attention distraction phenomenon and a lack of compatibility with existing techniques for efficient gradient clipping.
To deal with these two issues, we propose the Re-Attention Mechanism and Phantom Clipping, respectively.
We believe that our work not only casts new light on training DP Transformers but also promotes a modular treatment to advance research in the field of differentially private deep learning.
;Comment: ICML 2024
Ding, Youlong,Wu, Xueyang,Meng, Yining,Luo, Yonggang,Wang, Hao,Pan, Weike, 2024, Delving into Differentially Private Transformer
SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models
methods salience efficiency quantization language slim-llm llms
Prediction Models for Suicide Attempts among Adolescents Using Machine Learning Techniques
using data ml prediction 0 suicide adolescents
