Dokumentdetails
ID

oai:arXiv.org:2405.18194

Thema
Computer Science - Machine Learnin... Computer Science - Cryptography an...
Autor
Ding, Youlong Wu, Xueyang Meng, Yining Luo, Yonggang Wang, Hao Pan, Weike
Kategorie

Computer Science

Jahr

2024

Auflistungsdatum

28.08.2024

Schlüsselwörter
learning transformer training
Metrisch

Zusammenfassung

Deep learning with differential privacy (DP) has garnered significant attention over the past years, leading to the development of numerous methods aimed at enhancing model accuracy and training efficiency.

This paper delves into the problem of training Transformer models with differential privacy.

Our treatment is modular: the logic is to `reduce' the problem of training DP Transformer to the more basic problem of training DP vanilla neural nets.

The latter is better understood and amenable to many model-agnostic methods.

Such `reduction' is done by first identifying the hardness unique to DP Transformer training: the attention distraction phenomenon and a lack of compatibility with existing techniques for efficient gradient clipping.

To deal with these two issues, we propose the Re-Attention Mechanism and Phantom Clipping, respectively.

We believe that our work not only casts new light on training DP Transformers but also promotes a modular treatment to advance research in the field of differentially private deep learning.

;Comment: ICML 2024

Ding, Youlong,Wu, Xueyang,Meng, Yining,Luo, Yonggang,Wang, Hao,Pan, Weike, 2024, Delving into Differentially Private Transformer

Dokumentieren

Öffnen

Teilen

Quelle

Artikel empfohlen von ES/IODE AI

Lung cancer risk and exposure to air pollution: a multicenter North China case–control study involving 14604 subjects
lung cancer case–control air pollution never-smokers nomogram model controls lung-related 14604 subjects north polluted consistent smokers quit exposure lung cancer risk air people factor smoking pollution study history