Détail du document
Identifiant
oai:arXiv.org:2410.08727
Sujet
Statistics - Machine Learning Computer Science - Machine Learnin...Auteur
Achilli, Beatrice Ventura, Enrico Silvestri, Gianluigi Pham, Bao Raya, Gabriel Krotov, Dmitry Lucibello, Carlo Ambrogioni, LucaCatégorie
Computer Science
Année
2024
Date de référencement
16/10/2024
Mots clés
learning machine dataRésumé
Generative diffusion processes are state-of-the-art machine learning models deeply connected with fundamental concepts in statistical physics.
Depending on the dataset size and the capacity of the network, their behavior is known to transition from an associative memory regime to a generalization phase in a phenomenon that has been described as a glassy phase transition.
Here, using statistical physics techniques, we extend the theory of memorization in generative diffusion to manifold-supported data.
Our theoretical and experimental findings indicate that different tangent subspaces are lost due to memorization effects at different critical times and dataset sizes, which depend on the local variance of the data along their directions.
Perhaps counterintuitively, we find that, under some conditions, subspaces of higher variance are lost first due to memorization effects.
This leads to a selective loss of dimensionality where some prominent features of the data are memorized without a full collapse on any individual training point.
We validate our theory with a comprehensive set of experiments on networks trained both in image datasets and on linear manifolds, which result in a remarkable qualitative agreement with the theoretical predictions.
Achilli, Beatrice,Ventura, Enrico,Silvestri, Gianluigi,Pham, Bao,Raya, Gabriel,Krotov, Dmitry,Lucibello, Carlo,Ambrogioni, Luca, 2024, Losing dimensions: Geometric memorization in generative diffusion
Systematic Feature Design for Cycle Life Prediction of Lithium-Ion Batteries During Formation
cycle prediction framework feature design features life formation
