Paper published in a book (Scientific congresses, symposiums and conference proceedings)
Efficient Hessian-based DNN Optimization via Chain-Rule Approximation
Temperoni, Alessandro; Dalle Lucca Tosi, Mauro; Theobald, Martin
2023In Proceedings of the 6th Joint International Conference on Data Science Management of Data (10th ACM IKDD CODS and 28th COMAD)
Peer reviewed
 

Files


Full Text
Efficient Hessian-based DNN Optimization.pdf
Author preprint (886.4 kB)
Request a copy

All documents in ORBilu are protected by a user license.

Send to



Details



Abstract :
[en] Learning non use-case specific models has been shown to be a challenging task in Deep Learning (DL). Hyperparameter tuning requires long training sessions that have to be restarted any time the network or the dataset changes and are not affordable by most stakeholders in industry and research. Many attempts have been made to justify and understand the source of the use-case specificity that distinguishes DL problems. To this date, second-order optimization methods have been partially shown to be effective in some cases but have not been sufficiently investigated in the context of learning and optimization. In this work, we present a chain rule for the efficient approximation of the Hessian matrix (i.e., the second-order derivatives) of the weights across the layers of a Deep Neural Network (DNN). We show the application of our approach for weight optimization during DNN training, as we believe that this is a step that particularly suffers from the enormous variety of the optimizers provided by state-of-the-art libraries such as Keras and PyTorch. We demonstrate—both theoretically and empirically—the improved accuracy of our approximation technique and that the Hessian is a useful diagnostic tool which helps to more rigorously optimize training. Our preliminary experiments prove the efficiency as well as the improved convergence of our approach which both are crucial aspects for DNN training.
Disciplines :
Computer science
Author, co-author :
Temperoni, Alessandro ;  University of Luxembourg > Faculty of Science, Technology and Medicine (FSTM) > Department of Computer Science (DCS)
Dalle Lucca Tosi, Mauro ;  University of Luxembourg > Faculty of Science, Technology and Medicine (FSTM) > Department of Computer Science (DCS)
Theobald, Martin ;  University of Luxembourg > Faculty of Science, Technology and Medicine (FSTM) > Department of Computer Science (DCS)
External co-authors :
no
Language :
English
Title :
Efficient Hessian-based DNN Optimization via Chain-Rule Approximation
Publication date :
2023
Event name :
6th Joint International Conference on Data Science & Management of Data
Event date :
from 04-01-2023 to 07-01-2023
Audience :
International
Main work title :
Proceedings of the 6th Joint International Conference on Data Science Management of Data (10th ACM IKDD CODS and 28th COMAD)
Pages :
297--298
Peer reviewed :
Peer reviewed
FnR Project :
FNR12252781 - Data-driven Computational Modelling And Applications, 2017 (01/09/2018-28/02/2025) - Andreas Zilian
Available on ORBilu :
since 06 March 2023

Statistics


Number of views
59 (16 by Unilu)
Number of downloads
0 (0 by Unilu)

Scopus citations®
 
0
Scopus citations®
without self-citations
0

Bibliography


Similar publications



Contact ORBilu