Paper published in a book (Scientific congresses, symposiums and conference proceedings)
Highway Network Block with Gates Constraints for Training Very Deep Networks
Oyedotun, Oyebade; Shabayek, Abd El Rahman; Aouada, Djamila et al.
2018In 2018 IEEE International Conference on Computer Vision and Pattern Recognition Workshop, June 18-22, 2018
Peer reviewed
 

Files


Full Text
Oyebade_CVPRW.pdf
Author preprint (583.62 kB)
Download

All documents in ORBilu are protected by a user license.

Send to



Details



Keywords :
deep learning; neural network; image classification
Abstract :
[en] In this paper, we propose to reformulate the learning of the highway network block to realize both early optimization and improved generalization of very deep networks while preserving the network depth. Gate constraints are duly employed to improve optimization, latent representations and parameterization usage in order to efficiently learn hierarchical feature transformations which are crucial for the success of any deep network. One of the earliest very deep models with over 30 layers that was successfully trained relied on highway network blocks. Although, highway blocks suffice for alleviating optimization problem via improved information flow, we show for the first time that further in training such highway blocks may result into learning mostly untransformed features and therefore a reduction in the effective depth of the model; this could negatively impact model generalization performance. Using the proposed approach, 15-layer and 20-layer models are successfully trained with one gate and a 32-layer model using three gates. This leads to a drastic reduction of model parameters as compared to the original highway network. Extensive experiments on CIFAR-10, CIFAR-100, Fashion-MNIST and USPS datasets are performed to validate the effectiveness of the proposed approach. Particularly, we outperform the original highway network and many state-ofthe- art results. To the best our knowledge, on the Fashion-MNIST and USPS datasets, the achieved results are the best reported in literature.
Disciplines :
Computer science
Author, co-author :
Oyedotun, Oyebade ;  University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT)
Shabayek, Abd El Rahman ;  University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT)
Aouada, Djamila  ;  University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT)
Ottersten, Björn ;  University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT)
External co-authors :
no
Language :
English
Title :
Highway Network Block with Gates Constraints for Training Very Deep Networks
Publication date :
19 June 2018
Event name :
2018 IEEE International Conference on Computer Vision and Pattern Recognition Workshop
Event date :
June 18-22, 2018
Audience :
International
Main work title :
2018 IEEE International Conference on Computer Vision and Pattern Recognition Workshop, June 18-22, 2018
Peer reviewed :
Peer reviewed
Focus Area :
Computational Sciences
FnR Project :
FNR11295431 - Automatic Feature Selection For Visual Recognition, 2016 (01/02/2017-31/01/2021) - Oyebade Oyedotun
Funders :
FNR AFR: R-AGR- 0424-05-D/Bjorn Ottersten
Available on ORBilu :
since 12 May 2018

Statistics


Number of views
216 (25 by Unilu)
Number of downloads
214 (17 by Unilu)

Scopus citations®
 
11
Scopus citations®
without self-citations
3

Bibliography


Similar publications



Contact ORBilu