Reference : Highway Network Block with Gates Constraints for Training Very Deep Networks
Scientific congresses, symposiums and conference proceedings : Paper published in a book
Engineering, computing & technology : Computer science
Computational Sciences
http://hdl.handle.net/10993/35691
Highway Network Block with Gates Constraints for Training Very Deep Networks
English
Oyedotun, Oyebade mailto [University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > >]
Shabayek, Abd El Rahman mailto [University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > >]
Aouada, Djamila mailto [University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > >]
Ottersten, Björn mailto [University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > >]
19-Jun-2018
2018 IEEE International Conference on Computer Vision and Pattern Recognition Workshop, June 18-22, 2018
Yes
International
Utah
US
2018 IEEE International Conference on Computer Vision and Pattern Recognition Workshop
June 18-22, 2018
[en] deep learning ; neural network ; image classification
[en] In this paper, we propose to reformulate the learning of the highway network block to realize both early optimization and improved generalization of very deep networks while preserving the network depth. Gate constraints are duly employed to improve optimization, latent representations and parameterization usage in order to efficiently learn hierarchical feature transformations which are crucial for the success of any deep network. One of the earliest
very deep models with over 30 layers that was successfully trained relied on highway network blocks. Although, highway blocks suffice for alleviating optimization problem via improved information flow, we show for the first time that further in training such highway blocks may result into learning mostly untransformed features and therefore a reduction in the effective depth of the model; this could negatively impact model generalization performance. Using the
proposed approach, 15-layer and 20-layer models are successfully trained with one gate and a 32-layer model using three gates. This leads to a drastic reduction of model parameters as compared to the original highway network. Extensive experiments on CIFAR-10, CIFAR-100, Fashion-MNIST and USPS datasets are performed to validate the effectiveness of the proposed approach. Particularly, we outperform the original highway network and many state-ofthe-
art results. To the best our knowledge, on the Fashion-MNIST and USPS datasets, the achieved results are the best reported in literature.
FNR AFR: R-AGR- 0424-05-D/Bjorn Ottersten
Researchers ; Professionals ; Students ; General public
http://hdl.handle.net/10993/35691
FnR ; FNR11295431 > Oyebade Oyedotun > AVR > Automatic Feature Selection for Visual Recognition > 01/02/2017 > 31/01/2021 > 2016

File(s) associated to this reference

Fulltext file(s):

FileCommentaryVersionSizeAccess
Open access
Oyebade_CVPRW.pdfAuthor preprint569.94 kBView/Open

Bookmark and Share SFX Query

All documents in ORBilu are protected by a user license.