[en] A validation data set plays a pivotal role in tweaking a machine learning model trained in a supervised manner. Many existing algorithms select a part of available data by using random sampling to produce a validation set. However, this approach can be prone to overfitting. One should follow careful data splitting to have reliable training and validation sets that can produce a generalized model with a good performance for the unseen (test) data. Data splitting based on resampling techniques involves repeatedly drawing samples from the available data. Hence, resampling methods can give better generalization power to a model, because they can produce and use many training and/or validation sets. These techniques are computationally expensive, but with increasingly available high-performance computing facilities, one can exploit them. Though a multitude of resampling methods exist, investigation of their influence on the generality of deep learning (DL) algorithms is limited due to its non-linear black-box nature. This paper contributes by: (1) investigating the generalization capability of the four most popular resampling methods: k-fold cross-validation (k-CV), repeated k-CV (Rk-CV), Monte Carlo CV (MC-CV) and bootstrap for creating training and validation data sets used for developing, training and validating DL based point cloud classifiers (e.g., PointNet; Qi et al., 2017a), (2) justifying Mean Square Error (MSE) as a statistically consistent estimator, and (3) exploring the use of MSE as a reliable performance metric for supervised DL. Experiments in this paper are performed on both synthetic and real-world aerial laser scanning (ALS) point clouds.
Disciplines :
Engineering, computing & technology: Multidisciplinary, general & others
Author, co-author :
Nurunnabi, Abdul Awal Md ; University of Luxembourg > Faculty of Science, Technology and Medicine (FSTM) > Department of Engineering (DoE)
Teferle, Felix Norman ; University of Luxembourg > Faculty of Science, Technology and Medicine (FSTM) > Department of Engineering (DoE)
External co-authors :
no
Language :
English
Title :
Resampling methods for a reliable validation set in deep learning based point cloud classification
Publication date :
June 2022
Event name :
ISPRS Congress, 2022
Event organizer :
ISPRS
Event place :
Nice, France
Event date :
from 06-06-2022 to 11-06-2022
Audience :
International
Main work title :
Resampling methods for a reliable validation set in deep learning based point cloud classification
AHN3: Actueel Hoogtebestand Nederland. Available at: https://app.pdok.nl/ahn3-downloadpage/.
Beasley, W. H., Rodgers, J. L., 2009. Re-Sampling methods. Millsap: The SAGE Handbook of Quantitative Methods in Psychology, 362-386.
Becker, C., Hani, N., Rosinskaya, E., Angelo, E. D., Strecha, C., 2017. Classification of aerial photogrammetric 3D point clouds., ISPRS Ann. Photogramm. Remote Sens. Spat. Inf. Sci., Vol. IV-1/W1, 2017.
Boos, D. D., Stefanski, L. A., 2013. Essential Statistical Inference: Theory and Methods. New York: Springer.
Efron, B., 1979. Bootstrap methods: Another look at the jackknife. Annals of Statistics, 7: 1-26.
Efron, B., 1982. The Jackknife, the Bootstrap, and Other Resampling Plans. Philadelphia: Society for Industrial and Applied Mathematics.
Efron, B., Tibshirani, R. J. 1993. An Introduction to the Bootstrap. Springer-Science + Business Media.
Fisher, R. A., 1935. Design of Experiments. Edinburgh, Oliver and Boyd.
Good, P. I., 2010. Permutation, Parametric and Bootstrap Tests of Hypotheses. Springer, New York.
Good, P. I., 2013. Introduction to Statistics through Resampling Methods and R, Wiley & Sons, NJ.
Goodfellow, I., Yoshua B., Aaron, C., 2016. Deep Learning. MIT press.
Guo, L., Chehata, N., Mallet, C., Boukir, S., 2011. Relevance of airborne lidar and multispectral image data for urban scene classification using Random Forests. ISPRS J. Photogramm. Remote Sens., 66(1): 56-66.
Guo, Y., et al., 2020. Deep learning for 3D point clouds: A survey. IEEE Trans. Pattern Anal. Mach. Intell., 1-27.
Hackel, T., Savinov, N., Ladicky, L., Wegner, J. D., Schindler, K., Pollefeys, M., 2017. Semantic3d.net: A new large-scale point cloud classification benchmark. arXiv:1704.03847.
Hastie, T., Tibshirani, R., Friedman, J., 2017. The Elements of Statistical Learning. New York: Springer.
Hu, Q., et al., 2020. RandLA-Net: Efficient semantic segmentation of large-scale point clouds. IEEE CVPR, 11108-11117.
Ioffe, S., Szegedy, C., 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. Int. Conf. Mach. Learn. ICML, 448-456.
Jaderberg, M., Simonyan, K., Zisserman, A., Kavukcuoglu, K., 2015. Spatial transformer networks. ArXiv :1506.02025.
James, G., Witten, D., Hastie, T., Tibshirani, R. 2015. An Introduction to Statistical Learning. New York: Springer.
Krizhevsky, A., Sutskever, I., Hinton, G. E., 2012. Imagenet classification with deep convolutional neural networks. Adv. Neural Inf. Process. Syst., 25.
LeCun et al., 1989. Backpropagation applied to handwritten zip code recognition, Neural Comput, 1(4), 541-551.
Li, N., Kähler, O., Pfeifer, N., 2021. A comparison of deep learning methods for airborne lidar point clouds classification. IEEE J Sel Top Appl Earth Obs Remote Sens, 14: 6467-6486.
Lyons, M. B., Keith, D. A., Phinn, S. R., Mason, T. J., Elith, J. (2018). A comparison of resampling methods for remote sensing classification and accuracy assessment. Remote Sens. Environ, 208, 145-153.
Manly, B., 2020. Randomization, Bootstrap and Monte Carlo Methods in Biology. Boca Raton, FL: Chapman and Hall/CRC.
Michelucci, U., 2018. Applied deep Learning-A Case Based Approach to Understanding Deep Learning Networks. APRESS Media, LLC: New York.
Niemeyer, J., Rottensteiner, F., Soergel, U., 2014. Contextual classification of lidar data and building object detection in urban areas. ISPRS J. Photogramm. Remote Sens. 87, 152-165.
Nurunnabi, A., West, G., Belton, D., 2015. Outlier detection and robust normal-curvature estimation in mobile laser scanning 3D point cloud data. Pattern Recognit. 48(4), 1404-1419.
Nurunnabi, A., Teferle, F. N., Li, J., Lindenbergh, R., Hunegnaw, A., 2021a. An efficient deep learning approach for ground point filtering in aerial laser scanning point clouds. Int. Arch. of the Photogramm. Remote Sens. and Spat. Info. Sci., Vol. XLIII-B1-2021, 31-38, XXIV-ISPRS Congress, 5-9 July.
Nurunnabi, A., Teferle, F. N., Li, J., Lindenbergh, R., Parvaz, S., 2021b. Investigation of PointNet for semantic segmentation of large-scale outdoor point clouds. Int. Arch. of the Photogramm. Remote Sens. and Spat. Info. Sci., Vol. XLVI-4/W5, 397-404.
Nurunnabi, A., Teferle, F. N., Laefer, D. F., Lindenbergh, R., Hunegnaw, A., 2022. A two-step feature extraction algorithm: application to deep learning for point cloud classification. Int. Arch. of the Photogramm. Remote Sens. and Spat. Info. Sci., Vol. XLVI-2/W1, 401-408.
Qi, C. R., Su, H., Mo, K., Guibas, L. J., 2017a. Pointnet: Deep learning on point sets for 3d classification and segmentation. IEEE CVPR, 652-660.
Qi, C. R., Yi, L., Su, H., Guibas, L. J., 2017b. PointNet++ : Deep hierarchical feature learning on point sets in a metric space. ArXiv:1706.02413.
Quenouille, M. H., 1949. Approximate tests of correlation in time-series. J R Stat Soc Series B, Stat Methodol, 11: 68-84.
Ramezan, C. A., Warner, T. A., Maxwell, A. E., 2019. Evaluation of sampling and cross-validation tuning strategies for regional-scale machine learning classification. Remote Sens., 11, 1-21.
Secord, J., Zakhor, A., 2007. Tree detection in urban regions using aerial lidar and image data. IEEE Geosci. Remote. Sens. Lett., 2: 196-200.
Stehman, S.V., 2009. Sampling designs for accuracy assessment of land cover. Int J Remote Sens., 30, 5243-5272.
Su, Y., et al., 2022. DLA-Net: Learning dual local attention features for semantic segmentation of large-scale building facade point clouds. Pattern Recognit., 123, 108372.
Thomas, H., Qi, C. R., Deschaud, J. E., Marcotegui, B., Goulette, F., Guibas, L. J., 2019. KPConv: Flexible and deformable convolution for point clouds. IEEE Int. Conf. Computer Vision, 6411-6420.
Tran, G., Nguyen, D., Milenkovic, M., Pfeifer, N., 2015. Potential of full waveform airborne laser scanning data for urban area classification-Transfer of classification approaches between missions. Int. Arch. of the Photogramm. Remote Sens. and Spat. Info. Sci., 40(7): 1317.
Tsamardinos, I., Greasidou, E., Borboudakis, G., 2018. Bootstrapping the out-of-sample predictions for efficient and accurate cross-validation. Mach Learn, 107(12):1895-1922.
Vabalas, A., Emma G., Poliakoff, E., Casson, A., J., 2019. Machine learning algorithm validation with a limited sample size. PLoS One, 14(11): e0224365. doi: 10.1371/journal.pone.0224365.
Wang, F., et al., 2021. Applying different resampling strategies in machine learning models to predict head-cut gully erosion susceptibility. Alex. Eng. J., 60(6): 5813-5829.
Weber, K. T., Langille, J., 2007. Improving classification accuracy Assessments with statistical bootstrap resampling techniques. GIsci Remote Sens, 2007, 44(3), p. 237-250.
Wu, W., Qi, Z., Fuxin, L., 2019. PointConv: Deep convolutional networks on 3d point clouds. IEEE CVPR, 9621-9630.
Zhen, Z., Quackenbush, L.J., Stehman, S.V., Zhang, L., 2013. Impact of training and validation sample selection on classification accuracy and accuracy assessment when using reference polygons in object-based classification. Int J Remote Sens., 34 (19), 6914-6930.
