Automatic Analysis, Representation and Reconstruction of Textured 3D Human Scans

Various practical applications in computer vision are related to the human body. These involve representing and modelling the body shape, pose, clothing and appearance with mathematical and statistical tools requiring datasets of examples, representative of the variation in the data. Three-dimensional (3D) data is especially important as it allows to simulate the physical world directly, for example to analyse and lift ambiguities in other prevalent data modalities, such as images. However, existing datasets of 3D human scans show limitations in their size, diversity, quality or annotation. This reduces their applicability in tackling research questions around the 3D human body. Two particular applications of interest that remain unanswered are the estimation of body shape under clothing, and the completion of textured shape of missing or defective data.
This thesis proposes three main contributions. First, 3DBodyTex, a dataset of 3D human scans, which complements alternative datasets with real scans, body and clothing scans, hundreds of subjects, high-resolution texture information, dense annotations and aligned body shapes under the clothing. The aim is to enable and facilitate new research possibilities with learning-based methods, in 3D or using derived modalities. Second, to build this dataset automatically from raw scans, multiple robust 3D processing methods are proposed. These involve pose estimation, pose fitting, tight body shape fitting, and body shape estimation under clothing. The proposed methods show competitive or improved results on existing benchmarks and new proposed benchmarks based on 3DBodyTex. In particular, an alternative method is proposed to estimate the body shape under clothing from a single scan. On independent benchmarks, it is competitive with, or better than, methods requiring a full time sequence of scans. Third, the task of shape and texture completion of 3D human scans is tackled. A new method is proposed that completes the shape and the texture sequentially, and automatically identifies the missing regions. In particular, partial convolutions are extended to texture images (UV maps) for inpainting the colour of a 3D scan using a convolutional neural network. A new benchmark, based on 3DBodyTex, is proposed for the evaluation.