3D Reconstruction; Shape Completion; Texture-Inpainting; Implicit Function; Signed Distance Function
Abstract :
[en] Reconstructing 3D human body shapes from 3D partial textured scans remains a fundamental task for many computer vision and graphics applications – e.g., body animation, and virtual dressing. We propose a new neural network architecture for 3D body shape and highresolution texture completion – TSCom-Net – that can reconstruct the full geometry from mid-level to high-level partial input scans. We decompose the overall reconstruction task into two stages – first, a joint implicit learning network (SCom-Net and TCom-Net) that takes a voxelized scan and its occupancy grid as input to reconstruct the full body shape and predict vertex textures. Second, a high-resolution texture completion network, that utilizes the predicted coarse vertex textures to inpaint the missing parts of the partial ‘texture atlas’. A Thorough experimental evaluation on 3DBodyTex.V2 dataset shows that our method achieves competitive results with respect to the state-of-the-art while generalizing to different types and levels of partial shapes. The proposed method has also ranked second in the track1 of SHApe Recovery from Partial textured 3D scans (SHARP [37 , 2]) 2022 1 challenge1.
Research center :
Interdisciplinary Centre for Security, Reliability and Trust (SnT) > Computer Vision Imaging & Machine Intelligence (CVI²)
Disciplines :
Computer science
Author, co-author :
Karadeniz, Ahmet Serdar ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > CVI2
Ali, Sk Aziz ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > CVI2
Kacem, Anis ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > CVI2
Dupont, Elona ; University of Luxembourg > Faculty of Science, Technology and Medicine (FSTM) > CVI2
Aouada, Djamila ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > CVI2
External co-authors :
no
Language :
English
Title :
TSCom-Net: Coarse-to-Fine 3D Textured Shape Completion Network
Publication date :
2022
Event name :
European Conference on Computer Vision Workshops
Event place :
Tel-Aviv, Israel
Event date :
from 23-10-2022 to 27-10-2022
Main work title :
TSCom-Net: Coarse-to-Fine 3D Textured Shape Completion Network
SHARP 2021, the 2nd shape recovery from partial textured 3d scans. https://cvi2. uni.lu/sharp2021/. Accessed 23 July 2022
SHARP 2022 Repository, the repository of the 3rd shape recovery from partial textured 3d scans. https://gitlab.uni.lu/cvi2/cvpr2022-sharp-workshop. Accessed 23 July 2022
SHARP 2022, the 3rd shape recovery from partial textured 3d scans. https://cvi2. uni.lu/sharp2022/. Accessed 23 July 2022
Ali, S.A., Golyanik, V., Stricker, D.: NRGA: gravitational approach for non-rigid point set registration. In: International Conference on 3D Vision (3DV) (2018)
Ali, S.A., Kahraman, K., Reis, G., Stricker, D.: RPSRNet: end-to-end trainable rigid point set registration network using Barnes-hut 2d-tree representation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2021)
Ali, S.A., Yan, S., Dornisch, W., Stricker, D.: FoldMatch: accurate and high fidelity garment fitting onto 3d scans. In: IEEE International Conference on Image Processing (ICIP) (2020)
Arora, H., Mishra, S., Peng, S., Li, K., Mahdavi-Amiri, A.: Multimodal shape completion via implicit maximum likelihood estimation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2022)
Chibane, J., Alldieck, T., Pons-Moll, G.: Implicit functions in feature space for 3d shape reconstruction and completion. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2020)
Chibane, J., Pons-Moll, G.: Implicit feature networks for texture completion from partial 3D Data. In: Bartoli, A., Fusiello, A. (eds.) ECCV 2020. LNCS, vol. 12536, pp. 717–725. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-660963 48
Deng, J., Cheng, S., Xue, N., Zhou, Y., Zafeiriou, S.: UV-GAN: adversarial facial UV map completion for pose-invariant face recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7093–7102 (2018)
Golyanik, V., Reis, G., Taetz, B., Strieker, D.: A framework for an accurate point cloud based registration of full 3d human body scans. In: International Conference on Machine Vision and Applications (ICMVA) (2017)
Groueix, T., Fisher, M., Kim, V.G., Russell, B.C., Aubry, M.: 3D-CODED: 3D correspondences by deep deformation. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11206, pp. 235–251. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01216-8 15
Gurumurthy, S., Agrawal, S.: High fidelity semantic shape completion for point clouds using latent optimization. In: IEEE Winter Conference on Applications of Computer Vision (WACV) (2019)
Han, X.F., Laga, H., Bennamoun, M.: Image-based 3d object reconstruction: stateof-the-art and trends in the deep learning era. IEEE Trans. Pattern Anal. Mach. Intell. 43(5), 1578–1604 (2019)
Han, X., Li, Z., Huang, H., Kalogerakis, E., Yu, Y.: High-resolution shape completion using deep neural networks for global structure and local geometry inference. In: IEEE International Conference on Computer Vision (ICCV) (2017)
Hanocka, R., Hertz, A., Fish, N., Giryes, R., Fleishman, S., Cohen-Or, D.: MeshCNN: a network with an edge. ACM Trans. Graph. (TOG) 38(4), 1–12 (2019)
Hasler, N., Ackermann, H., Rosenhahn, B., Thormählen, T., Seidel, H.P.: Multilinear pose and body shape estimation of dressed subjects from image sets. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition (2010)
Kanazawa, A., Black, M.J., Jacobs, D.W., Malik, J.: End-to-end recovery of human shape and pose. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)
Kanazawa, A., Tulsiani, S., Efros, A.A., Malik, J.: Learning category-specific mesh reconstruction from image collections. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11219, pp. 386–402. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01267-0 23
Lazova, V., Insafutdinov, E., Pons-Moll, G.: 360-degree textures of people in clothing from a single image. In: International Conference on 3D Vision (3DV) (2019)
Li, X., et al.: Self-supervised single-view 3D reconstruction via semantic consistency. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12359, pp. 677–693. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58568-6 40
Liu, G., et al.: Image inpainting for irregular holes using partial convolutions. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11215, pp. 89–105. Springer, Cham (2018). https://doi.org/10.1007/978-3-03001252-6 6
Liu, Z., Tang, H., Lin, Y., Han, S.: Point-voxel CNN for efficient 3d deep learning. In: Advances in Neural Information Processing Systems, vol. 32 (2019)
Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., Black, M.J.: SMPL: a skinned multi-person linear model. ACM Trans. Graph. (TOG) 34(6), 1–16 (2015)
Lorensen, W.E., Cline, H.E.: Marching cubes: a high resolution 3d surface construction algorithm. ACM SIGGRAPH Comput. Graph. 21(4), 163–169 (1987)
Malik, J., et al.: HandVoxNet: deep voxel-based network for 3d hand shape and pose estimation from a single depth map. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2020)
Mescheder, L., Oechsle, M., Niemeyer, M., Nowozin, S., Geiger, A.: Occupancy networks: learning 3d reconstruction in function space. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2019)
Park, J.J., Florence, P., Straub, J., Newcombe, R., Lovegrove, S.: DeepSDF: learning continuous signed distance functions for shape representation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2019)
Patel, P., Huang, C.H.P., Tesch, J., Hoffmann, D.T., Tripathi, S., Black, M.J.: Agora: avatars in geography optimized for regression analysis. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13468–13478 (2021)
Prokudin, S., Black, M.J., Romero, J.: SMPLpix: neural avatars from 3d human models. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1810–1819 (2021)
Qi, C.R., Su, H., Mo, K., Guibas, L.J.: PointNet: deep learning on point sets for 3d classification and segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)
Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: International Conference on Medical Image Computing and Computer-assisted Intervention (2015)
Saint, A., et al.: 3dbodytex: textured 3d body dataset. In: International Conference on 3D Vision (3DV) (2018)
Saint, A., Cherenkova, K., Gusev, G., Aouada, D., Ottersten, B., et al.: BODY FITR: robust automatic 3d human body fitting. In: IEEE International Conference on Image Processing (ICIP) (2019)
Saint, A., Kacem, A., Cherenkova, K., Aouada, D.: 3DBooSTeR: 3D body shape and texture recovery. In: Bartoli, A., Fusiello, A. (eds.) ECCV 2020. LNCS, vol. 12536, pp. 726–740. Springer, Cham (2020). https://doi.org/10.1007/978-3-03066096-3 49
Saint, A., et al.: SHARP 2020: the 1st shape recovery from partial textured 3d scans challenge results. In: Bartoli, A., Fusiello, A. (eds.) ECCV 2020. LNCS, vol. 12536, pp. 741–755. Springer, Cham (2020). https://doi.org/10.1007/978-3-03066096-3 50
Sarkar, K., Varanasi, K., Stricker, D.: Learning quadrangulated patches for 3d shape parameterization and completion. In: International Conference on 3D Vision (3DV) (2017)
Sinha, A., Unmesh, A., Huang, Q., Ramani, K.: SurfNet: generating 3d shape surfaces using deep residual networks. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)
Tatarchenko, M., Dosovitskiy, A., Brox, T.: Octree generating networks: efficient convolutional architectures for high-resolution 3d outputs. In: IEEE International Conference on Computer Vision (ICCV) (2017)
Tian, Y., Zhang, H., Liu, Y., Wang, L.: Recovering 3d human mesh from monocular images: a survey. arXiv preprint arXiv:2203.01923 (2022)
Wang, N., et al.: Pixel2Mesh: generating 3d mesh models from single RGB images. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11215, pp. 55–71. Springer, Cham (2018). https://doi.org/10.1007/978-3-03001252-6 4
Wang, T.C., Liu, M.Y., Zhu, J.Y., Tao, A., Kautz, J., Catanzaro, B.: Highresolution image synthesis and semantic manipulation with conditional GANs. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)
Yan, X., Lin, L., Mitra, N.J., Lischinski, D., Cohen-Or, D., Huang, H.: Shapeformer: transformer-based shape completion via sparse representation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2022)
Yuan, W., Khot, T., Held, D., Mertz, C., Hebert, M.: PCN: point completion network. In: International Conference on 3D Vision (3DV) (2018)
Zheng, Z., Yu, T., Dai, Q., Liu, Y.: Deep implicit templates for 3d shape representation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2021)
Zheng, Z., Yu, T., Wei, Y., Dai, Q., Liu, Y.: DeepHuman: 3d human reconstruction from a single image. In: IEEE International Conference on Computer Vision (ICCV) (2019)
