Full 3D Reconstruction of Non-Rigidly Deforming Objects

in ACM Transactions on Multimedia Computing, Communications, & Applications (2018)

Enhancement of Dynamic Depth Scenes by Upsampling for Precise Super-Resolution (UP-SR)

in Computer Vision and Image Understanding (2016)

Multi-frame super-resolution is the process of recovering a high resolution image or video from a set of captured low resolution images. Super-resolution approaches have been largely explored in 2-D ... [more ▼]

Multi-frame super-resolution is the process of recovering a high resolution image or video from a set of captured low resolution images. Super-resolution approaches have been largely explored in 2-D imaging. However, their extension to depth videos is not straightforward due to the textureless nature of depth data, and to their high frequency contents coupled with fast motion artifacts. Recently, few attempts have been introduced where only the super-resolution of static depth scenes has been addressed. In this work, we propose to enhance the resolution of dynamic depth videos with non-rigidly moving objects. The proposed approach is based on a new data model that uses densely upsampled, and cumulatively registered versions of the observed low resolution depth frames. We show the impact of upsampling in increasing the sub-pixel accuracy and reducing the rounding error of the motion vectors. Furthermore, with the proposed cumulative motion estimation, a high registration accuracy is achieved between non-successive upsampled frames with relative large motions. A statistical performance analysis is derived in terms of mean square error explaining the effect of the number of observed frames and the effect of the super-resolution factor at a given noise level. We evaluate the accuracy of the proposed algorithm theoretically and experimentally as function of the SR factor, and the level of contaminations with noise. Experimental results on both real and synthetic data show the effectiveness of the proposed algorithm on dynamic depth videos as compared to state-of-art methods. [less ▲]

Real-Time Non-Rigid Multi-Frame Depth Video Super-Resolution

in IEEE Conference on Computer Vision and Pattern Recognition Workshop (CVPRW'15), (Best paper award) (2015, June 12)

This paper proposes to enhance low resolution dynamic depth videos containing freely non–rigidly moving objects with a new dynamic multi–frame super–resolution algorithm. Existent methods are either ... [more ▼]

This paper proposes to enhance low resolution dynamic depth videos containing freely non–rigidly moving objects with a new dynamic multi–frame super–resolution algorithm. Existent methods are either limited to rigid objects, or restricted to global lateral motions discarding radial displacements. We address these shortcomings by accounting for non–rigid displacements in 3D. In addition to 2D optical flow, we estimate the depth displacement, and simultaneously correct the depth measurement by Kalman filtering. This concept is incorporated efficiently in a multi–frame super–resolution framework. It is formulated in a recursive manner that ensures an efficient deployment in real–time. Results show the overall improved performance of the proposed method as compared to alternative approaches, and specifically in handling relatively large 3D motions. Test examples range from a full moving human body to a highly dynamic facial video with varying expressions. [less ▲]

RGB-D Multi-View System Calibration for Full 3D Scene Reconstruction

in 22nd International Conference on Pattern Recognition (ICPR'14) (2014)

One of the most crucial requirements for building a multi-view system is the estimation of relative poses of all cameras. An approach tailored for a RGB-D cameras based multi-view system is missing. We ... [more ▼]

One of the most crucial requirements for building a multi-view system is the estimation of relative poses of all cameras. An approach tailored for a RGB-D cameras based multi-view system is missing. We propose BAICP+ which combines Bundle Adjustment (BA) and Iterative Closest Point (ICP) algorithms to take into account both 2D visual and 3D shape information in one minimization formulation to estimate relative pose parameters of each camera. BAICP+ is generic enough to take different types of visual features into account and can be easily adapted to varying quality of 2D and 3D data. We perform experiments on real and simulated data. Results show that with the right weighting factor BAICP+ has an optimal performance when compared to BA and ICP used independently or sequentially. [less ▲]

Range Image Pixel Matching Method

Patent (2013)

Mutli-Frame Super-Resolution by Enhanced Shift & Add

in 8th International Symposium on Image and Signal Processing and Analysis (2013)

A critical step in multi-frame super-resolution is the registration of frames based on their motion. We improve the performance of current state-of-the-art super-resolution techniques by proposing a more ... [more ▼]

A critical step in multi-frame super-resolution is the registration of frames based on their motion. We improve the performance of current state-of-the-art super-resolution techniques by proposing a more robust and accurate registration as early as in the initialization stage of the high resolution estimate. Indeed, we solve the limitations on scale and motion inherent to the classical Shift & Add approach by upsampling the low resolution frames up to the super-resolution factor prior to estimating motion or to median filtering. This is followed by an appropriate selective optimization, leading to an enhanced Shift & Add. Quantitative and qualitative evaluations have been conducted at two levels; the initial estimation and the final optimized super-resolution. Results show that the proposed algorithm outperforms existing state-of-art methods. [less ▲]

Depth Super-Resolution by Enhanced Shift and Add

in Computer Analysis of Images and Patterns, 15th International Conference, CAIP 2013, York, UK, August 27-29, 2013, Proceedings, Part II (2013)

Real-Time Distance-Dependent Mapping for a Hybrid ToF Multi-Camera Rig

in IEEE Journal of Selected Topics in Signal Processing (2012), 6(5), 1-12

Bilateral Filter Evaluation Based on Exponential Kernels

in Pattern Recognition (ICPR), 2012 21st International Conference on (2012)

The well-known bilateral filter is used to smooth noisy images while keeping their edges. This filter is commonly used with Gaussian kernel functions without real justification. The choice of the kernel ... [more ▼]

The well-known bilateral filter is used to smooth noisy images while keeping their edges. This filter is commonly used with Gaussian kernel functions without real justification. The choice of the kernel functions has a major effect on the filter behavior. We propose to use exponential kernels with L1 distances instead of Gaussian ones. We derive Stein's Unbiased Risk Estimate to find the optimal parameters of the new filter and compare its performance with the conventional one. We show that this new choice of the kernels has a comparable smoothing effect but with sharper edges due to the faster, smoothly decaying kernels. [less ▲]

A New Multi-Lateral Filter for Real-Time Depth Enhancement

in 8th IEEE International Conference on Advanced Video and Signal-Based Surveillance (2011)