3D Image Registration And Depth Enhancement Of Depth Images Captured By
Intel’s Tof Camera
Abstract’ Image registration refers to determining a geometrical transformation that brings together points in one view of an object and corresponding points in another view of that object, thus, finding the best alignment between the two. The Iterative Closest Point (ICP) algorithm is considered as classical algorithm for image registration. This paper provides a comparative study of using traditional ICP algorithm against using ICP algorithm along with Quaternion for image registration. The use of Quaternion provides much better visual perception to the registered image. Depth refers to distance of the surface of scene object from a viewpoint. Depth Map is an image that contains information related to depth of the image. Depth enhancement is an image processing technique in which the noise in the depth map is reduced. The Time Of Flight (TOF) camera by Intel is a low resolution camera, and requires enhancement of the depth images, captured by it, for better visualization. Bilateral filter, an edge preserving noise reducing filter, is generally used for enhancing these images. This paper proposes the use of wiener filter over bilateral filter for providing better visual perception to the depth images taken by the camera. The comparison between proposed designs and the existing ones is also provided.
Index Terms ‘ Image Registration, ICP, Depth Enhancement, Bilateral Filter, Weiner Filter, Time-of-Flight Camera.
I. INTRODUCTION
Image registration is a technique of image processing that aligns different views of an object into a single integrated image. The different views of an object may be captured from different cameras and may come from different photographs, which may be translated and even rotated. Image registration helps in overlaying them in order to bring them to the same coordinate system. This technique finds its use in computer vision, 3D reconstruction, face recognition, medical imaging, military automatic target recognition system, etc. [1]
Gold standard for this technique is considered as ICP algorithm as it is robust, insusceptible to noise and easily adapts to rigid shape matching [2]. It is a recursive algorithm that tries to determine a transformation between the two views of an object which provides a geometric correspondence between them.
ICP algorithm reduces the difference between the sets of points of the two views of object. In this, one set of points called target is kept fixed whereas the other set of points called source is transformed iteratively.
Quaternion, described by Irish Mathematician, William Rowan Hamilton, in 1843, is a number system extending complex numbers. It is also called hypercomplex or supercomplex number, defined as:
TOF camera is a camera which finds the distance of a surface of an image from a viewpoint based on the known speed of light.
The Intel’s TOF camera is of low resolution. So, there arises a requirement to enhance the depth images captured by the camera. Enhancement of depth image includes removal of noise while preserving its edges.
Bilateral filter is a non linear smoothing filter that reduces the noise in images and preserves the edges at the same time [3]. In this, each pixel’s intensity value is replaced by a weighted average of intensity values of pixels in neighborhood of the pixel. The weight is based on Gaussian distribution and depends on Euclidean distance of pixels as well as range difference of pixels. The equation for bilateral filtering is given by:
where, BF[I]p is Bilateral filtered image
1/Wp is Normalization factor
G??s is Gaussian distribution in space domain
G??r is Gaussian distribution in intensity domain
p is Central pixel position
q is Neighborhood pixel position
Ip is Central pixel value
Iq is Neighborhood pixel value
Wiener filter is a linear time invariant filter used for restoration of desired image by filtering a noisy image. This restoration of image is done by minimizing mean square error between the original image and the restored image and thus filtering out noise from the image [4]. The mean square error is given as:
The frequency response of Weiner filter is given as:
II. REVIEW OF IMAGE REGISTRATION AND DEPTH ENHANCEMENT
Various approaches of ICP are available ranging from classical approach to fast ICP approach. The steps of classic ICP algorithm are given as follows for a set of source points, denoted by D and a set of target points denoted by M [1]:
1. Find the closest points in the target set of points corresponding to each point in source set of points. So, for each source point represented as di ‘ D, the closest point mj ‘ M, needs to be computed based on the given equation:
t can be written as:
Figure 2: Translation matrix having dx, dy and dz as translations.
Rx, Ry and Rz can be written as:
Figure 1: Rotation Matrices about x, y and z axis
2. Using these points, (di, mj), estimate the rotation and translation parameters, denoted by a = (R, t), that best align them.
3. Apply the obtained transformation to the source set of points, D.
4. Reiterate the process till the error between the two successive iterations gets minimized and becomes lower than a predetermined threshold.
Figure 3: Estimating correspondence between source and target points in ICP calculated as mj for each source point di’ = (Rdi + t).
Techniques to enhance depth involve the use of Bilateral filter and, in some cases, Joint Bilateral Filter [3] has found its use. This filter accounts both space weight and intensity weight. Space weight corresponds to G??s (||p ‘ q||) in the equation of Bilateral filter. Space corresponds to ??s which is the spatial extent of the kernel, that is, the size of the considered neighborhood. Intensity weight corresponds to G??r(|Ip – Iq|) in the equation of Bilateral filter. Range corresponds to ??r which is the minimum amplitude of an edge. In the spatial domain, pixels are said to be close to one another, if they occupy nearby spatial location. In intensity domain, pixels are said to be similar to one another, if they have nearby intensity values in a perceptual way. These weights are calculated by various Gaussian kernels. The operation is defined such that the pixels lying far from the central pixel, of desired window of image taken into consideration, in the space domain as well as in the intensity domain are neglected whereas the pixels lying in proximity of the central pixel, of desired window of the image taken into consideration, in space and intensity domain are used in the smoothing operation. This helps Bilateral filter in preserving edges of the image.
III. PROPOSED METHOD FOR IMAGE REGISTRATION
AND DEPTH ENHANCEMENT
The various approaches seen in ICP improve the robustness and speed of the algorithm, as it is an iterative algorithm. The technique proposed in this paper illustrates the use of Quaternion with ICP algorithm for improving the visual perception of the registered image. This technique uses Quaternion to represent the rotation matrix R of the transformation. The three rows of the matrix R based on Quaternion are given as follows:
For testing the proposed design of ICP with Quaternion, Figure 4 and 5 were taken as a test images. Figure 4 is the target image while Figure 5 is a source image which is rotated at an angle of 30??. The two images were then registered by classical ICP algorithm to provide the registered image as shown in Figure 6. The two images, Figure 4 and Figure 5 were then registered by using Quaternion with ICP to provide the registered image as shown in Figure 7. As is evident from both the registered images, the image registered by using Quaternion with ICP, Figure 7, has better viewing quality than the image registered by classic ICP algorithm, Figure 6.
Figure 4: Target image taken as test image for image registration.
Figure 5: Source image taken as test image for image registration. The target image is rotated at an angle of 30??.
Figure 6: Registered image using classical ICP having low contrast compared to Figure 7.
Figure 7: Registered image using Quaternion with ICP having high contrast compared to Figure 6.
The Bilateral filter, along with its variants, is used to enhance the depth images. This paper proposes a technique of enhancing the depth images captured by low resolution depth camera like Intel’s TOF camera. The technique proposed involves filtering the image by Bilateral filter and the output of the filter is provided to Weiner filter. Bilateral filter smoothens the image and preserves its edges while the Weiner filter applies restoration to the image and thus enhances the image providing a better visual aspect to the image as compared to the image filtered only by Bilateral filter.
For testing the proposed technique for depth enhancement, Figure 8 was taken as a test image. This image is captured by Intel’s TOF camera. It was then filtered by Bilateral filter to smoothen it and the filtered image obtained is as shown in Figure 9. Figure 9 was provided as input to the Weiner filter for applying restoration on it and the output enhanced depth image is as shown in Figure 10. The enhanced image obtained on applying Weiner filter over the Bilateral filtered image, Figure 10, is, evidently, visually appealing as compared to the enhanced depth image as obtained by applying Bilateral filter alone, Figure 9.
Figure 8: Depth image taken as test image for enhancement.
Figure 9: Depth image output from Bilateral Filter.
Figure 10: Depth image output from Weiner over Bilateral Filter having better visual perception as compared to Figure 9.
IV. CONCLUSION
This paper shows the study and comparison of various techniques, existing and proposed, in terms of better visual perception for image registration as well as depth image enhancement. The effectiveness of the proposed techniques for both registered image and enhanced depth image is illustrated by practical results.
This work provides timing optimization, using the same amount of resources. Further scope of this work may include minimizing time required for the computation of registered and enhanced depth images. This will make the technique optimized as visually better results will be obtained in less amount of time.
...(download the rest of the essay above)