Essay: Removing Photography Artifacts Using Flash and No-flash Image Pairs”Removing Artifacts with Flash and No-flash Image Pairs for HDR Photos



Removing Photography Artifacts Using Flash &  No-flash Image Pairs

Dr. Prashant R. Deshmukh1, Ankita R. Borkar2,

Student, Computer Science and Engineering, Sipna C.O.E.T., Amravati, India2

p_rdeshmukh@yahoo.com1,borkarankita972@gmail.com

 Abstract –  In this paper, we are going to take a novel view based on a photography approach in which we are concerning  the study of removing Photography Artifacts like removal of glare or the flash  hot spot and remove reflection ,Highlights caused by glossy surface . so here we are using both Ambient and Flash images to. In this proposed scheme system we are going to focus on different approaches for remove the Photography artifact. We are presenting a novel Weighted Mapping scheme where we compute weighted averaging to remove glare and also introduce  techniques  like Diffusion tensor , cross diffusion  tensors using gradient fields of both ambient and flash image to remove the reflection. We used these tensors to remove reflections in an image We analyze some experimental results that demonstrate the proficiency of algorithms to produce improved quality of flash images having glare and also reflection.

Keywords – Glare, flash, Gradient field , reflection removal, cross Diffusion tensor, high dynamic range.

                               I .  Introduction

 This over the years, a number of schemes  have been  introduced  for removing glare and reflection  which defined as a uncomfortably of bright light. In photography artifact the glare is a sensation in rcaused by bright light that can take a different numbers of forms and   Flash highlight  can also cause an unwanted discomfort, exasperation or even pain, and this sensation is another form of glare. In HDR image, glare is a sensation caused by bright or flash of light. In addition flashes caused disagreeable reflections. Often one sees the reflection of an object that mendacity exterior the field of view of the camera but is robustly lit by the flash[1-4]. One also sees strong highlights due to straight reflection of the flash itself by glossy objects in the scene. Former work has shown that a flash and ambient image pair can be used to de-noise and enhance the ambient image and to accomplish enhanced image fusion. Our objective is to use a flash and ambient image pair to produce a high quality flash image by reducing glare region and reflection photography artifacts. Our purpose is the design of the glare free and reflection free image by using some different techniques like Weight mapping approach where we are going to used two image one is flash one is without flash so called Ambient image and apply the weighted average method to get glare free image and similarly we need ambient and flash image to remove reflection where we are using techniques like gradient field ,diffusion tensor and cross diffusion tensor using the single threshold for the across all pixels of  image cannot account for variations in high lighting and reflection through the image[5,6,7,8].

II.   Literature Survey

Until the year 1997, P.E Debevec &  Malik introduced a scheme that present a method which has been convalescing HDR radiance maps on various photographs of the scene. The algorithm implies that different exposed photographs has been recover by  function of the imaging process. [1]. In  year 1997, Nayar, Fang and Boult represent the techniques for extrication the reflection components using color an also polarization in this paper  reflection produced strong highlights in the brightness image which contain minute quantity of glare reflection of light[2]. In year 2002, R. Fattal et. al. introduced  a new techniques on conventional display for interpretation  HDR images. This approach is simply easy and robust ,Where one can find the gradient field by diminution the magnitudes of large gradients of the luminance high dynamic range image by depletion [3]. E. Eisemann & F. Durand has planned in year 2004 to enhanced qualityof  Flash photography by the via intrinsic relighting in which they take photographs shot in dark  region area by merging  that picture taken with the available light and also with the flash. They also preserve original lighting. By using bilateral filter the images is break up into detail and large scale [4].

   R. Raskar, K.Tan, R. Feris, J.Yu , and M.Turk, also present work on numerous flash imaging in year 2004. Where they take non-photorealistic interpretation to capture and create shape features of real world scenes using a camera with multiple depth discontinuities in the scene. The projective geometric relationship of the camera-_ash setup exploited uncovering of depth discontinuities and. The resulting images get is more better and clear to the 3D structure of the imaged scenes [5].

   G. Petschnigg et al. Anticipated  method for both flash and no flash images in year 2004 in which they represent the Digital photography which has been made it possible to quickly capture a pair of images of low-light environments with flash and without flash to take ambient illumination[6]. Amit  Agrawal et al. propose to use a flash and ambient image to produce better flash images also present a novel gradient projection scheme based on a gradient consistency model take contained Gx along x, Gy along y axis of variable gradient field for each  pixel and flash intensity exposure space shows the  method for adaptively sampling to minimize the quantity images for any given scene. [7].

   In the year 2007, E. Talvala, A. Adams, M. Horowitz, & M. Levoy anticipated  record of a high dynamic range image, by taking picture at one snapshot or a sequence snapshots, is limited by the presence of veiling glare the contrast has been reduced everywhere within the some field of the view and such glare arises from multiple scattering of light inside the camera’s body . In this paper, two methods used for removing glare are deco volution by a measured glare spread function and selectively block the light that contributes to veiling glare with deco volution. [8].

     In year 2008 , R. Raskar et al. introduced the Glare aware photography in camera lenses of 4D ray sampling to minimized glare effects on image Phenomenon. This scheme explore masks for uniform and non-uniform ray sampling technique to show how a practical solution is used to analyse the 4D statistics without significantly compromising image resolution [9].                      P. Didyk, R. Mantiuk, M. Hein, and H. Seidel utilize the full potential of new high dynamic range displays, the video sequences in a system of bright luminous objects improved the quality is proposed in year 2008 .Which may classifies saturated regions as lights, reflections and diffuse surfaces by using a semi-automatic classifier [10].

         In year 2008 , Rolf Adelsberger  et.al. introduced system that employs an infrared time-of-flight rangefinder to find the distance to each object and a enhanced video projector to modulate the illumination. While this approach is limited by the power of our light source that produced illumination physically  plausible. [11] O. Gallo et. al. projected an scheme which is capable of dealing with a large amount of movement in the scene with a reference image previously selected from the stack of images in year 2009 . It create the HDR image by average radiance estimates of all regions and also compensate for camera calibration error [12].

     In year 2011 , C. Lau, M. Rouf, M. Trentacoste, R. Mantiuk and W. Heidrich. Proposed a novel view approach of  High Dynamic Range capture, which is basically based on  photography approach. Where firslyt optically encoding done with both the high light information into LDR image that can be captured with a conventional image sensor and low dynamic range part of the scene and high light information into LDR image that can be captured with a conventional image sensor. This process overcome by using a cross-screen or star filter then decode, in software [13].

    Kanchit Pamanee, Angela Duangchit, Roungsan Chaisricharoen has anticipated a new method recovering information from the surrounding area of glare centroid in 2013 and 3DRS is assigned to extract luminance information and then  separate them from intensity of edges form by material to find out edge features in animated images from Macroblock of the previous non-glare affected frames. For enhancing  the distorted MB tone mapping method is applied[14].

     Neha Hial and Somesh Dewangan introduced  an approach for edge-suppressing operations on an image,which  depends on affine transformation of gradient fields using cross projection tensor method which  derived from another image[15].

Proposed Methodology

Our proposed work is basically a framework design with two folded method using Ambient and Flash image which as shown below :-

 1. Using weighted Mapping method for Removal of Glare

 2. Using several method like diffusion tensor ,gradient and cross diffusion for Removal of Reflection

Removal Glare or flash hotspot

In this method we are going to apply Weighted average method to both Ambient and flash image and select the minimum among the three planes would give flashy region or whiteness present in image and then we apply the weighted Average method are shown as follows:-                                                                                         

         I  =  w × A + (1-w) × F,

  Let,   W = min (I, [], 3)     

  I  =  A + F

 Where, w is the weight , A is the Ambient and  F is Flash image .

Algorithm:- Glare Removal Algorithm

 Input:-   Ambient and Flash image of same object

Output:-  Glare free image.

 Step 1:-  An input carrier Ambient and Flash image will be selected.

Step 2:-  Determine the flash hot spot or glare region in Flash image using weighted average method

Step 3:- Find weight which indicates the undesired portion or region  with ambient image and negative weight indicates the desired region with flash image .

Step 4:-  Add both Ambient and Flash image using weight mapping techniques.

Step 5:-  After adding both image we get Glare free image or artefact free image.

 Step 6:-  End.

Reflection Removal

  In this section we are applying diffusion tensor  and cross diffusion tensor by computing gradient field of both ambient and flash image.We will initiate process by showing how to compute the gradient at an image . Then, we compute actually does encode the gradient direction and magnitude.We form the gradient vector by combining the partial derivative of the image in the x direction and the y direction.

                 Î”I= (∂I/∂x, ∂I/∂y)

                    We are also using the Diffusion Tensor , where Df  is a diffusion tensor field contain at each pixel, Df  (x, y) is a 2 × 2 symmetric matrix, positive definite. Df  is then obtained as

Df   = [â– (Df11&Df12@Df21&Df22)]

Where,

               D(y, x) =  [â– (d11(y,x) &d12(y,x)@d21(y,x) &d22(y,x))]

is a 2×2 symmetric, positive-definite matrix at each pixel or a field of diffusion tensors using it we also compute eigen values and eigen vector of diffusion tensor matrix. Due to reflection artifact ,the estimation of the gradient using finite differences, along with  small non-zero value is used as a threshold to decide for homogeneity  which vary across the image, if the image has spatially varying illumination and then used Cross Diffusion Tensor method as follows:-

D11  =   L1 × x1 2 +  L2× y12

D12  =   L1 × x 1 x2  + L2 × y1 y2

D22  =  L1 × x22  +  L2 × y22

 Algorithm: Reflection Removal Algorithm

 Input:   Ambient and Flash image of same object

Output:  Reflection free image.

 Step 1:  An input carrier Ambient and Flash image will be selected.

Step 2:  Convert RGB to YUV  of ambient and flash image

Step 3:  Apply diffusion tensor both images

Step 4:  Calculate eigen values and eigen vector of diffusion tensor matrix.

Step 5:  Apply thresholding term

Step 6:  Determine Cross Diffusion tensors.

Step 7:  Calculate gradient of Ambient image and Subtract that gradient with Cross diffusion tensor gradients.

Step 8:   Convert YUV to RGB

Step 9:  Finally we get Reflection free image.

Step10:  End.

Experimental results

 In the experimental results ,to find out noise or artifact like glare and reflection we were simply used the term PSNR which stands for peak signal to noise ratio . It used to measure the quality of images. Higher the PSNR value, better the quality of image. It is estimated in decibels (db). So we are taking PSNR of artefact or noisy image like glare or reflection with Resultant recovered image.

  PSNR = 10.log10 ( MAX 2I /MSE )

Where ,MSE stands for the mean square error. Lower the value of MSE, better the quality of image. Now see the following input image of computer taken with flash or no flash:-

 

Fig. 1  Ambient image  (left) and Flash image (right)

   

    Fig. 2  Grayscale image of Flash image with (w) and (1-w)        

                           

            Fig. 3  3D visualisation of (w)weight

           Fig. 4 3D visualisation of (1-w)weight

                

TABLE I

PSNR for different Image types

Ambient Image Type Size of Ambient Image Flash Image Type Size of Flash Image Processing Time for Glare free image PSNR

.bmp 2560*1920 .bmp 2560*1920 4.0021 19.9685

.png 2560*1920 .png 2560*1920 11.3193 19.9685

.jpg 2560*1920 .bmp 2560*1920 4.976 19.9693

       

                               Fig. 5  Comparison of experimental result

     From above table. 1 we can see that when we take input  .jpg as ambient image and .bmp as flash image we

get maximum PSNR as 19.9693 as compared to others. Finally we get glare free image as show below :-

                                     Fig.6  Resultant image (Glarefree image)

       Now similarly we compute PSNR for reflection where we experiment on different parameters like sigma ,threshold by changing its value and also we take different image types like .jpg , .png , .bmp etc. Which shows in above table 1 and fig.5.

 

                       Input Images

   

          Fig. 7  Ambient and flash image

                  Fig.8  Reflection free image

Amb-ient

Image type Flash Image Type Size of both Image Iteration Processing Time PSNR

(T) (σ)

.png .png 401*301 5 5 0.4 23.72845 23.9066

.jpg .bmp 401*301 5 5 0.8 21.0125 24.6817

 

                           Fig.9  Recovered reflection

 In fig.8 and fig.9, we get Reflection free and Recovered reflection image by taking diffusion tensor with its eigen values and eigen vector so here we changes the values of threshold as 5 ,15 ,25 which shown below in table 2 and table 3 means we are increasing threshold but we are getting better result when threshold  value is minimum and we also experiment by changing  the value of sigma as 0.4 , 0.6, 0.8 . After changing value of sigma and threshold we estimate PSNR of Flash image and the resultant image.

TABLE II

PSNR after changing Threshold value

                        

     Threshold

Image Types

T = 5

(PSNR)

T = 15

(PSNR)

T = 25

(PSNR)

.png 24.5978 22.6706 22.3934

.jpg 24.5554 22.7620 22.4607

.jpg & .bmp 24.6817 22.5487 22.3450

TABLE III

                     Comparison result with Threshold 5,15, 25

Threshold PSNR

5 24.6817

15 22.7620

25 22.4607

Fig. 10  Comparison result with Threshold 5,15, 25

TABLE IV

                     Comparison result with Threshold & Sigma

                       Fig. 11  Comparison of experimental result

 

   We have noticed  that  in above  table 4 and fig. 10 when sigma is  0.4 and threshold  is 5 we get 23.6817  PSNR value which is less than  the value  24.6817 when sigma set as 0.8 with threshold as 5 .

Conclusion   

   The main objective of proposed scheme is to remove Flash photography artifact like glare and reflection from  so  here we are taken flash photography and have addressed three well-known problems: over-illumination or under-illumination at a given flash intensity, reflections or highlights also refered as glare . We reduced  these  glare by exploiting information in the ambient image. We used weighted average method to remove glare for different image types and used cross diffusion tensor method to remove reflection by changing threshold and sigma value for different images type like .bmp,.png ,.jpg etc. We noticed that we get good quality of image when threshold value is minimum. Our approach is conceptually simple and can easily handle any type of color images . we hope in future our method can remove all problem of photography artifacts easily.

References

 [1] P. E. Debevec and J. Malik “Recovering high dynamic range radiance maps from photographs”. In Proc. Of the 24th annual conference on Computer graphics and interactive techniques, 369–378 , 1997 .

[2] S.K. Nayar,,X.S. Fang, and T. Boult, “Separation  of reflection components using color and polarization.” Int’l. Journal of Computer Vision 21, 3 (Feb.),163–186,1997 .

[3] R. Fattal, D. Lischinski, and M. Werman, “Gradient domain high dynamic range compression.” ACM Transactions on Graphics 21, 3, 249– 256, 2002 .

[4] E. Eisemann and F. Durand “Flash  photography enhancement via intrinsic relighting”. ACM Transactions on Graphics 23, 3 (Aug.), 673– 678 , 2004.

 [5]  R. Raskar, K.Tan, R. Feris, M.Turk and J.Yu,. “Non-photorealistic camera : depth edge detection and stylized rendering using multi-flash imaging.” ACM Transactions on Graphics 23, 3, 679–688 , 2004.

 [6]  G. Petschnigg , M. Agrawala, H.Hoppe, R.Szeliski, M.Cohen & K .Toyama” Digital photography with flash and no-flash image.pairs.” ACM Transactions on Graphics 23, 3 (Aug.) , 2004, 664–672.

[7]  Amit Agrawal, Ramesh Raskar, Shree K. Nayar , & Yuanzhen Li .” Removing Photography Artifacts using Gradient Projection & Flash Exposure Sampling”, 2006.

[8]  E. Talvala, A. Adams, M. Horowitz, and M. Levoy. “Veiling glare in high dynamic range imaging.” ACM Trans. Graph. ,, 26(3):37, 2007.

 [9]  A. Agrawal, C. A. Wilson, Ramesh Raskar & A. Veeraraghavan. “Glare  aware  photography : 4D ray sampling for reducing glare effects of camera lenses.” ACM Trans. Graph., 27(3 ) . 2008 .

[10]  M. Hein, and H. Seidel. R. Mantiuk, & P. Didyk,  â€œEnhancement of bright video features for HDR displays.” Computer Graphics Forum  27(4):1265–1274,2008.

 [11]  Remo Ziegler, Rolf Adelsberger, Marc Levoy, & Markus Gross,” Spatially Adaptive Photographic Flash” ETH Zurich, Department of  Computer Science, Technical Report No., 612, 2008.

[12]  O. Gallo, N. Gelfand , W. Chen, M. Tico, & K. Pulli. “Artifactfree   high dynamic range imaging.” In Proc. ICCP, 2009.

[13] C. Lau, M. Rouf, M. Trentacoste, R. Mantiuk & W. Heidrich. “Glare Encoding of High Dynamic Range Images,” in Computer Vision and Pattern Recognition (CVPR),  pp. 289-296, 2011.

[14] Chaisricharoen  ,Roungsan & Angela Duangchit ,“Vision Restoration of Dangerous Head Light Glare Affected Scenario,” in 13th International Symposium on Communications and Information Technologies (ISCIT), , pp. 733-736 , 2013.

[15] Somesh Dewangan & Neha Hial ,” Detection of Shadow and Its Removal along with the Edge Suppression” International Journal of Science and Research (IJSR) , ISSN:2319-7064,2013.