Video Demoireing using Focused-Defocused Dual-Camera System

Moire patterns, unwanted color artifacts in images and videos, arise from the interference between spatially high-frequency scene contents and the spatial discrete sampling of digital cameras. Existing demoireing methods primarily rely on single-camera image/video processing, which faces two critical challenges: 1) distinguishing moire patterns from visually similar real textures, and 2) preserving tonal consistency and temporal coherence while removing moire artifacts. To address these issues, we propose a dual-camera framework that captures synchronized videos of the same scene: one in focus (retaining high-quality textures but may exhibit moire patterns) and one defocused (with significantly reduced moire patterns but blurred textures). We use the defocused video to help distinguish moire patterns from real texture, so as to guide the demoireing of the focused video. We propose a frame-wise demoireing pipeline, which begins with an optical flow based alignment step to address any discrepancies in displacement and occlusion between the focused and defocused frames. Then, we leverage the aligned defocused frame to guide the demoireing of the focused frame using a multi-scale CNN and a multi-dimensional training loss. To maintain tonal and temporal consistency, our final step involves a joint bilateral filter to leverage the demoireing result from the CNN as the guide to filter the input focused frame to obtain the final output. Experimental results demonstrate that our proposed framework largely outperforms state-of-the-art image and video demoireing methods.

This figure shows our proposed frame-wise processing pipeline. In the \(\textbf{alignment}\) step, we warp the defocused frame \(\bf{I_D}\) to align with the focused frame \(\bf{I_F}\) and use the pixels of \(\bf{I_F}\) to fill the occluded regions to obtain the alignment result \(\bf{I_A}\). In the \(\textbf{demoireing}\) step, \(\bf{I_F}\) is fed into the demoireing network to remove the moire patterns with \(\bf{I_A}\) as guidance. The network is trained by a multi-dimensional loss. In the \(\textbf{recovery}\) step, \(\bf{I_F}\) is filtered by the joint bilateral filter with \(\bf{I_R}\) as guidance, to produce the final output \(\bf{I_O}\) with tonal and temporal consistency.

This figure shows network structures of (a) the demoireing network and (b) the discriminator in the paired adversarial loss.

This figure shows our results on DualSyntheticVideo dataset. The regions marked with blue and orange boxes are enlarged.

This figure shows our results on DualReal dataset. The regions marked with green and red boxes are enlarged.