Self-Supervised Scene Dynamic Recovery from Rolling Shutter Images and Events

1Wuhan University, 2Peking University
Corresponding authors
Event Frame
RS Video
Scene Dynamic Recovery
by SelfUnroll


Abstract

Scene Dynamic Recovery (SDR) by inverting distorted Rolling shutter (RS) images to an undistorted high frame-rate Global Shutter (GS) video is a severely ill-posed problem due to the missing temporal dynamic information in both RS intra-frame scanlines and inter-frame exposures, particularly when prior knowledge about camera/object motions is unavailable. Commonly used artificial assumptions on scenes/motions and data-specific characteristics are prone to producing sub-optimal solutions in real-world scenarios. To address this challenge, we propose an event-based SDR network within a self-supervised learning paradigm, i.e., SelfUnroll. We leverage the extremely high temporal resolution of event cameras to provide accurate inter/intra-frame dynamic information. Specifically, an Event-based Inter/intra-frame Compensator (E-IC) is proposed to predict the per-pixel dynamic between arbitrary time intervals, including the temporal transition and spatial translation. Exploring connections in terms of RS-RS, RS-GS, and GS-RS, we explicitly formulate mutual constraints with the proposed E-IC, resulting in supervisions without ground-truth GS images. Extensive evaluations over synthetic and real datasets demonstrate that the proposed method achieves state-of-the-art and shows remarkable performance for event-based RS2GS inversion in real-world scenarios.




Method



More Results in Real-World Scenarios

Single GS Frame Reconstrction

To evaluate the effectiveness of different methods (CVR, EvUnroll and Ours) of RS correction, the upper portion of the image displays the reconstructed effect when the mouse is hovered over it, while the lower portion shows the original input RS image. This allows for a clearer comparison of the correction effect.


Event Frame
RS Video
CVR
EvUnroll
Ours

Continuous-time GS Video Reconstrction

Our method can generate continuous-time GS video at arbitrary time from rolling shutter images and events. We compare Our SelfUnoll-M with CVR and EvUnroll on 10X (1st row), 50X (2nd row), 100X (3rd row) reconstructions (nX represents the reconstructions n GS frames from one RS frame).

<
Event Frame
RS Video
CVR
EvUnroll
Ours





BibTeX

@article{wang2023self,
        title={Self-Supervised Scene Dynamic Recovery from Rolling Shutter Images and Events},
        author={Wang, Yangguang and Zhang, Xiang and Lin, Mingyuan and Yu, Lei and Shi, Boxin and Yang, Wen and Xia, Gui-Song},
        journal={arXiv},
        year={2023}
      }