Physics-guided infrared spatiotemporal noise modeling based on hybrid neural representation

Collecting real-world infrared noisy-clean video pairs to train deep video denoising networks is challenging; therefore, these networks typically rely on synthetic data generated from noise models. Existing models primarily focus on the spatial distribution of noise but often fail to accurately capture the complexity of temporal variations in infrared noise within dynamic scenes, which limits the performance of denoising networks. To address this issue, we propose an infrared spatiotemporal noise modeling framework (IRSTN) based on hybrid neural representation, which leverages unpaired video data to simulate real-world noise. The key of IRSTN lies in its independent and compact representation of the spatial and temporal distributions of noise. Specifically, IRSTN first obtains spatial embeddings by introducing physical-based noise prior to capture the spatial context of noise; secondly, it generates temporal embeddings using position encoding of the frame index to describe the temporal correlations of noise. Subsequently, IRSTN constructs hybrid neural representations of noise that deeply integrate spatial and temporal embeddings while implicitly modeling the complex spatiotemporal distribution of infrared noise through recurrent adversarial learning. Furthermore, by constraining the consistency of noise intensity in both the forward and backward recursions, it effectively suppresses temporal artifacts that may appear in the generated noisy videos. To validate the effectiveness of IRSTN, we collected a real-world infrared video denoising dataset for training and benchmarking. Qualitative experiments indicate that the infrared noise generated by IRSTN is highly similar to real noise in terms of spatiotemporal distribution. Extensive denoising experiments demonstrate that IRSTN endows infrared video denoising networks with highly competitive performance in real-world scenarios.

IEEE Sensors Journal [Under Review]