Super resolution reconstruction of noisy images based on dense residual connected U-shaped networks
-
摘要: 现有的图像超分辨率重建网络难以适用于煤矿井下噪声密集的应用场景，且多数网络通过增加深度提升性能会导致无法有效提取关键特征、高频信息丢失等问题。针对上述问题，提出了一种密集残差连接U型网络，用于对低分辨率噪声图像进行超分辨率重建。在特征提取路径中引入基于密集残差连接的去噪模块，通过密集连接的方式对图像特征进行充分提取，再利用残差学习的特点对低分辨率噪声图像进行有效去噪；在重建路径中引入残差特征注意力蒸馏模块，通过在残差块中融入增强特征注意力块，对不同空间的特征赋予不同的权重，加强网络对于图像关键特征的提取能力，同时减少图像细节特征在残差块中的损失，从而更好地恢复图像细节信息。在煤矿井下图像数据集及公共数据集上进行了对比实验，结果表明：在客观评价指标上，所提网络的结构相似度、图像感知相似度均优于对比网络，且在复杂度及运行速度上有着较好的均衡；在主观视觉效果上，所提网络重建的图像基本消除了原有图像噪声，有效恢复了图像的细节特征。Abstract: The existing image super-resolution reconstruction networks are difficult to apply to noise intensive application scenarios in coal mines. Most networks improve performance by increasing depth, which leads to problems such as ineffective extraction of key features and loss of high-frequency information. In order to solve the above problems, a dense residual connected U-shaped network is proposed for super-resolution reconstruction of low resolution noisy images. The denoising module based on dense residual connections is introduced in the feature extraction path, fully extracting image features through dense connections. The features of residual learning are used to effectively denoise low resolution noisy images. The residual feature attention distillation module is introduced in the reconstruction path, by incorporating enhanced feature attention blocks into the residual blocks, different weights are assigned to features in different spaces to enhance the network's capability to extract key image features. The loss of image detail features is reduced in the residual blocks, thus better restoring image detail information. Comparative experiments are conducted on coal mine underground image datasets and public datasets, and the results show that in terms of objective evaluation index, structure similarity and image perception similarity of the proposed network are superior to the comparison network. It has a good balance in complexity and running speed. In terms of subjective visual effects, the image reconstructed by the proposed network basically eliminates the original image noise and effectively restores the detailed features of the image.
-
图 5 不同网络在Noise−Urban100上的图像超分辨率重建效果对比
Figure 5. Comparison of image super resolution reconstruction effect of different networks on Noise-Urban100
图 6 不同网络在Noise−B100上的图像超分辨率重建效果对比
Figure 6. Comparison of image super resolution reconstruction effect of different networks on Noise-B100
图 7 不同网络在Noise−场景1上的图像超分辨率重建效果对比
Figure 7. Comparison of image super resolution reconstruction effect of different networks on Noise-scenario 1
图 8 不同网络在Noise−场景2上的图像超分辨率重建效果对比
Figure 8. Comparison of image super resolution reconstruction effect of different networks on Noise-scenario 2
表 1 含有不同数量RFL的网络在Noise−Set14上的LPIPS和SSIM
Table 1. LPIPS and SSIM of network with different numbers of residual feature fusion layer on Noise-Set14
RFL数量 0 1 2 3 4 LPIPS 0.585 0.528 0.492 0.470 0.461 SSIM 0.627 0.669 0.717 0.752 0.763 
下载: 导出CSV
表 2 含有不同数量RFL的网络在Noise−B100上的LPIPS和SSIM
Table 2. LPIPS and SSIM of network with different numbers of residual feature fusion layer on Noise-B100
RFL数量 0 1 2 3 4 LPIPS 0.632 0.598 0.563 0.532 0.525 SSIM 0.650 0.665 0.671 0.682 0.688
下载: 导出CSV
表 3 消融实验结果
Table 3. Results of ablation experiments
DRCDM RFAM PSNR SSIM × × 30.12 0.8968 √ × 30.50 0.9306 × √ 30.42 0.9289 √ √ 30.58 0.9315
下载: 导出CSV
表 4 不同网络在测试集上的LPIPS对比
Table 4. Comparison of LPIPS of different networks on test set
测试集 缩放因子 LPIPS Bicubic ESPCN EDSR RCAN DBPN CSNLN NLSN EFDN Ours Noise−Set5 4 0.716 0.535 0.693 0.582 0.678 0.547 0.545 0.542 0.357 8 0.626 0.525 0.653 0.557 0.564 0.539 0.537 0.529 0.453 Noise−Set14 4 0.782 0.521 0.740 0.629 0.742 0.562 0.563 0.559 0.461 8 0.692 0.603 0.708 0.623 0.672 0.579 0.577 0.575 0.540 Noise−Urban100 4 0.894 0.671 0.715 0.681 0.707 0.677 0.675 0.672 0.492 8 0.782 0.681 0.798 0.712 0.718 0.675 0.672 0.673 0.612 Noise−B100 4 0.708 0.610 0.654 0.642 0.683 0.618 0.616 0.613 0.525 8 0.810 0.723 0.795 0.776 0.788 0.721 0.719 0.717 0.685 Noise−场景1 4 0.823 0.623 0.714 0.588 0.677 0.615 0.542 0.523 0.502 8 0.799 0.633 0.702 0.655 0.625 0.725 0.622 0.630 0.615 Noise−场景2 4 0.816 0.556 0.742 0.596 0.764 0.645 0.566 0.526 0.510 8 0.795 0.645 0.756 0.637 0.755 0.655 0.678 0.636 0.622
下载: 导出CSV
表 5 不同网络在测试集上的SSIM对比
Table 5. Comparison of SSIM of different networks on test set
测试集 缩放因子 SSIM Bicubic ESPCN EDSR RCAN DBPN CSNLN NLSN EFDN Ours Noise−Set5 4 0.599 0.697 0.602 0.675 0.608 0.692 0.695 0.697 0.736 8 0.565 0.672 0.552 0.612 0.587 0.632 0.638 0.640 0.712 Noise−Set14 4 0.567 0.707 0.592 0.611 0.588 0.630 0.637 0.636 0.763 8 0.538 0.647 0.508 0.598 0.551 0.647 0.653 0.655 0.701 Noise−Urban100 4 0.698 0.801 0.708 0.788 0.711 0.789 0.792 0.795 0.877 8 0.531 0.710 0.605 0.658 0.649 0.718 0.719 0.721 0.785 Noise−B100 4 0.563 0.651 0.617 0.635 0.573 0.645 0.648 0.651 0.688 8 0.496 0.522 0.472 0.495 0.493 0.525 0.528 0.531 0.559 Noise−场景1 4 0.814 0.789 0.846 0.855 0.823 0.845 0.865 0.845 0.878 8 0.768 0.723 0.756 0.745 0.755 0.767 0.774 0.792 0.802 Noise−场景2 4 0.717 0.723 0.712 0.746 0.742 0.748 0.789 0.768 0.799 8 0.623 0.633 0.625 0.645 0.665 0.672 0.674 0.682 0.701
下载: 导出CSV
表 6 不同网络的复杂度和运行速度对比
Table 6. Comparison of complexity and running speed of different networks
网络 参数量/106个 每秒浮点运算次数/109 每张图像耗时/ms SSIM EDSR 43.1 1 212 1 520 0.846 RCAN 16.0 352 960 0.855 DBPN 10.4 325 756 0.823 CSNLN 7.2 110 655 0.845 NLSN 6.7 105 602 0.865 EFDN 1.2 87 450 0.845 Ours 4.8 101 465 0.866
下载: 导出CSV
-
[1] 程德强,陈杰,寇旗旗,等. 融合层次特征和注意力机制的轻量化矿井图像超分辨率重建方法[J]. 仪器仪表学报,2022,43(8):73-84.CHENG Deqiang,CHEN Jie,KOU Qiqi,et al. Lightweight super-resolution reconstruction method based on hierarchical features fusion and attention mechanism for mine image[J]. Chinese Journal of Scientific Instrument,2022,43(8):73-84. [2] ZHUANG Cheng,LI Minqi,ZHANG Kaibing,et al. Multi-level landmark-guided deep network for face super-resolution[J]. Neural Networks,2022,152:276-286. doi: 10.1016/j.neunet.2022.04.026 [3] TAO Hongjiu,TANG Xinjian,LIU Jian,et al. Super resolution remote sensing image processing algorithm based on wavelet transform and interpolation[C]. Conference on Image Processing and Pattern Recognition in Remote Sensing,Barcelona,2003:259-263. [4] YANG Qi,ZHANG Yanzhu,ZHAO Tiebiao. Example-based image super-resolution via blur kernel estimation and variational reconstruction[J]. Pattern Recognition Letters,2019,117:83-89. doi: 10.1016/j.patrec.2018.12.008 [5] KANG Xuejing,DUAN Peiqi,XU Ruyu. Single image super-resolution based on mapping-vector clustering and nonlinear pixel-reconstruction[J]. Signal Processing:Image Communication,2022,100. DOI: 10.1016/j.image.2021.116501. [6] 高青青,赵建伟,周正华. 基于递归多尺度卷积网络的图像超分辨率重建[J]. 模式识别与人工智能,2020,33(11):972-980.GAO Qingqing,ZHAO Jianwei,ZHOU Zhenghua. Image super-resolution reconstruction based on recursive multi-scale convolutional networks[J]. Pattern Recognition and Artificial Intelligence,2020,33(11):972-980. [7] YANG Shuyuan,LIU Zhizhou,WANG Min,et al. Multitask dictionary learning and sparse representation based single-image super-resolution reconstruction[J]. Neurocomputing,2011,74(17):3193-3203. doi: 10.1016/j.neucom.2011.04.014 [8] DONG Chao,LOY C C,HE Kaiming,et al. Learning a deep convolutional network for image super-resolution[C]. 13th European Conference on Computer Vision,Zurich,2014:184-199. [9] DONG Chao,LOY C C,TANG Xiao'ou,et al. Accelerating the super-resolution convolutional neural network[C]. 14th European Conference on Computer Vision,Amsterdam,2016:391-407. [10] SHI Wenzhe,CABALLERO J,HUSZAR F,et al. Real-Time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]. IEEE Conference on Computer Vision and Pattern Recognition,Las Vegas,2016:1874-1883. [11] HE Kaiming,ZHANG Xiangyu,REN Shaoqing,et al. Deep residual learning for image recognition[C]. IEEE Conference on Computer Vision and Pattern Recognition,Las Vegas,2016:770-778. [12] KIM J,LEE J K,LEE K M. Accurate image super-resolution using very deep convolutional networks[J]. IEEE Conference on Computer Vision and Pattern Recognition,Las Vegas,2016:1646-1654. [13] LIM B,SON S,KIM H,et al. Enhanced deep residual networks for single image super-resolution[C]. 30th IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops,Honolulu,2017:1132-1140. [14] SEGU M,TONIONI A,TOMBARI F. Batch normalization embeddings for deep domain generalization[J]. Pattern Recognition,2023,135. DOI: 10.1016/j.patcog.2022.109115. [15] ZHANG Yulun,LI Kunpeng,LI Kai,et al. Image super-resolution using very deep residual channel attention networks[C]. 15th European Conference on Computer Vision,Munich,2018:294-310. [16] ZHANG Yulun,TIAN Yapeng,KONG Yu,et al. Residual dense network for image super-resolution[C]. IEEE/CVF Conference on Computer Vision and Pattern Recognition,Salt Lake City,2018:2472-2481. [17] CHEN Liangliang,KOU Qiqi,CHENG Deqiang,et al. Content-guided deep residual network for single image super-resolution[J]. Optik,2020,202. DOI: 10.1016/j.ijleo.2019.163678. [18] 程德强,郭昕,陈亮亮,等. 多通道递归残差网络的图像超分辨率重建[J]. 中国图象图形学报,2021,26(3):605-618. doi: 10.11834/jig.200108CHENG Deqiang,GUO Xin,CHEN Liangliang,et al. Image super-resolution reconstruction from multi-channel recursive residual networks[J]. Journal of Image and Graphics,2021,26(3):605-618. doi: 10.11834/jig.200108 [19] WOO S,PARK J,LEE J,et al. CBAM:convolutional block attention module[C]. 15th European Conference on Computer Vision,Munich,2018:3-19. [20] LIU Jie,TANG Jie,WU Gangshan. Residual feature distillation network for lightweight image super-resolution[C]. European Conference on Computer Vision Workshops,Glasgow,2020:41-55. [21] BEVILACQUA M,ROUMY A,GUILLEMOT C,et al. Low-complexity single-Image super-resolution based on nonnegative neighbor embedding[C]. 23rd British Machine Vision Conference,Surrey,2012. DOI: 10.5244/C.26.135. [22] ROMANO Y,PROTTER M,ELAD M. Single image interpolation via adaptive nonlocal sparsity-based modeling[J]. IEEE Transactions on Image Processing,2014,23(7):3085-3098. doi: 10.1109/TIP.2014.2325774 [23] LIU Yun,CHENG Mingming,HU Xiaowen,et al. Richer convolutional features for edge detection[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence,2019,41(8):1939-1946. doi: 10.1109/TPAMI.2018.2878849 [24] HUANG Jiabin,SINGH A,AHUJA N. Single image super-resolution from transformed self-exemplars[C]. IEEE Conference on Computer Vision and Pattern Recognition,Boston,2015:5197-5206. [25] ZHANG Kai,ZUO Wangmeng,CHEN Yunjin,et al. Beyond a Gaussian denoiser:residual learning of deep CNN for image denoising[J]. IEEE Transactions on Image Processing,2017,26(7):3142-3155. doi: 10.1109/TIP.2017.2662206 [26] ZHANG Kai,ZUO Wangmeng,ZHANG Lei. FFDNet:toward a fast and flexible solution for CNN-based image denoising[J]. IEEE Transactions on Image Processing,2018,27(9):4608-4622. doi: 10.1109/TIP.2018.2839891 [27] HARIS M,SHAKHNAROVICH G,UKITA N. Deep back-projection networks for super-resolution[C]. IEEE/CVF Conference on Computer Vision and Pattern Recognition,Salt Lake City,2018:1664-1673. [28] MEI Yiqun,FAN Yuchen,ZHOU Yuqian,et al. Image super-resolution with cross-scale non-local attention and exhaustive self-exemplars mining[C]. IEEE/CVF Conference on Computer Vision and Pattern Recognition,Seattle,2020:5689-5698. [29] MEI Yiqun,FAN Yuchen,ZHOU Yuqian. Image super-resolution with non-local sparse attention[C]. IEEE/CVF Conference on Computer Vision and Pattern Recognition,Nashville,2021:3516-3525. [30] WANG Yan. Edge-enhanced feature distillation network for efficient super-resolution[C]. IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops,New Orleans,2022:776-784. -
点击查看大图
图(8) / 表(6)
计量
- 文章访问数: 54
- HTML全文浏览量: 18
- PDF下载量: 4
- 被引次数: 0
