Volume 49 Issue 9
Sep.  2023
Turn off MathJax
Article Contents
Article Navigation >  Journal of Mine Automation  > 2023  >  49(9): 73-83
ZHOU Libing, CHEN Xiaojing, JIA Wenqi, et al. Visible and infrared image fusion algorithm for underground personnel detection[J]. Journal of Mine Automation，2023，49（9）：73-83.  doi: 10.13272/j.issn.1671-251x.2023070025
Citation: ZHOU Libing, CHEN Xiaojing, JIA Wenqi, et al. Visible and infrared image fusion algorithm for underground personnel detection[J]. Journal of Mine Automation，2023，49（9）：73-83.  doi: 10.13272/j.issn.1671-251x.2023070025
shu

Visible and infrared image fusion algorithm for underground personnel detection

doi: 10.13272/j.issn.1671-251x.2023070025
  • 1.

    CCTEG Changzhou Research Institute, Changzhou 213015, China

  • 2.

    Tiandi(Changzhou) Automation Co., Ltd., Changzhou 213015, China

  • 3.

    School of Mechanical and Electrical Engineering, Nanjing University of Aeronautics and Astronautics, Nanjing 210016, China

  • Received Date: 2023-07-07
  • Rev Recd Date: 2023-09-16
    • Available Online: 2023-09-28
    Abstract
  • The working environment and lighting conditions of mining intelligent vehicles are complex. When detecting underground personnel, infrared reflection information and detailed texture information can be fused into visible light images by fusing visible and infrared images to improve the target detection effect. Traditional visible and infrared image fusion methods can lead to blurring of image edges and textures as the number of decomposition layers increases, and the fusion time also increases. At present, deep learning based fusion methods for visible and infrared images are difficult to balance the features in visible and infrared images, resulting in blurred detail information in the fused images. In order to solve the above problems, the image fusion algorithm based on multiple attention modules (IFAM) is proposed. Firstly, convolutional neural networks are used to extract image features from visible and infrared images. Secondly, the extracted features are cross fused using spatial attention and channel attention modules. The fusion weights of the output features of the two attention modules are calculated using the gradient information in the features. The output features of the two attention modules are fused based on the weights. Finally, the image features are restored through deconvolution transformation to obtain the final fused image. The fusion results on the RoadScene dataset and TNO dataset indicate that the IFAM fused image contains both background texture information from visible light images and personnel contour feature information from infrared images. The fusion results on the underground dataset indicate that in low lighting environments, infrared images can compensate for the shortcomings of visible light and are not affected by other light sources in the environment. In low lighting conditions, the personnel contour in the fused image is still obvious. The comparative analysis results show that the information entropy (EN), standard deviation (SD), gradient fusion metric (QAB/F), visual information fidelity for fusion (VIFF), and the union structural similarity index measure (SSIMu) of the image after IFAM fusion are 4.901 3, 88.521 4, 0.169 3, 1.413 5, and 0.806 2, respectively. The overall performance is superior to similar algorithms such as LLF-IOI and NDM.

     

    • FullText(HTML)
  • loading
    • References(40)
  • [1]
    周李兵. 煤矿井下无轨胶轮车无人驾驶系统研究[J]. 工矿自动化,2022,48(6):36-48.

    ZHOU Libing. Research on unmanned driving system of underground trackless rubber-tyred vehicle in coal mine[J]. Journal of Mine Automation,2022,48(6):36-48.
    [2]
    MA Jiayi,MA Yong,LI Chang. Infrared and visible image fusion methods and applications:a survey[J]. Information Fusion,2019,45:153-178. doi: 10.1016/j.inffus.2018.02.004
    [3]
    WANG Zhishe,XU Jiawei,JIANG Xiaolin,et al. Infrared and visible image fusion via hybrid decomposition of NSCT and morphological sequential toggle operator[J]. Optik,2020,201. DOI: 10.1016/j.ijleo.2019.163497.
    [4]
    MALLAT S G. A theory for multiresolution signal decomposition:the wavelet representation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence,1989,11(7):674-693. doi: 10.1109/34.192463
    [5]
    肖杨,张凌雪,付玲. Contourlet变换及方向滤波器组设计相关问题[J]. 科技风,2009(23):221. doi: 10.3969/j.issn.1671-7341.2009.23.199

    XIAO Yang,ZHANG Lingxue,FU Ling. Problems related to Contourlet transformation and directional filter bank design[J]. Technology Wind,2009(23):221. doi: 10.3969/j.issn.1671-7341.2009.23.199
    [6]
    DA CUNHA A L,ZHOU Jianping,DO M N. The nonsubsampled contourlet transform:theory,design,and applications[J]. IEEE Transactions on Image Processing,2006,15(10):3089-3101. doi: 10.1109/TIP.2006.877507
    [7]
    罗娟,王立平. 基于非下采样Contourlet变换耦合特征选择机制的可见光与红外图像融合算法[J]. 电子测量与仪器学报,2021,35(7):163-169.

    LUO Juan,WANG Liping. Infrared and visible image fusion algorithm based on nonsubsampled contourlet transform coupled with feature selection mechanism[J]. Journal of Electronic Measurement and Instrumentation,2021,35(7):163-169.
    [8]
    GAO Ce,QI Donghao,ZHANG Yanchao,et al. Infrared and visible image fusion method based on ResNet in a nonsubsampled contourlet transform domain[J]. IEEE Access,2021,9:91883-91895. doi: 10.1109/ACCESS.2021.3086096
    [9]
    詹玲超,刘瑾. 基于非下采样Contourlet变换红外和可见光图像的融合方法[J]. 数字技术与应用,2016(10):45-46.

    ZHAN Lingchao,LIU Jin. Infrared and visible image fusion method based on nonsubsampled contourlet transform[J]. Digital Technology and Application,2016(10):45-46.
    [10]
    FARBMAN Z,FATTAL R,LISCHINSKI D,et al. Edge-preserving decompositions for multi-scale tone and detail manipulation[J]. ACM Transactions on Graphics,2008,27(3):1-10.
    [11]
    ZHANG Yu,LIU Yu,SUN Peng,et al. IFCNN:a general image fusion framework based on convolutional neural network[J]. Information Fusion,2020,54:99-118. doi: 10.1016/j.inffus.2019.07.011
    [12]
    LI Hui,WU Xiaojun,KITTLER J. RFN-Nest:an end-to-end residual fusion network for infrared and visible images[J]. Information Fusion,2021,73:72-86. doi: 10.1016/j.inffus.2021.02.023
    [13]
    WANG Zhishe,WANG Junyao,WU Yuanyuan,et al. UNFusion:a unified multi-scale densely connected network for infrared and visible image fusion[J]. IEEE Transactions on Circuits and Systems for Video Technology,2022,32(6):3360-3374. doi: 10.1109/TCSVT.2021.3109895
    [14]
    LIU Yu,CHEN Xun,CHENG Juan,et al. Infrared and visible image fusion with convolutional neural networks[J]. International Journal of Wavelets,Multiresolution and Information Processing,2018,16(3). DOI: 10.1142/S0219691318500182.
    [15]
    LI Hui,WU Xiaojun. DenseFuse:a fusion approach to infrared and visible images[J]. IEEE Transactions on Image Processing,2018,28(5):2614-2623.
    [16]
    罗迪,王从庆,周勇军. 一种基于生成对抗网络与注意力机制的可见光和红外图像融合方法[J]. 红外技术,2021,43(6):566-574.

    LUO Di,WANG Congqing,ZHOU Yongjun. A visible and infrared image fusion method based on generative adversarial networks and attention mechanism[J]. Infrared Technology,2021,43(6):566-574.
    [17]
    王志社,邵文禹,杨风暴,等. 红外与可见光图像交互注意力生成对抗融合方法[J]. 光子学报,2022,51(4):318-328.

    WANG Zhishe,SHAO Wenyu,YANG Fengbao,et al. Infrared and visible image fusion method via interactive attention based generative adversarial network[J]. Acta Photonica Sinica,2022,51(4):318-328.
    [18]
    MA Jiayi,ZHANG Hao,SHAO Zhenfeng,et al. GANMcC:a generative adversarial network with multiclassification constraints for infrared and visible image fusion[J]. IEEE Transactions on Instrumentation and Measurement,2021,70:1-14.
    [19]
    RAO Dongyu,WU Xiaojun,XU Tianyang. TGFuse:an infrared and visible image fusion approach based on transformer and generative adversarial network[EB/OL]. [2023-06-20]. https://arxiv.org/abs/2201.10147.
    [20]
    LI Jing,ZHU Jianming,LI Chang,et al. CGTF:convolution-guided transformer for infrared and visible image fusion[J]. IEEE Transactions on Instrumentation and Measurement,2022,71:1-14.
    [21]
    JIANG Yifan,GONG Xinyu,LIU Ding,et al. EnlightenGAN:deep light enhancement without paired supervision[J]. IEEE Transactions on Image Processing,2021,30:2340-2349. doi: 10.1109/TIP.2021.3051462
    [22]
    秦沛霖,张传伟,周李兵,等. 煤矿井下无人驾驶无轨胶轮车目标3D检测研究[J]. 工矿自动化,2022,48(2):35-41.

    QIN Peilin,ZHANG Chuanwei,ZHOU Libing,et al. Research on 3D target detection of unmanned trackless rubber-tyred vehicle in coal mine[J]. Industry and Mine Automation,2022,48(2):35-41.
    [23]
    WOO S,PARK J,LEE J-Y,et al. CBAM:convolutional block attention module[C]. 15th European Conference on Computer Vision,Munich,2018:3-19.
    [24]
    梁美彦,张倩楠,任竹云,等. 基于注意力机制的结肠癌病理学图像识别研究[J]. 测试技术学报,2022,36(2):93-100.

    LIANG Meiyan,ZHANG Qiannan,REN Zhuyun,et al. Research on identification of colon pathology image based on attention mechanism[J]. Journal of Test and Measurement Technology,2022,36(2):93-100.
    [25]
    牛悦,王安南,吴胜昔. 基于注意力机制和级联金字塔网络的姿态估计[J/OL]. 华东理工大学学报(自然科学版):1-11[2023-06-20]. DOI: 10.14135/j.cnki.1006-3080.20220715003.

    NIU Yue,WANG Annan,WU Shengxi. Pose estimation based on attention module and CPN[J/OL]. Journal of East China University of Science and Technology(Natural Science Edition):1-11[2023-06-20]. DOI: 10.14135/j.cnki.1006-3080.20220715003.
    [26]
    ZAGORUYKO S,KOMODAKIS N. Paying more attention to attention:improving the performance of convolutional neural networks via attention transfer[EB/OL]. [2023-06-20]. https://arxiv.org/abs/1612.03928v2.
    [27]
    陈舞,孙军梅,李秀梅. 融合多尺度残差和注意力机制的特发性肺纤维化进展预测[J]. 中国图象图形学报,2022,27(3):812-826.

    CHEN Wu,SUN Junmei,LI Xiumei. Multi-scale residual and attention mechanism fusion based prediction for the progression of idiopathic pulmonary fibrosis[J]. Journal of Image and Graphics,2022,27(3):812-826.
    [28]
    李国梁,向文豪,张顺利,等. 基于残差网络和注意力机制的红外与可见光图像融合算法[J]. 无人系统技术,2022,5(2):9-21.

    LI Guoliang,XIANG Wenhao,ZHANG Shunli,et al. Infrared and visible image fusion algorithm based on residual network and attention mechanism[J]. Unmanned Systems Technology,2022,5(2):9-21.
    [29]
    XU Han,MA Jiayi,JIANG Junjun,et al. U2Fusion:a unified unsupervised image fusion network[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence,2020,44(1):502-518.
    [30]
    LIN T-Y,MAIRE M,BELONGIE S,et al. Microsoft COCO:common objects incontext[C]. 13th European Conference on Computer Vision,Zurich,2014:740-755.
    [31]
    ROBERTS J W,VAN AARDT J A,AHMED F B. Assessment of image fusion procedures using entropy,image quality,and multispectral classification[J]. Journal of Applied Remote Sensing,2008,2(1). DOI: 10.1117/1.2945910.
    [32]
    XYDEAS C,PETROVIC V. Objective image fusion performance measure[J]. Electronics Letters,2000,36(4):308-309. doi: 10.1049/el:20000267
    [33]
    HAN Yu,CAI Yunze,CAO Yin,et al. A new image fusion performance metric based on visual information fidelity[J]. Information Fusion,2013,14(2):127-135. doi: 10.1016/j.inffus.2011.08.002
    [34]
    吴明辉. 联合特征提取方法的图像融合技术研究[D]. 武汉:武汉大学,2021.

    WU Minghui. Research on image fusion based on joint feature extraction[D]. Wuhan:Wuhan University,2021.
    [35]
    XU Han. RoadScene:a new dataset of aligned infrared and visible images[DB/OL]. [2023-06-07]. https://github.com/hanna-xu/RoadScene.
    [36]
    TOET A. TNO image fusion datase[DB/OL]. [2023-06-26]. https://figshare.com/articles/dataset/TNO_Image_Fusion_Dataset/1008029.
    [37]
    DU Jiao,LI Weisheng,XIAO Bin. Anatomical-functional image fusion by information of interest in local Laplacian filtering domain[J]. IEEE Transactions on Image Processing,2017,26(12):5855-5866. doi: 10.1109/TIP.2017.2745202
    [38]
    LIU Zhe,SONG Yuqing,SHENG V S,et al. MRI and PET image fusion using the nonparametric density model and the theory of variable-weight[J]. Computer Methods and Programs in Biomedicine,2019,175:73-82. doi: 10.1016/j.cmpb.2019.04.010
    [39]
    YIN Ming,LIU Xiaoning,LIU Yu,et al. Medical image fusion with parameter-adaptive pulse coupled neural network in nonsubsampled shearlet transform domain[J]. IEEE Transactions on Instrumentation and Measurement,2018,68(1):49-64.
    [40]
    KANG Jiayin,LU Wu,ZHANG Wenjuan. Fusion of brain PET and MRI images using tissue-aware conditional generative adversarial network with joint loss[J]. IEEE Access,2020,8:6368-6378. doi: 10.1109/ACCESS.2019.2963741
    • Relative Articles
    • Supplements(0)
    • Cited By
  • 加载中

Catalog

    通讯作者: 陈斌, bchen63@163.com
    • 1. 

      沈阳化工大学材料科学与工程学院 沈阳 110142

    1. 本站搜索
    2. 百度学术搜索
    3. 万方数据库搜索
    4. CNKI搜索

    Figures(12)  / Tables(4)

    Get Citation
    PDF
    XML

    Article Metrics

    Article views (1183) PDF downloads(56) Cited by()
    Proportional views
    Related

    苏ICP备 05016349号-2

    Supported by: Beijing Renhe Information Technology Co. Ltd.  Visits：    

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return