Volume 50 Issue 1
Jan.  2024
Turn off MathJax
Article Contents
Abstract
References
Related Articles
CAO Xiangang, LI Hu, WANG Peng, et al. A coal foreign object detection method based on cross modal attention fusion[J]. Journal of Mine Automation，2024，50（1）：57-65. DOI: 10.13272/j.issn.1671-251x.2023110035
Citation: CAO Xiangang, LI Hu, WANG Peng, et al. A coal foreign object detection method based on cross modal attention fusion[J]. Journal of Mine Automation，2024，50（1）：57-65. DOI: 10.13272/j.issn.1671-251x.2023110035
shu

A coal foreign object detection method based on cross modal attention fusion

  • 1.

    School of Mechanical Engineering, Xi'an University of Science and Technology, Xi'an 710054, China

  • 2.

    Shaanxi Provincial Key Laboratory of Intelligent Testing of Mine Mechanical and Electrical Equipment, Xi'an 710054, China

More Information
  • Received Date: November 10, 2023
  • Revised Date: January 20, 2024
  • Available Online: January 30, 2024
    Graphical Abstract
    • Abstract
  • The RGB image of coal foreign objects lacks target space and edge information, the color and texture between the object to be detected and the background are similar, the contrast is low, and there are overlapping and occlusion phenomena among the objects to be detected, resulting in insufficient feature extraction of coal foreign objects, and the existing foreign object detection methods are difficult to achieve ideal results. In order to solve the above problems, a coal foreign object detection method based on cross modal attention fusion is proposed. By introducing Depth images to construct a dual feature pyramid network (DFPN) for RGB images and Depth images, a shallow feature extraction strategy is adopted to extract low-level features of Depth images. Basic features such as deep edges and deep textures are used to assist deep features of RGB images, effectively obtaining complementary information between the two features. It thereby enriches the spatial and edge information of foreign object features and improves detection precision. A cross modal attention fusion module (CAFM) based on coordinate attention and improved spatial attention is constructed to synergistically optimize and fuse RGB features and Depth features. It enhances the network's attention to the visible parts of occluded foreign objects in the feature map, and improves the precision of occluded foreign object detection. Finally, regional convolutional neural network (R-CNN) is used to output the classification, regression, and segmentation results of coal foreign objects. The experimental results show that in terms of detection precision, the average segmentation precision AP of the proposed method is 3.9% higher than the better Mask transformer in the two-stage model. In terms of detection efficiency, the proposed method has a single frame detection time of 110.5 ms, which can meet the real-time requirements of foreign object detection. The coal foreign object detection method based on cross modal attention fusion can assist color, shape, and texture features with spatial features. It accurately recognizes the differences between coal foreign objects and between coal foreign objects and conveyor belts, effectively improves the detection precision of complex feature foreign objects. It reduces false alarms and missed detections, and achieves precise detection and pixel level segmentation of coal foreign objects under complex features.
    • FullText(HTML)
    • References (25)
  • [1]
    刘峰,郭林峰,赵路正. 双碳背景下煤炭安全区间与绿色低碳技术路径[J]. 煤炭学报,2022,47(1):1-15.

    LIU Feng,GUO Linfeng,ZHAO Luzheng. Research on coal safety range and green low-carbon technology path under the dual-carbon background[J]. Journal of China Coal Society,2022,47(1):1-15.
    [2]
    刘峰,曹文君,张建明,等. 我国煤炭工业科技创新进展及“十四五”发展方向[J]. 煤炭学报,2021,46(1):1-15.

    LIU Feng,CAO Wenjun,ZHANG Jianming,et al. Current technological innovation and development direction of the 14(th) Five-Year Plan period in China coal industry[J]. Journal of China Coal Society,2021,46(1):1-15.
    [3]
    曹现刚,刘思颖,王鹏,等. 面向煤矸分拣机器人的煤矸识别定位系统研究[J]. 煤炭科学技术,2022,50(1):237-246. DOI: 10.3969/j.issn.0253-2336.2022.1.mtkxjs202201024

    CAO Xiangang,LIU Siying,WANG Peng,et al. Research on coal gangue identification and positioning system based on coal-gangue sorting robot[J]. Coal Science and Technology,2022,50(1):237-246. DOI: 10.3969/j.issn.0253-2336.2022.1.mtkxjs202201024
    [4]
    LI Man,DUAN Yong,HE Xianli,et al. Image positioning and identification method and system for coal and gangue sorting robot[J]. International Journal of Coal Preparation and Utilization,2022,42(4/6):1759-1777.
    [5]
    赵跃民,张亚东,周恩会,等. 清洁高效干法选煤研究进展与展望[J]. 中国矿业大学学报,2022,51(3):607-616. DOI: 10.3969/j.issn.1000-1964.2022.3.zgkydxxb202203023

    ZHAO Yuemin,ZHANG Yadong,ZHOU Enhui,et al. Research progress and prospect of clean and efficient dry coal separation[J]. Journal of China University of Mining & Technology,2022,51(3):607-616. DOI: 10.3969/j.issn.1000-1964.2022.3.zgkydxxb202203023
    [6]
    葛世荣,郝尚清,张世洪,等. 我国智能化采煤技术现状及待突破关键技术[J]. 煤炭科学技术,2020,48(7):28-46.

    GE Shirong,HAO Shangqing,ZHANG Shihong,et al. Status of intelligent coal mining technology and potential key technologies in China[J]. Coal Science and Technology,2020,48(7):28-46.
    [7]
    WANG Yuanbin,WANG Yujing,DANG Langfei. Video detection of foreign objects on the surface of belt conveyor underground coal mine based on improved SSD[J]. Journal of Ambient Intelligence and Humanized Computing,2023,14:5507-5516. DOI: 10.1007/s12652-020-02495-w
    [8]
    郝帅,张旭,马旭,等. 基于CBAM−YOLOv5的煤矿输送带异物检测[J]. 煤炭学报,2022,47(11):4147-4156.

    HAO Shuai,ZHANG Xu,MA Xu,et al. Foreign object detection in coal mine conveyor belt based on CBAM-YOLOv5[J]. Journal of China Coal Society,2022,47(11):4147-4156.
    [9]
    ZHANG Kanghui,WANG Weidong,LYU Ziqi,et al. Computer vision detection of foreign objects in coal processing using attention CNN[J]. Engineering Applications of Artificial Intelligence,2021,102. DOI: 10.1016/j.engappai.2021.104242.
    [10]
    任志玲,朱彦存. 改进CenterNet算法的煤矿皮带运输异物识别研究[J]. 控制工程,2023,30(4):703-711.

    REN Zhiling,ZHU Yancun. Research on foreign object detection of coal mine belt transportation with improved CenterNet algorithm[J]. Control Engineering of China,2023,30(4):703-711.
    [11]
    程德强,徐进洋,寇旗旗,等. 融合残差信息轻量级网络的运煤皮带异物分类[J]. 煤炭学报,2022,47(3):1361-1369.

    CHENG Deqiang,XU Jinyang,KOU Qiqi,et al. Lightweight network based on residual information for foreign body classification on coal conveyor belt[J]. Journal of China Coal Society,2022,47(3):1361-1369.
    [12]
    SONG Xinhang,JIANG Shuqiang,HERRANZ L,et al. Learning effective RGB-D representations for scene recognition[J]. IEEE Transactions on Image Processing,2019,28(2):980-993. DOI: 10.1109/TIP.2018.2872629
    [13]
    BALTRUŠAITIS T,AHUJA C,MORENCY L-P. Multimodal machine learning:a survey and taxonomy[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence,2019,41(2):423-443. DOI: 10.1109/TPAMI.2018.2798607
    [14]
    GAO Mingliang,JIANG Jun,ZOU Guofeng,et al. RGB-D-based object recognition using multimodal convolutional neural networks:a survey[J]. IEEE Access,2019,7:43110-43136. DOI: 10.1109/ACCESS.2019.2907071
    [15]
    LIN T-Y,DOLLAR P,GIRSHICK R B,et al. Feature pyramid networks for object detection[C]. IEEE Conference on Computer Vision and Pattern Recognition,Honolulu,2017:936-944.
    [16]
    HE Kaiming,ZHANG Xiangyu,REN Shaoqing,et al. Deep residual learning for image recognition[C]. IEEE Conference on Computer Vision and Pattern Recognition,Las Vegas,2016:770-778.
    [17]
    MA Jiayi,MA Yong,LI Chang. Infrared and visible image fusion methods and applications:a survey[J]. Information Fusion,2019,45:153-178. DOI: 10.1016/j.inffus.2018.02.004
    [18]
    RAMACHANDRAM D,TAYLOR G W. Deep multimodal learning:a survey on recent advances and trends[J]. IEEE Signal Processing Magazine,2017,34(6):96-108. DOI: 10.1109/MSP.2017.2738401
    [19]
    HOU Qibin,ZHOU Daquan,FENG Jiashi. Coordinate attention for efficient mobile network design[C]. IEEE/CVF Conference on Computer Vision and Pattern Recognition,Nashville,2021:13713-13722.
    [20]
    REN Shaoqing,HE Kaiming,GIRSHICK R,et al. Faster R-CNN:towards real-time object detection with region proposal networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence,2017,39(6):1137-1149. DOI: 10.1109/TPAMI.2016.2577031
    [21]
    WANG Xinlong,ZHANG Rufeng,KONG Tao,et al. SOLOv2:dynamic and fast instance segmentation[EB/OL]. [2023-09-12]. https://arxiv.org/abs/2003.10152.
    [22]
    CHEN Hao,SUN Kunyang,TIAN Zhi,et al. BlendMask:top-down meets bottom-up for instance segmentation[C]. IEEE/CVF Conference on Computer Vision and Pattern Recognition,Seattle,2020:8570-8578.
    [23]
    HE Kaiming,GKIOXARI G,DOLLÁR P,et al. Mask R-CNN[J]. IEEE Transactions on Pattern Analysis & Machine Intelligence,2017,42(2):2980-2988.
    [24]
    HUANG Zhaojin,HUANG Lichao,GONG Yongchao,et al. Mask scoring R-CNN[C]. IEEE Conference on Computer Vision and Pattern Recognition,Washington,2019:6409-6418.
    [25]
    KE Lei,DANELLJAN M,LI Xia,et al. Mask transfiner for high-quality instance segmentation[C]. IEEE/CVF Conference on Computer Vision and Pattern Recognition,New Orleans,2022:4402-4411.
    • Related Articles

  • Related Articles

    [1]TAN Donggui, YUAN Yiping, FAN Panpan. Health status evaluation of CNN-GRU mine motor based on adaptive multi-scale attention mechanism[J]. Journal of Mine Automation, 2024, 50(2): 138-146. DOI: 10.13272/j.issn.1671-251x.2023110024
    [2]CHENG Deqiang, ZHENG Lijuan, LIU Jingjing, KOU Qiqi, JIANG He. Quantitative analysis of coal particle size based on bi-level routing attention mechanism[J]. Journal of Mine Automation, 2024, 50(2): 9-17. DOI: 10.13272/j.issn.1671-251x.2023100002
    [3]CAO Zhengyuan, JIANG Wei, FANG Chenghui. Intelligent detection method for coal flow foreign objects based on dual attention generative adversarial network[J]. Journal of Mine Automation, 2023, 49(12): 56-62. DOI: 10.13272/j.issn.1671-251x.18094
    [4]LI Zhongfei, FENG Shiyong, GUO Jun, ZHANG Yunhe, XU Feixiang. Lightweight safety helmet wearing detection fusing coordinate attention and multiscale feature[J]. Journal of Mine Automation, 2023, 49(11): 151-159. DOI: 10.13272/j.issn.1671-251x.2023080123
    [5]JI Liang. Coal mine image instance segmentation method based on improved SOLOv2[J]. Journal of Mine Automation, 2023, 49(11): 115-120. DOI: 10.13272/j.issn.1671-251x.2023030017
    [6]ZHU Fuwen, HOU Zhihui, LI Mingzhen. Lightweight multi-scale cross channel attention coal flow detection network[J]. Journal of Mine Automation, 2023, 49(8): 100-105. DOI: 10.13272/j.issn.1671-251x.2023030045
    [7]DU Jingyi, DANG Mengke, QIAO Lei, WEI Meiting, HAO Le. Drill pipe counting method based on improved spatial-temporal graph convolution neural network[J]. Journal of Mine Automation, 2023, 49(1): 90-98. DOI: 10.13272/j.issn.1671-251x.2022030098
    [8]RAO Tianrong, PAN Tao, XU Huijun. Unsafe action recognition in underground coal mine based on cross-attention mechanism[J]. Journal of Mine Automation, 2022, 48(10): 48-54. DOI: 10.13272/j.issn.1671-251x.17949
    [9]YE Ou, DOU Xiaoyi, FU Yan, DENG Jun. Coal block detection method integrating lightweight network and dual attention mechanism[J]. Journal of Mine Automation, 2021, 47(12): 75-80. DOI: 10.13272/j.issn.1671-251x.2021030075
    [10]ZHANG Shan, WU Jian-ping, ZHENG Hong-bo, ZHU Yu-peng. Research of Drill Information Management[J]. Journal of Mine Automation, 2009, 35(7): 30-33.

  • Cited by

    Periodical cited type(5)

    1. 李润泽, 郭星歌, 杨发展, 赵培培, 谢国龙. 基于改进YOLOv8s的矿用输送带异物检测方法. 工矿自动化. 2025(06) 本站查看
    2. 曹现刚, 刘依哲, 吴旭东, 王鹏, 张烨. 煤矸分拣机器人的煤矸动态称重方法. 工矿自动化. 2025(06) 本站查看
    3. 柳小波，范立鹏，秦丽杰，王连成，张兴帆. 机器视觉技术在矿山行业的应用现状与展望. 有色金属(矿山部分). 2025(02): 1-15 .
    4. 董海峰，林志豪，李宗祥. 基于改进RegNet的煤矿传送带异物识别方法研究. 信息技术与信息化. 2025(04): 188-191 .
    5. 徐慈强，贾运红，田原. 基于MES-YOLOv5s的综采工作面大块煤检测算法. 工矿自动化. 2024(03): 42-47+141 . 本站查看

    Other cited types(1)

Catalog

    Get Citation
    PDF
    XML

    Article Metrics

    Article views (719) PDF downloads (75) Cited by(6)
    Related

    苏ICP备 05016349号-2

    Supported by: Beijing Renhe Information Technology Co., Ltd. Visits：1336386    Baidu Analytics

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return