Volume 49 Issue 11
Nov.  2023
Turn off MathJax
Article Contents
Abstract
References
Related Articles
WANG Yu, YU Chunhua, CHEN Xiaoqing, et al. Recognition of unsafe behaviors of underground personnel based on multi modal feature fusion[J]. Journal of Mine Automation，2023，49（11）：138-144. DOI: 10.13272/j.issn.1671-251x.2023070055
Citation: WANG Yu, YU Chunhua, CHEN Xiaoqing, et al. Recognition of unsafe behaviors of underground personnel based on multi modal feature fusion[J]. Journal of Mine Automation，2023，49（11）：138-144. DOI: 10.13272/j.issn.1671-251x.2023070055
shu

Recognition of unsafe behaviors of underground personnel based on multi modal feature fusion

  • 1.

    School of Mining Engineering, University of Science and Technology Liaoning, Anshan 114051, China

  • 2.

    Lingang Group Beipiao Baoguo Iron Mining Co., Ltd., Chaoyang 122102, China

More Information
  • Received Date: July 15, 2023
  • Revised Date: October 26, 2023
  • Available Online: November 26, 2023
    Graphical Abstract
    • Abstract
  • The use of artificial intelligence technology for real-time recognition of underground personnel's behavior is of great significance for ensuring safe production in mines. The RGB modal based behavior recognition methods is susceptible to video image background noise. The bone modal based behavior recognition methods lacks visual feature information of humans and objects. In order to solve the above problems, a multi modal feature fusion based underground personnel unsafe behavior recognition method is proposed by combining the two methods. The SlowOnly network is used to extract RGB modal features. The YOLOX and Lite HRNet networks are used to obtain bone modal data. The PoseC3D network is used to extract bone modal features. The early and late fusion of RGB modal features and bone modal features are performed. The recognition results for unsafe behavior of underground personnel are finally obtained. The experimental results on the NTU60 RGB+D public dataset under the X-Sub standard show the following points. In the behavior recognition model based on a single bone modal, PoseC3D has a higher recognition accuracy than GCN (graph convolutional network) methods, reaching 93.1%. The behavior recognition model based on multimodal feature fusion has a higher recognition accuracy than the recognition model based on a single bone modal, reaching 95.4%. The experimental results on a self-made underground unsafe behavior dataset show that the behavior recognition model based on multimodal feature fusion still has the highest recognition accuracy in complex underground environments, reaching 93.3%. It can accurately recognize similar unsafe behaviors and multiple unsafe behaviors.
    • FullText(HTML)
    • References (21)
  • [1]
    吴爱祥,王勇,张敏哲,等. 金属矿山地下开采关键技术新进展与展望[J]. 金属矿山,2021(1):1-13. DOI: 10.19614/j.cnki.jsks.202101001

    WU Aixiang,WANG Yong,ZHANG Minzhe,et al. New development and prospect of key technology in underground mining of metal mines[J]. Metal Mine,2021(1):1-13. DOI: 10.19614/j.cnki.jsks.202101001
    [2]
    张涵,王峰. 基于矿工不安全行为的煤矿生产事故分析及对策[J]. 煤炭工程,2019,51(8):177-180.

    ZHANG Han,WANG Feng. Countermeasure and analysis on accidents of mines based on staff's unsafe behaviors[J]. Coal Engineering,2019,51(8):177-180.
    [3]
    李国清,王浩,侯杰,等. 地下金属矿山智能化技术进展[J]. 金属矿山,2021(11):1-12. DOI: 10.19614/j.cnki.jsks.202111001

    LI Guoqing,WANG Hao,HOU Jie,et al. Progress of intelligent technology in underground metal mines[J]. Metal Mine,2021(11):1-12. DOI: 10.19614/j.cnki.jsks.202111001
    [4]
    WANG Xiaolong,GIRSHICK R,GUPTA A,et al. Non-local neural networks[C]. IEEE/CVF Conference on Computer Vision and Pattern Recognition,Salt Lake City,2018:7794-7803.
    [5]
    LIN Tianwei,ZHAO Xu,SU Haisheng,et al. BSN:boundary sensitive network for temporal action proposal generation[C]. European Conference on Computer Vision,Munich,2018:3-21.
    [6]
    GU Chunhui,SUN Chen,ROSS D A,et al. AVA:a video dataset of spatio-temporally localized atomic visual actions[C]. IEEE/CVF Conference on Computer Vision and Pattern Recognition,Salt Lake City,2018:6047-6056.
    [7]
    YAN Sijie,XIONG Yuanjun,LIN Dahua. Spatial temporal graph convolutional networks for skeleton-based action recognition[C]. AAAI Conference on Artificial Intelligence,New Orleans,2018:7444-7452.
    [8]
    党伟超,张泽杰,白尚旺,等. 基于改进双流法的井下配电室巡检行为识别[J]. 工矿自动化,2020,46(4):75-80. DOI: 10.13272/j.issn.1671-251x.2019080074

    DANG Weichao,ZHANG Zejie,BAI Shangwang,et al. Inspection behavior recognition of underground power distribution room based on improved two-stream CNN method[J]. Industry and Mine Automation,2020,46(4):75-80. DOI: 10.13272/j.issn.1671-251x.2019080074
    [9]
    刘浩,刘海滨,孙宇,等. 煤矿井下员工不安全行为智能识别系统[J]. 煤炭学报,2021,46(增刊2):1159-1169. DOI: 10.13225/j.cnki.jccs.2021.0670

    LIU Hao,LIU Haibin,SUN Yu,et al. Intelligent recognition system of unsafe behavior of underground coal miners[J]. Journal of China Coal Society,2021,46(S2):1159-1169. DOI: 10.13225/j.cnki.jccs.2021.0670
    [10]
    黄瀚,程小舟,云霄,等. 基于DA-GCN的煤矿人员行为识别方法[J]. 工矿自动化,2021,47(4):62-66. DOI: 10.13272/j.issn.1671-251x.17721

    HUANG Han,CHENG Xiaozhou,YUN Xiao,et al. DA-GCN-based coal mine personnel action recognition method[J]. Industry and Mine Automation,2021,47(4):62-66. DOI: 10.13272/j.issn.1671-251x.17721
    [11]
    曹虎晨,姚善化,王仲根. 基于边界约束的煤矿井下尘雾图像去雾算法[J]. 工矿自动化,2022,48(6):139-146.

    CAO Huchen,YAO Shanhua,WANG Zhonggen. Defogging algorithm of underground coal mine dust and fog image based on boundary constraint[J]. Journal of Mine Automation,2022,48(6):139-146.
    [12]
    FEICHTENHOFER C,FAN Haoqi,MALIK J,et al. SlowFast networks for video recognition[C]. IEEE/CVF International Conference on Computer Vision,Seoul,2019:6201-6210.
    [13]
    GE Zheng,LIU Songtao,WANG Feng,et al. YOLOX:exceeding YOLO series in 2021[EB/OL]. [2023-06-20]. https://arxiv.org/abs/2107.08430.
    [14]
    YU Changqian,XIAO Bin,GAO Changxin,et al. Lite-HRNet:a lightweight high-resolution network[C]. IEEE/CVF Conference on Computer Vision and Pattern Recognition,2021:10440-10450.
    [15]
    DUAN Haodong,ZHAO Yue,CHEN Kai,et al. Revisiting skeleton-based action recognition[C]. IEEE/CVF Conference on Computer Vision and Pattern Recognition,New Orleans,2022:2959-2968.
    [16]
    REDMON J,FARHADI A. YOLOv3:an incremental improvement[EB/OL]. [2023-06-20]. https://arxiv.org/abs/1804.02767.
    [17]
    LIN T-Y,MAIRE M,BELONGIE S,et al. Microsoft COCO:common objects in context[C]. European Conference on Computer Vision,Zurich,2014:740-755.
    [18]
    SUN Ke,XIAO Bin,LIU Dong,et al. Deep high-resolution representation learning for human pose estimation[C]. IEEE/CVF Conference on Computer Vision and Pattern Recognition,Long Beach,2019:5686-5696.
    [19]
    MA Ningning,ZHANG Xiangyu,ZHENG Haitao,et al. Shufflenet V2:practical guidelines for efficient CNN architecture design[C]. 15th European Conference on Computer Vision,Munich,2018:122-138.
    [20]
    SHAHROUDY A,LIU Jun,NG T-T,et al. NTU RGB + D:a large scale dataset for 3D human activity analysis[C]. IEEE/CVF Conference on Computer Vision and Pattern Recognition,Las Vegas,2016:1010-1019.
    [21]
    SHI Lei,ZHANG Yifan,CHENG Jian,et al. Two-stream adaptive graph convolutional networks for skeleton-based action recognition[C]. IEEE/CVF Conference on Computer Vision and Pattern Recognition,Long Beach,2019:12018-12027.
    • Related Articles

  • Related Articles

    [1]LI Xiaosi, MA Jianmin, WANG Yingying. Evolution of smart mine construction and its development tendency[J]. Journal of Mine Automation, 2019, 45(9): 65-69.. DOI: 10.13272/j.issn.1671-251x.2019040032
    [2]LU Xinming, KAN Shuting, ZHANG Xingli. Cloud computation and platform technology for mine Internet of things[J]. Journal of Mine Automation, 2018, 44(2): 1-5. DOI: 10.13272/j.issn.1671-251x.17308
    [3]ZHANG Shen, HU Qingsong, WANG Gang. Consideration about construction of standards for mine Internet of things[J]. Journal of Mine Automation, 2018, 44(1): 1-5.
    [4]SHI Weizu. Accelerating construction of mine Internet of things[J]. Journal of Mine Automation, 2017, 43(11): 1-4. DOI: 10.13272/j.issn.1671-251x.2017.11.001
    [5]HE Yaoyi. Discussion on evaluation index system and architecture of smart mine[J]. Journal of Mine Automation, 2017, 43(9): 16-20. DOI: 10.13272/j.issn.1671-251x.2017.09.003
    [6]DING Enjie, SHI Weizu, ZHANG Shen, ZHAO Xiaohu. Top-down design of mine Internet of things[J]. Journal of Mine Automation, 2017, 43(9): 1-11. DOI: 10.13272/j.issn.1671-251x.2017.09.001
    [7]YAO Jianquan, DING Enjie, ZHANG Shen, WANG Gang. Prospect of perception mine internet of things and its development trend[J]. Journal of Mine Automation, 2016, 42(9): 1-5. DOI: 10.13272/j.issn.1671-251x.2016.09.001
    [8]ZHANG She. Service carrying platform of mine Internet of things and mine buys service[J]. Journal of Mine Automation, 2016, 42(4): 7-11. DOI: 10.13272/j.issn.1671-251x.2016.04.002
    [9]WANG Li. Study on concept and key technologies of smart mine[J]. Journal of Mine Automation, 2014, 40(6): 37-41.
    [10]WANG Hui-bi. Design of Virtual Mine System Based on Virtual Reality Technology[J]. Journal of Mine Automation, 2009, 35(5): 53-56.

  • Cited by

    Periodical cited type(3)

    1. 陈生凯，明平田，申宁. 青海某金矿安全监管视频AI智能系统建设. 黄金. 2025(01): 20-23+36+6 .
    2. 张珂. 井工煤矿智能化基础理论及关键技术. 煤矿机电. 2024(02): 35-38+45 .
    3. 江鹤，程德强，乙夫迪，汪鹏，崔文，寇旗旗. 新一代信息技术在智能矿山中的研究与应用综述. 工矿自动化. 2024(11): 1-16 . 本站查看

    Other cited types(5)

Catalog

    Get Citation
    PDF
    XML

    Article Metrics

    Article views (1699) PDF downloads (114) Cited by(8)
    Related

    苏ICP备 05016349号-2

    Supported by: Beijing Renhe Information Technology Co., Ltd. Visits：1255670    Baidu Analytics

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return