Unsafe action recognition in underground coal mine based on cross-attention mechanism

RAO Tianrong; PAN Tao; XU Huijun

doi:10.13272/j.issn.1671-251x.17949

Volume 48 Issue 10

Oct. 2022

Turn off MathJax

Article Contents

Abstract

References

Journal of Mine Automation > 2022 > 48(10): 48-54. > DOI: 10.13272/j.issn.1671-251x.17949

RAO Tianrong, PAN Tao, XU Huijun. Unsafe action recognition in underground coal mine based on cross-attention mechanism[J]. Journal of Mine Automation，2022，48(10)：48-54. DOI: 10.13272/j.issn.1671-251x.17949

Citation:

PDF (9598 KB)

Unsafe action recognition in underground coal mine based on cross-attention mechanism

CHN Energy Information Technology Co., Ltd., Beijing 100011, China

More Information

Received Date: May 17, 2022
Revised Date: October 07, 2022
Available Online: October 21, 2022

Abstract

Abstract

The real-time video monitoring and alarming of unsafe actions of coal mine personnel is an important means to improve the level of safety in production. The coal mine underground environment is complex, and the monitoring video quality is poor. The conventional action recognition method based on image features or human body key point features is limited in application in the underground coal mine. An action recognition model of multi-feature fusion based on cross-attention mechanism is proposed to recognize unsafe actions of coal mine personnel. For segment video images, the 3D ResNet101 model is adopted to extract image features. The openpose algorithm and ST-GCN (space-time graph convolutional network) are adopted to extract human body key point features. The cross-attention mechanism is used to fuse the image features and human key point features. The fused features are spliced respectively with the image features or human key point features processed by the self-attention mechanism to obtain the final action recognition features. The recognition features is processed by the fully connected layer and the normalized exponential function softmax to obtain action recognition result. Based on the public data sets HMDB51 and UCF101, and the self-built coal mine video dataset, the action recognition experiment is carried out. The results show that the cross-attention mechanism can make the action recognition model more effective in fusing image features and human key point features, and greatly improve the recognition accuracy. At present, SlowFast is the most widely used action recognition model. Compared with the SlowFast, the recognition accuracy of the action recognition model of multi-feature fusion based on cross-attention mechanism has been improved by 1.8% and 0.9% for HMDB51 and UCF101 datasets respectively. The recognition accuracy on the self-built dataset has increased by 6.7%. It is verified that the action recognition model of multi-feature fusion based on cross-attention mechanism is more suitable for the recognition of unsafe actions in the complex coal mine environment.

FullText(HTML)

References (20)

References

[1]	党伟超,史云龙,白尚旺,等. 基于条件变分自编码器的井下配电室巡检行为检测[J]. 工矿自动化,2021,47(12):98-105. DOI: 10.13272/j.issn.1671-251x.2021030087 DANG Weichao,SHI Yunlong,BAI Shangwang,et al. Inspection behavior detection of underground power distribution room based on conditional variational auto-encoder[J]. Industry and Mine Automation,2021,47(12):98-105. DOI: 10.13272/j.issn.1671-251x.2021030087
[2]	王国法,任怀伟,赵国瑞,等. 煤矿智能化十大“痛点”解析及对策[J]. 工矿自动化,2021,47(6):1-11. DOI: 10.13272/j.issn.1671-251x.17808 WANG Guofa,REN Huaiwei,ZHAO Guorui,et al. Analysis and countermeasures of ten 'pain points' of intelligent coal mine[J]. Industry and Mine Automation,2021,47(6):1-11. DOI: 10.13272/j.issn.1671-251x.17808
[3]	SIMONYAN K, ZISSERMAN A. Two-streamconvolutional networks for action recognition in videos[Z]. arXiv Preprint, arXiv:1406.2199v2.
[4]	WANG Limin, XIONG Yuanjun, WANG Zhe, et al. Temporal segment networks: towards good practices for deep action recognition[C]. European Conference on Computer Vision, Amsterdam, 2016: 20-36.
[5]	JI Lin, GAN Chuang, HAN Song. TSM: temporal shift module for efficient video understanding[C]. The IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, 2019: 7083-7093.
[6]	LIU Kun, LIU Wu, GAN Chuang, et al. T-C3D: temporal convolutional 3D network for real-time action recognition[C]. The AAAI Conference on Artificial Intelligence, New Orleans, 2018: 7138-7145.
[7]	FEICHTENHOFER C, FAN H, MALIK J, et al. Slowfast networks for video recognition[C]. The IEEE International Conference on Computer Vision, Long Beach, 2019: 6202-6211.
[8]	党伟超,张泽杰,白尚旺,等. 基于改进双流法的井下配电室巡检行为识别[J]. 工矿自动化,2020,46(4):75-80. DOI: 10.13272/j.issn.1671-251x.2019080074 DANG Weichao,ZHANG Zejie,BAI Shangwang,et al. Inspection behavior recognition of underground power distribution room based on improved two-stream CNN method[J]. Industry and Mine Automation,2020,46(4):75-80. DOI: 10.13272/j.issn.1671-251x.2019080074
[9]	刘浩,刘海滨,孙宇,等. 煤矿井下员工不安全行为智能识别系统[J]. 煤炭学报,2021,46(增刊2):1159-1169. DOI: 10.13225/j.cnki.jccs.2021.0670 LIU Hao,LIU Haibin,SUN Yu,et al. Intelligent recognition system of unsafe behavior of underground coal miners[J]. Journal of China Coal Society,2021,46(S2):1159-1169. DOI: 10.13225/j.cnki.jccs.2021.0670
[10]	张立亚. 基于图像识别的煤矿井下安全管控技术[J]. 煤矿安全,2021,52(2):165-168. DOI: 10.13347/j.cnki.mkaq.2021.02.032 ZHANG Liya. Safety control technology of coal mine based on image recognition[J]. Safety in Coal Mines,2021,52(2):165-168. DOI: 10.13347/j.cnki.mkaq.2021.02.032
[11]	YAN Sijie, XIONG Yuanjun, LIN Dahua. Spatial temporal graph convolutional networks for skeleton-based action recognition[C]. The AAAI Conference on Artificial Intelligence, New Orleans, 2018: 7444-7452.
[12]	SHI Lei, ZHANG Yifan, CHENG Jian, et al. Two-stream adaptive graph convolutional networks for skeleton-based action recognition[C]. The IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, 2019: 12026-12035.
[13]	黄瀚,程小舟,云霄,等. 基于DA−GCN的煤矿人员行为识别方法[J]. 工矿自动化,2021,47(4):62-66. DOI: 10.13272/j.issn.1671-251x.17721 HUANG Han,CHENG Xiaozhou,YUN Xiao,et al. DA-GCN-based coal mine personnel action recognition method[J]. Industry and Mine Automation,2021,47(4):62-66. DOI: 10.13272/j.issn.1671-251x.17721
[14]	王璇,吴佳奇,阳康,等. 煤矿井下人体姿态检测方法[J]. 工矿自动化,2022,48(5):79-84. DOI: 10.13272/j.issn.1671-251x.17867 WANG Xuan,WU Jiaqi,YANG Kang,et al. Human posture detection method in coal mine[J]. Journal of Mine Automation,2022,48(5):79-84. DOI: 10.13272/j.issn.1671-251x.17867
[15]	HARA K, KATAOKA H, SATOH Y. Can spatiotemporal 3D CNNs retrace the history of 2D CNNs and imagenet[C]. The IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, 2018: 6546-6555.
[16]	CAO Zhe, SIMON T, WEI S-E, et al. Realtime multi-person 2D pose estimation using part affinity fields[C]. The IEEE International Conference on Computer Vision, Honolulu, 2017: 7291-7299.
[17]	WANG Xiaolong, GIRSHICK R, GUPTA A, et al. Non-local neural networks[C]. The IEEE International Conference on Computer Vision, Salt Lake City, 2018: 7794-7803.
[18]	VELICKOVIC P, CUCURULL G, CASANOVA A, et al. Graph attention networks[Z]. arXiv Preprint, arXiv: 1710.10903.
[19]	KUEHNE H, JHUANG H, GARROTE E, et al. HMDB: a large video database for human motion recognition[C]. International Conference on Computer Vision, Barcelona, 2011: 2556-2563.
[20]	SOOMORO K, ZAMIR A R, SHAH M. UCF101: a dataset of 101 human actions classes from videos in the wild[Z]. arXiv Preprint, arXiv: 1212.0402.

[1]	TAN Donggui, YUAN Yiping, FAN Panpan. Health status evaluation of CNN-GRU mine motor based on adaptive multi-scale attention mechanism[J]. Journal of Mine Automation, 2024, 50(2): 138-146. DOI: 10.13272/j.issn.1671-251x.2023110024
[2]	CHENG Deqiang, ZHENG Lijuan, LIU Jingjing, KOU Qiqi, JIANG He. Quantitative analysis of coal particle size based on bi-level routing attention mechanism[J]. Journal of Mine Automation, 2024, 50(2): 9-17. DOI: 10.13272/j.issn.1671-251x.2023100002
[3]	CAO Xiangang, LI Hu, WANG Peng, WU Xudong, XIANG Jingfang, DING Wentao. A coal foreign object detection method based on cross modal attention fusion[J]. Journal of Mine Automation, 2024, 50(1): 57-65. DOI: 10.13272/j.issn.1671-251x.2023110035
[4]	CAO Zhengyuan, JIANG Wei, FANG Chenghui. Intelligent detection method for coal flow foreign objects based on dual attention generative adversarial network[J]. Journal of Mine Automation, 2023, 49(12): 56-62. DOI: 10.13272/j.issn.1671-251x.18094
[5]	LI Zhongfei, FENG Shiyong, GUO Jun, ZHANG Yunhe, XU Feixiang. Lightweight safety helmet wearing detection fusing coordinate attention and multiscale feature[J]. Journal of Mine Automation, 2023, 49(11): 151-159. DOI: 10.13272/j.issn.1671-251x.2023080123
[6]	ZHU Fuwen, HOU Zhihui, LI Mingzhen. Lightweight multi-scale cross channel attention coal flow detection network[J]. Journal of Mine Automation, 2023, 49(8): 100-105. DOI: 10.13272/j.issn.1671-251x.2023030045
[7]	YE Ou, DOU Xiaoyi, FU Yan, DENG Jun. Coal block detection method integrating lightweight network and dual attention mechanism[J]. Journal of Mine Automation, 2021, 47(12): 75-80. DOI: 10.13272/j.issn.1671-251x.2021030075
[8]	LAI Cheng-yu. Research of Integrated Congestion Control Mechanism[J]. Journal of Mine Automation, 2012, 38(9): 75-78.
[9]	LI Jian, LIU He-ping, LI Jie. Research of Routing Mechanism of Mine WS[J]. Journal of Mine Automation, 2012, 38(6): 48-51.
[10]	XU Chang-sheng, ZHANG Yao-qing. A Self-Learning Mechanism Based on Objective and its Applicatio[J]. Journal of Mine Automation, 1995, 21(4): 32-35.

Cited By

Cited by

Periodical cited type(12)

1.	郭孝园，朱美强，田军，朱贝贝. 基于深度学习的煤矿井下人员不安全行为检测与识别. 工矿自动化. 2025(03): 138-147 . 本站查看
2.	杨艺，杨艳磊，王田，王科平. 基于多重信息自注意力的综采工作面目标行为识别. 煤炭学报. 2025(02): 1425-1442 .
3.	苏晨阳，武文红，牛恒茂，石宝，郝旭，王嘉敏，高勒，汪维泰. 深度学习的工人多种不安全行为识别方法综述. 计算机工程与应用. 2024(05): 30-46 .
4.	孙晴，杨超宇. 基于多模态的井下登高作业专人扶梯检测方法. 工矿自动化. 2024(05): 142-150 . 本站查看
5.	邬心怡，邓志良，刘云平，董娟，李嘉琦. 基于交叉注意力机制的多特征行人重识别. 南京信息工程大学学报. 2024(04): 461-471 .
6.	冯兴龙，吴田，万亚旭，肖宾，方春华，黎鹏，赵慧敏. 基于人-物交互关系检测的带电作业人员行为识别方法研究. 中国安全生产科学技术. 2024(09): 205-211 .
7.	卢纪峰，杨超宇. 基于M3CFC-YOLOv7-tiny的矿工乘坐架空乘人装置违章行为识别研究. 煤矿安全. 2024(11): 250-256 .
8.	田枫，卫宁彬，刘芳，韩玉祥，赵玲，张思睿，马贵宝. 基于时空-动作自适应融合网络的油田作业行为识别. 智能系统学报. 2024(06): 1407-1418 .
9.	李善华，肖涛，李肖利，杨发展，姚勇，赵培培. 基于DRCA-GCN的矿工动作识别模型. 工矿自动化. 2023(04): 99-105+112 . 本站查看
10.	毛清华，郭文瑾，翟姣，王荣泉，尚新芒，李世坤，薛旭升. 煤矿带式输送机异常状态视频AI识别技术研究. 工矿自动化. 2023(09): 36-46 . 本站查看
11.	高祺，易妍，王燕萍. 一种电网综合场景技能评估方法. 科技与创新. 2023(20): 86-88 .
12.	郝明月，闵冰冰，张新建，赵作鹏，吴晨，王欣. 基于改进YOLOv5s的矿工排队检测方法. 工矿自动化. 2023(11): 160-166 . 本站查看

Other cited types(14)

XU Huijun

CHN Energy Information Technology Co., Ltd., Beijing 100011, China

Get Citation

PDF

XML

Article Metrics

Article views (625) PDF downloads (124) Cited by(26)

Unsafe action recognition in underground coal mine based on cross-attention mechanism