Research on video AI recognition technology for abnormal state of coal mine belt conveyors
-
摘要: 传统的带式输送机异常状态识别采用人工巡检或机械综合保护系统进行检测，人工巡检劳动强度大、效率低、难以准确发现故障等，机械综合保护系统易造成误判，识别效果不佳，已无法满足煤炭行业智能化需求。随着机器视觉、深度学习和工业以太网技术发展，视频AI技术成为煤矿带式输送机异常状态智能识别的研究热点。分析了采用视频AI技术识别煤矿带式输送机输送带跑偏、托辊故障、人员入侵、人员不安全行为、堆煤及异物等异常状态的研究现状，指出目前煤矿带式输送机异常状态视频AI识别技术存在视频图像数据集构建耗时长、异常状态识别精度不高、视频信息传输延时大3个主要问题。针对视频图像数据集构建耗时长问题，提出加强基于半监督、无监督及小样本学习的视频AI识别算法研究、基于生成模型等方式扩充数据集的解决思路；针对异常状态识别精度不高问题，提出加强数据去模糊方法研究、利用生成对抗网络等算法均衡正负样本和改进AI识别算法的解决思路；针对视频信息传输延时大问题，提出构建“云−边−端”协同的带式输送机异常状态视频AI识别系统架构，合理部署高带宽、低延时的网络通信系统的解决思路。从高性能视频AI识别算法，高带宽、低延时视频通信技术，“云−边−端”高效协同的视频AI识别系统和健全视频AI识别技术标准4个方面展望了带式输送机异常状态视频AI识别技术的发展趋势。Abstract: Traditional belt conveyor abnormal state recognition uses manual inspection or mechanical comprehensive protection system for detection. The manual inspection is labor-intensive, inefficient, and difficult to accurately detect faults. Mechanical comprehensive protection system is prone to misjudgment and poor recognition effect. The above methods can no longer meet the needs of coal industry intelligence. With the development of machine vision, deep learning, and industrial Ethernet technology, video AI technology has become a research hotspot for intelligent recognition of abnormal states of coal mine belt conveyors. This paper analyzes the current research status of using video AI technology to identify abnormal states of coal mine belt conveyors, such as belt deviation, idler failure, personnel invasion, unsafe behavior of personnel, coal stacking, and foreign objects. It is pointed out that there are three main problems in the current video AI recognition technology for abnormal states of coal mine belt conveyors: long construction time-consumption of video image datasets, low precision of abnormal state recognition, and large time delay in video information transmission. To address the issue of long construction time-consumption of video image datasets, a solution is proposed to strengthen the research on video AI recognition algorithms based on semi supervised, unsupervised, and small sample learning, and to expand the dataset based on generative models. To address the issue of low precision of abnormal state recognition, a solution is proposed to strengthen research on data deblurring methods, and to utilize algorithms such as generative adversarial networks to balance positive and negative samples, and improve AI recognition algorithms. To address the issue of large time delay in video information transmission, a solution is proposed to build a "cloud-edge-end" collaborative video AI recognition system architecture for abnormal states of belt conveyors, and to deploy a high bandwidth and low time delay network communication system. This article looks forward to the development trend of video AI recognition technology for abnormal states of belt conveyors from four aspects: high-performance video AI recognition algorithms, high bandwidth and low time delay video communication technology, "cloud-edge-end" efficient collaborative video AI recognition system, and sound video AI recognition technology standards.
-
图 1 基于视频AI技术的输送带跑偏识别
Figure 1. Recognition of belt deviation based on video AI technology
图 2 融合红外数据与RGB数据的托辊故障视频AI识别
Figure 2. Idler fault video AI recognition based on fusion of infrared data and RGB data
图 3 基于视频AI技术的人员入侵危险区域识别效果
Figure 3. Personnel intrusing dangerous region recognition based on video AI technology
图 4 基于视频AI技术的输送带堆煤识别效果
Figure 4. Coal pile recognition of belt based on video AI technology
图 5 基于视频AI技术的输送带异物识别效果
Figure 5. Foreign object recognition of belt based on video AI technology
图 6 煤矿带式输送机异常状态识别精度不高问题的解决思路
Figure 6. Solutions to the problem of low precision of abnormal state recognition of coal mine belt conveyor
-
[1] 王国法,杜毅博. 智慧煤矿与智能化开采技术的发展方向[J]. 煤炭科学技术,2019,47(1):1-10.WANG Guofa,DU Yibo. Development direction of intelligent coal mine and intelligent mining technology[J]. Coal Science and Technology,2019,47(1):1-10. [2] 李鑫. 矿用带式输送机巡检机器人研究及行走机构设计[D]. 太原: 太原理工大学, 2022.LI Xin. Research on inspection robot of mine belt conveyor and design of its walking mechanism[D]. Taiyuan: Taiyuan University of Science and Technology, 2022. [3] 刘建思,尹丽菊,潘金凤,等. 基于参数化对数图像处理模型的光照不均匀图像的边缘检测算法[J]. 激光与光电子学进展,2021,58(22):140-149.LIU Jiansi,YIN Liju,PAN Jinfeng,et al. Edge detection algorithm for unevenly illuminated images based on parameterized logarithmic image processing model[J]. Laser & Optoelectronics Progress,2021,58(22):140-149. [4] KUSNIK D, SMOLKA B. On the robust technique of mixed Gaussian and impulsive noise reduction in color digital images[C]. International Conference on Information, Intelligence, Systems and Applications, Corfu, 2015: 940-945. [5] 王锴,曾祥进,黎新,等. 输送带跑偏检测方法研究[J]. 工矿自动化,2023,49(3):23-30,52.WANG Kai,ZENG Xiangjin,LI Xin,et al. Research on conveyor belt deviation detection method[J]. Journal of Mine Automation,2023,49(3):23-30,52. [6] 王星. 基于视觉的煤矿井下带式输送机异常状态监测方法研究[D]. 太原: 太原科技大学, 2017.WANG Xing. Research on monitoring method for abnormal state of coal mine belt conveyor based on vision[D]. Taiyuan: Taiyuan University of Science and Technology, 2017. [7] 李占利,赵文博. 基于视觉计算的胶带运输机跑偏监测[J]. 煤矿安全,2014,45(5):118-121.LI Zhanli,ZHAO Wenbo. Deviation monitoring of coal belt conveyor based on visual computing[J]. Safety in Coal Mines,2014,45(5):118-121. [8] ZHANG Mengchao,SHI Hao,YU Yan,et al. A computer vision based conveyor deviation detection system[J]. Applied Sciences,2020,10(7):2402. DOI: 10.3390/app10072402. [9] MARIA S,WIOLETTA K,PAWEL S,et al. Procedures of detecting damage to a conveyor belt with use of an inspection legged robot for deep mine infrastructure[J]. Minerals,2021,11(10):1040. DOI: 10.3390/min11101040. [10] 王磊,张磊,董妍,等. 基于FPGA的输送带跑偏检测系统设计[J]. 煤矿机械,2019,40(7):27-30.WANG Lei,ZHANG Lei,DONG Yan,et al. Design of conveyor belt deviation detecting system based on FPGA[J]. Coal Mine Machinery,2019,40(7):27-30. [11] LIU Yi, MIAO Changyun, LI Xiangguo, et al. Research on deviation detection of belt conveyor based on inspection robot and deep learning[J]. Complexity, 2021, 2021. DOI: 10.1155/2021/3734560. [12] 林俊,党伟超,潘理虎,等. 基于计算机视觉的井下输送带跑偏检测方法[J]. 煤矿机械,2019,40(10):169-171.LIN Jun,DANG Weichao,PAN Lihu,et al. Deviation monitoring method of underground conveyor belt based on computer vision[J]. Coal Mine Machinery,2019,40(10):169-171. [13] 韩涛,黄友锐,张立志,等. 基于图像识别的带式输送机输煤量和跑偏检测方法[J]. 工矿自动化,2020,46(4):17-22.HAN Tao,HUANG Yourui,ZHANG Lizhi,et al. Detection method of coal quantity and deviation of belt conveyor based on image recognition[J]. Industry and Mine Automation,2020,46(4):17-22. [14] ZENG Chan,ZHENG Junfeng,LI Jiangyun. Real-time conveyor belt deviation detection algorithm based on multi-scale feature fusion network[J]. Algorithms,2019,12(10):205. DOI: 10.3390/a12100205. [15] ZHANG Mengchao, JIANG Kai, CAO Yueshuai, et al. A deep learning-based method for deviation status detection in intelligent conveyor belt system[J]. Journal of Cleaner Production, 2022, 363. DOI: 10.1016/j.jclepro.2022.132575. [16] 杨志方, 张立亚, 郝博南, 等. 基于双流融合网络的运输机皮带跑偏检测方法[J/OL]. 煤炭科学技术: 1-10[2023-08-04]. https://doi.org/10.13199/j.cnki.cst.2023-0215.YANG Zhifang, ZHANG Liya, HAO Bonan, et al. Conveyor belt deviation detection method based on dual flow network[J/OL]. Coal Science and Technology: 1-10[2023-08-04]. https://doi.org/10.13199/j.cnki.cst.2023-0215. [17] 张瑞,汤心溢,李争. 低照度短波红外图像增强算法[J]. 红外与毫米波学报,2020,39(6):818-824.ZHANG Rui,TANG Xinyi,LI Zheng. Research on low illumination shortwave infrared image enhancement algorithm[J]. Journal of Infrared and Millimeter Waves,2020,39(6):818-824. [18] PATHAK S, SEJWAR V. Optimized noisy image segmentation using genetic algorithm[C]. International Conference on Intelligent Computing and Control Systems, Madurai, 2017: 1311-1316. [19] SZREK J,WODECKI J,BAEJ R,et al. An inspection robot for belt conveyor maintenance in underground mine-infrared thermography for overheated idlers detection[J]. Applied Sciences,2020,10(14):4984. DOI: 10.3390/app10144984. [20] SIAMI M,BARSZCZ T,WODECKI J,et al. Design of an infrared image processing pipeline for robotic inspection of conveyor systems in opencast mining sites[J]. Energies,2022,15(18):6771. DOI: 10.3390/en15186771. [21] SZURGACZ D,ZHIRONKIN S,VTH S,et al. Thermal imaging study to determine the operational condition of a conveyor belt drive system structure[J]. Energies,2021,14(11):3258. DOI: 10.3390/en14113258. [22] 马宏伟,杨文娟,张旭辉. 基于红外热像的带式输送机监测与预警系统[J]. 激光与红外,2017,47(4):448-452.MA Hongwei,YANG Wenjuan,ZHANG Xuhui. Monitoring and warning system of belt conveyor based on infrared thermography[J]. Laser & Infrared,2017,47(4):448-452. [23] 董瑞佳. 基于迁移学习和DenseNet的带式输送机托辊故障检测方法[J]. 煤炭技术,2023,42(1):250-252.DONG Ruijia. Fault detection method of belt conveyor idler based on transfer learning and densenet[J]. Coal Technology,2023,42(1):250-252. [24] 刘宇琦. 基于深度学习的托辊异常检测方法研究[D]. 西安: 西安科技大学, 2020.LIU Yuqi. Research on abnormal detection method of idler based on deep learning[D]. Xi'an: Xi'an University of Science and Technology, 2020. [25] LIU Yi, MIAO Changyun, LI Xiangguo, et al. Research on the fault analysis method of belt conveyor idlers based on sound and thermal infrared image features[J]. Measurement, 2021, 186. DOI: 10.1016/j.measurement.2021.110177. [26] CARVALHO R,NASCIMENTO R,DANGELO T,et al. A UAV-based framework for semi-automated thermographic inspection of belt conveyors in the mining industry[J]. Sensors,2020,20(8):2243. DOI: 10.3390/s20082243. [27] DABEK P,SZREK J,ZIMROZ R,et al. An automatic procedure for overheated idler detection in belt conveyors using fusion of infrared and RGB images acquired during UGV robot inspection[J]. Energies,2022,15(2):601. DOI: 10.3390/en15020601. [28] 邹斐. 煤矿井下运动目标检测与跟踪研究[D]. 西安: 西安科技大学, 2018.ZOU Fei. The detecting and tracking methods of moving target in coal mine tunnel[D]. Xi'an: Xi'an University of Science and Technology, 2018. [29] SI Lei,WANG Zhongbin,XU Rongxin,et al. Image enhancement for surveillance video of coal mining face based on single-scale retinex algorithm combined with bilateral filtering[J]. Symmetry,2017,9(6). DOI: 10.3390/sym9060093. [30] 秦晓伟. 基于视频端AI算法实现的矿井斜巷联动联控系统研究[D]. 淮南: 安徽理工大学, 2022.QIN Xiaowei. Research on joint control system of mine inclined lane based on AI algorithm in video terminal[D]. Huainan: Anhui University of Science and Technology. 2022. [31] 董观利,宋春林. 基于视频的矿井行人越界检测系统[J]. 工矿自动化,2017,43(2):29-34.DONG Guanli,SONG Chunlin. Underground pedestrian crossing detection system based on video[J]. Industry and Mine Automation,2017,43(2):29-34. [32] 李晓建. 矿井人员目标检测与跟踪算法的研究与实现[D]. 青岛: 山东科技大学, 2020.LI Xiaojian. The research and realization of mine personnel target detection and tracking algorithm[D]. Qingdao: Shandong University of Science and Technology, 2020. [33] 田隽,钱建生,厉丹,等. 基于多摄像机的矿井危险区域目标匹配算法[J]. 中国矿业大学学报,2010,39(1):139-144.TIAN Jun,QIAN Jiansheng,LI Dan,et al. Moving object matching algorithm in danger zones of coal mine based on multiple cameras[J]. Journal of China University of Mining & Technology,2010,39(1):139-144. [34] 杨世超. 基于Faster−RCNN的矿井人员识别检测[J]. 信息记录材料,2020,21(12):236-238.YANG Shichao. Mine personnel identification and detection based on Faster-RCNN[J]. Information Recording Materials,2020,21(12):236-238. [35] LIU Xiaoyang,LIU Jinqiang. Gait recognition method of underground coal mine personnel based on densely connected convolution network and stacked convolutional autoencoder[J]. Entropy,2020,22(6):695. DOI: 10.3390/e22060695. [36] HAO Bonan, ZHANG Liya, PENG Ran. An improved vibe algorithm to detect personnel underground in coal mines[J]. Journal of Physics: Conference Series, 2021, 2025（1）. DOI: 10.1088/1742-6596/2025/1/012032. [37] 饶天荣,潘涛,徐会军. 基于交叉注意力机制的煤矿井下不安全行为识别[J]. 工矿自动化,2022,48(10):48-54.RAO Tianrong,PAN Tao,XU Huijun. Unsafe action recognition in underground coal mine based on cross-attention mechanism[J]. Journal of Mine Automation,2022,48(10):48-54. [38] 赵小虎,黄程龙. 基于Kinect的矿井人员违规行为识别算法研究[J]. 湖南大学学报(自然科学版),2020,47(4):92-98.ZHAO Xiaohu,HUANG Chenglong. Research on identification algorithm of mine person's violation behavior based on Kinect[J]. Journal of Hunan University(Natural Sciences),2020,47(4):92-98. [39] 陈庆峰. 矿井皮带区域矿工不安全行为识别方法的研究[D]. 徐州: 中国矿业大学, 2019.CHEN Qingfeng. Research on recognition method of unsafe behavior of miners in mine belt area[D]. Xuzhou: China University of Mining and Technology, 2019. [40] 李雯静,刘鑫. 基于深度学习的井下人员不安全行为识别与预警系统研究[J]. 金属矿山,2023(3):177-184.LI Wenjing,LIU Xin. Research on underground personnel unsafe behavior identification and early warning system based on deep learning[J]. Metal Mine,2023(3):177-184. [41] 刘浩,刘海滨,孙宇,等. 煤矿井下员工不安全行为智能识别系统[J]. 煤炭学报,2021,46(增刊2):1159-1169.LIU Hao,LIU Haibin,SUN Yu,et al. Intelligent recognition system of unsafe behavior of underground coal miners[J]. Journal of China Coal Society,2021,46(S2):1159-1169. [42] 黄瀚,程小舟,云霄,等. 基于DA−GCN的煤矿人员行为识别方法[J]. 工矿自动化,2021,47(4):62-66.HUANG Han,CHENG Xiaozhou,YUN Xiao,et al. DA-GCN-based coal mine personnel action recognition method[J]. Industry and Mine Automation,2021,47(4):62-66. [43] 张立亚. 矿山智能视频分析与预警系统研究[J]. 工矿自动化,2017,43(11):16-20.ZHANG Liya. Research on intelligent video analysis and early warning system for mine[J]. Industry and Mine Automation,2017,43(11):16-20. [44] 许军,吕俊杰,杨娟利,等. 基于图像处理的溜槽堆煤预警研究[J]. 煤炭技术,2017,36(12):232-234.XU Jun,LYU Junjie,YANG Juanli,et al. Research on early warning of coal chute blocking based on image processing[J]. Coal Technology,2017,36(12):232-234. [45] 袁敦鹏. 基于三维点云的带式输送机跑偏及堆煤监测技术研究[D]. 徐州: 中国矿业大学, 2022.YUAN Dunpeng. Study on the belt conveyor deviation and coal stacking monitoring based on three-dimensional point cloud[D]. Xuzhou: China University of Mining and Technology, 2022. [46] 彭利泽. 基于视觉的传输带堆煤体积监测系统设计[D]. 太原: 中北大学, 2021.PENG Lize. Design of volume monitoring system for coal stacking in transmission belt based on vision[D]. Taiyuan: North University of China, 2021. [47] 游磊,朱兴林,秦伟,等. 基于曲面重建的带式输送机堆煤识别方法[J]. 工矿自动化,2021,47(6):45-50.YOU Lei,ZHU Xinglin,QIN Wei,et al. Coal stacking identification method of belt conveyor based on surface reconstruction[J]. Industry and Mine Automation,2021,47(6):45-50. [48] 姜文涛,王梓民,张驰. 基于曲量场空间的皮带堆煤识别[J]. 传感器与微系统,2021,40(1):140-143.JIANG Wentao,WANG Zimin,ZHANG Chi. Coal pile recognition based on curved space field[J]. Transducer and Microsystem Technologies,2021,40(1):140-143. [49] 孙旭. 基于机器视觉的带式输送机异物实时检测与定位研究[D]. 徐州: 中国矿业大学, 2022.SUN Xu. Study on the machine vision for belt conveyor real-time foreign object detection and positioning[D]. Xuzhou: China University of Mining and Technology, 2022. [50] 任国强,韩洪勇,李成江,等. 基于FastYOLOv3算法的煤矿胶带运输异物检测[J]. 工矿自动化,2021,47(12):128-133.REN Guoqiang,HAN Hongyong,LI Chengjiang,et al. Foreign object detection in coal mine belt transportation based on FastYOLOv3 algorithm[J]. Industry and Mine Automation,2021,47(12):128-133. [51] 薛旭升,杨星云,齐广浩,等. 煤矿带式输送机分拣机器人异物识别与定位系统设计[J]. 工矿自动化,2022,48(12):33-41.XUE Xusheng,YANG Xingyun,QI Guanghao,et al. Design of foreign object recognition and positioning system for sorting robot of coal mine belt conveyor[J]. Journal of Mine Automation,2022,48(12):33-41. [52] 程德强,徐进洋,寇旗旗,等. 融合残差信息轻量级网络的运煤皮带异物分类[J]. 煤炭学报,2022,47(3):1361-1369.CHENG Deqiang,XU Jinyang,KOU Qiqi,et al. Lightweight network based on residual information for foreign body classification on coal conveyor belt[J]. Journal of China Coal Society,2022,47(3):1361-1369. [53] 杜京义,陈瑞,郝乐,等. 煤矿带式输送机异物检测[J]. 工矿自动化,2021,47(8):77-83.DU Jingyi,CHEN Rui,HAO Le,et al. Coal mine belt conveyor foreign object detection[J]. Industry and Mine Automation,2021,47(8):77-83. [54] 王卫东,张康辉,吕子奇,等. 基于深度学习的煤中异物机器视觉检测[J]. 矿业科学学报,2021,6(1):115-123.WANG Weidong,ZHANG Kanghui,LYU Ziqi,et al. Machine vision detection of foreign objects in coal using deep learning[J]. Journal of Mining Science and Technology,2021,6(1):115-123. [55] 胡璟皓,高妍,张红娟,等. 基于深度学习的带式输送机非煤异物识别方法[J]. 工矿自动化,2021,47(6):57-62,90.HU Jinghao,GAO Yan,ZHANG Hongjuan,et al. Research on the identification method of non-coal foreign object of belt conveyor based on deep learning[J]. Industry and Mine Automation,2021,47(6):57-62,90. [56] 吴守鹏,丁恩杰,俞啸. 基于改进FPN的输送带异物识别方法[J]. 煤矿安全,2019,50(12):127-130.WU Shoupeng,DING Enjie,YU Xiao. Foreign body identification of belt based on improved FPN[J]. Safety in Coal Mines,2019,50(12):127-130. [57] 郝帅,张旭,马旭,等. 基于CBAM−YOLOv5的煤矿输送带异物检测[J]. 煤炭学报,2022,47(11):4147-4156.HAO Shuai,ZHANG Xu,MA Xu,et al. Foreign object detection in coal mine conveyor belt based on CBAM-YOLOv5[J]. Journal of China Coal Society,2022,47(11):4147-4156. [58] MAO Qinghua,LI Shikun,HU Xin,et al. Coal mine belt conveyor foreign objects recognition method of improved YOLOv5 algorithm with defogging and deblurring[J]. Energies,2022,15(24):9504. DOI: 10.3390/en15249504. [59] DAI Lili, ZHANG Xu, GARDONI P, et al. A new machine vision detection method for identifying and screening out various large foreign objects on coal belt conveyor lines[J]. Complex & Intelligent Systems, 2023: 1-14. [60] 王燕,郭潇樯,刘新华. 带式输送机大块异物视觉检测系统设计[J]. 机械科学与技术,2021,40(12):1939-1943.WANG Yan,GUO Xiaoqiang,LIU Xinhua. Design of visual detection system for large foreign body in belt conveyor[J]. Mechanical Science and Technology for Aerospace Engineering,2021,40(12):1939-1943. [61] 毛清华,李世坤,胡鑫,等. 基于改进YOLOv7的煤矿带式输送机异物识别[J]. 工矿自动化,2022,48(12):26-32.MAO Qinghua,LI Shikun,HU Xin,et al. Foreign object recognition of belt conveyor in coal mine based on improved YOLOv7[J]. Journal of Mine Automation,2022,48(12):26-32. [62] 姜雨辰. 遥感图像目标检测中的小样本数据扩充方法研究[D]. 长沙: 国防科技大学, 2021.JIANG Yuchen. Research on small sample data augmentation in remote sensing images object detection[D]. Changsha: National University of Defense Technology, 2021. -
下载:
点击查看大图
图(7)
计量
- 文章访问数: 1148
- HTML全文浏览量: 80
- PDF下载量: 82
- 被引次数: 0
