带式输送机上散状物料堆积视频实时检测

唐俊; 李敬兆; 石晴; 刘阳; 宋世现; 任成成

doi:10.13272/j.issn.1671-251x.2022050078

带式输送机上散状物料堆积视频实时检测

doi: 10.13272/j.issn.1671-251x.2022050078

唐俊^1,,
李敬兆^1, ,,
石晴²,
刘阳²,
宋世现²,
任成成²

1.
安徽理工大学电气与信息工程学院, 安徽淮南　232001
2.
淮北合众机械设备有限公司, 安徽淮北　235000

基金项目: 国家自然科学基金项目（51874010）；淮北市重大科技专项项目（Z2020004）。

详细信息

作者简介:
唐俊（1998—），男，安徽合肥人，硕士研究生，主要研究方向为嵌入式系统和深度学习，E-mail：1592267145@qq.com

通讯作者:
李敬兆（1964—），男，安徽淮南人，教授，博士，博士研究生导师，主要研究方向为智能控制，E-mail：254662583@qq.com

中图分类号: TD56/67
计量
- 文章访问数: 290
- HTML全文浏览量: 27
- PDF下载量: 49
- 被引次数: 0
出版历程
- 收稿日期: 2022-05-29
- 修回日期: 2022-09-29
- 网络出版日期: 2022-08-16

Video real-time detection of bulk material accumulation on belt conveyor

1.
School of Electrical and Information Engineering, Anhui University of Science and Technology, Huainan 232001 China
2.
Huaibei Hezhong Mechanical Equipment Co., Ltd., Huaibei 235000, China

摘要

摘要: 针对非接触式散状物料堆积检测方法存在检测速度慢、在图像模糊场景下检测精度低、深度学习模型内存需求大等问题，提出了一种基于轻量化Mask−RCNN（掩码−区域卷积神经网络）的带式输送机上散状物料堆积视频实时检测方法。首先，通过暗通道先验算法对采集的图像进行预处理，以减少运输装载过程中粉尘造成的图像雾化现象，提高图像边缘特征。针对传统的Mask−RCNN的主干网络ResNet无法满足在嵌入式平台上对散状物料堆积进行实时检测的需求问题，将去雾预处理后的图像输入到基于MobileNetV2+特征金字塔网络（FPN）的主干网络中进行特征提取，生成特征图，并对主干网络进行轻量化设计，以部署在嵌入式平台上，对实时采集图像数据进行实例分割。为更精确地找到分割物体的边缘，提出了在传统Mask−RCNN的掩码分支中添加边缘损失的方法，利用全卷积网络层生成掩码，结合Scharr算子构造边缘损失函数，融合目标分类、边界框回归、语义信息得到实例分割图像。最后，通过判断散状物料堆积掩码内的像素值是否超过预设阈值实现散状物料堆积检测。实验结果表明：所提方法的模型内存需求降低到以ResNet101为主干网络的模型的1/5，经图像去雾预处理后的平均精度均值提高了8%，单张图像平均检测时间为0.56 s，检测精度可达91.8%。
- 矿用带式输送机 /
- 散状物料运输 /
- 物料堆积 /
- 视频实时检测 /
- 图像处理 /
- 暗通道先验算法 /
- 掩码−区域卷积神经网络 /
- 轻量化主干网络
Abstract: The non-contact bulk material accumulation detection method has problems, such as slow detection speed, low detection precision in image fuzzy scene, large memory requirement of deep learning model. In order to solve the above problems, a video real-time detection method of bulk material accumulation on belt conveyor based on lightweight Mask-RCNN (mask region-based convolutional neural network) is proposed. Firstly, the collected image is preprocessed by the dark channel prior algorithm to reduce the image fogging phenomenon caused by dust in the transportation and loading process and improve the image edge features. The traditional Mask-RCNN backbone network ResNet can not meet the requirement of real-time detection of bulk material accumulation on an embedded platform. In order to solve this problem, the defogging preprocessed image is input into the backbone network based on MobileNetV2 + feature pyramid network (FPN) for feature extraction. The feature graph is generated. The backbone network is designed to be lightweight. The backbone network is deployed on the embedded platform to collect image data in real-time for instance segmentation. In order to find the edge of the segmented object more accurately, a method of adding edge loss in the mask branch of traditional Mask-RCNN is proposed. The mask is generated by using full convolutional network layer. The edge loss function is constructed by combining the Scharr operator. The segmentation image is obtained by fusing object classification, bounding box regression and semantic information. Finally, the bulk material accumulation detection is realized by judging whether the pixel value in the bulk material accumulation mask exceeds a preset threshold value. The experimental results show that the memory requirement of the proposed method is reduced to 1/5 of that of the model taking ResNet 101 as the backbone network. The average precision mean value after image defogging pre-processing is increased by 8%. The average detection time of one image is 0.56 s, the detection precision can reach 91.8%.
- mine belt conveyor /
- transportation of bulk materials /
- material accumulation /
- video real-time detection /
- image processing /
- dark channel prior algorithm /
- mask region-based convolutional neural network /
- lightweight backbone network

HTML全文

图 1 散状物料堆积检测模型架构

Figure 1. Structure of detection model for bulk material accumulation

下载: 全尺寸图片幻灯片

图 2 轻量化Mask−RCNN网络结构

Figure 2. Lightweight Mask-RCNN network structure

下载: 全尺寸图片幻灯片

图 3 暗通道先验算法去雾预处理前后的图像对比

Figure 3. Comparison of images before and after defogging preprocessing by the dark channel prior algorithm

下载: 全尺寸图片幻灯片

图 4 损失函数曲线

Figure 4. Loss function curves

下载: 全尺寸图片幻灯片

图 5 Sobel算子与Scharr算子的堆煤边缘提取结果

Figure 5. Edge extraction results of coal pile of Sobel operator and Scharr operator

下载: 全尺寸图片幻灯片

图 6 是否添加边缘损失函数散装物料堆积检测对比

Figure 6. Comparison of bulk material accumulation detection whether to add edge loss function

下载: 全尺寸图片幻灯片

图 7 不同主干网络散装物料堆积检测对比

Figure 7. Comparison of bulk material accumulation detection in different backbone networks

下载: 全尺寸图片幻灯片

表 1 暗通道先验算法去雾预处理前后图像的$ {P_{{\rm{mA}}}} $对比

Table 1. Comparison of $ {P_{{\rm{mA}}}} $ values of images before and after defogging preprocessing by dark channel prior algorithm　　%

主干网络是否经过图像预处理 $ {P_{{\rm{mA}}}} $

ResNet101 否 82.6
ResNet101 是 89.1
MobileNetV2 否 79.2
MobileNetV2 是 87.3

下载: 导出CSV

表 2 不同主干网络上改进前后实例分割结果对比

Table 2. Comparison of instance segmentation results before and after improvement on different Backbones

Backbone 平均检测时间/s 模型内存/MB AP50/% AP75/%

ResNet 50 1.02 168 85.4 71.3
ResNet101 1.54 276 92.6 87.2
MobileNetV2 0.56 54 91.8 86.3

下载: 导出CSV

参考文献(18)

[1]	杜京义,陈瑞,郝乐,等. 煤矿带式输送机异物检测[J]. 工矿自动化,2021,47(8):77-83. doi: 10.13272/j.issn.1671-251x.2021040026 DU Jingyi,CHEN Rui,HAO Le,et al. Coal mine belt conveyor foreign object detection[J]. Industry and Mine Automation,2021,47(8):77-83. doi: 10.13272/j.issn.1671-251x.2021040026
[2]	游磊,朱兴林,秦伟,等. 基于曲面重建的带式输送机堆煤识别方法[J]. 工矿自动化,2021,47(6):45-50. YOU Lei,ZHU Xinglin,QIN Wei,et al. Coal stacking identification method of belt conveyor based on surface reconstruction[J]. Industry and Mine Automation,2021,47(6):45-50.
[3]	姜文涛,王梓民,张驰. 基于曲量场空间的皮带堆煤识别[J]. 传感器与微系统,2021,40(1):140-143. doi: 10.13873/J.1000-9787(2021)01-0140-04 JIANG Wentao,WANG Zimin,ZHANG Chi. Coal pile recognition based on curved space field[J]. Transducer and Microsystem Technologies,2021,40(1):140-143. doi: 10.13873/J.1000-9787(2021)01-0140-04
[4]	张涛,吴高镇. 带式输送机故障巡检机器人系统设计[J]. 工矿自动化,2018,44(10):72-76. doi: 10.13272/j.issn.1671-251x.2018060026 ZHANG Tao,WU Gaozhen. Design of fault inspection robot system for belt conveyor[J]. Industry and Mine Automation,2018,44(10):72-76. doi: 10.13272/j.issn.1671-251x.2018060026
[5]	王瀚哲. 基于图像识别的带式输送机监护系统[J]. 煤矿机械,2015,36(5):290-292. doi: 10.13436/j.mkjx.201505120 WANG Hanzhe. Monitor system of belt conveyor based on image recognition[J]. Coal Mine Machinery,2015,36(5):290-292. doi: 10.13436/j.mkjx.201505120
[6]	彭利泽. 基于视觉的传输带堆煤体积监测系统设计[D]. 太原: 中北大学, 2021. PENG Lize. Design of volume monitoring system for coal stacking in transmission belt based on vision [D]. Taiyuan: North University of China, 2021.
[7]	HE K, GKIOXARI G, DOLLAR P, et al. Mask R-CNN[C]. The IEEE International Conference on Computer Vision, Venice, 2017.
[8]	南晓虎,丁雷. 深度学习的典型目标检测算法综述[J]. 计算机应用研究,2020,37(增刊2):15-21. NAN Xiaohu,DING Lei. Review of typical target detection algorithms for deep learning[J]. Application Research of Computers,2020,37(S2):15-21.
[9]	HE K,SUN J,TANG X. Single image haze removal using dark channel prior[J]. IEEE Transactions on Pattern Analysis & Machine Intelligence,2011,33(12):2341-2353.
[10]	靳欢欢,王双亭,姚继峰,等. 暗通道先验去雾算法优化探讨[J]. 测绘科学,2015,40(11):142-145,167. doi: 10.16251/j.cnki.1009-2307.2015.11.029 JIN Huanhuan,WANG Shuangting,YAO Jifeng,et al. Optimization of haze removal based on dark channel prior[J]. Science of Surveying and Mapping,2015,40(11):142-145,167. doi: 10.16251/j.cnki.1009-2307.2015.11.029
[11]	马宇超,付华良,吴鹏,等. 深度网络自适应优化的Mask R−CNN模型在铸件表面缺陷检测中的应用研究[J]. 现代制造工程,2022(4):112-118. MA Yuchao,FU Hualiang,WU Peng,et al. Research on the Mask R-CNN model of deep network adaptive optimization in the detection of casting surface defects[J]. Modern Manufacturing Engineering,2022(4):112-118.
[12]	哈马友吉,任万春,张秤,等. 基于轻量级网络MobileNet V2的二极管玻壳缺陷识别[J]. 传感器与微系统,2022,41(4):153-155,160. HAMA Youji,REN Wanchun,ZHANG Cheng,et al. Defect recognition of diode glass shells based on lightweight network MobileNet V2[J]. Transducer and Microsystem Technologies,2022,41(4):153-155,160.
[13]	魏秀业,程海吉,贺妍,等. 基于特征融合与ResNet的行星齿轮箱故障诊断[J]. 电子测量与仪器学报,2022,36(5):213-222. doi: 10.13382/j.jemi.B2105065 WEI Xiuye,CHENG Haiji,HE Yan,et al. Fault diagnosis of planetary gearboxes based on feature fusion and ResNet[J]. Journal of Electronic Measurement and Instrumentation,2022,36(5):213-222. doi: 10.13382/j.jemi.B2105065
[14]	田枫,白欣宇,刘芳,等. 一种轻量化油田危险区域入侵检测算法[J]. 智能系统学报,2022,17(3):634-642. TIAN Feng,BAI Xinyu,LIU Fang,et al. A lightweight intrusion detection algorithm for hazardous areas in oilfields[J]. CAAI Transactions on Intelligent Systems,2022,17(3):634-642.
[15]	LIN T Y, P DOLLÁR, GIRSHICK R, et al. Feature pyramid networks for object detection[EB/OL]. [2022-03-25]. https://ui.adsabs.harvard.edu/abs/2016arXiv161203144L/abstract.
[16]	王安,王芳荣,郭柏苍,等. 基于边缘检测的视差图效果优化[J]. 计算机应用与软件,2019,36(7):236-241. doi: 10.3969/j.issn.1000-386x.2019.07.040 WANG An,WANG Fangrong,GUO Baicang,et al. Disparity map optimization based on edge detection[J]. Computer Applications and Software,2019,36(7):236-241. doi: 10.3969/j.issn.1000-386x.2019.07.040
[17]	张睿萍, 宁芊, 雷印杰, 等. 基于Mask R−CNN的生活垃圾检测[J/OL]. 计算机工程与科学: 1-8[2022-07-01]. http://www.cnki.com.cn/Article/CJFDTotal-JSJK20220308000.htm. ZHANG Ruiping, NING Qian, LEI Yinjie, et al. Garbage detection based on Mask R-CNN[J/OL]. Computer Engineering and Science: 1-8 [2022-07-01]. http://www.cnki.com.cn/Article/CJFDTotal-JSJK20220308000.htm.
[18]	李晓玲, 刘广钟, 乔大雷. 改进Mask RCNN在海面船舶实例分割中的应用[J]. 船舶工程, 2021, 43（12）: 166-171. LI Xiaoling, LIU Guangzhong, QIAO Dalei. Application of improved Mask RCNN in offshore ship instance segmentation [J]. Ship Engineering, 2021, 43 （12）: 166-171.