留言板

尊敬的读者、作者、审稿人, 关于本刊的投稿、审稿、编辑和出版的任何问题, 您可以本页添加留言。我们将尽快给您答复。谢谢您的支持!

姓名
邮箱
手机号码
标题
留言内容
验证码

基于深度神经网络的综采工作面视频目标检测

杨艺 付泽峰 高有进 崔科飞 王科平

杨艺,付泽峰,高有进,等. 基于深度神经网络的综采工作面视频目标检测[J]. 工矿自动化,2022,48(8):33-42.  doi: 10.13272/j.issn.1671-251x.2022040003
引用本文: 杨艺,付泽峰,高有进,等. 基于深度神经网络的综采工作面视频目标检测[J]. 工矿自动化,2022,48(8):33-42.  doi: 10.13272/j.issn.1671-251x.2022040003
YANG Yi, FU Zefeng, GAO Youjin, et al. Video object detection of the fully mechanized working face based on deep neural network[J]. Journal of Mine Automation,2022,48(8):33-42.  doi: 10.13272/j.issn.1671-251x.2022040003
Citation: YANG Yi, FU Zefeng, GAO Youjin, et al. Video object detection of the fully mechanized working face based on deep neural network[J]. Journal of Mine Automation,2022,48(8):33-42.  doi: 10.13272/j.issn.1671-251x.2022040003

基于深度神经网络的综采工作面视频目标检测

doi: 10.13272/j.issn.1671-251x.2022040003
基金项目: 河南省科技攻关计划项目(212102210390);河南省煤矿智能开采技术创新中心支撑项目(2021YD01)。
详细信息
    作者简介:

    杨艺(1980-),男,湖北利川人,副教授,博士,主要研究方向为深度学习、强化学习和智能控制,E-mail:yangyi@hpu.edu.cn

    通讯作者:

    付泽峰(1995-),男,江西抚州人,硕士研究生,主要研究方向为信息处理与网络控制,E-mail:18864770547@163.com

  • 中图分类号: TD67

Video object detection of the fully mechanized working face based on deep neural network

  • 摘要: 综采工作面环境较复杂,地形狭长,多目标多设备经常出现在同一场景当中,使得目标检测难度加大。目前应用于煤矿井下的目标检测方法存在特征提取难度较大、泛化能力较差、检测目标类别较为单一等问题,且主要应用于巷道、井底车场等较为空旷场景,较少应用于综采工作面场景。针对上述问题,提出了一种基于深度神经网络的综采工作面视频目标检测方法。首先,针对综采工作面环境复杂多变、光照不均、煤尘大等不利条件,针对性挑选包含各角度、各环境条件下的综采工作面关键设备和人员的监控视频,并进行剪辑、删选,制作尽可能涵盖工作面现场各类场景的目标检测数据集。然后,通过对 YOLOv4模型进行轻量化改进,构建了LiYOLO目标检测模型。该模型利用CSPDarknet、SPP、PANet等加强特征提取模块对视频特征进行充分提取,使用6分类YoloHead进行目标检测,对综采工作面环境动态变化、煤尘干扰等具有较好的鲁棒性。最后,将LiYOLO目标检测模型部署到综采工作面,应用Gstreamer对视频流进行管理,同时使用TensorRT对模型进行推理加速,实现了多路视频流的实时检测。与YOLOv3、YOLOv4模型相比,LiYOLO目标检测模型具有良好的检测能力,能够满足综采工作面视频目标检测的实时性和精度要求,在综采工作面数据集上的平均准确率均值为96.48%,召回率为95%,同时视频检测帧率达67帧/s。工程应用效果表明,LiYOLO目标检测模型可同时检测、展示6路视频,且对于不同场景下的检测目标都有较好的检测效果。

     

  • 图  1  综采工作面视频目标检测流程

    Figure  1.  Flow of video object detection in fully mechanized working face

    图  2  不同条件下的综采工作面图像

    Figure  2.  Images of fully mechanized working face under different conditions

    图  3  数据集标注示例

    Figure  3.  Example of dataset annotation

    图  4  LiYOLO模型结构

    Figure  4.  LiYOLO model structure

    图  5  改进前后 的YoloHead

    Figure  5.  YoloHead before and after improved

    图  6  YOLOv4模型的mAP和损失变化曲线

    Figure  6.  mAP and loss variation curves of YOLOv4 model

    图  7  LiYOLO模型的mAP和损失变化曲线

    Figure  7.  mAP and loss variation curves of LiYOLO model

    图  8  3种模型对不同场景下设备及行人的检测效果

    Figure  8.  Detection effect of three models for devices and pedestrians in different scenes

    图  9  LiYOLO模型工程部署过程

    Figure  9.  Project deployment process of LiYOLO model

    图  10  多路视频检测效果

    Figure  10.  Multi-video detection effect

    表  1  不同条件下的图像采集数量

    Table  1.   Number of image samples under different conditions

    位置环境图像采集数量/张
    无尘轻微严重
    端头顺光4 5242 2632 263
    逆光1 132565565
    中部顺光22 62211 31311 313
    逆光5 6572 8282 828
    下载: 导出CSV

    表  2  标签分类

    Table  2.   Classification of labels

    序号标签名称序号标签名称
    1Groove(线槽) 4Roller(滚筒)
    2Conveyer(刮板输送机)5Person(人)
    3Shearer(采煤机)6face_guard(护帮板)
    下载: 导出CSV

    表  3  主要实验结果对比

    Table  3.   Comparison of main experimental results %

    模型mAPRecall
    YOLOv481.6990
    LiYOLO96.4895
    下载: 导出CSV

    表  4  检测时间

    Table  4.   Detection time

    模型检测时间/ms传输帧率/(帧·s−1
    YOLOv329.927.9
    YOLOv416.259.1
    LiYOLO16.166.8
    下载: 导出CSV

    表  5  未加速与加速后模型FPS对比

    Table  5.   Comparison of FPS between the unaccelerated model and the accelerated model 帧/s

    未加速FPS加速后FPS
    1路1路4路6路
    55.285.420.8×413.9×6
    下载: 导出CSV
  • [1] 王国法,刘峰,庞义辉,等. 煤矿智能化−煤炭工业高质量发展的核心技术支撑[J]. 煤炭学报,2019,44(2):349-357. doi: 10.13225/j.cnki.jccs.2018.2041

    WANG Guofa,LIU Feng,PANG Yihui,et al. Coal mine intellectualization:the core technology of high quality development[J]. Journal of China Coal Society,2019,44(2):349-357. doi: 10.13225/j.cnki.jccs.2018.2041
    [2] 高有进,杨艺,常亚军,等. 综采工作面智能化关键技术现状与展望[J]. 煤炭科学技术,2021,49(8):1-22. doi: 10.13199/j.cnki.cst.2021.08.001

    GAO Youjin,YANG Yi,CHANG Yajun,et al. Status and prospect of key technologies of intelligentization of fully mechanized coal mining face[J]. Coal Science and Technology,2021,49(8):1-22. doi: 10.13199/j.cnki.cst.2021.08.001
    [3] 王道元,王俊,孟志斌,等. 煤矿安全风险智能分级管控与信息预警系统[J]. 煤炭科学技术,2021,49(10):136-144. doi: 10.13199/j.cnki.cst.2021.10.019

    WANG Daoyuan,WANG Jun,MENG Zhibin,et al. Intelligent hierarchical management and control and information pre-warning system of coal mine safety risk[J]. Coal Science and Technology,2021,49(10):136-144. doi: 10.13199/j.cnki.cst.2021.10.019
    [4] 郭金刚,李化敏,王祖洸,等. 综采工作面智能化开采路径及关键技术[J]. 煤炭科学技术,2021,49(1):128-138. doi: 10.13199/j.cnki.cst.2021.01.007

    GUO Jingang,LI Huamin,WANG Zuguang,et al. Path and key technologies of intelligent mining in fully-mechanized coal mining face[J]. Coal Science and Technology,2021,49(1):128-138. doi: 10.13199/j.cnki.cst.2021.01.007
    [5] 王国法,任怀伟,庞义辉,等. 煤矿智能化(初级阶段)技术体系研究与工程进展[J]. 煤炭科学技术,2020,48(7):1-27. doi: 10.13199/j.cnki.cst.2020.07.001

    WANG Guofa,REN Huaiwei,PANG Yihui,et al. Research and engineering progress of intelligent coal mine technical system in early stages[J]. Coal Science and Technology,2020,48(7):1-27. doi: 10.13199/j.cnki.cst.2020.07.001
    [6] 任怀伟,孟祥军,李政,等. 8 m大采高综采工作面智能控制系统关键技术研究[J]. 煤炭科学技术,2017,45(11):37-44.

    REN Huaiwei,MENG Xiangjun,LI Zheng,et al. Study on key technology of intelligent control system applied in 8 m large mining height fully-mechanized face[J]. Coal Science and Technology,2017,45(11):37-44.
    [7] DALAL N, TRIGGS B. Histograms of oriented gradients for human detection [EB/OL]. (2017-02-23)[2022-02-20]. https://blog.csdn.net/yurnm/article/details/56673837.
    [8] LOWE D G. Distinctive image features from scale-invariant keypoints[J]. International Journal of Computer Vision,2004,60(2):91-110. doi: 10.1023/B:VISI.0000029664.99615.94
    [9] FELZENSZWALB P, MCALLESTER D, RAMANAN D. A discriminatively trained, multiscale, deformable part model[EB/OL]. [2022-01-20]. https://ieeexplore.ieee.org/document/4587597/footnotes#footnotes.
    [10] 孙继平,贾倪. 矿井视频图像中人员目标匹配与跟踪方法[J]. 中国矿业大学学报,2015,44(3):540-548. doi: 10.13247/j.cnki.jcumt.000264

    SUN Jiping,JIA Ni. Human target matching and tracking method in coal mine video[J]. Journal of China University of Mining & Technology,2015,44(3):540-548. doi: 10.13247/j.cnki.jcumt.000264
    [11] 徐美华,龚露鸣,郭爱英,等. 基于自适应CtF DPM特征提取的快速行人检测模型[J]. 复旦大学学报(自然科学版),2018,57(4):453-461.

    XU Meihua,GONG Luming,GUO Aiying,et al. A fast pedestrian detection model based on adaptive CtF DPM feature extraction[J]. Journal of Fudan University(Natural Science),2018,57(4):453-461.
    [12] 张银萍. 煤矿地面轨道运输环境感知系统研究[D]. 徐州: 中国矿业大学, 2020.

    ZHANG Yinping. Study on environmental perception system of coal mine ground rail transportation[D]. Xuzhou: China University of Mining and Technology, 2020.
    [13] 卢万杰,付华,赵洪瑞. 基于深度学习算法的矿用巡检机器人设备识别[J]. 工程设计学报,2019,26(5):527-533. doi: 10.3785/j.issn.1006-754X.2019.05.005

    LU Wanjie,FU Hua,ZHAO Hongrui,et al. Equipment recognition of mining patrol robot based on deep learning algorithm[J]. Chinese Journal of Engineering Design,2019,26(5):527-533. doi: 10.3785/j.issn.1006-754X.2019.05.005
    [14] 林俊,党伟超,潘理虎,等. 基于计算机视觉的井下输送带跑偏检测方法[J]. 煤矿机械,2019,40(10):169-171. doi: 10.13436/j.mkjx.201910057

    LIN Jun,DANG Weichao,PAN Lihu,et al. Deviation monitoring method of underground conveyor belt based on computer vision[J]. Coal Mine Machinery,2019,40(10):169-171. doi: 10.13436/j.mkjx.201910057
    [15] 董昕宇,师杰,张国英. 基于参数轻量化的井下人体实时检测算法[J]. 工矿自动化,2021,47(6):71-78. doi: 10.13272/j.issn.1671-251x.2021010035

    DONG Xinyu,SHI Jie,ZHANG Guoying. Real-time detection algorithm of underground human body based on lightweight parameters[J]. Industry and Mine Automation,2021,47(6):71-78. doi: 10.13272/j.issn.1671-251x.2021010035
    [16] 南柄飞, 郭志杰, 王凯, 等. 基于视觉显著性的煤矿井下关键目标对象实时感知研究[J/OL]. 煤炭科学技术: 1-11[2022-07-15]. http://kns.cnki.net/kcms/detail/11.2402.TD.20210512.1304.004.html.

    NAN Bingfei, GUO Zhijie, WANG Kai, et al. Real-time perception method of target ROI in coal mine underground based on visual saliency[J/OL]. Coal Science and Technology: 1-11[2022-07-15]. http://kns.cnki.net/kcms/detail/11.2402.TD.20210512.1304.004.html.
    [17] 韩江洪,沈露露,卫星,等. 基于轻量级CNN的井下视觉识别策略[J]. 合肥工业大学学报(自然科学版),2020,43(11):1469-1475,1562.

    HAN Jianghong,SHEN Lulu,WEI Xing,et al. Downhole visual recognition strategy based on lightweight CNN[J]. Journal of Hefei University of Technology(Natural Science),2020,43(11):1469-1475,1562.
    [18] BOCHKOVSKIY A, WANG C Y, LIAO H Y M. YOLOv4: optimal speed and accuracy of object detection[EB/OL]. [2022-01-20]. https://doi.org/10.48550/arXiv.2004.10934.
    [19] REDMON J, FARHADI A. YOLO9000: better, faster, stronger[EB/OL]. [2022-01-22]. https://wenku.baidu.com/view/d74b46407b3e0912a21614791711cc7931b778d6.html.
    [20] HE Kaiming,ZHANG Xiangyu,REN Shaoqing,et al. Spatial pyramid pooling in deep convolutional networks for visual recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence,2014,37(9):1904-1916.
    [21] LIU S, QI L, QIN H F, et al. Path aggregation network for instance segmentation[EB/OL]. [2022-01-15]. https://ieeexplore.ieee.org/document/8579011.
  • 加载中
图(10) / 表(5)
计量
  • 文章访问数:  241
  • HTML全文浏览量:  23
  • PDF下载量:  54
  • 被引次数: 0
出版历程
  • 收稿日期:  2022-04-01
  • 修回日期:  2022-08-09
  • 网络出版日期:  2022-08-09

目录

    /

    返回文章
    返回