留言板

尊敬的读者、作者、审稿人, 关于本刊的投稿、审稿、编辑和出版的任何问题, 您可以本页添加留言。我们将尽快给您答复。谢谢您的支持!

姓名
邮箱
手机号码
标题
留言内容
验证码

基于改进YOLOv5s+DeepSORT的煤块行为异常识别

张旭辉 闫建星 张超 万继成 王利欣 胡成军 王力 王东

张旭辉,闫建星,张超,等. 基于改进YOLOv5s+DeepSORT的煤块行为异常识别[J]. 工矿自动化,2022,48(6):77-86, 117.  doi: 10.13272/j.issn.1671-251x.17915
引用本文: 张旭辉,闫建星,张超,等. 基于改进YOLOv5s+DeepSORT的煤块行为异常识别[J]. 工矿自动化,2022,48(6):77-86, 117.  doi: 10.13272/j.issn.1671-251x.17915
ZHANG Xuhui, YAN Jianxing, ZHANG Chao, et al. Coal block abnormal behavior identification based on improved YOLOv5s + DeepSORT[J]. Journal of Mine Automation,2022,48(6):77-86, 117.  doi: 10.13272/j.issn.1671-251x.17915
Citation: ZHANG Xuhui, YAN Jianxing, ZHANG Chao, et al. Coal block abnormal behavior identification based on improved YOLOv5s + DeepSORT[J]. Journal of Mine Automation,2022,48(6):77-86, 117.  doi: 10.13272/j.issn.1671-251x.17915

基于改进YOLOv5s+DeepSORT的煤块行为异常识别

doi: 10.13272/j.issn.1671-251x.17915
基金项目: 国家自然科学基金项目(51834006);陕西省重点研发计划项目(2018ZDCXL-GY-06-04)。
详细信息
    作者简介:

    张旭辉(1972—),男,陕西凤翔人,教授,博士,博士研究生导师,研究方向为煤矿机电设备智能检测与控制,E-mail:zhangxh@xust.edu.cn

    通讯作者:

    闫建星(1995—),男,陕西榆林人,硕士研究生,研究方向为智能检测与控制,E-mail:yanjianxing2013@163.com

  • 中图分类号: TD76

Coal block abnormal behavior identification based on improved YOLOv5s + DeepSORT

  • 摘要: 煤块检测方法主要包括传统图像检测方法和深度学习目标检测方法。传统图像检测方法检测精度不高、实时性较差、无法对堆煤进行准确判断;深度学习目标检测方法虽然可以实现实时检测,但没有对煤块的数量、滞留和堵塞状态进行识别,而且识别模型参数较多。针对上述问题,提出了一种基于改进YOLOv5s+DeepSORT的煤块行为异常识别方法。首先通过摄像头和巡检机器人采集煤矿综采工作面带式输送机上煤块视频图像,并制作数据集。然后利用MobileNetV3_YOLOv5s_AF−FPN模型进行煤块图像目标检测:通过MobileNetV3替换原始YOLOv5s主干特征提取网络,减少参数量,提高推理速度;将YOLOv5s中原有的特征金字塔网络改进为增强特征金字塔网络(AF−FPN),以提高YOLOv5s网络对多尺度煤块目标的检测性能。利用DeepSORT进行煤块多目标跟踪:将改进YOLOv5s模型检测后的煤块图像作为DeepSORT的输入进行多目标跟踪,利用DeepSORT对煤块进行状态估计、数据关联匹配和跟踪器参数更新,确定跟踪结果,并对连续跟踪的煤块进行ID编码,对当前帧的煤块数量进行计数。最后在目标跟踪器中取出连续跟踪的目标,设置距离阈值,判断其是否滞留;设置数量阈值,判断其是否堵塞,最终实现煤块滞留和堵塞行为异常识别。利用自建dkm_data2021数据集对基于改进YOLOv5s+DeepSORT的煤块行为异常识别方法的可靠性进行实验验证,结果表明:改进YOLOv5s模型相比YOLOv5s模型平均检测精度提高了1.45%,参数量减少了35.3%,推理加速了12.7%,平均漏检率降低了11.08%,平均误检率降低了11.54%;基于改进YOLOv5s+DeepSORT的煤块行为异常识别方法检测精度为80.1%,可准确识别煤块滞留、堵塞状态,验证了该方法的可靠性。

     

  • 图  1  YOLOv5s网络结构

    Figure  1.  YOLOv5s network structure

    图  2  改进后的YOLOv5s网络结构

    Figure  2.  Improved YOLOv5s network structure

    图  3  Bneck网络结构

    Figure  3.  Bneck network structure

    图  4  MobileNetV3_YOLOv5s_AF−FPN网络结构

    Figure  4.  MobileNetV3_YOLOv5s_AF-FPN network structure

    图  5  AAM网络结构

    Figure  5.  AAM network structure

    图  6  FEM网络结构

    Figure  6.  FEM network structure

    图  7  煤块行为异常识别方法流程

    Figure  7.  Flow of coal block abnormal behavior identification method

    图  8  DeepSORT多目标跟踪算法流程

    Figure  8.  DeepSORT multi-target tracking algorithm flow

    图  9  YOLOv5s模型与MobileNetV3_YOLOv5s_AF−FPN模型的煤块检测效果对比

    Figure  9.  Comparison of coal detection effect of YOLOv5s model and MobileNetV3_YOLOv5s_AF-FPN model

    图  10  煤块正常跟踪

    Figure  10.  Coal block normal tracking

    图  12  煤块堵塞

    Figure  12.  Coal block blockage

    图  11  煤块滞留

    Figure  11.  Coal block retention

    表  1  MobileNetV3_Large结构

    Table  1.   MobileNetV3_Large structure

    Input ShapeOperatorSEAFStride
    2242×3Conv2dHS2
    1122×16Bneck,3×3RE1
    1122×16Bneck,3×3RE2
    562×24Bneck,3×3RE1
    562×24Bneck,3×3RE2
    282×40Bneck,3×3RE1
    282×40Bneck,3×3RE1
    282×40Bneck,3×3HS2
    142×80Bneck,3×3HS1
    142×80Bneck,3×3HS1
    142×80Bneck,3×3HS1
    142×80Bneck,3×3HS1
    142×112Bneck,3×3HS1
    142×112Bneck,5×5HS1
    72×160Bneck,5×5HS2
    72×160Bneck,5×5HS1
    72×160Conv2d,1×1HS1
    72×160Pool, 7×71
    12×960Conv2d,1×1HS1
    12×1280Conv2d,1×11
    下载: 导出CSV

    表  2  特征提取网络实验对比

    Table  2.   Comparison of feature extraction network experiments

    模型召回率平均精度参数量/M平均漏检率平均误检率推理时间/ms
    YOLOv5s0.7850.8217.090.3340.02618.9
    MobileNetV3_ YOLOv5s0.7660.7953.560.3650.02715.0
    下载: 导出CSV

    表  3  特征融合网络实验对比

    Table  3.   Comparison of feature fusion network experiments

    模型召回率平均精度参数量/M平均漏检率平均误检率推理时间/ms
    YOLOv5s0.7850.8297.090.3340.02618.9
    YOLOv5s_AF−FPN0.8240.8708.120.3650.02715.0
    MobileNetV3_YOLOv5s0.7660.7953.560.3650.02715.0
    MobileNetV3_YOLOv5s_AF−FPN0.8100.8414.590.2970.02316.5
    下载: 导出CSV

    表  4  多目标跟踪结果对比

    Table  4.   Comparison of multi-target tracking results

    模型MOTA/%MOTP/%漏检数误检数推理速度/
    (帧·s−1
    YOLOv5s+DeepSORT60.576.51195734
    MobileNetV3_YOLOv5s_
    AF−FPN+DeepSORT
    63.480.1954240
    下载: 导出CSV
  • [1] 王国法,赵国瑞,任怀伟. 智慧煤矿与智能化开采关键核心技术分析[J]. 煤炭学报,2019,44(1):34-41. doi: 10.13225/j.cnki.jccs.2018.5034

    WANG Guofa,ZHAO Guorui,REN Huaiwei. Analysis on key technologies of intelligent coal mine and intelligent mining[J]. Journal of China Coal Society,2019,44(1):34-41. doi: 10.13225/j.cnki.jccs.2018.5034
    [2] 王国法,徐亚军,张金虎,等. 煤矿智能化开采新进展[J]. 煤炭科学技术,2021,49(1):1-10.

    WANG Guofa,XU Yajun,ZHANG Jinhu,et al. New development of intelligent mining in coal mines[J]. Coal Science and Technology,2021,49(1):1-10.
    [3] 张渤,谢金辰,张后斌. 矿井下输送带大块物体检测[J]. 煤炭技术,2021,40(4):154-156. doi: 10.13301/j.cnki.ct.2021.04.046

    ZHANG Bo,XIE Jinchen,ZHANG Houbin. Detection of large objects in transportation belt under mine[J]. Coal Technology,2021,40(4):154-156. doi: 10.13301/j.cnki.ct.2021.04.046
    [4] 许军,吕俊杰,杨娟利,等. 基于图像处理的溜槽堆煤预警研究[J]. 煤炭技术,2017,36(12):232-234.

    XU Jun,LYU Junjie,YANG Juanli,et al. Research on early warning of coal chute blocking based on image processing[J]. Coal Technology,2017,36(12):232-234.
    [5] 张立亚. 矿山智能视频分析与预警系统研究[J]. 工矿自动化,2017,43(11):16-20.

    ZHANG Liya. Research on intelligent video analysis and early warning system for mine[J]. Industry and Mine Automation,2017,43(11):16-20.
    [6] 吴帅,徐勇,赵东宁. 基于深度卷积网络的目标检测综述[J]. 模式识别与人工智能,2018,31(4):335-346. doi: 10.16451/j.cnki.issn1003-6059.201804005

    WU Shuai,XU Yong,ZHAO Dongning. Survey of object detection based on deep convolutional networks[J]. Pattern Recognition and Artificial Intelligence,2018,31(4):335-346. doi: 10.16451/j.cnki.issn1003-6059.201804005
    [7] 管皓,薛向阳,安志勇. 深度学习在视频目标跟踪中的应用进展与展望[J]. 自动化学报,2016,42(6):834-847. doi: 10.16383/j.aas.2016.c150705

    GUAN Hao,XUE Xiangyang,AN Zhiyong. Advances on application of deep learning for video object tracking[J]. Acta Automatica Sinica,2016,42(6):834-847. doi: 10.16383/j.aas.2016.c150705
    [8] 罗海波,许凌云,惠斌,等. 基于深度学习的目标跟踪方法研究现状与展望[J]. 红外与激光工程,2017,46(5):14-20.

    LUO Haibo,XU Lingyun,HUI Bin,et al. Status and prospect of target tracking based on deep learning[J]. Infrared and Laser Engineering,2017,46(5):14-20.
    [9] 南柄飞, 郭志杰, 王凯, 等. 基于视觉显著性的煤矿井下关键目标对象实时感知研究[J/OL]. 煤炭科学技术: 1-11[2022-01-12]. https://kns.cnki. net/kcms/detail/11.2402.TD.20210512.1304.004.html.

    NAN Bingfei, GUO Zhijie, WANG Kai, et al. Real-time method of target ROI in coal mine underground based on visual saliency [J/OL]. Coal Science and Technology: 1-11[2022-01-12]. https://kns.cnki.net/kcms/detail/11.2402.TD.20210512.1304.004.html.
    [10] 杜京义,郝乐,王悦阳,等. 一种煤矿井下输煤大块物检测方法[J]. 工矿自动化,2020,46(5):63-68.

    DU Jingyi,HAO Le,WANG Yueyang,et al. A detection method for large blocks in underground coal transportation[J]. Industry and Mine Automation,2020,46(5):63-68.
    [11] WANG Yujing,WANG Yuanbin,DANG Langfei. Video detection of foreign objects on the surface of belt conveyor underground coal mine based on improved SSD[J]. Journal of Ambient Intelligence and Humanized Computing,2020,46(7):1-10.
    [12] 胡璟皓,高妍,张红娟,等. 基于深度学习的带式输送机非煤异物识别方法[J]. 工矿自动化,2021,47(6):57-62,90. doi: 10.13272/j.issn.1671-251x.2021020041

    HU Jinghao,GAO Yan,ZHANG Hongjuan,et al. Recognition method of non-coal foreign objects of belt conveyor based on deep learning[J]. Industry and Mine Automation,2021,47(6):57-62,90. doi: 10.13272/j.issn.1671-251x.2021020041
    [13] 叶鸥,窦晓熠,付燕,等. 融合轻量级网络和双重注意力机制的煤块检测方法[J]. 工矿自动化,2021,47(12):75-80.

    YE Ou,DOU Xiaoyi,FU Yan,et al. Coal block detection method integrating lightweight network and dual attention mechanism[J]. Industry and Mine Automation,2021,47(12):75-80.
    [14] 张伟,庄幸涛,王雪力,等. DS−YOLO:一种部署在无人机终端上的小目标实时检测算法[J]. 南京邮电大学学报(自然科学版),2021,41(1):86-98.

    ZHANG Wei,ZHUANG Xingtao,WANG Xueli,et al. DS-YOLO:a real-time small object detection algorithm on UAVs[J]. Journal of Nanjing University of Posts and Telecommunications ( Natural Science Edition),2021,41(1):86-98.
    [15] MITTAL S. A survey on optimized implementation of deep learning models on the NVIDIA Jetson platform[J]. Journal of Systems Architecture,2019,97:428-442. doi: 10.1016/j.sysarc.2019.01.011
    [16] WANG Junfan,CHEN Yi,GAO Mingyu,et al. Improved YOLOv5 network for real-time multi-scale traffic sign detection[J]. IEEE Sensors Journal,2021,38(8):1724-1733.
    [17] WANG Chenyao, LIAO Hongyuan, WU Yuehua, et al. CSPNet: a new backbone that can enhance learning capability of CNN[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Seattle, 2020: 390-391.
    [18] HE Kaiming,ZHANG Xingyu,REN Shaoqing,et al. Spatial pyramid pooling in deep convolutional networks for visual recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence,2015,37(9):1904-1916. doi: 10.1109/TPAMI.2015.2389824
    [19] LIU Shu, QI Lu, QIN Haifang, et al. Path aggregation network for instance segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018: 8759-8768.
    [20] BEWLEY A, GE Z, OTT L, et al. Simple online and real time tracking[C]//2016 IEEE International Conference on Image Processing ( ICIP), Phoenix, 2016: 3464-3468.
  • 加载中
图(12) / 表(4)
计量
  • 文章访问数:  43
  • HTML全文浏览量:  4
  • PDF下载量:  15
  • 被引次数: 0
出版历程
  • 收稿日期:  2022-04-12
  • 修回日期:  2022-06-10
  • 网络出版日期:  2022-06-28

目录

    /

    返回文章
    返回