Volume 48 Issue 8
Aug.  2022
Turn off MathJax
Article Contents
Article Navigation >  Journal of Mine Automation  > 2022  >  48(8): 33-42
YANG Yi, FU Zefeng, GAO Youjin, et al. Video object detection of the fully mechanized working face based on deep neural network[J]. Journal of Mine Automation，2022，48(8)：33-42.  doi: 10.13272/j.issn.1671-251x.2022040003
Citation: YANG Yi, FU Zefeng, GAO Youjin, et al. Video object detection of the fully mechanized working face based on deep neural network[J]. Journal of Mine Automation，2022，48(8)：33-42.  doi: 10.13272/j.issn.1671-251x.2022040003
shu

Video object detection of the fully mechanized working face based on deep neural network

doi: 10.13272/j.issn.1671-251x.2022040003
  • 1.

    School of Electrical Engineering and Automation, Henan Polytechnic University, Jiaozuo 454003, China

  • 2.

    Henan Key Laboratory of Intelligent Detection and Control of Coal Mine Equipment, Henan Polytechnic University, Jiaozuo 454003, China

  • 3.

    Zhengzhou Coal Mining Machinery Group Co., Ltd., Zhengzhou 450000, China

  • 4.

    Zhengzhou Coal Machine Hydraulic Electric Control Group Company Limited, Zhengzhou 450013, China

  • Received Date: 2022-04-01
  • Rev Recd Date: 2022-08-09
    • Available Online: 2022-08-09
    Abstract
  • The environment of the fully mechanized working face is complex. The terrain is long and narrow. The multi-object and multi-equipment often appear in the same scene, which makes object detection more difficult. At present, the object detection method applied to the underground coal mine has the problems of high difficulty in characteristic extraction, poor generalization capability, and relatively single detection object category. The existing method is mainly applied to open scenes such as a roadway, a shaft bottom station, and is rarely applied to scenes of a fully mechanized working face. In order to solve the above problems, a video object detection method based on deep neural network is proposed. Firstly, in view of the unfavorable conditions such as complex and changeable environments, uneven illumination, and much coal dust in the fully mechanized working face, the monitoring videos are selected which containing key equipment and personnel of the fully mechanized working face at various angles and under various environmental conditions. By editing, deleting and selecting, an object detection data set covering various scenes of the working face site as much as possible is produced. Secondly, the LiYOLO object detection model is constructed by lightweight improvement of YOLOv4 model. The model fully extracts video characteristics by using CSPDarknet, SPP, PANet and other enhanced characteristic extraction modules. This model uses 6-classification YoloHead for object detection, which has good robustness to the dynamic change of environment and coal dust interference in fully mechanized working face. Finally, the LiYOLO object detection model is deployed to the fully mechanized working face. While the video stream is managed by Gstreamer, TensorRT is used to accelerate the reasoning of the model, and realize the real-time detection of multi-channel video streams. Compared with the YOLOv3 and YOLOv4 models, the LiYOLO object detection model has good detection capability, and can meet the real-time and precision requirements of video object detection in the fully mechanized working face. The mean average precision on the data set of fully mechanized working face is 96.48%, the recall rate is 95%, and the frame rate of video detection can reach 67 frames/s. The engineering application results show that the LiYOLO object detection model can detect and display 6-channel videos at the same time. The model has relatively good detection effect for detection of objects in different scenes.

     

    • FullText(HTML)
  • loading
    • References(21)
  • [1]
    王国法,刘峰,庞义辉,等. 煤矿智能化−煤炭工业高质量发展的核心技术支撑[J]. 煤炭学报,2019,44(2):349-357. doi: 10.13225/j.cnki.jccs.2018.2041

    WANG Guofa,LIU Feng,PANG Yihui,et al. Coal mine intellectualization:the core technology of high quality development[J]. Journal of China Coal Society,2019,44(2):349-357. doi: 10.13225/j.cnki.jccs.2018.2041
    [2]
    高有进,杨艺,常亚军,等. 综采工作面智能化关键技术现状与展望[J]. 煤炭科学技术,2021,49(8):1-22. doi: 10.13199/j.cnki.cst.2021.08.001

    GAO Youjin,YANG Yi,CHANG Yajun,et al. Status and prospect of key technologies of intelligentization of fully mechanized coal mining face[J]. Coal Science and Technology,2021,49(8):1-22. doi: 10.13199/j.cnki.cst.2021.08.001
    [3]
    王道元,王俊,孟志斌,等. 煤矿安全风险智能分级管控与信息预警系统[J]. 煤炭科学技术,2021,49(10):136-144. doi: 10.13199/j.cnki.cst.2021.10.019

    WANG Daoyuan,WANG Jun,MENG Zhibin,et al. Intelligent hierarchical management and control and information pre-warning system of coal mine safety risk[J]. Coal Science and Technology,2021,49(10):136-144. doi: 10.13199/j.cnki.cst.2021.10.019
    [4]
    郭金刚,李化敏,王祖洸,等. 综采工作面智能化开采路径及关键技术[J]. 煤炭科学技术,2021,49(1):128-138. doi: 10.13199/j.cnki.cst.2021.01.007

    GUO Jingang,LI Huamin,WANG Zuguang,et al. Path and key technologies of intelligent mining in fully-mechanized coal mining face[J]. Coal Science and Technology,2021,49(1):128-138. doi: 10.13199/j.cnki.cst.2021.01.007
    [5]
    王国法,任怀伟,庞义辉,等. 煤矿智能化(初级阶段)技术体系研究与工程进展[J]. 煤炭科学技术,2020,48(7):1-27. doi: 10.13199/j.cnki.cst.2020.07.001

    WANG Guofa,REN Huaiwei,PANG Yihui,et al. Research and engineering progress of intelligent coal mine technical system in early stages[J]. Coal Science and Technology,2020,48(7):1-27. doi: 10.13199/j.cnki.cst.2020.07.001
    [6]
    任怀伟,孟祥军,李政,等. 8 m大采高综采工作面智能控制系统关键技术研究[J]. 煤炭科学技术,2017,45(11):37-44.

    REN Huaiwei,MENG Xiangjun,LI Zheng,et al. Study on key technology of intelligent control system applied in 8 m large mining height fully-mechanized face[J]. Coal Science and Technology,2017,45(11):37-44.
    [7]
    DALAL N, TRIGGS B. Histograms of oriented gradients for human detection [EB/OL]. （2017-02-23）[2022-02-20]. https://blog.csdn.net/yurnm/article/details/56673837.
    [8]
    LOWE D G. Distinctive image features from scale-invariant keypoints[J]. International Journal of Computer Vision,2004,60(2):91-110. doi: 10.1023/B:VISI.0000029664.99615.94
    [9]
    FELZENSZWALB P, MCALLESTER D, RAMANAN D. A discriminatively trained, multiscale, deformable part model[EB/OL]. [2022-01-20]. https://ieeexplore.ieee.org/document/4587597/footnotes#footnotes.
    [10]
    孙继平,贾倪. 矿井视频图像中人员目标匹配与跟踪方法[J]. 中国矿业大学学报,2015,44(3):540-548. doi: 10.13247/j.cnki.jcumt.000264

    SUN Jiping,JIA Ni. Human target matching and tracking method in coal mine video[J]. Journal of China University of Mining & Technology,2015,44(3):540-548. doi: 10.13247/j.cnki.jcumt.000264
    [11]
    徐美华,龚露鸣,郭爱英,等. 基于自适应CtF DPM特征提取的快速行人检测模型[J]. 复旦大学学报(自然科学版),2018,57(4):453-461.

    XU Meihua,GONG Luming,GUO Aiying,et al. A fast pedestrian detection model based on adaptive CtF DPM feature extraction[J]. Journal of Fudan University(Natural Science),2018,57(4):453-461.
    [12]
    张银萍. 煤矿地面轨道运输环境感知系统研究[D]. 徐州: 中国矿业大学, 2020.

    ZHANG Yinping. Study on environmental perception system of coal mine ground rail transportation[D]. Xuzhou: China University of Mining and Technology, 2020.
    [13]
    卢万杰,付华,赵洪瑞. 基于深度学习算法的矿用巡检机器人设备识别[J]. 工程设计学报,2019,26(5):527-533. doi: 10.3785/j.issn.1006-754X.2019.05.005

    LU Wanjie,FU Hua,ZHAO Hongrui,et al. Equipment recognition of mining patrol robot based on deep learning algorithm[J]. Chinese Journal of Engineering Design,2019,26(5):527-533. doi: 10.3785/j.issn.1006-754X.2019.05.005
    [14]
    林俊,党伟超,潘理虎,等. 基于计算机视觉的井下输送带跑偏检测方法[J]. 煤矿机械,2019,40(10):169-171. doi: 10.13436/j.mkjx.201910057

    LIN Jun,DANG Weichao,PAN Lihu,et al. Deviation monitoring method of underground conveyor belt based on computer vision[J]. Coal Mine Machinery,2019,40(10):169-171. doi: 10.13436/j.mkjx.201910057
    [15]
    董昕宇,师杰,张国英. 基于参数轻量化的井下人体实时检测算法[J]. 工矿自动化,2021,47(6):71-78. doi: 10.13272/j.issn.1671-251x.2021010035

    DONG Xinyu,SHI Jie,ZHANG Guoying. Real-time detection algorithm of underground human body based on lightweight parameters[J]. Industry and Mine Automation,2021,47(6):71-78. doi: 10.13272/j.issn.1671-251x.2021010035
    [16]
    南柄飞, 郭志杰, 王凯, 等. 基于视觉显著性的煤矿井下关键目标对象实时感知研究[J/OL]. 煤炭科学技术: 1-11[2022-07-15]. http://kns.cnki.net/kcms/detail/11.2402.TD.20210512.1304.004.html.

    NAN Bingfei, GUO Zhijie, WANG Kai, et al. Real-time perception method of target ROI in coal mine underground based on visual saliency[J/OL]. Coal Science and Technology: 1-11[2022-07-15]. http://kns.cnki.net/kcms/detail/11.2402.TD.20210512.1304.004.html.
    [17]
    韩江洪,沈露露,卫星,等. 基于轻量级CNN的井下视觉识别策略[J]. 合肥工业大学学报(自然科学版),2020,43(11):1469-1475,1562.

    HAN Jianghong,SHEN Lulu,WEI Xing,et al. Downhole visual recognition strategy based on lightweight CNN[J]. Journal of Hefei University of Technology(Natural Science),2020,43(11):1469-1475,1562.
    [18]
    BOCHKOVSKIY A, WANG C Y, LIAO H Y M. YOLOv4: optimal speed and accuracy of object detection[EB/OL]. [2022-01-20]. https://doi.org/10.48550/arXiv.2004.10934.
    [19]
    REDMON J, FARHADI A. YOLO9000: better, faster, stronger[EB/OL]. [2022-01-22]. https://wenku.baidu.com/view/d74b46407b3e0912a21614791711cc7931b778d6.html.
    [20]
    HE Kaiming,ZHANG Xiangyu,REN Shaoqing,et al. Spatial pyramid pooling in deep convolutional networks for visual recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence,2014,37(9):1904-1916.
    [21]
    LIU S, QI L, QIN H F, et al. Path aggregation network for instance segmentation[EB/OL]. [2022-01-15]. https://ieeexplore.ieee.org/document/8579011.
    • Relative Articles
    • Supplements(0)
    • Cited By
  • 加载中

Catalog

    通讯作者: 陈斌, bchen63@163.com
    • 1. 

      沈阳化工大学材料科学与工程学院 沈阳 110142

    1. 本站搜索
    2. 百度学术搜索
    3. 万方数据库搜索
    4. CNKI搜索

    Figures(10)  / Tables(5)

    Get Citation
    PDF
    XML

    Article Metrics

    Article views (246) PDF downloads(54) Cited by()
    Proportional views
    Related

    苏ICP备 05016349号-2

    Supported by: Beijing Renhe Information Technology Co. Ltd.  Visits：    

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return