A fully mechanized working face inspection system based on SLAM and virtual reality
-
摘要: 针对综采工作面巡检机器人由于缺乏尺度信息导致可靠性较低的问题，将虚拟现实（VR）技术引入综采工作面巡检中，设计了基于即时定位与地图构建（SLAM）和VR的综采工作面巡检系统。该系统包括位于井下的巡检机器人子系统和位于地面的VR实时渲染子系统2个部分。巡检机器人子系统利用激光SLAM技术实现实时三维扫描，并建立三维地图，同时利用全景相机实时捕获综采工作面的场景，将实时获取的激光点云及全景视频传输到VR实时渲染子系统。VR实时渲染子系统采用GPU加速技术对激光点云进行着色，通过对Unreal三维引擎渲染部分进行定制化开发，实现对激光点云的实时渲染，并将激光点云投屏到VR眼镜上。远程操作人员通过VR眼镜实时获取现场三维场景，通过操作手柄远程控制巡检机器人动作，从而实现基于第一视角的综采工作面巡检。井下工业性试验结果表明，该系统可实现视角自由切换，并对场景进行放大，从而能够更好地观察到细节部分，精确性和可靠性更高；采用GPU加速技术进行点云着色，处理时间明显比CPU处理时间短，GPU实时性更高，整个系统的延时能满足巡检任务需求。Abstract: The reliability of the inspection robot in the fully mechanized working face is low due to the lack of scale information. In order to solve the above problem, virtual reality (VR) technology is introduced into the inspection of fully mechanized working face. A fully mechanized working face inspection system based on simultaneous localization and mapping (SLAM) and VR is designed. The system includes two parts: an inspection robot subsystem located underground and a VR real-time rendering subsystem located on the ground. The inspection robot subsystem utilizes laser SLAM technology to achieve real-time 3D scanning and establish a 3D map. At the same time, a panoramic camera is used to capture the scene of the fully mechanized working face in real-time. The real-time obtained laser point cloud and panoramic video are transmitted to the VR real-time rendering subsystem. The VR real-time rendering subsystem uses GPU acceleration technology to color laser point clouds. By customizing the rendering part of the Unreal 3D engine, real-time rendering of the laser point cloud is achieved, and the laser point cloud is projected onto the VR glasses. Remote operators obtain real-time 3D scenes through VR glasses, and remotely control the movements of the inspection robot through the operating handle. The fully mechanized working face inspection based on the first perspective is achieved. The underground industrial test results show that the system can achieve free switching of perspectives and zoom in on the scene. It enables better observation of details, with higher accuracy and reliability. By using GPU acceleration technology for point cloud coloring, the processing time is significantly shorter than CPU processing time. GPU has higher real-time performance, and the entire system's latency can meet the requirements of inspection tasks.
-
Key words:
- fully mechanized working face /
- inspection robot /
- laser SLAM /
- VR /
- GPU acceleration /
- real time rendering /
- 3D perception
-
图 1 基于SLAM和VR的综采工作面巡检系统结构
Figure 1. Structure of fully mechanized working face inspection system based on SLAM and VR
图 6 球幕坐标与经纬图坐标的变换
Figure 6. The transformation of spherical screen coordinates and latitude map coordinates
表 1 CPU与GPU处理时间对比
Table 1. Comparison of CPU and GPU processing time
scan序号 1 2 3 4 5 CPU处理
时间/s121 127 200 167 134 GPU处理
时间/s0.5 0.45 0.3 0.39 0.36 
下载: 导出CSV
-
[1] 王国法,范京道,徐亚军,等. 煤炭智能化开采关键技术创新进展与展望[J]. 工矿自动化,2018,44(2):5-12.WANG Guofa,FAN Jingdao,XU Yajun,et al. Innovation progress and prospect on key technologies of intelligent coal mining[J]. Industry and Mine Automation,2018,44(2):5-12. [2] 张守祥,张学亮,张磊,等. 综采巡检机器人关键技术研究[J]. 煤炭科学技术,2022,50(1):247-255.ZHANG Shouxiang,ZHANG Xueliang,ZHANG Lei,et al. Research on key technology of patrol robot in fully-mechanized mining face[J]. Coal Science and Technology,2022,50(1):247-255. [3] 梁占泽. 矿用带式输送机巡检机器人驱动系统设计[J]. 工矿自动化,2021,47(4):108-112. doi: 10.13272/j.issn.1671-251x.2021010003LIANG Zhanze. Driving system design of inspection robot for mine belt conveyor[J]. Industry and Mine Automation,2021,47(4):108-112. doi: 10.13272/j.issn.1671-251x.2021010003 [4] 毛浩,薛忠新,范生军,等. 张家峁煤矿回风巷道智能巡检机器人系统[J]. 煤矿安全,2021,52(7):107-111. doi: 10.13347/j.cnki.mkaq.2021.07.019MAO Hao,XUE Zhongxin,FAN Shengjun,et al. Intelligent inspection robot system for return air roadway in Zhangjiamao Coal Mine[J]. Safety in Coal Mines,2021,52(7):107-111. doi: 10.13347/j.cnki.mkaq.2021.07.019 [5] 商德勇,崔栓伟,周丹,等. 薄煤层巡检机器人行走机构跨沟性能分析及试验研究[J]. 煤炭工程,2017,49(4):136-138.SHANG Deyong,CUI Shuanwei,ZHOU Dan,et al. Ditch-crossing performance analysis and experimental study on inspection robot walking mechanism in thin coal seam[J]. Coal Engineering,2017,49(4):136-138. [6] 张树生,马静雅,岑强,等. 煤矿综采工作面巡检机器人系统研究[J]. 煤炭科学技术,2019,47(10):136-140.ZHANG Shusheng,MA Jingya,CEN Qiang,et al. Research on inspection robot system for fully-mechanized mining face in coal mine[J]. Coal Science and Technology,2019,47(10):136-140. [7] 李森,王峰,刘帅,等. 综采工作面巡检机器人关键技术研究[J]. 煤炭科学技术,2020,48(7):218-225.LI Sen,WANG Feng,LIU Shuai,et al. Study on key technology of patrol robots for fully-mechanized mining face[J]. Coal Science and Technology,2020,48(7):218-225. [8] 郝勇,袁智. 综采工作面自动巡检机器人系统设计[J]. 煤炭科学技术,2020,48(8):145-149. doi: 10.13199/j.cnki.cst.2020.08.018HAO Yong,YUAN Zhi. Design of automatic inspection robot system for fully-mechanized coal mining face[J]. Coal Science and Technology,2020,48(8):145-149. doi: 10.13199/j.cnki.cst.2020.08.018 [9] YASHIN G A, TRINITATOVA D, AGISHEV R T, et al. Aerovr: virtual reality-based teleoperation with tactile feedback for aerial manipulation[C]. The 19th International Conference on Advanced Robotics, Belo Horizonte, 2019: 767-772. [10] BARENTINE C, MCNAY A, PFAFFENBICHLER R, et al. A VR teleoperation suite with manipulation assist[C]. The 16th ACM/IEEE International Conference on Human-Robot Interaction, Boulder, 2021: 442-446. [11] GUZZI J, ABBATE G, PAOLILLO A, et al. Interacting with a conveyor belt in virtual reality using pointing gestures[C]. The 17th ACM/IEEE International Conference on Human-Robot Interaction, Sapporo, 2022: 1194-1195. [12] KENNEL-MAUSHART F, PORANNE R, COROS S. Multi-arm payload manipulation via mixed reality[C]. International Conference on Robotics and Automation, Philadelphia, 2022. DOI: 10.1109/ICRA46639.2022.9811580. [13] KUO C-Y, HUANG C-C, TSAI C-H, et al. Development of an immersive SLAM-based VR system for teleoperation of a mobile manipulator in an unknown environment[J]. Computers in Industry, 2021, 132. DOI: 10.1016/j.compind.2021.103502. [14] WONSICK M, KELEȘTEMUR T, ALT S, et al. Telemanipulation via virtual reality interfaces with enhanced environment models[C]. IEEE/RSJ International Conference on Intelligent Robots and Systems, Prague, 2021: 2999-3004. [15] 迟焕磊,袁智,曹琰,等. 基于数字孪生的智能化工作面三维监测技术研究[J]. 煤炭科学技术,2021,49(10):153-161.CHI Huanlei,YUAN Zhi,CAO Yan,et al. Study on digital twin-based smart fully-mechanized coal mining workface monitoring technology[J]. Coal Science and Technology,2021,49(10):153-161. [16] 李首滨,李森,张守祥,等. 综采工作面智能感知与智能控制关键技术与应用[J]. 煤炭科学技术,2021,49(4):28-39. doi: 10.13199/j.cnki.cst.2021.04.004LI Shoubin,LI Sen,ZHANG Shouxiang,et al. Key technology and application of intelligent perception and intelligent control in fully mechanized mining face[J]. Coal Science and Technology,2021,49(4):28-39. doi: 10.13199/j.cnki.cst.2021.04.004 [17] 杨生华,周永昌,芮丰,等. 薄煤层开采与成套装备技术的发展趋势[J]. 煤炭科学技术,2020,48(3):49-58. doi: 10.13199/j.cnki.cst.2020.03.003YANG Shenghua,ZHOU Yongchang,RUI Feng,et al. Development trend of thin coal seam mining and complete equipment technology[J]. Coal Science and Technology,2020,48(3):49-58. doi: 10.13199/j.cnki.cst.2020.03.003 [18] 亓玉浩,关士远. 基于激光SLAM的综采工作面实时三维建图方法[J]. 工矿自动化,2022,48(11):139-144. doi: 10.13272/j.issn.1671-251x.2022060047QI Yuhao,GUAN Shiyuan. Real-time 3D mapping method of fully mechanized working face based on laser SLAM[J]. Journal of Mine Automation,2022,48(11):139-144. doi: 10.13272/j.issn.1671-251x.2022060047 [19] 任伟. 综采工作面多目全景摄像仪的研制与应用[J]. 煤炭工程,2022,54(12):102-108.REN Wei. Development and application of multi view panoramic camera in fully mechanized mining face[J]. Coal Engineering,2022,54(12):102-108. [20] CHENG Xianwei, ZHAO Hui, KANDEMIR M, et al. AMOEBA: a coarse grained reconfigurable architecture for dynamic GPU scaling[C]. The 34th ACM International Conference on Supercomputing, 2020: 1-13. DOI: 10.1145/3392717.3392738. [21] CHENG Xianwei, ZHAO Hui, KANDEMIR M, et al. Alleviating bottlenecks for DNN execution on GPUs via opportunistic computing[C]. The 21st International Symposium on Quality Electronic Design, Santa Clara, 2020. DOI: 10.1109/ISQED48828.2020.9136967. [22] KIRK D B, HWU W-M W. Programming massively parallel processors[M]. 3rd ed. San Francisco: Morgan Kaufmann, 2016. -
点击查看大图
图(10) / 表(2)
计量
- 文章访问数: 165
- HTML全文浏览量: 19
- PDF下载量: 29
- 被引次数: 0
