YANG Jiajia, ZHANG Chuanwei, ZHOU Libing, et al. Research on the targetless automatic calibration method for mining LiDAR and camera[J]. Journal of Mine Automation,2024,50(10):53-61, 89. DOI: 10.13272/j.issn.1671-251x.2024070056
Citation: YANG Jiajia, ZHANG Chuanwei, ZHOU Libing, et al. Research on the targetless automatic calibration method for mining LiDAR and camera[J]. Journal of Mine Automation,2024,50(10):53-61, 89. DOI: 10.13272/j.issn.1671-251x.2024070056

Research on the targetless automatic calibration method for mining LiDAR and camera

More Information
  • Received Date: July 15, 2024
  • Revised Date: October 07, 2024
  • Available Online: August 15, 2024
  • The realization of autonomous driving for mining vehicles relies on accurate environmental perception, and the combination of LiDAR and cameras can provide richer and more accurate environmental information. To ensure effective fusion of LiDAR and cameras, external parameter calibration is necessary. Currently, most mining intrinsically safe onboard LiDARs are 16-line LiDARs, which generate relatively sparse point clouds. To address this issue, this paper proposed a targetless automatic calibration method for mining LiDAR and camera. Multi-frame point cloud fusion was utilized to obtain fused frame point clouds, increasing point cloud density and enriching point cloud information. Then, effective targets such as vehicles and traffic signs in the scene were extracted using panoramic segmentation. By establishing a corresponding relationship between the centroids of 2D and 3D effective targets, a coarse calibration was completed. In the fine calibration process, the effective target point clouds were projected onto the segmentation mask after inverse distance transformation using the coarse-calibrated external parameters, constructing an objective function based on the matching degree of effective target panoramic information. The optimal external parameters were obtained by maximizing the objective function using a particle swarm algorithm. The effectiveness of the method was validated from three aspects: quantitative, qualitative, and ablation experiments. ① In the quantitative experiments, the translation error was 0.055 m, and the rotation error was 0.394°. Compared with the method based on semantic segmentation technology, the translation error was reduced by 43.88%, and the rotation error was reduced by 48.63%. ② The qualitative results showed that the projection effects in the garage and mining area scenes were highly consistent with the true values of the external parameters, demonstrating the stability of the method. ③ Ablation experiments indicated that multi-frame point cloud fusion and the weight coefficients of the objective function significantly improved calibration accuracy. When using fused frame point clouds as input compared to single-frame point clouds, the translation error was reduced by 50.89%, and the rotation error was reduced by 53.76%. Considering the weight coefficients, the translation error was reduced by 36.05%, and the rotation error was reduced by 37.87%.
  • [1]
    王国法. 煤矿智能化最新技术进展与问题探讨[J]. 煤炭科学技术,2022,50(1):1-27. DOI: 10.3969/j.issn.0253-2336.2022.1.mtkxjs202201001

    WANG Guofa. New technological progress of coal mine intelligence and its problems[J]. Coal Science and Technology,2022,50(1):1-27. DOI: 10.3969/j.issn.0253-2336.2022.1.mtkxjs202201001
    [2]
    陈晓晶. 井工煤矿运输系统智能化技术现状及发展趋势[J]. 工矿自动化,2022,48(6):6-14,35.

    CHEN Xiaojing. Current status and development trend of intelligent technology of underground coal mine transportation system[J]. Journal of Mine Automation,2022,48(6):6-14,35.
    [3]
    宋秦中,胡华亮. 基于CNN算法的井下无人驾驶无轨胶轮车避障方法[J]. 金属矿山,2023(10):168-174.

    SONG Qinzhong,HU Hualiang. Obstacle avoidance method for underground unmanned trackless rubber-tyred vehicle based on CNN algorithm[J]. Metal Mine,2023(10):168-174.
    [4]
    张宏伟,高亚男,王宇,等. 燃料受限条件下矿区无人驾驶卡车路径最优化策略研究[J]. 金属矿山,2024(8):140-145.

    ZHANG Hongwei,GAO Yanan,WANG Yu,et al. Study on route optimization strategy of unmanned truck in mining area under fuel constraint condition[J]. Metal Mine,2024(8):140-145.
    [5]
    胡青松,孟春蕾,李世银,等. 矿井无人驾驶环境感知技术研究现状及展望[J]. 工矿自动化,2023,49(6):128-140.

    HU Qingsong,MENG Chunlei,LI Shiyin,et al. Research status and prospects of perception technology for unmanned mining vehicle driving environment[J]. Journal of Mine Automation,2023,49(6):128-140.
    [6]
    ZHANG Qilong,PLESS R. Extrinsic calibration of a camera and laser range finder (improves camera calibration)[C]. IEEE/RSJ International Conference on Intelligent Robots and Systems,Sendai,2004. DOI: 10.1109/IROS.2004.1389752.
    [7]
    ZHOU Lipu,DENG Zhidong. Extrinsic calibration of a camera and a lidar based on decoupling the rotation from the translation[C]. IEEE Intelligent Vehicles Symposium,Madrid,2012:642-648.
    [8]
    WANG Weimin,SAKURADA K,KAWAGUCHI N. Reflectance intensity assisted automatic and accurate extrinsic calibration of 3D LiDAR and panoramic camera using a printed chessboard[J]. MDPI AG,2017(8). DOI: 10.3390/RS9080851.
    [9]
    SIM S,SOCK J,KWAK K. Indirect correspondence-based robust extrinsic calibration of LiDAR and camera[J]. Sensors,2016,16(6). DOI: 10.3390/s16060933.
    [10]
    LIAO Qinghai,CHEN Zhenyong,LIU Yang,et al. Extrinsic calibration of lidar and camera with polygon[C]. IEEE International Conference on Robotics and Biomimetics,Kuala Lumpur,2018. DOI: 10.1109/ROBIO.2018.8665256.
    [11]
    徐孝彬,曹晨飞,张磊,等. 基于四面体特征的面阵激光雷达与相机标定方法[J]. 光子学报,2024 ,53(7):176-190.

    XU Xiaobin,CAO Chenfei,ZHANG Lei,et al. Planar array lidar and camera calibration method based on tetrahedral features[J]. Acta Photonica Sinica,2024,53(7):176-190.
    [12]
    谢婧婷,蔺小虎,王甫红,等. 一种点线面约束的激光雷达和相机标定方法[J]. 武汉大学学报(信息科学版),2021,46(12):1916-1923.

    XIE Jingting,LIN Xiaohu,WANG Fuhong,et al. Extrinsic calibration method for LiDAR and camera with joint point-line-plane constraints[J]. Geomatics and Information Science of Wuhan University,2021,46(12):1916-1923.
    [13]
    PANDEY G,MCBRIDE J R,SAVARESE S,et al. Automatic extrinsic calibration of vision and lidar by maximizing mutual information[J]. Journal of Field Robotics,2015,32(5):696-722. DOI: 10.1002/rob.21542
    [14]
    ZHAO Yipu,WANG Yuanfang,TSAI Y. 2D-image to 3D-range registration in urban environments via scene categorization and combination of similarity measurements[C]. IEEE International Conference on Robotics and Automation ,Stockholm,2016:1866-1872.
    [15]
    JIANG Peng,OSTEEN P,SARIPALLI S. SemCal:semantic LiDAR-camera calibration using neural mutual information estimator[C]. IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems,Karlsruhe,2021. DOI: 10.48550/arXiv.2109.10270.
    [16]
    MA Tao,LIU Zhizheng,YAN Guohang,et al. CRLF:automatic calibration and refinement based on line feature for LiDAR and camera in road scenes[EB/OL]. (2021-03-08)[2024-06-22]. https://arxiv.org/abs/2103.04558v1.
    [17]
    ZHU Yufeng,LI Chenghui,ZHANG Yubo. Online camera-LiDAR calibration with sensor semantic information[C]. IEEE International Conference on Robotics and Automation,Paris,2020. DOI: 10.1109/ICRA40945.2020.9196627.
    [18]
    ISHIKAWA R,OISHI T,IKEUCHI K. LiDAR and camera calibration using motions estimated by sensor fusion odometry[C]. IEEE/RSJ International Conference on Intelligent Robots and Systems,Madrid,2018:7342-7349.
    [19]
    WANG Li,XIAO Zhipeng,ZHAO Dawei,et al. Automatic extrinsic calibration of monocular camera and LIDAR in natural scenes[C]. IEEE International Conference on Information and Automation,Wuyishan,2018:997-1002.
    [20]
    SCHNEIDER N,PIEWAK F,STILLER C,et al. RegNet:multimodal sensor registration using deep neural networks[C]. IEEE Intelligent Vehicles Symposium,Los Angeles,2017:1803-1810.
    [21]
    IYER G,RAM R K,MURTHY J K,et al. CalibNet:geometrically supervised extrinsic calibration using 3D spatial transformer networks[C]. IEEE/RSJ International Conference on Intelligent Robots and Systems,Madrid,2018:1110-1117.
    [22]
    LYU Xudong,WANG Shuo,YE Dong. CFNet:lidar-camera registration using calibration flow network[J]. Sensors,2021. DOI: 10.48550/arXiv.2104.11907.
    [23]
    WANG Weimin,NOBUHARA S,NAKAMURA R,et al. SOIC:semantic online initialization and calibration for LiDAR and camera[EB/OL]. (2023-03-09)[2024-06-22]. https://arxiv.org/abs/2003.04260v1.
    [24]
    JAIN J,LI Jiachen,CHIU M,et al. OneFormer:one transformer to rule universal image segmentation[C]. IEEE/CVF Conference on Computer Vision and Pattern Recognition,Vancouver,2023:2989-2998.
    [25]
    XIAO Zeqi,ZHANG Wenwei,WANG Tai,et al. Position-guided point cloud panoptic segmentation transformer[EB/OL]. (2023-03-23)[2024-06-22]. https://arxiv.org/abs/2303.13509v1.
  • Related Articles

    [1]LI Junshi. Development of automatic test platform of electro-hydraulic control system of hydraulic support[J]. Journal of Mine Automation, 2018, 44(4): 13-19. DOI: 10.13272/j.issn.1627-251x.2017110051
    [2]WANG Shuming. Design of transparent communication network system of electro-hydraulic control system of hydraulic support[J]. Journal of Mine Automation, 2017, 43(12): 6-10. DOI: 10.13272/j.issn.1671-251x.2017.12.002
    [3]NIU Jianfeng, BAI Yongsheng, GUO Dinghe. Software defect management of electro hydraulic control system of hydraulic support[J]. Journal of Mine Automation, 2016, 42(11): 67-71. DOI: 10.13272/j.issn.1671-251x.2016.11.016
    [4]CHAI Wen, TIAN Muqin, SONG Jiancheng, XU Chunyu, JIANG Chunyue, YANG Shihua, YIN Weibing, TIAN Weidong, ZHAO Xudong. Research of emergency stop control scheme for electro-hydraulic control system of hydraulic support[J]. Journal of Mine Automation, 2015, 41(4): 13-17. DOI: 10.13272/j.issn.1671-251x.2015.04.004
    [5]BI Dongzhu. Design of remote electro-hydraulic control system of overhead monorail for cable handling[J]. Journal of Mine Automation, 2014, 40(11): 6-9. DOI: 10.13272/j.issn.1671-251x.2014.11.002
    [6]JIANG Chunyue, TIAN Muqin, SONG Jiancheng, XU Chunyu, YANG Shihua, CHAI Wen, YANG Dong, DONG Guangzhuai. Design of hydraulic support controller for automatic working face[J]. Journal of Mine Automation, 2014, 40(9): 1-5. DOI: 10.13272/j.issn.1671-251x.2014.09.001
    [7]YANG Shihua, SONG Jiancheng, TIAN Muqin, XU Chunyu, DONG Gugangzhuai, YIN Weibing. Development of running status monitoring system for hydraulic support based on dual RS485 buses[J]. Journal of Mine Automation, 2014, 40(8): 1-5. DOI: 10.13272/j.issn.1671-251x.2014.08.001
    [8]WU Qing, LI Yong, GONG Fuzhang. Keys of electro-hydraulic control of support used for coal plough unit[J]. Journal of Mine Automation, 2014, 40(5): 97-99. DOI: 10.13272/j.issn.1671-251x.2014.05.025
    [9]ZHAO Bin. Design of electro-hydraulic control system of withdraw shield support on fully-mechanized face[J]. Journal of Mine Automation, 2013, 39(12): 91-94. DOI: 10.7526/j.issn.1671-251X.2013.12.022
    [10]ZHANG Wen-jie, TIAN Mu-qin, SONG Jian-cheng, XU Chun-yu, XIN Chun-hong. Design of Emergency Stop Function of Electro-hydraulic Control System of Hydraulic Supports[J]. Journal of Mine Automation, 2012, 38(8): 4-7.

Catalog

    ZHAO Ruiqi

    1. On this Site
    2. On Google Scholar
    3. On PubMed

    Article Metrics

    Article views (823) PDF downloads (49) Cited by()
    Related

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return