Volume 49 Issue 1
Feb.  2023
Turn off MathJax
Article Contents
Abstract
References
Related Articles
GUO Jun. Research on distributed storage of 3D stack grid model of coal mine geology based on HDF5[J]. Journal of Mine Automation，2023，49(1)：153-161. DOI: 10.13272/j.issn.1671-251x.18056
Citation: GUO Jun. Research on distributed storage of 3D stack grid model of coal mine geology based on HDF5[J]. Journal of Mine Automation，2023，49(1)：153-161. DOI: 10.13272/j.issn.1671-251x.18056
shu

Research on distributed storage of 3D stack grid model of coal mine geology based on HDF5

  • 1.

    Research Institute of Mine Big Data, CCTEG Chinese Institute of Coal Science, Beijing 100013, China

  • 2.

    State Key Laboratory of High Efficient Mining and Clean Utilization of Coal Resources, Beijing 100013, China

More Information
  • Received Date: November 06, 2022
  • Revised Date: January 07, 2023
  • Available Online: January 17, 2023
    Graphical Abstract
    • Abstract
  • The realization of multi-resolution expression and multi-parameter fusion of coal mine geological environment by using true 3D gridded geological model is one of the key contents of coal mine geological big data research. The core issues are the organization, storage and management of 3D geological model data. Aiming at the data scale, distributed storage and query performance of 3D geological grid model in coal mines, a distributed storage scheme of 3D stack grid model based on HDF5 is proposed. In terms of grid data organization, the 3D geological model data is compressed and organized in blocks by using the stack grid model. The problem of large-scale geological grid model data organization is solved by data segmentation. The data segmentation also concentrates the data with similar space in the adjacent hard disk sector or storage device. It is conducive to improving the efficiency of data scheduling. In terms of data storage, HDF5 is used as the persistence layer of storage to store all original data. The memory database Redis is used to store hot data, HDF5 metadata and other related information. In terms of Web services, H5Serv is used to send and receive HDF5 data. In terms of HDF5 distribution, network file system (NFS) is used to realize the sharing of HDF5 data between different node servers. Rsync and Inotify are used to realize real-time synchronization of HDF5 data in different node servers. Nginx is used to realize load balancing of reverse proxy and data service nodes during access. The Docker container technology is used to uniformly deploy the data node service and Nginx service. The JupyterLab interactive analysis platform is used to realize the scheduling and management of real-time data resources. The experimental results show that the data organization of the geological model based on the stack grid and the distributed storage based on HDF5 can realize the effective storage management and spatial query of 3D geological grid model of the coal mine. Compared with the voxel model and octree model, the data volume of the stack grid model is small. It is convenient to realize the spatial quick query of the geological interface. The spatial query performance is better than the relational database MySQL and the non-relational database MongoDB. The stack grid model is more suitable for the grid expression and data organization of the coal measures sedimentary stratigraphic structure. The file storage based on HDF5 is significantly more space-saving than MySQL and MongoDB database storage. The main reason is that the DataSet of HDF5 can directly store data blocks without additional storage information. The data organization and storage scheme based on stack grid model and HDF5 can provide references for the effective storage management of 3D geological grid model in coal mines.
    • FullText(HTML)
    • References (21)
  • [1]
    王国法. 煤矿智能化最新技术进展与问题探讨[J]. 煤炭科学技术,2022,50(1):1-27. DOI: 10.3969/j.issn.0253-2336.2022.1.mtkxjs202201001

    WANG Guofa. New technological progress of coal mine intelligence and its problems[J]. Coal Science and Technology,2022,50(1):1-27. DOI: 10.3969/j.issn.0253-2336.2022.1.mtkxjs202201001
    [2]
    焦玉勇. 新一代信息科技如何支持城市透明地质建设?[J]. 地球科学,2022,47(10):3918-3918.

    JIAO Yuyong. How does the cutting-edge information technology support the construction of urban transparent geology?[J]. Earth Science,2022,47(10):3918-3918.
    [3]
    吴立新,史文中. 论三维地学空间构模[J]. 地理与地理信息科学,2005,21(1):1-4. DOI: 10.3969/j.issn.1672-0504.2005.01.001

    WU Lixin,SHI Wenzhong. On three dimensional geosciences spatial modeling[J]. Geography and Geo-Information Science,2005,21(1):1-4. DOI: 10.3969/j.issn.1672-0504.2005.01.001
    [4]
    AARNES J,KROGSTAD S,LIE K. Multiscale mixed/mimetic methods on corner-point grids[J]. Computers & Geosciences,2008,12(3):297-315.
    [5]
    吴立新,陈学习,车德福,等. 一种基于GTP的地下真3D集成表达的实体模型[J]. 武汉大学学报(信息科学版),2007,32(4):331-335. DOI: 10.3969/j.issn.1671-8860.2007.04.014

    WU Lixin,CHEN Xuexi,CHE Defu,et al. A GTP-based entity model for underground real 3D integral representation[J]. Geomatics and Information Science of Wuhan University,2007,32(4):331-335. DOI: 10.3969/j.issn.1671-8860.2007.04.014
    [6]
    WATSON C,RICHARDSON J,WOOD B,et al. Improving geological and process model integration through TIN to 3D grid conversion[J]. Computers & Geosciences,2015,82:45-54.
    [7]
    GRACIANO A,ANTONIO J R,FRANCISCO R F. Real-time visualization of 3D terrains and subsurface geological structures[J]. Advances in Engineering Software,2018,115(1):314-326.
    [8]
    CAUMON G,COLLON-DROUAILLET P,VESLUD C L C D,et al. Surface-based 3D modeling of geological structures[J]. Mathematical Geosciences,2009,41(8):927-945. DOI: 10.1007/s11004-009-9244-2
    [9]
    ANDRIYCHENKO V,FAN C T,LI Y H. File transfer and synchronization over multiple clients/server environment[J]. Frontiers in Artificial Intelligence and Applications,2015,274:2133-2142.
    [10]
    雷德龙,郭殿升,陈崇成,等. 基于MongoDB的矢量空间数据云存储与处理系统[J]. 地球信息科学学报,2014,16(4):507-516.

    LEI Delong,GUO Diansheng,CHEN Chongcheng,et al. Vector spatial data cloud storage and processing based on MongoDB[J]. Journal of Geo-Information Science,2014,16(4):507-516.
    [11]
    MURTY M S,MALLESWARARAO N N. Loading,searching and retrieving data from local data nodes on HDFS[J]. International Journal of Data Science,2020,5:178-198.
    [12]
    COLLETTE A. Python and HDF5: unlocking scientific data[M]. Sebastopol: O'Reilly Media Inc., 2013.
    [13]
    葛钰,李洪赭,李赛飞. 一种Web服务器集群自适应动态负载均衡设计与实现[J]. 计算机与数字工程,2020,48(12):3002-3007. DOI: 10.3969/j.issn.1672-9722.2020.12.036

    GE Yu,LI Hongzhe,LI Saifei. Design and implementation of adaptive dynamic load balancing for Web server clusters[J]. Computer & Digital Engineering,2020,48(12):3002-3007. DOI: 10.3969/j.issn.1672-9722.2020.12.036
    [14]
    POTDAR A M,NARAYAN D G,KENGOND S,et al. Performance evaluation of docker container and virtual machine[J]. Procedia Computer Science,2020,171:1419-1428. DOI: 10.1016/j.procs.2020.04.152
    [15]
    MARZUN S M,SAVADI A,TOOSI A N,et al. Cross-MapReduce:data transfer reduction in geo-distributed MapReduce[J]. Future Generation Computer Systems,2021,115:188-200. DOI: 10.1016/j.future.2020.09.009
    [16]
    VOLZ W R. Gigabyte volume viewing using split software/hardware interpolation[C]. IEEE Symposium on Volume Visualization （VV 2000）, Salt Lake City, 2000: 15-22.
    [17]
    左珍德. 基于Rsync的结构化数据库实时高速备份研究及工具开发[D]. 广州: 华南理工大学, 2017.

    ZUO Zhende. Research and tool development of real-time high-speed backup for structured database based on Rsync[D]. Guangzhou: South China University of Technology, 2017.
    [18]
    林跃,冯薇桦,孙源泽. 基于Docker的容器虚拟化技术[J]. 中国新通信,2020,22(9):68. DOI: 10.3969/j.issn.1673-4866.2020.09.049

    LIN Yue,FENG Weihua,SUN Yuanze. Container virtualization technology based on Docker[J]. China New Telecommunications,2020,22(9):68. DOI: 10.3969/j.issn.1673-4866.2020.09.049
    [19]
    杨阳,王品,杜少华. NFS在分布式数控系统中的应用与改进[J]. 计算机系统应用,2015,24(6):202-206.

    YANG Yang,WANG Pin,DU Shaohua. Application and improvement of NFS in distributed CNC system[J]. Computer Systems & Applications,2015,24(6):202-206.
    [20]
    高原. 基于Nginx的web服务器负载均衡策略研究[D]. 海口: 海南大学, 2019.

    GAO Yuan. Research on load balance strategy of web server based on Nginx[D]. Haikou: Hainan University, 2019.
    [21]
    贺宗平,张晓东,刘玉. 基于Jupyter交互式分析平台的微服务架构[J]. 计算机系统应用,2019,28(8):63-70. DOI: 10.15888/j.cnki.csa.007017

    HE Zongping,ZHANG Xiaodong,LIU Yu. Microservice architecture for Jupyter-based interactive analysis platform[J]. Computer Systems & Applications,2019,28(8):63-70. DOI: 10.15888/j.cnki.csa.007017
    • Related Articles

  • Related Articles

    [1]YANG Xiaohui, YANG Bin, SONG Yongbao. Application of next generation network in coal mine communication system[J]. Journal of Mine Automation, 2015, 41(3): 105-107. DOI: 10.13272/j.issn.1671-251x.2015.03.027
    [2]YANG Xiaokang, LI Hequn, GAO Qiaomei, CHENG Hong. Research of single phase-shift close-loop control for bi-directional full-bridge DC-DC converter[J]. Journal of Mine Automation, 2014, 40(4): 63-67. DOI: 10.13272/j.issn.1671-251x.2014.04.015
    [3]YU Qing-lin, WANG Dong, TANG Dong-xu, LI Cui-la. Design of Mine-used Combination Switch Based on PLC[J]. Journal of Mine Automation, 2011, 37(3): 19-21.
    [4]GU Jun, HUO Zhen-long, XU Xi-liang, WANG Wei. Research of Mine-used Service System of Dispatching and Communication Based on Soft Switch Platform and Its Implementatio[J]. Journal of Mine Automation, 2011, 37(1): 5-9.
    [5]Meng Run-quan, WANG Zhen-min, XU Chun-yu. An Implementation Strategy of Soft-switching for Power Converter of SRD[J]. Journal of Mine Automation, 2009, 35(12): 46-49.
    [6]NIU Ben~, SONG Shu-zhong~, MA Jian-wei~, ZHU Jin-hong~. Research of a New Type of Soft-switching Buck Converter[J]. Journal of Mine Automation, 2009, 35(12): 38-42.
    [7]JIAO Wen-liang, LI Li-hua. Development of Dynamic Braking Power Supply of High-frequency Switch Based on SPWM[J]. Journal of Mine Automation, 2009, 35(5): 78-80.
    [8]NIU Yao-hong. Probe on Common Fault in Use of Soft-start Switch[J]. Journal of Mine Automation, 2008, 34(5): 128-130.
    [9]LIU Wen-yan~, KAN Quan-fu~. Study of a Half-bridge Converter of Single-stage PFC[J]. Journal of Mine Automation, 2007, 33(1): 27-29.
    [10]YANG Jin-zhong. Study of PLC Monitoring and Control System for Mine Used Switch Element[J]. Journal of Mine Automation, 2000, 26(6): 27-28.

  • Cited by

    Periodical cited type(7)

    1. 尚国银. 煤矿外因火灾防治与应急处置技术研究. 中国安全科学学报. 2024(S1): 274-279 .
    2. 王彦文，张旭然，高彦，王寅生. 基于热路模型的矿用高压电缆内因火灾预警研究. 矿业科学学报. 2022(02): 225-232 .
    3. 任晓伟，王振兴，王锐，康付如，王洋，韩东洋，王伟峰，齐龙辉，徐勇勇. 煤矿电缆火灾危险性研究. 煤矿安全. 2022(09): 151-156 .
    4. 陈晓晶. 基于“云-边-端”协同的煤矿火灾智能化防控体系建设. 煤炭科学技术. 2022(12): 136-143 .
    5. 方书博，罗浩，洪骁，郑雷. 隧道机器人电缆火灾参数仿真与试验研究. 电气应用. 2021(09): 41-49 .
    6. 苟宝洋，吴兵，马一飞，苏敬亮，贾泽鹏. 基于Simtec的巴黎圣母院火灾数值模拟. 消防科学与技术. 2020(01): 49-51 .
    7. 康文杰，周亮宇，王德明，李金帅. 煤最短自然发火期快速预测方法. 工矿自动化. 2019(05): 31-34 . 本站查看

    Other cited types(2)

Catalog

    Get Citation
    PDF
    XML

    Article Metrics

    Article views (194) PDF downloads (23) Cited by(9)
    Related

    苏ICP备 05016349号-2

    Supported by: Beijing Renhe Information Technology Co., Ltd. Visits：1253115    Baidu Analytics

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return