Design and Implementation of Big Data Training Project Platform Based on Hyper-converged Architecture

Abstract

Building high quality big data training courses is the necessary foundation and urgent need for training big data talents, and it is also an urgent task at present. In this study, hyper-converged architecture technology is used to design and implement a big data training project platform for college students, providing different simulation scenarios for big data training. The system software architecture design adopts the principle of loose coupling and hierarchical design, from bottom to top, it is mainly divided into data acquisition layer, physical layer, device virtualization layer, data layer, business processing layer, service layer, User Interface layer. The platform deploys private cloud OpenStack on CentOS7. The big data analysis process and technology are the data acquisition module (Scrapy and NoSQL), data cleaning module (Kettle), data mining and analysis module (Python scripts), and data visualization module (Power BI). The big data training project platform based on Openstack provides reference for the construction of related platforms.