Optimizing large-scale data processing in the digital economy using high-performance computing techniques

Abstract

In order to solve the current issues of not making the most of scheduling strategies, uneven load, and using too much energy in high-performance computing for the digital economy, this paper creates a task scheduling algorithm that is based on both a genetic-ant colony algorithm and an energy-aware task scheduling algorithm. The first algorithm leverages the robust search space solution capability of the genetic algorithm to transform the optimal solution into an initial pheromone, thereby enhancing global searchability and feedback. The latter reduces overall energy consumption through ETF and NCM operations and allocates appropriate computing nodes for each computing task. The CSI 300 index is used as the research data to carry out empirical analysis of large-scale data processing optimization. Under the scheduling strategy optimized by the algorithm in this paper, the performance of the task scheduling strategy is improved by 2.6 times compared with that of the unoptimized one, and the difference between the capacity share of the memory storage layer and the SSD storage layer is no more than 6%, so as to achieve the load balance between the storage layers. The average CPU utilization remains low at 18.12%, with the running total job power consumption primarily concentrated in the 10w-38w range, which accounts for about 56% of the overall power consumption.