An improved hybrid grey wolf optimization algorithm (IHGWO) is proposed to solve the problem of population diversity, imbalance of exploration and development capabilities, and premature convergence. The algorithm benefits from particle swarm optimization and a dimension learning-based hunting search strategy. In the particle swarm algorithm search strategy, linear variable social learning and self-learning are introduced to improve the population's ability to communicate information. The individual position, current iteration optimal position, and optimal population position of grey wolves are combined to update the individual position information, thus strengthening the communication between individuals and the population. In the dimension learning-based hunting search strategy, neighborhoods are built for each search member, and neighborhood members can share information, balance global and local searches, and maintain diversity. To validate the algorithm, 23 typical benchmark functions, CEC2022 benchmark functions, and engineering problem sinusoidal low-order-polynomial prediction of positioning error of numerical control machine tools are used to optimize the algorithm's parameters. Results are compared with those from four other algorithms and analyzed using Friedman's statistical test. Experimental and statistical tests reveal that the IHGWO algorithm has the best overall benchmark function rating, with an overall effectiveness of 87.23%. In the engineering parameter optimization problem, the mean square error, root mean square error, and goodness of fit of the prediction equation after IHGWO algorithm optimization are 95.3761, 9.7661, and 97.47%, respectively. These numerical values are superior to those of the compared algorithms, effectively demonstrating the comprehensive performance and applicability of the algorithm.