PurposeTo improve the accuracy of stock price trend prediction in the field of quantitative financial trading, this paper takes the prediction accuracy as the goal and avoid the enormous number of network structures and hyperparameter adjustments of long-short-term memory (LSTM).Design/methodology/approachIn this paper, an adaptive genetic algorithm based on individual ordering is used to optimize the network structure and hyperparameters of the LSTM neural network automatically.FindingsThe simulation results show that the accuracy of the rise and fall of the stock outperform than the model with LSTM only as well as other machine learning models. Furthermore, the efficiency of parameter adjustment is greatly higher than other hyperparameter optimization methods.Originality/value(1) The AGA-LSTM algorithm is used to input various hyperparameter combinations into genetic algorithm to find the best hyperparameter combination. Compared with other models, it has higher accuracy in predicting the up and down trend of stock prices in the next day. (2) Adopting real coding, elitist preservation and self-adaptive adjustment of crossover and mutation probability based on individual ordering in the part of genetic algorithm, the algorithm is computationally efficient and the results are more likely to converge to the global optimum.