Stock prediction is an exciting issue and is very much needed by investors and business people to develop their assets. The main difficulties in predicting stock prices are dynamic movements, high volatility, and noises caused by company performance and external influences. The traditional method used by investors is the technical analysis based on statistics, valuation of previous stock portfolios, and news from the mass media and social media. Deep learning can predict stock price movements more accurately than traditional methods. As a solution to the issue of stock prediction, we offer the Exponential Moving Average Gated Recurrent Unit (EMAGRU) model and demonstrate its utility. The EMAGRU architecture contains two stacked GRUs arranged in parallel. The inputs and outputs are the EMA10 and EMA20, formed from the closing prices over ten years. We also combine the AntiReLU and ReLU activation functions into the model so that EMAGRU has 6 model variants. Our proposed model produced low losses and high accuracy. RMSE, MEPA, MAE, R2 and were 0.0060, 0.0064, 0.0050, and 0.9976 for EMA10, and 0.0050, 0.0058, 0.0045, and 0.9982 for EMA20, respectively.