Predicting students' academic achievements is an essential issue in education, which can benefit many stakeholders, for instance, students, teachers, managers, etc. Compared with online courses such as MOOCs, students' academicrelated data in the face-to-face physical teaching environment is usually sparsity, and the sample size is relatively small. It makes building models to predict students' performance accurately in such an environment even more challenging. This paper proposes a Two-Way Neural Network (TWNN) model based on the bidirectional recurrent neural network and graph neural network to predict students' next semester's course performance using only their previous course achievements. Extensive experiments on a real dataset show that our model performs better than the baselines in many indicators.