Graphite/polymer composites are brittle materials that are prone to producing cracks and concavities on machined surfaces, and their surface quality shows greater randomness. This work aims to overcome the large fluctuations in the machined surface quality of graphite/polymer composites, realize the prediction of machined surface roughness under different machining conditions and optimize the process parameters. A graphite/polymer composite material was cut orthogonally using different machining parameters, and the machined surface roughness of the cut samples was measured by a noncontact surface profiler to obtain training samples for Artificial Neural Network (ANN). In this study, a trained radial basis function neural network was used to predict the machined surface roughness, and the prediction accuracy was more than 93%. A Genetic Algorithm (GA) was used to optimize the established ANN, and then grey relational analysis was used to compare the accuracy of the GA optimization results. The ANN prediction after GA optimization showed that the lowest machined surface roughness of the graphite/polymer composites was 1.81 μm, and the corresponding optimal cutting speed, cutting depth, tool rake angle, and rounded edge radius were 11.2 m/min, 0.1 mm, 6.85°, and 11.16 μm, respectively. A verification experiment showed that the lowest machined surface roughness was obtained when the above process parameters were selected, which was only 1.95 μm, and the prediction error of the ANN was approximately 7%. The combination of a GA and an ANN can accurately predict the surface roughness of graphite/polymer composite materials and optimize the process parameters.