Domestic sewage in rural regions is mainly treated by small-scale treatment terminals in China. The large quantities and high dispersion of these terminals render the chemical measurement of effluent to be a time and energy intensive work and further hinder the efficient surveillance of terminals’ performance. After a thorough investigation of 136 operating terminals, this study successfully employs two artificial neural network (ANN) models to predict effluent total nitrogen (TN) and COD (R2 both higher than 0.8) by setting some easily detectable parameters, e.g., pH and conductivity, as inputs. To prevent ANN models getting stuck on local optima and enhance the model performance, genetic algorithm (GA) and particle swarm optimization (PSO) are introduced into ANN, respectively. By comparison, ANN-PSO excels in modelling both TN and COD. The root mean square error (RMSE) and R2 of ANN-PSO in modelling TN are 9.14 and 0.90, respectively, in the training stage, and 11.54 and 0.90, respectively, in the validation stage. The RMSE and R2 of ANN-PSO in modelling COD are 22.10 and 0.90, respectively, in the training stage, and 26.57 and 0.85, respectively, in the validation stage. This is the first study to provide performance prediction models that are available for different terminals. Two established ANN-PSO models show great practical significance in monitoring huge amounts of terminals despite the slight sacrifice of models’ accuracy caused by the great heterogeneity of different terminals.