The widespread of rumors on social media, carrying unreal or even malicious information, brings negative effects on society and individuals, which makes the automatic detection of rumors become particularly important. Most of the previous studies focused on text mining using supervised models based on feature engineering or deep learning models. In recent years, another parallel line of works, which focuses on the spatial structure of message propagation, provides an alternative and promising solution. However, these existing methods in this parallel line largely overlooked the temporal structure information associated with the spatial structure in message propagation. Actually the addition of temporal structure information can make the message propagations be classified from the perspective of spatial-temporal structure, a more fine-grained perspective. Under these observations, this paper proposes a spatial-temporal structure neural network for rumor detection, termed as STS-NN, which treats the spatial structure and the temporal structure as a whole to model the message propagation. All the STS-NN units are parameter shared and consist of three components, including spatial capturer, temporal capturer and integrator, to capture the spatial-temporal information for the message propagation. The results show that our approach obtains better performance than baselines in both rumor classification and early detection.