Excellent rat behavior observation methods help promote scientific research in neuroscience, social sciences, and pharmacy. Almost all traditional rat behavior observation methods track rats in the fixed environment or through intrusive devices or markers, which may have an impact on rats. Recently, deep learning methods have achieved great success in the field of computer vision because of their powerful ability to feature extraction. However, it is disadvantageous that deep learning methods require a large number of labeled images as a training dataset to adjust its deep neural networks. In this paper, in order to apply the deep learning method to rat behavior observation, we adopted two transfer learning methods to reduce dataset and realized detecting rats in various environments without any intrusive devices or markers. In addition, with the help of track by detection method, we have completed the long-term tracking of multiple rats. We also proposed global category non-maximum suppression to classify rat postures accurately with deep neural networks, which provides researchers with more experimental attempts. In the observation of three rats for one hour, tracking identity definition only happens 34 times and the classification accuracy rate is 89.09%.INDEX TERMS Rats behavior observation, deep learning, transfer learning, tracking.