Cervical cancer is the second most common malignancy in women, while is prevented through diagnosing and treating cervical precancerous lesions. Clinically, histopathological image analysis is recognized as the gold standard for diagnosis. However, the diagnosis of cervical precancerous lesions is challenging due to the massive size of whole slide images and subjective grading without precise quantification criteria. Most existing computer aided diagnosis approaches are patches-based, first learning patch-wise features and then aggregating these local features to infer the final prediction. Cropping pathology images into patches restrains the contextual information available to those networks, causing failing to learn clinically relevant structural representations. To address the above problems, this paper proposes a novel weakly supervised learning method called general attention network (GANet) for grading cervical precancerous lesions. A bag-of-instances pattern is introduced to overcome the limitation of the high resolution of whole slide images. Moreover, based on two transformer blocks, the proposed model is able to encode the dependencies among bags and instances that are beneficial to capture much more informative contexts, and thus produce more discriminative WSI descriptors. Finally, extensive experiments are conducted on a public cervical histology dataset and the results show that GANet achieves the state-of-the-art performance.