Unsupervised extractive document summarization aims to select important sentences from a document without using labeled summaries during training. Existing methods are mostly graph-based with sentences as nodes and edge weights measured by sentence similarities. In this work, we find that transformer attentions can be used to rank sentences for unsupervised extractive summarization. Specifically, we first pre-train a hierarchical transformer model using unlabeled documents only. Then we propose a method to rank sentences using sentence-level self-attentions and pre-training objectives. Experiments on CNN/DailyMail and New York Times datasets show our model achieves state-of-the-art performance on unsupervised summarization. We also find in experiments that our model is less dependent on sentence positions. When using a linear combination of our model and a recent unsupervised model explicitly modeling sentence positions, we obtain even better results.

Contrastive learning models have achieved great success in unsupervised visual representation learning, which maximize the similarities between feature representations of different views of the same image, while minimize the similarities between feature representations of views of different images. In text summarization, the output summary is a shorter form of the input document and they have similar meanings. In this paper, we propose a contrastive learning model for supervised abstractive text summarization, where we view a document, its gold summary and its model generated summaries as different views of the same mean representation and maximize the similarities between them during training. We improve over a strong sequence-to-sequence text generation model (i.e., BART) on three different summarization datasets. Human evaluation also shows that our model achieves better faithfulness ratings compared to its counterpart without contrastive objectives. We release our code at https://github.com/xssstory/SeqCo.

Tangerine Peel has rich medicinal value, known as ' one kilogram of tangerine peel, one kilogram of gold '. However, the value of tangerine peels in different years is different, and there is no significant difference in the appearance of tangerine peels in different years. Identifying their authenticity has brought trouble to the industry. Generally speaking, the characteristics of tangerine peel can be identified through the texture, color and oil parcel points on the surface of tangerine peel. However, compared with the feature recognition of other Chinese medicinal materials, there is no significant difference in the shape of tangerine peel in different years, and the color is similar. Therefore, the feature extraction of tangerine peel is more complicated and the recognition is more difficult. The existing deep learning algorithms face great challenges in efficient and high accuracy recognition. In response to this challenge, this paper builds a new lightweight tangerine peel recognition algorithm TPRA (Tangerine Peel Recognition Algorithm) based on ResNet50. This algorithm uses a variety of methods to optimize the generalization ability of the model and improve the recognition accuracy. Firstly, TPRA adopts mixed data enhancement, including traditional data enhancement, deep convolution generation confrontation network DCGAN, and Mosaic data enhancement to enhance the richness of sample images in the dataset, reduced the data of each batch regularization (Batch Normal), and enhanced the performance of algorithm identification. Secondly, TPRA introduced the attention mechanism module CBAM (Convolutional Block Attention Module) combined with the cross stage partial network CSPNet (Cross Stage Partial Network) to propose an improved ResNet50 model, which adjusts the position of the maximum pooling layer and disassembles the large convolution kernel to effectively avoid overfitting. The experimental results showed that the accuracy of the algorithm can reach 98.8%, and the effect was better than that of Alexnet, VGG16 and Resnet50. TPRA provided a new method for the identification of peel years.

Although coronaviruses have RNA proofreading functions, a large number of variants still exist as quasispecies. Identified coronaviruses might just be the tip of the iceberg, and potentially more fatal variants of concern (VOCs) may emerge over time. These VOCs may exhibit increased pathogenicity, infectivity, transmissibility, angiotensin-converting enzyme 2 (ACE2) binding affinity, and antigenicity, causing an increased threat to public health. In this article, we developed PhyloTransformer, a Transformer-based self-supervised discriminative model, which can model genetic mutations that may lead to viral reproductive advantage. We trained PhyloTransformer on 1,765,297 severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) sequences to infer fitness advantages, by directly modeling the amino acid sequence mutations. PhyloTransformer utilizes advanced techniques from natural language processing, including the Fast Attention Via positive Orthogonal Random features approach (FAVOR+) and the Masked Language Model (MLM), which enable efficient and accurate intra-sequence dependency modeling over the entire RNA sequence. We measured the prediction accuracy of novel mutations and novel combinations using our method and baseline models that only take local segments as input. We found that PhyloTransformer outperformed every baseline method with statistical significance. In order to identify mutations associated with altered glycosylation that might be favored during viral evolution, we predicted the occurrence of mutations in each nucleotide of the receptor binding motif (RBM) and predicted modifications of N-glycosylation sites. We anticipate that the viral mutations predicted by PhyloTransformer may identify potential mutations of threat to guide therapeutics and vaccine design for effective targeting of future SARS-CoV-2 variants.

Coronaviruses are enveloped non-segmented positive-sense RNA viruses. Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) has caused an ongoing pandemic infecting 219 million people as of October 19, 2021, with a 3.6% mortality rate. Although coronaviruses have RNA proofreading functions, a large number of variants still exist as quasispecies. Natural selection can generate favorable mutations with improved fitness advantages, including pathogenicity, infectivity, transmissibility, angiotensin-converting enzyme 2 (ACE2) binding affinity, and antigenicity. However, the identified coronaviruses might just be the tip of the iceberg, and potentially more fatal variants of concern (VOCs) may emerge over time. Understanding the patterns of emerging VOCs and forecasting mutations that may potentially lead to gain of function or immune escape is urgently required. Here we developed PhyloTransformer, which is a Transformer-based discriminative model that engages a multi-head self-attention mechanism to model genetic mutations that may lead to viral reproductive advantage. In order to identify complex dependencies between the elements of each input sequence, PhyloTransformer utilizes advanced modeling techniques, including a novel Fast Attention Via positive Orthogonal Random features approach (FAVOR+) from Performer, and the Masked Language Model (MLM) from Bidirectional Encoder Representations from Transformers (BERT). PhyloTransformer was trained with 1,765,297 genetic sequences retrieved from the Global Initiative for Sharing All Influenza Data (GISAID) database. Firstly, we compared the prediction accuracy of novel mutations and novel combinations using extensive baseline models, including a Transformer-based local model, called Local Transformer, and other local models, such as ResNet-18, multilayer perceptron, logistic regression, KNN, random forest, and gradient boosting; we found that PhyloTransformer outperformed every baseline method with statistical significance. Secondly, we examined predictions of mutations in each nucleotide of the receptor binding motif (RBM), which is a specific sequence of amino acids from the SARS-CoV-2 spike protein that mediates the binding of spike protein to ACE2. Our predictions displayed preciseness and accuracy: our model predicted a total of two mutations in the RBM, and these two mutations precisely coincided with two of the four important mutations presented in seminal bench studies. Thirdly, we predicted modifications of N-glycosylation sites to help identify mutations associated with altered glycosylation that might be favored during viral evolution. We anticipate that the viral mutations predicted by PhyloTransformer may shed light on potential new mutations that may lead to fitness advantages of SARS-CoV-2 variants. Thus, our predicted variants may guide therapeutics and vaccine design for effective targeting of future SARS-CoV-2 variants.