Textual representations play an important role in the field of natural language processing (NLP). The efficiency of NLP tasks, such as text comprehension and information extraction, can be significantly improved with proper textual representations. As neural networks are gradually applied to learn the representation of words and phrases, fairly efficient models of learning short text representations have been developed, such as the continuous bag of words (CBOW) and skip-gram models, and they have been extensively employed in a variety of NLP tasks. Because of the complex structure generated by the longer text lengths, such as sentences, algorithms appropriate for learning short textual representations are not applicable for learning long textual representations. One method of learning long textual representations is the Long Short-Term Memory (LSTM) network, which is suitable for processing sequences. However, the standard LSTM does not adequately address the primary sentence structure (subject, predicate and object), which is an important factor for producing appropriate sentence representations. To resolve this issue, this paper proposes the dependency-based LSTM model (D-LSTM). The D-LSTM divides a sentence representation into two parts: a basic component and a supporting component. The D-LSTM uses a pre-trained dependency parser to obtain the primary sentence information and generate supporting components, and it also uses a standard LSTM model to generate the basic sentence components. A weight factor that can adjust the ratio of the basic and supporting components in a sentence is introduced to generate the sentence representation. Compared with the representation learned by the standard LSTM, the sentence representation learned by the D-LSTM contains a greater amount of useful information. The experimental results show that the D-LSTM is superior to the standard LSTM for sentences involving compositional knowledge (SICK) data.

No abstract

The efficiency of natural language processing (NLP) tasks, such as text classification and information retrieval, can be significantly improved with proper sentence representations. Neural networks such as convolutional neural network (CNN) and recurrent neural network (RNN) are gradually applied to learn the representations of sentences and are suitable for processing sequences. Recently, bidirectional encoder representations from transformers (BERT) has attracted much attention because it achieves state-of-the-art performance on various NLP tasks. However, these standard models do not adequately address a general linguistic fact, that is, different sentence components serve diverse roles in the meaning of a sentence. In general, the subject, predicate, and object serve the most crucial roles as they represent the primary meaning of a sentence. Additionally, words in a sentence are also related to each other by syntactic relations. To emphasize on these issues, we propose a sentence representation model, a modification of the pre-trained bidirectional encoder representations from transformers (BERT) network via component focusing (CF-BERT). The sentence representation consists of a basic part which refers to the complete sentence, and a component-enhanced part, which focuses on subject, predicate, object, and their relations. For the best performance, a weight factor is introduced to adjust the ratio of both parts. We evaluate CF-BERT on two different tasks: semantic textual similarity and entailment classification. Results show that CF-BERT yields a significant performance gain compared to other sentence representation methods.

Sentence representations play an important role in the field of natural language processing. While word representation has been applied to many natural language processing tasks, sentence representation has not been applied as widely due to the more complex structure and richer syntactic information of sentences. To learn sentence representations with structural and syntactic information, we propose a new model called the part-of-speech-based Long Short-Term Memory network (pos-LSTM) model. The pos-LSTM model generates a structural representation using the standard LSTM model and a syntactic representation using the part of speech; then, the pos-LSTM model obtains the final sentence representation by combining these two representations. The experimental results from 20 sentence similarity tasks and an entailment classification task show that the pos-LSTM model can better capture the syntactic information of sentences and generate higher quality sentence representations than traditional models. INDEX TERMS Natural language processing, sentence representation, neural networks.

It is important to assess the insulation aging state of transformers, which can not only avoid equipment faults, but also improve the economic benefits of transformer operation. The main negative effect of transformer insulation aging is the decline of insulation mechanical properties, which will eventually lead to transformer fault. In this paper, the transformer vibration characteristic test platform is built and the vibration characteristic variables are defined to describe the vibration characteristic quantitatively. Then, the vibration characteristics and vibration variables of transformer tank surface under different aging time are studied by accelerated aging test in the laboratory. Finally, the vibration signals of transformers in service under different operation time are measured. And the vibration characteristics and vibration variables of in-service transformers under different operation time are analyzed and studied. This paper directly aims at the mechanical characteristics of insulation, and studies the relationship between insulation aging state and vibration of transformer. It provides a basis for more effective and direct monitoring of insulation aging state.