PatentNet: multi-label classification of patent documents using deep learning based language understanding

Abstract

Patent classification is an expensive and time-consuming task that has conventionally been performed by domain experts. However, the increase in the number of filed patents and the complexity of the documents make the classification task challenging. The text used in patent documents is not always written in a way to efficiently convey knowledge. Moreover, patent classification is a multi-label classification task with a large number of labels, which makes the problem even more complicated. Hence, automating this expensive and laborious task is essential for assisting domain experts in managing patent documents, facilitating reliable search, retrieval, and further patent analysis tasks. Transfer learning and pre-trained language models have recently achieved state-of-the-art results in many Natural Language Processing tasks. In this work, we focus on investigating the effect of fine-tuning the pre-trained language models, namely, BERT, XLNet, RoBERTa, and ELECTRA, for the essential task of multi-label patent classification. We compare these models with the baseline deep-learning approaches used for patent classification. We use various word embeddings to enhance the performance of the baseline models. The publicly available USPTO-2M patent classification benchmark and M-patent datasets are used for conducting experiments. We conclude that fine-tuning the pre-trained language models on the patent text improves the multi-label patent classification performance. Our findings indicate that XLNet performs the best and achieves a new state-of-the-art classification performance with respect to precision, recall, F1 measure, as well as coverage error, and LRAP.

VisFormers—Combining Vision and Transformers for Enhanced Complex Document Classification

“…Multiple ML algorithms and NLP techniques are evaluated, with support vector machine (SVM) classifiers achieving over 96% accuracy in multi-class resume classification. Aroush et al [25] use deep learning and already-trained language models to handle the complex and timeconsuming task of patent classification. Using datasets like USPTO-2M and M-patent, their work investigates the fine-tuning of models like BERT, XLNet, RoBERTa, and ELECTRA to achieve cutting-edge performance on multi-label patent categorization.…”

Artificial Intelligence-Enabled Chatbots in Mental Health: A Systematic Review

“…Language recognition relies heavily on Natural Language Processing (NLP), which is used by Google and Apple's Siri. The ability of technology to recognize human natural language textual data and speech-based commands depends on two fundamental elements: natural language understanding (NLU) and natural language generation (NLG) [20].…”

Discovery of topic evolution path and semantic relationship based on patent entity representation

“…Zhang and Yu (2020) use word2vec to form word vectors for patent documents and use a linear algorithm to the semantic representations of patent phrases. Haghighian Roudsari et al. (2021) perform a pre-training model BERT on patent text for the subsequent classification task.…”