The Performance of BERT as Data Representation of Text Clustering

Abstract

Text clustering is the task of grouping a set of texts so that text in the same group will be more similar than those from a different group. The process of grouping text manually requires a significant amount of time and labor. Therefore, automation utilizing machine learning is necessary. The standard method used to represent textual data is Term Frequency Inverse Document Frequency (TFIDF). However, TFIDF cannot consider the position and context of a word in a sentence. Bidirectional Encoder Representation from Transformers (BERT) model can produce text representation that incorporates the position and context of a word in a sentence. This research analyzed the performance of the BERT model as data representation for text. Moreover, various feature extraction and normalization methods are also applied for the data representation of the BERT model. To examine the performances of BERT, we use four clustering algorithms, i.e., k-means clustering, eigenspace-based fuzzy c-means, deep embedded clustering, and improved deep embedded clustering. Our simulations show that BERT outperforms the standard TFIDF method in 28 out of 36 metrics. Furthermore, different feature extraction and normalization produced varied performances. The usage of these feature extraction and normalization must be altered depending on the text clustering algorithm used.

The Teacher's Role in Motivating Students in Physics Learning Using the Learning Model Backward by Design

“…The Backward by Design model is a learning model that starts with the teacher conveying the learning objectives at the beginning of the class to the strategic planning process and clear instructions are achieved (Subakti et al, 2021). Research conducted by Martha (2021) results, namely the teacher determines the learning objectives to be carried out supported by relevant reference sources, enduring understanding, essential questions, writing down indicators or core knowledge, and accompanied by guiding questions that can help students understand the material to be studied.…”

“…The learning model also serves as a guide for educators to design interesting learning activities so that they can foster student learning motivation. One learning model that can be applied is the Backward by Design model, which is a learning model that begins with setting learning objectives and a frame of mind, namely by determining the main idea of learning (Subakti et al, 2021). In this research, the Backward by Design model was used in Physics lessons to increase student motivation in learning physics.…”

“…The application of the Backward by Design model can make students actively involved from the beginning of learning to the end so that students have a high enthusiasm for learning so that conditions in the class come alive and can foster student learning motivation (Wiggins & Mc Tighe, 2005). The advantages of the Backward by Design model according to Subakti et al (2021) are 1) It can attract attention and trigger curiosity from students, 2) Students understand the goals and directions of learning that are carried out because they are conveyed 3) Make students look active during the learning process. The drawbacks of this model are that the desired results may not be by expected expectations, students will have difficulty dealing with validity and reliability issues, and teacher effectiveness is measured more from student achievement in class through formulated assessments rather than the ability to relate knowledge, skills, and interest in student learning (Cho & Allen,2005).…”

Word Embedding for Unknown Words: Adding New Words Into Bert’s Vocabulary

“…Modelos de linguagem que apresentam arquitetura com base em 'transformer models' e 'attention mechanisms' (Vaswani et al, 2017) como BERT ® (Bidirectional Encoder Representations from Transformers) (Devlin et al, 2019) e GPT-3 ® (Brown et al, 2020), representam o estado da arte em NLP e vem sendo usados por apresentarem resultados superiores às técnicas precedentes de modelos sequenciais, como, LSTM (Long Short-T erm Memory), RNN (Recurrent Neural Network), GRU (Gated Recurrent Unit), entre outros. Tendo seu sucesso atrelado ao conceito de transferência de conhecimento (transfer learning) permitindo que modelos treinados em grandes corpus de textos genéricos possam ser aplicados em tarefas específicas sem a necessidade de replicação extensa do treinamento (Qiu et al, 2020;Subakti, Murfi, & Hariadi, 2022).…”