Self-Attention Mechanism of RoBERTa to Improve QAS for e-health Education

Suwarningsih, Wiwin; Pratama, Raka Aditya; Rahadika, Fadhil Yusuf; Purnomo, Mochamad Havid Albar

doi:10.1109/ic2ie53219.2021.9649363

Cited by 1 publication

(1 citation statement)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…From our previous research experience, RoBERTa showed the better results than ALBERT for Indonesian [27], but the training time tended to be long (restricted resources). Thus, we did long training for RoBERTa with loss results like the graph in the picture above.…”

Section: Results and Analysis 31 Long Training Robertamentioning

confidence: 94%

RoBERTa: language modelling in building Indonesian question-answering systems

et al. 2022

Self Cite

View full text Add to dashboard Cite

This research aimed to evaluate the performance of the A Lite BERT (ALBERT), efficiently learning an encoder that classifies token replacements accurately (ELECTRA) and a robust optimized BERT pretraining approach (RoBERTa) models to support the development of the Indonesian language question and answer system model. The evaluation carried out used Indonesian, Malay and Esperanto. Here, Esperanto was used as a comparison of Indonesian because it is international, which does not belong to any person or country and this then make it neutral. Compared to other foreign languages, the structure and construction of Esperanto is relatively simple. The dataset used was the result of crawling Wikipedia for Indonesian and Open Super-large Crawled ALMAnaCH coRpus (OSCAR) for Esperanto. The size of the token dictionary used in the test used approximately 30,000 sub tokens in both the SentencePiece and byte-level byte pair encoding methods (ByteLevelBPE). The test was carried out with the learning rates of 1e-5 and 5e-5 for both languages in accordance with the reference from the bidirectional encoder representations from transformers (BERT) paper. As shown in the final result of this study, the ALBERT and RoBERTa models in Esperanto showed the results of the loss calculation that were not much different. This showed that the RoBERTa model was better to implement an Indonesian question and answer system.

show abstract

Section: Results and Analysis 31 Long Training Robertamentioning

confidence: 94%