Hybrid approach for aligning parallel sentences for languages without a written form using standard Malay and Malay dialects

Khaw, Yen-Min Jasmina; Tan, Tien-Ping

doi:10.1109/ialp.2014.6973524

Cited by 2 publications

(1 citation statement)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We propose a hybrid distance-statistical-based phrase alignment algorithm that uses Levenshtein distance and statistical approach. The alignment algorithm was improved from Khaw and Tan [63] to include phrase matching.…”

Section: Aligning Transcribed Dialect Words and Phrasesmentioning

confidence: 99%

Hybrid Distance-Statistical-Based Phrase Alignment for Analyzing Parallel Texts in Standard Malay and Malay Dialects

Yen Min,

Tan,

Malancon

2024

MJCS

View full text Add to dashboard Cite

Parallel texts corpora are essential resources in linguistics and natural language processing, especially in translation and multilingual information retrieval. The publicly available parallel text corpora are limited to certain genres, types and domains. Furthermore, the parallel dialect text is scarce, even though they are important in the analysis and study of a dialect. Collecting parallel dialect text is challenging because dialects typically appear in the form of speech and very limited dialectic texts exist. Moreover, there is no standard orthography in most dialects. The contributions of this paper are threefold. First, the paper describes a methodology in acquiring a parallel text corpus of Standard Malay and Malay dialects, particularly Kelantan Malay and Sarawak Malay. Second, we propose a hybrid of distance based and statistical-based alignment algorithm to align words and phrases the parallel text. The results show that the precision and recall values of the proposed alignment algorithm are more than 95% and better than the state-of the-art GIZA++. Third, the alignment obtained were compared to find out the lexical similarities and differences between Standard Malay and the two studied Malay dialects, contributing valuable insights into the linguistic variations within the Malay language family.

show abstract

Section: Aligning Transcribed Dialect Words and Phrasesmentioning

confidence: 99%

Hybrid Distance-Statistical-Based Phrase Alignment for Analyzing Parallel Texts in Standard Malay and Malay Dialects

Yen Min,

Tan,

Malancon

2024

MJCS

View full text Add to dashboard Cite

show abstract

A Mixed Malay–English Language COVID-19 Twitter Dataset: A Sentiment Analysis

Kong

Juwono

Ngu

et al. 2023

BDCC

View full text Add to dashboard Cite

Social media has evolved into a platform for the dissemination of information, including fake news. There is a lot of false information about the current situation of the Coronavirus Disease 2019 (COVID-19) pandemic, such as false information regarding vaccination. In this paper, we focus on sentiment analysis for Malaysian COVID-19-related news on social media such as Twitter. Tweets in Malaysia are often a combination of Malay, English, and Chinese with plenty of short forms, symbols, emojis, and emoticons within the maximum length of a tweet. The contributions of this paper are twofold. Firstly, we built a multilingual COVID-19 Twitter dataset, comprising tweets written from 1 September 2021 to 12 December 2021. In particular, we collected 108,246 tweets, with over 67% in Malay language, 27% in English, 2% in Chinese, and 4% in other languages. We then manually annotated and assigned the sentiment of 11,568 tweets into three-class sentiments (positive, negative, and neutral) to develop a Malay-language sentiment analysis tool. For this purpose, we applied a data compression method using Byte-Pair Encoding (BPE) on the texts and used two deep learning approaches, i.e., the Multilingual Bidirectional Encoder Representation for Transformer (M-BERT) and convolutional neural network (CNN). BPE tokenization is used to encode rare and unknown words into smaller meaningful subwords. With the CNN, we converted the labeled tweets into image files. Our experiments explored different BPE vocabulary sizes with our BPE-Text-to-Image-CNN and BPE-M-BERT models. The results show that the optimal vocabulary size for BPE is 12,000; any values beyond that would not contribute much to the F1-score. Overall, our results show that BPE-M-BERT slightly outperforms the CNN model, thereby showing that the pre-trained M-BERT network has the advantage for our multilingual dataset.

show abstract

Hybrid approach for aligning parallel sentences for languages without a written form using standard Malay and Malay dialects

Cited by 2 publications

References 3 publications

Hybrid Distance-Statistical-Based Phrase Alignment for Analyzing Parallel Texts in Standard Malay and Malay Dialects

Hybrid Distance-Statistical-Based Phrase Alignment for Analyzing Parallel Texts in Standard Malay and Malay Dialects

A Mixed Malay–English Language COVID-19 Twitter Dataset: A Sentiment Analysis

Contact Info

Product

Resources

About