The Use of Hidden Markov Model in Natural ARABIC Language Processing: a survey

Suleiman, Dima; Awajan, Arafat; Etaiwi, Wael

doi:10.1016/j.procs.2017.08.363

Cited by 28 publications

(7 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It is a common practice to choose the elements according to the uniform distribution: π i ≈ 1/N, a ij ≈ 1/N, and b j (k) ≈ 1/M but these values must be randomized to avoid that the algorithm becomes stuck at a local maximum. • We calculate the parameters α t (i), β t (i), γ t (i, j) and γ t (i) by applying the corresponding expressions in Equations (3), (5), (7) and (8).…”

Section: Reestimating the Modelmentioning

confidence: 99%

“…Word alignment was also studied in connection with statistical translation by using a HMM by Vogel et al [23]. More recent applications of HMM to linguistics include part-of-speech tagging, i.e., the labeling of the different words according to their grammatical category [8,9]. The Baum-Welch algorithm has also been used to identify spoken phrases in VoIP calls [24].…”

Section: Applications To Linguistics Of Hmm and Other Network Modelsmentioning

confidence: 99%

“…Applications of HMMs to the field of Natural Language Processing (NLP) has also flourished in recent years as its has been shown for different layers of NLP such as speech tagging and morphological analysis. By using this approach, successful results for many languages such as Arabic and Persian have been obtained [8,9]. For these reasons, it seems promising to extend these analyses to other sources of text that still cannot be deciphered because they are written in an unknown script and with a unique linguistic structure.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

A Hidden Markov Model for the Linguistic Analysis of the Voynich Manuscript

Acedo¹

2019

MCA

View full text Add to dashboard Cite

Hidden Markov models are a very useful tool in the modeling of time series and any sequence of data. In particular, they have been successfully applied to the field of mathematical linguistics. In this paper, we apply a hidden Markov model to analyze the underlying structure of an ancient and complex manuscript, known as the Voynich manuscript, which remains undeciphered. By assuming a certain number of internal states representations for the symbols of the manuscripts, we train the network by means of the α and β -pass algorithms to optimize the model. By this procedure, we are able to obtain the so-called transition and observation matrices to compare with known languages concerning the frequency of consonant andvowel sounds. From this analysis, we conclude that transitions occur between the two states with similar frequencies to other languages. Moreover, the identification of the vowel and consonant sounds matches some previous tentative bottom-up approaches to decode the manuscript.

show abstract

Section: Reestimating the Modelmentioning

confidence: 99%

Section: Applications To Linguistics Of Hmm and Other Network Modelsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A Hidden Markov Model for the Linguistic Analysis of the Voynich Manuscript

Acedo¹

2019

MCA

View full text Add to dashboard Cite

show abstract

“…• We calculate the parameters α t (i), β t (i), γ t (i, j) and γ t (i) by applying the corresponding expressions in Eqs. (3), (5), (7) and (8). • For i = 0, 1, .…”

Section: Reestimating the Modelmentioning

confidence: 99%

“…Applications of HMMs to the emergent field of Natural Language Processing (NLP) has also flourished in recent years as it has been shown its applicability to different layers of NLP such as speech tagging and morphological analysis. By using this approach successful results for many languages such as Arabic and Persian have been obtained [8,9]. For these reasons, it seems promising to extend these analyses to other sources of text that cannot still be deciphered because they are written in an unknown script and with a unique linguistic structure.…”

Section: Introductionmentioning

confidence: 99%

A Hidden Markov Model for the Linguistic Analysis of the Voynich’s Manuscript

Acedo¹

2018

Preprint

View full text Add to dashboard Cite

Hidden markov models are a very useful tool in the modelling of time series and any sequence of data. In particular, they have been successfully applied to the field of mathematical linguistics. In this paper, we apply a hidden markov model to analyze the underlying structure of an ancient and complex manuscript, known as Voynich's manuscript, that remains still undeciphered. By assuming a certain number of internal states representations for the symbols of the manuscript we train the network by means of the $\alpha$ and $\beta$-pass algorithms to optimize the model. By this procedure, we are able to obtain the so-called transition and observation matrices in order to compare with known languages concerning the frequency of consonant and vowel sounds. From this analysis, we conclude that transitions occur between the two states with similar frequencies to other languages. Moreover, the identification of the vowel and consonant sounds matches some previous tentative bottom-up approaches to decode the manuscript.

show abstract