Automated classification of the world′s languages: a description of the method and preliminary results

Abstract

An approach to the classification of languages through automated lexical comparison is described. This method produces near-expert classifications. At the core of the approach is the Automated Similarity Judgment Program (ASJP). ASJP is applied to 100-item lists of core vocabulary from 245 globally distributed languages. The output is 29,890 lexical similarity percentages for the same number of paired languages. Percentages are used as a database in a program originally designed for generating phylogenetic trees in biology. This program yields branching structures (ASJP trees) reflecting the lexical similarity of languages. ASJP trees for languages of the sample spoken in Middle and South America show that the method is capable of grouping together on distinct branches languages of non-controversial genetic groups. In addition, ASJP sub-branching for each of nine respective genetic groups -Mayan, Mixe-Zoque, Otomanguean, Huitotoan-Ocaina, Tacanan, Chocoan, Muskogean, Indo-European, and Austro-Asiatic -agrees substantially with subgrouping for those groups produced by expert historical linguists. Among many other uses, ASJP can be applied to search for possible relationships among languages heretofore not observed or only provisionally recognized. Preliminary ASJP analysis reveals several such possible relationships for languages of Middle and South America. Expanding the ASJP database to all of the world's languages for which 100-word lists can be assembled is a realistic goal that could be achieved in a relatively short period of time, maybe less than a year. STUF, Berlin 61 (2008) 4, 285-308 * We are grateful to André Müller for setting up the website for the references to the sources of the Swadesh lists.We would also like to thank Patience Epps for providing comments relating to the classification of languages of South America. Others who responded with comments to a first draft of this paper or in other important ways deserve our gratitude as well. These include

Sound Symbolism in Basic Vocabulary

“…In the transcription of wordlists of individual languages a simplified system is used, as described in [6]. The symbols in this system are listed and briefly described in Tables 1-2.…”

“…These items were selected not for their susceptibility to sound symbolism but rather for their phonological stability across time: specifically, they were identified in [5] as the concepts for which the words were most similar phonologically in languages known to be related by common descent. The database is therefore unbiased with respect to sound symbolism, because it was developed for the purpose of producing automated language classifications [6,7] and for investigating other issues of a historical linguistic nature, including the identification of linguistic homelands [8] and the calculation of dates for the break-up of proto-languages using a technique similar to glottochronology. Consequently, it provides an opportunity to make a pilot study of sound symbolism in basic vocabulary, which is to our knowledge the first of its kind.…”

English as a Global Language

“…9 The measures of spoken and common native language concern the probability that two people at random for a country pair will share the same spoken language or the same native language, respectively, as the case may be. The measure of linguistic proximity refers instead to similarities of a limited list of words with identical meanings based on expert judgments, where these judgments come from the Automatic Similarity Judgment Program, an international project by ethnolinguists and ethnostatisticians (see Brown et al, 2008). The measure of common official language is the usual binary one.…”

Returns to Local and Foreign Language Skills - Causal Evidence from Spain

“…From the word-by-word distances, a cardinal measure of phonetic distance is derived by averaging and normalizing to take into account potential similarities in phonetic inventories, that might induce a certain similarity by chance. For computational details see Brown et al (2008).…”