Unsupervised learning of natural languages

Abstract

We address the problem, fundamental to linguistics, bioinformatics, and certain other disciplines, of using corpora of raw symbolic sequential data to infer underlying rules that govern their production. Given a corpus of strings (such as text, transcribed speech, chromosome or protein sequence data, sheet music, etc.), our unsupervised algorithm recursively distills from it hierarchically structured patterns. The ADIOS (automatic distillation of structure) algorithm relies on a statistical method for pattern extraction and on structured generalization, two processes that have been implicated in language acquisition. It has been evaluated on artificial context-free grammars with thousands of rules, on natural languages as diverse as English and Chinese, and on protein data correlating sequence with function. This unsupervised algorithm is capable of learning complex syntax, generating grammatical novel sentences, and proving useful in other fields that call for structure discovery from raw data, such as bioinformatics.computational linguistics ͉ grammar induction ͉ language acquisition ͉ machine learning ͉ protein classification M any types of sequential symbolic data possess structure that is (i) hierarchical and (ii) context-sensitive. Natural-language text and transcribed speech are prime examples of such data: a corpus of language consists of sentences defined over a finite lexicon of symbols such as words. Linguists traditionally analyze the sentences into recursively structured phrasal constituents (1); at the same time, a distributional analysis of partially aligned sentential contexts (2) reveals in the lexicon clusters that are said to correspond to various syntactic categories (such as nouns or verbs). Such structure, however, is not limited to the natural languages; recurring motifs are found, on a level of description that is common to all life on earth, in the base sequences of DNA that constitute the genome. We introduce an unsupervised algorithm that discovers hierarchical structure in any sequence data, on the basis of the minimal assumption that the corpus at hand contains partially overlapping strings at multiple levels of organization. In the linguistic domain, our algorithm has been successfully tested both on artificialgrammar output and on natural-language corpora such as ATIS (3), CHILDES (4), and the Bible (5). In bioinformatics, the algorithm has been shown to extract from protein sequences syntactic structures that are highly correlated with the functional properties of these proteins. The ADIOS Algorithm for Grammar-Like Rule InductionIn a machine learning paradigm for grammar induction, a teacher produces a sequence of strings generated by a grammar G 0 , and a learner uses the resulting corpus to construct a grammar G, aiming to approximate G 0 in some sense (6). Recent evidence suggests that natural language acquisition involves both statistical computation (e.g., in speech segmentation) and rule-like algebraic processes (e.g., in structured generalization) (7-11). Modern computatio...

Viral photosynthetic reaction center genes and transcripts in the marine environment

“…These results also increase the credibility of our binning method for assigning 'viral-like' versus 'non-viral' notations, as the identified motifs were found almost exclusively in the 'viral-like' file (statistically significant based on hypergeometric distribution; see Methods and Supplementary Table S3). Interestingly, some of these viral-specific motifs, as well as others in other regions, were also identified independently by the unsupervised de novo motif extraction (MEX) algorithm (Solan et al, 2005;Kunik et al, 2007; a list of different viral-specific D1 peptides identified by the MEX algorithm is presented in Supplementary File S3).…”

Coevolution of learning and data-acquisition mechanisms: a model for cognitive evolution

“…Onnis et al [54] showed that when sentences that include common words are presented sequentially or only one sentence apart, segmentation is better than when the same corpus of sentences is presented in a scrambled random order. Note that this result would not follow in models that did not require a learning window, for example, in models where all the data are first acquired and then analysed, as in most computational models for word learning [17][18][19]. Moreover, recent analyses of child-directed speech show that parents and carers behave as if they know that this proximity of sentences with common words is necessary.…”

“…input mechanisms in social learning [16]), but these mechanisms are usually not incorporated into learning models as a way of guiding data selection. For example, computational models for language acquisition use large datasets of child-directed speech without using attentional or communicational cues for data selection [17][18][19]. We believe that much of the learning is already determined by the selection of data to acquire.…”

An empirical generative framework for computational modeling of language acquisition

“…(Here we intentionally gloss over the distinction between acceptability and grammaticality.) This consideration suggests that the generative performance of a grammar could be measured by two figures : RECALL, defined as the proportion of unfamiliar sentences that a parser based on the grammar accepts, and PRECISION, defined as the proportion of novel sentences generated by the grammar that are deemed acceptable by native-speaker subjects, preferably in a blind, controlled test (Solan et al, 2005). These definitions of recall and precision are related but not identical to those used in NLP (Klein & Manning, 2002).…”