Recurrent-Neural-Network-Based Boolean Factor Analysis and Its Application to Word Clustering

Abstract

The objective of this paper is to introduce a neural-network-based algorithm for word clustering as an extension of the neural-network-based Boolean factor analysis algorithm (Frolov , 2007). It is shown that this extended algorithm supports even the more complex model of signals that are supposed to be related to textual documents. It is hypothesized that every topic in textual data is characterized by a set of words which coherently appear in documents dedicated to a given topic. The appearance of each word in a document is coded by the activity of a particular neuron. In accordance with the Hebbian learning rule implemented in the network, sets of coherently appearing words (treated as factors) create tightly connected groups of neurons, hence, revealing them as attractors of the network dynamics. The found factors are eliminated from the network memory by the Hebbian unlearning rule facilitating the search of other factors. Topics related to the found sets of words can be identified based on the words' semantics. To make the method complete, a special technique based on a Bayesian procedure has been developed for the following purposes: first, to provide a complete description of factors in terms of component probability, and second, to enhance the accuracy of classification of signals to determine whether it contains the factor. Since it is assumed that every word may possibly contribute to several topics, the proposed method might be related to the method of fuzzy clustering. In this paper, we show that the results of Boolean factor analysis and fuzzy clustering are not contradictory, but complementary. To demonstrate the capabilities of this attempt, the method is applied to two types of textual data on neural networks in two different languages. The obtained topics and corresponding words are at a good level of agreement despite the fact that identical topics in Russian and English conferences contain different sets of keywords.

Neural Autoassociative Memories for Binary Vectors: A Survey

“…Revealing these stable states can be used for data mining, e.g. for binary factor analysis [36,37]. Research of complex (possibly hierarchical) structure of stable states (discussed in terms of cores and fringes of neural assemblies) may appear useful both for modeling brain function and for applications.…”

“…The "neurowindow" approach of [74] may be considered as using multiple thresholds to activate cores or fringes. Revealing the stable states corresponding to emergent assemblies is used for data mining (binary factor analysis) in [36,37].…”

Introduction to Pattern Recognition and Bioinformatics

“…They are also suitable in data rich environments and are typically used for extracting embedded knowledge in the form of rules, quantitative evaluation of these rules, clustering, self-organization, classification, and regression. They have an advantage, over other types of machine learning algorithms, for scaling [37,56,89,107,131,198,210,357,375,376,380]. Investigations have also been made in the area of pattern recognition using genetic algorithms [22,266].…”

Formation of Similarity-Reflecting Binary Vectors with Random Binary Projections

“…Search, classification, and clustering with respect to similarity (see [1][2][3] and bibliography therein) are widely used both in engineering systems (retrieval systems the Internet and intranet, expert systems based on precedent reasoning, etc.) and by people (memory, analogical reasoning, etc.)…”

“…The papers [21,22] consider "dense" binary vectors (with the half of unit components), and [23] investigates binary vectors with adjustable share of unit vectors (sparsity). Sparse binary vectors are required for a number of information processing paradigms, for example, in associative-projective neural networks [11,12,[27][28][29], and for some versions [30][31][32][33][34] of associative memory of matrix type [3,17,[30][31][32][33][34][35][36][37].…”