Large-Scale Clustering of cDNA-Fingerprinting Data

Abstract

Clustering is one of the main mathematical challenges in large-scale gene expression analysis. We describe a clustering procedure based on a sequential k-means algorithm with additional refinements that is able to handle high-throughput data in the order of hundreds of thousands of data items measured on hundreds of variables. The practical motivation for our algorithm is oligonucleotide fingerprinting-a method for simultaneous determination of expression level for every active gene of a specific tissue-although the algorithm can be applied as well to other large-scale projects like EST clustering and qualitative clustering of DNA-chip data. As a pairwise similarity measure between two p-dimensional data points, x and y, we introduce mutual information that can be interpreted as the amount of information about x in y, and vice versa. We show that for our purposes this measure is superior to commonly used metric distances, for example, Euclidean distance. We also introduce a modified version of mutual information as a novel method for validating clustering results when the true clustering is known. The performance of our algorithm with respect to experimental noise is shown by extensive simulation studies. The algorithm is tested on a subset of 2029 cDNA clones coming from 15 different genes from a cDNA library derived from human dendritic cells. Furthermore, the clustering of these 2029 cDNA clones is demonstrated when the entire set of 76,032 cDNA clones is processed.

DEFOG: A Practical Scheme for Deciphering Families of Genes

“…The comparison was based on computing a Jaccard score [38] for each solution and picking the solution with the best score. To further validate this solution, we reclustered the data using a second clustering method, which is a variant of K-means [34]. We obtained similar results (data not shown).…”

“…The oligonucleotides were labeled at the 5Ј-end by a kinase reaction using [(␣ 33 P]dATP (Amersham International) and T4 polynucleotide kinase (New England Biolabs). We carried out the hybridizations as described [23,34]. The intensities of the hybridization signals were measured by phosphor storage autoradiography (Fuji).…”

“…First, we use a sophisticated primer design procedure for constructing degenerate primers with a good balance between degeneracy and specificity. Second, we combine OFP technology and cluster analysis to obtain many different genes by low-redundancy sequencing.The degeneracy of the primer pairs used in the described experiments ranged between 2.1 ϫ 10 7 and 1.4 ϫ 10 10 -perhaps the highest degeneracy ever used successfully in PCR reactions [34]. Nevertheless, only 0.4% (4 of 924) of the sequenced clones carried genes other than ORs.…”

“…The degeneracy of the primer pairs used in the described experiments ranged between 2.1 ϫ 10 7 and 1.4 ϫ 10 10 -perhaps the highest degeneracy ever used successfully in PCR reactions [34]. Nevertheless, only 0.4% (4 of 924) of the sequenced clones carried genes other than ORs.…”

“…However, the use of such highly degenerate primers increases the probability of nonspecific matches during the PCR reaction. Furthermore, PCR-based libraries, as well as cDNA and shotgun libraries, usually have high redundancy (on average about 140-fold [25,34]), which makes a straightforward exhaustive sequencing of them inefficient and expensive.The DEFOG scheme, which we developed and tested on the human OR gene superfamily, aims to overcome the aforementioned difficulties as follows. First, we use a sophisticated primer design procedure for constructing degenerate primers with a good balance between degeneracy and specificity.…”