High Speed Document Clustering in Reconfigurable Hardware

Covington, G. Adam; Comstock, Charles L.G.; Levine, Andrew; Lockwood, John W.; Cho, Young

doi:10.1109/fpl.2006.311245

Cited by 6 publications

(3 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…They determined that the Manhattan distance would be the best fit for their hardware. Covington et al [8] showed the implementation of K-means mapped into integer arithmetic. They utilized a Cosine Theta distance metric since this metric (also known as the spherical distance) provides a better distance in high dimensional sparse data.…”

Section: Related Workmentioning

confidence: 99%

Intelligent Avionics with Advanced Clustering

Meier

Sproull

Covington

et al. 2008

2008 IEEE Aerospace Conference

View full text Add to dashboard Cite

In this paper, we consider tracking targets using multiple distributed sensor platforms. Rather than sending the tracks to a central location, such as a command and control center where information is exchanged between platforms, we consider a distributed solution. While fixed position single sensor tracking of a single target is considered straightforward, multiple sensors on the different platforms with overlapping coverage is complex because duplicate tracking data is generated for the same targets. Redundant information generates network messages that in turn overload the network performance, and may result in traffic congestion on limited avionic bandwidth wireless links that prevents time critical data from reaching its destination. In our approach, we identify similar tracking data to be distributed and send only one copy of the message.In order to identify redundant data, we use a clustering algorithm to evaluate the large volumes of sensor information. Distributed multiple target tracking (MTT) combines track observations from different sensors to identify the same target. Massive computation and communication is required for distributed real time MTT. In this paper, the K-means clustering algorithm is used to aggregate redundant tracks. Software simulations using Matlab and emulation tests using Emulab show significant improvement of the information quality by using clustering. An MTT system was prototyped with FPGA hardware to cluster high volumes of data with low latency in real time at the network layer.

show abstract

Section: Related Workmentioning

confidence: 99%

Intelligent Avionics with Advanced Clustering

Meier

Sproull

Covington

et al. 2008

2008 IEEE Aerospace Conference

View full text Add to dashboard Cite

show abstract

“…This algorithm can be efficiently implemented on a FPGA and makes use of embedded RAM and multipliers. Assume a dividend X and a divisor Y both in the range of [1,2). The divisor Y is decomposed into a higher order m-bit part Y h and a lower order m-bit part Y l so that Y h > 2 m Y l .…”

Section: Mean Update Unitmentioning

confidence: 99%

“…One application in network security [8] is to cluster network traffic for anomaly detection. Another application is for document clustering [2].…”

Section: Related Workmentioning

confidence: 99%

K-means Clustering for Multispectral Images Using Floating-Point Divide

Meulenaer

Gosset

Dormale

et al. 2007

15th Annual IEEE Symposium on Field-Programmable Custom Computing Machines (FCCM 2007)

View full text Add to dashboard Cite

Many signal processing algorithms can be accelerated using reconfigurable hardware. To achieve a good speedup compared to running software on a general purpose processor, fine-grained control over the bitwidth of each component in the datapath is desired. This goal can be achieved by using NU's variable precision floating-point library. To analyze the usefulness of the floating-point divide unit, we incorporate it into our previous implementation of the Kmeans clustering algorithm applied to multispectral satellite images. With the lack of a floating-point divide hardware implementation, the mean updating step in each iteration of the K-means algorithm had to be moved to the host computer for calculation. The new means calculated on the host then had to be moved back to the FPGA board for the next iteration of the algorithm. This added data transfer overhead between the host and the FPGA board. In this work, we use the new fp div module to implement the mean updating step in FPGA hardware. This greatly reduces the communication overhead between host and FPGA board and further accelerates run time. The Kmeans clustering example illustrates the use of the fp div, fix2float and float2fix modules seamlessly assembled together in a real application. It is the first implementation that has the complete K-means computation done in FPGA hardware. Our results show that the hardware implementation achieves a speedup of over 2150x for core computation time and about 11x for total run time including data transfer time. They also show that the divide in FPGA hardware is 100 times faster than in software. Moreover, implementing divide in the FPGA frees the host to work on other tasks concurrently with K-means clustering, thus providing further speedup by allowing the image analyst to exploit this coarse grained parallelism.

show abstract

Streaming Hierarchical Clustering for Concept Mining

Looks

Levine

Covington

et al. 2007

2007 IEEE Aerospace Conference

Self Cite

View full text Add to dashboard Cite

We are concerned with the general problem of concept mining -discovering useful associations, relationships, and groupings in large collections of data. Mathematical transformation algorithms have proven effective at reducing the content of multilingual, unstructured data into a vector that describes the content. Such methods are particularly desirable in fields undergoing information explosions, such as network traffic analysis, bioinformatics, and the intelligence community. In response, concept mining methodology is being extended to improve performance and permit hardware implementation -traditional methods are not sufficiently scalable. 1 23 4Hardware-accelerated systems have proven effective at automatically classifying such content when topics are known in advance. Our complete system builds on our past work in this area, presented in the Aerospace 2005 and 2006 conferences, where we described a novel algorithmic approach for extracting semantic content from unstructured text document streams.However, there is an additional need within the intelligence community to cluster related sets of content without advance training. To allow this function to happen at high speed, we have implemented a system that hierarchically clusters streaming content. The method, streaming hierarchical partitioning, is designed to be implemented in hardware and handle extremely high ingestion rates.As new documents are ingested, they are dynamically organized into a hierarchy, which has a fixed maximal size. Once this limit is reached, documents must consequently be excreted at a rate equaling their ingestion. The choice of documents to excrete is a point of interest -we present several autonomous heuristics for doing so intelligently, as well as a proposal for incorporating user interaction to focus attention on concepts of interest.A related desideratum is robust accommodation of concept 1 1 1-4244-

show abstract

High Speed Document Clustering in Reconfigurable Hardware

Cited by 6 publications

References 9 publications

Intelligent Avionics with Advanced Clustering

Intelligent Avionics with Advanced Clustering

K-means Clustering for Multispectral Images Using Floating-Point Divide

Streaming Hierarchical Clustering for Concept Mining

Contact Info

Product

Resources

About