ParallelTopics: A probabilistic approach to exploring document collections

Abstract

Scalable and effective analysis of large text corpora remains a challenging problem as our ability to collect textual data continues to increase at an exponential rate. To help users make sense of large text corpora, we present a novel visual analytics system, ParallelTopics, which integrates a state-of-the-art probabilistic topic model Latent Dirichlet Allocation (LDA) with interactive visualization. To describe a corpus of documents, ParallelTopics first extracts a set of semantically meaningful topics using LDA. Unlike most traditional clustering techniques in which a document is assigned to a specific cluster, the LDA model accounts for different topical aspects of each individual document. This permits effective full text analysis of larger documents that may contain multiple topics.To highlight this property of the model, ParallelTopics utilizes the parallel coordinate metaphor to present the probabilistic distribution of a document across topics. Such representation allows the users to discover single-topic vs. multi-topic documents and the relative importance of each topic to a document of interest. In addition, since * e-mail: wdou1@uncc.edu † e-mail:viztang@gmail.com ‡ e-mail:remco@cs.tufts.edu § e-mail: ribarsky@uncc.edu most text corpora are inherently temporal, ParallelTopics also depicts the topic evolution over time. We have applied ParallelTopics to exploring and analyzing several text corpora, including the scientific proposals awarded by the National Science Foundation and the publications in the VAST community over the years. To demonstrate the efficacy of ParallelTopics, we conducted several expert evaluations, the results of which are reported in this paper.

An Affordance-Based Framework for Human Computation and Human-Computer Collaboration

“…In presenting a preliminary framework for describing and comparing systems involving human and machine collaborators, we aspire to lay the foundation for a more rigorous analysis of the tools and approaches presented by our field, thereby paving the way for the construction of an increasingly robust understanding of analytical reasoning and how to best support insight generation. [41] CrowdSearch [80] ParallelTopics [23] Dissimilarity [50] VH+ML [28] Implicit tagging [62] reCAPTCHA [77] VizWiz [10] Phetch [74] ESP Game [73] KissKissBan [33] LabelMe [59] Ka-captcha [21] PeekABoom [76] MRI [12] iView [83] iVisClassifier [18] Saliency [37] RP Explorer [3] DimStiller [36] WireVis [46] Action trails [65] NetClinic [47] Trajectories [4] Risk assessment [51] Automatic transfer functions [57] MDX [66] Automated+viz [68] CzSaw [39] Fold.it [20] HRI scripts [17] Animated agents for VR [56] VA Model-learning [29] EyeSpy [6] MonoTrans2 [35] CastingWords [16] Click2Annotate [15] Wrangler [40] Soylent [8] Crowdsourced solutions [67] Crowdsourced design [81] Stress OutSourced …”

“…In ParallelTopics (see Fig. 2a), computational methods for manipulating large datasets have been used to help users navigate and make sense of massive text corpora [23]. It has also been utilized to refine classification models and performing dimension reduction [18,29,51], interactively cluster data [4], and automatically extract transfer functions from user-selected data [57].…”

“…That is, apart from human bias introduced during the programming of the system, (a) ParallelTopics [23] (b) Saliency-Assisted Navigation [37] (c) MDX [66] machines are able to operate and report on numerically or computationally significant information without experiential or sociocultural influence. In Visual Analytics, we have seen this affordance leveraged to help analysts direct their attention for natural disaster prediction [66] (see Fig.…”

“…The study of general human-computer collaboration offers a plethora of examples of successful human/machine teams [15,23,37,39,41,46,50,65,66,68,83]. Developments in supervised machine learning in the visualization community present several vetted techniques for human intervention into computationally complex tasks [3,4,12,18,28,29,36,47,51,57].…”

The Whole Story: Building the Complete History of a Place

“…Now available are a collection of interactive data analysis methods that fit well with interactive visualization. These methods include topic modeling for large text collections [1], image understanding for large image collections [2], extraction of 3D geometry (e.g., building location, shapes, volumes) from perspective images [3], GIS methods to extract and organize meaningful layers from maps, and other computer vision techniques that, for example, can understand unstructured video or precisely locate detailed snapshots of scenes.…”