Interactive Learning for Identifying Relevant Tweets to Support Real-time Situational Awareness

Abstract

InterfaceAutomated Label Verify Label Fig. 1. Our interactive learning framework allows users to train text relevance classifiers in real-time to improve situational awareness.In this example, a real-time tweet regarding a car accident is incorrectly classified as "Irrelevant". Through the SMART 2.0 interface, the user can view its label and correct it to "Relevant", thereby retraining and improving the classifier for incoming streaming data.Abstract-Various domain users are increasingly leveraging real-time social media data to gain rapid situational awareness. However, due to the high noise in the deluge of data, effectively determining semantically relevant information can be difficult, further complicated by the changing definition of relevancy by each end user for different events. The majority of existing methods for short text relevance classification fail to incorporate users' knowledge into the classification process. Existing methods that incorporate interactive user feedback focus on historical datasets. Therefore, classifiers cannot be interactively retrained for specific events or user-dependent needs in real-time. This limits real-time situational awareness, as streaming data that is incorrectly classified cannot be corrected immediately, permitting the possibility for important incoming data to be incorrectly classified as well. We present a novel interactive learning framework to improve the classification process in which the user iteratively corrects the relevancy of tweets in real-time to train the classification model on-the-fly for immediate predictive improvements. We computationally evaluate our classification model adapted to learn at interactive rates. Our results show that our approach outperforms state-of-the-art machine learning models. In addition, we integrate our framework with the extended Social Media Analytics and Reporting Toolkit (SMART) 2.0 system, allowing the use of our interactive learning framework within a visual analytics system tailored for real-time situational awareness. To demonstrate our framework's effectiveness, we provide domain expert feedback from first responders who used the extended SMART 2.0 system.Index Terms-Interactive machine learning, human-computer interaction, social media analytics, emergency/disaster management, situational awareness

City-level Geolocation of Tweets for Real-time Visual Analytics

“…The Social Media Analytics and Reporting Toolkit (SMART) [17,18] is a system for visual analysis of geotagged, publiclyavailable real-time tweets to enhance situational awareness and expedite emergency response. SMART has been used by over 300 first responders in 70 organizations for major events, such as presidential inaugurations and sports games.…”

“…In addition, SMART users have frequently indicated that they would prefer more data to less, even if it is inaccurate, since they might be able to identify relevant tweets that would otherwise not be present without geolocation inference. SMART's integrated interactive learning allows users to train its relevance filters in real-time using a human-in-the-loop learning solution to filter out noise [17].…”

“…Further, the utilization of state-of-the-art prediction models in real-time visual analytics systems has not been explored. In this paper, we report on our ongoing research and (i) adapt an existing geolocation prediction method, computationally improve its accuracy, and integrate it with SMART [17,18], a visual analytics system designed to facilitate situational awareness, (ii) demonstrate the utility of geolocation prediction for real-time systems, and (iii) evaluate the effectiveness of the integrated system using Twitter's public feed effective in September 2019.…”

Situational Awareness Enhanced through Social Media Analytics: A Survey of First Responders

“…SMART [6,9] was developed at Purdue University through an iterative, user-centered design by collecting feedback from the first responder community to ensure its usefulness. SMART provides a user-friendly, graphical web-based user interface, hiding the complexity of advanced algorithms filtering and summarizing social media data, which allows users to interactively explore, filter, and visualize trends and anomalies in real-time geo-tagged Twitter data.…”

“…There has been considerable research on the potential and use of social media data for situational awareness, such as disaster response, event monitoring, or public sentiment analysis [1][2][3][4][5]. In particular, researchers have provided visual analytics approaches combining visualization and data analysis [2][3][4][5][6][7], geoparsing for crisis-mapping [7,8], cloud-based mobile services for on-the-ground information [14], and data mining and natural language processing techniques such as clustering and named entity recognition [15].…”