Design Challenges/Solutions for Environments Supporting the Analysis of Social Media Data in Crisis Informatics Research

Batch to Real-Time: Incremental Data Collection & Analytics Platform

“…EPIC Analyze makes use of Apache Solr to answer analyst queries at interactive speeds since EPIC Analyze only ever displays 50 query results at any one time due to a carefully-designed pagination mechanism [3]. Therefore, retrieving small chunks of query results by Apache Solr does not slow down analysis tasks.…”

“…In this paper, we focus exclusively on the work we performed to enable real-time analysis of streaming social media data during times of crisis. There are, of course, additional challenges beyond the ones mentioned above; for instance, data intensive software systems require well-designed user interfaces to facilitate access to large data sets and to allow users to search, filter, sort, query, and analyze that data [3,6,9]. While we encountered these challenges when creating the IDCAP, we do not discuss them here.…”

“…With this new design, Project EPIC analysts will be able to filter, search, sample, sort, and perform exploratory analytics efficiently on large Twitter datasets using IDCAP. This analysis is currently supported in the IDCAPs front-end web app-the IDCA App-and will eventually be integrated into a future version of EPIC Analyze [3]. The IDCA App is discussed in Section 4.…”

“…We have also reported on EPIC Analyze [3], the data analysis environment we built to allow Project EPIC analysts to browse, search, sample, filter, sort, and annotate large Twitter data sets. EPIC Analyze is focused on the batch processing of crisis data after a crisis event is over.…”

“…The software architecture of a system provides a view of a system's components, the way those components interact, and their responsibilities [10,16]. Therefore, using the right set of technologies allows one to meet the design goals of the system and provides efficient generation of metrics and statistics and fast exploratory analysis on large datasets [3]. The rise of big data has triggered the demand for a scalable data analytics software architecture and a flexible data processing infrastructure to handle diverse and various analysis requests in real-time and later in the batch processing of large amounts of data.…”