Towards Interactive Data Exploration

Abstract

Enabling interactive visualization over new datasets at "human speed" is key to democratizing data science and maximizing human productivity. In this work, we first argue why existing analytics infrastructures do not support interactive data exploration and outline the challenges and opportunities of building a system specifically designed for interactive data exploration. Furthermore, we present the results of building IDEA, a new type of system for interactive data exploration that is specifically designed to integrate seamlessly with existing data management landscapes and allow users to explore their data instantly without expensive data preparation costs. Finally, we discuss other important considerations for interactive data exploration systems including benchmarking, natural language interfaces, as well as interactive machine learning.

Addressing the need for interactive, efficient, and reproducible data processing in ecology with the datacleanr R package

“…Ideally, researchers should be able to engage with their data, across scales and dimensions as diligently as needed, with as little effort as possible. Accordingly, interactive processing is increasingly called for and deemed critical [ 18 ] for ensuring best practices in data exploration, and quality control when dealing with high-volume data and beyond [e.g., 19 , 20 ]. Indeed, interactive exploration is increasingly provided through open-source graphing frameworks (e.g., plotly; https://plotly.com/ or, highcharts; https://highcharts.com/ ) and/or commercially-licensed software (e.g., Tableau®; https://tableau.com/ ).…”

ELII: A novel inverted index for fast temporal query, with application to a large Covid-19 EHR dataset

“…Even for those that are equipped with a graphical user interface, there is a general lack of systematic and dedicated support for temporal queries in both the query language and the user interface [19] . Near real-time response to temporal query is one of the most computationally challenging aspects for interactive cohort exploration [20] . Methods for exploring, querying and interacting with data need to be improved to cope with the size and complexity of data.…”