Towards a Holistic Integration of Spreadsheets with Databases: A Scalable Storage Engine for Presentational Data Management

Abstract

Spreadsheet software is the tool of choice for interactive ad-hoc data management, with adoption by billions of users. However, spreadsheets are not scalable, unlike database systems. On the other hand, database systems, while highly scalable, do not support interactivity as a first-class primitive. We are developing DATASPREAD, to holistically integrate spreadsheets as a frontend interface with databases as a back-end datastore, providing scalability to spreadsheets, and interactivity to databases, an integration we term presentational data management (PDM). In this paper, we make the first step towards this vision: developing a storage engine for PDM, studying how to flexibly represent spreadsheet data within a database and how to support and maintain access by position. We first conduct an extensive survey of spreadsheet use to motivate our functional requirements for a storage engine for PDM. We develop a natural set of mechanisms for flexibly representing spreadsheet data and demonstrate that identifying the optimal representation is NP-HARD; however, we develop an efficient approach to identify the optimal representation from an important and intuitive subclass of representations. We extend our mechanisms with positional access mechanisms that don't suffer from cascading update issues, leading to constant time access and modification performance. We evaluate these representations on a workload of typical spreadsheets and spreadsheet operations, providing up to 50% reduction in storage, and up to 50% reduction in formula evaluation time.

Enabling data science for the majority

“…Second, we can apply greedy decomposition instead of dynamic programming. Both these tricks give us a solution that is close to the overall optimal decomposition [13]. Our results show that the recursive-decomposition-based hybrid representation gives us a 2-3× reduction in storage or time to perform formula computation relative to a single table or triples-only representation [13].…”

“…Figure 7 displays one such decomposition, where the spreadsheet is decomposed into red and blue areas; each red area is stored as a separate table, while the blue areas are stored together in a single triples table. Unfortunately, we determined that identifying this optimal decomposition is NP-HARD via a reduction from the minimum edge-length partitioning of rectilinear polygons [13]. So, instead, we opt for a restricted space of decompositions called recursive decompositions that are formed by recursively decomposing areas of the spreadsheet (an analogy is to use scissors to cut a sheet of paper with vertical or horizontal cuts that cut the remaining region entirely).…”

“…So, instead, we opt for a restricted space of decompositions called recursive decompositions that are formed by recursively decomposing areas of the spreadsheet (an analogy is to use scissors to cut a sheet of paper with vertical or horizontal cuts that cut the remaining region entirely). It turns out that identifying the best recursive decomposition is in PTIME via dynamic programming [13]. While this is promising, the complexity is still O(n 5 ).…”

Characterizing Scalability Issues in Spreadsheet Software using Online Forums

“…Furthermore, not only are scalability issues very real problems plaguing those who want to analyze large amounts of data, but these issues touched on all four areas of actions users take to process data. This motivates the need for developing a spreadsheet application which continues to perform as expected on very large volumes of data [12].…”

“…The last class of problems are what we call scalability problems, i.e., situations where users are unable to complete standard operations in a reasonable amount of time when the scale or complexity of the dataset increases [7]. With the increasing availability of large volumes of data in most domains, we suspect that scalability problems are only going to become more common-as an example, our biologist collaborators routinely use spreadsheets, and can now generate large genomic (VCF) files, but are unable to open these files even simply to verify correctness in spreadsheet software [12].…”