Demonstration of VegaPlus: Optimizing Declarative Visualization Languages

VegaFusion: Automatic Server-Side Scaling for Interactive Vega Visualizations

“…Moritz et al proposed a comprehensive client-server architecture to dynamically partition a declarative visualization plan based on a cost model and a predictive component to prefetch the results of likely future user interactions [6], and VegaFusion can be considered the implementation of some of these ideas. VegaPlus [15] proposes a similar system to automatically convert parts of the Vega transformation pipelines into SQL statements for server-side processing, albeit with slightly different design goals (notably not R3, see below). Tapestry [9] uses a similar client-server architecture to scalably embed interactive volume rendering visualizations in web pages.…”

“…An SQL datastore was an alternative here, but converting VegaFusion queries into SQL would have made it much more challenging to implement the fine-grained caching we sought by forcing us to rely on the SQL datastore's harder-to-control internal caching mechanism: there is no way in SQL to specify which intermediate outputs a query engine should cache. The architecture of VegaPlus [15] is quite similar to VegaFusion except for this key design choice, because R3 was not a requirement for VegaPlus.…”

“…The most promising area is the expansion of the Runtime to directly access datastores which support efficient computation such as SQL databases or Spark clusters and where data might already be stored. Instead of pulling data out of such systems into DataFusion, the Runtime could dynamically compile portions of queries into SQL, as VegaPlus [15] does it, or other systems such as Dask [4] or Spark [16] and let upstream systems handle part of the computation, while the Runtime itself continues to manage the cache.…”