ConnectorX

Abstract

Data is often stored in a database management system (DBMS) but dataframe libraries are widely used among data scientists. An important but challenging problem is how to bridge the gap between databases and dataframes. To solve this problem, we present ConnectorX, a client library that enables fast and memory-efficient data loading from various databases to different dataframes. We first investigate why the loading process is slow and consumes large memory. We surprisingly find that the main overhead comes from the client-side rather than query execution or data transfer. We integrate several existing and new techniques to reduce the overhead and carefully design the system architecture and interface to make ConnectorX easy to extend to various databases and dataframes. Moreover, we propose server-side result partitioning that can be adopted by DBMSs in order to better support exporting data to data science tools. We conduct extensive experiments to evaluate ConnectorX and compare it with popular libraries. The results show that ConnectorX significantly outperforms existing solutions. ConnectorX is open sourced at: https://github.com/sfu-db/connector-x.

A Comparison of Decision Forest Inference Platforms from A Database Perspective

“…Therefore, for netsDB, the testing datasets were stored natively. For other platforms, the testing datasets, except for Epsilon and Criteo, were stored in tabular format in a PostgreSQL database installed on the same machine, with the database connection accelerated using the state-of-art Connec-torX library [54]. Besides, Epsilon has 2000 features, and Criteo has 1 million features, but PostgreSQL only supports up to 1600 columns [8].…”