Transactional Python for Durable Machine Learning: Vision, Challenges, and Feasibility

Chockchowwat, Supawit; Li, Zhaoheng; Park, Yongjoo

doi:10.1145/3595360.3595855

Proceedings of the Seventh Workshop on Data Management for End-to-End Machine Learning 2023

DOI: 10.1145/3595360.3595855

|View full text |Cite

Transactional Python for Durable Machine Learning: Vision, Challenges, and Feasibility

Supawit Chockchowwat

Zhaoheng Li

Yongjoo Park

Help me understand this report

Search citation statements

Order By: Relevance

Paper Sections

Select...

Citation Types

Supporting

Mentioning

Contrasting

Year Published

2023

Publication Types

Select...

Article1

Relationship

Self Cite0

Independent1

Authors

Journals

Cited by 1 publication

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

ElasticNotebook: Enabling Live Migration for Computational Notebooks

Li,

Gor,

Prabhu

et al. 2023

Proc. VLDB Endow.

View full text Add to dashboard Cite

Computational notebooks (e.g., Jupyter, Google Colab) are widely used for interactive data science and machine learning. In those frameworks, users can start a session , then execute cells (i.e., a set of statements) to create variables, train models, visualize results, etc. Unfortunately, existing notebook systems do not offer live migration: when a notebook launches on a new machine, it loses its state , preventing users from continuing their tasks from where they had left off. This is because, unlike DBMS, the sessions directly rely on underlying kernels (e.g., Python/R interpreters) without an additional data management layer. Existing techniques for preserving states, such as copying all variables or OS-level checkpointing, are unreliable (often fail), inefficient, and platform-dependent. Also, re-running code from scratch can be highly time-consuming. In this paper, we introduce a new notebook system, Elastic-Notebook, that offers live migration via checkpointing/restoration using a novel mechanism that is reliable, efficient, and platform-independent. Specifically, by observing all cell executions via transparent, lightweight monitoring, ElasticNotebook can find a reliable and efficient way (i.e., replication plan ) for reconstructing the original session state, considering variable-cell dependencies, observed runtime, variable sizes, etc. To this end, our new graph-based optimization problem finds how to reconstruct all variables (efficiently) from a subset of variables that can be transferred across machines. We show that ElasticNotebook reduces end-to-end migration and restoration times by 85%-98% and 94%-99%, respectively, on a variety (i.e., Kaggle, JWST, and Tutorial) of notebooks with negligible runtime and memory overheads of <2.5% and <10%.

show abstract

ElasticNotebook: Enabling Live Migration for Computational Notebooks

Li,

Gor,

Prabhu

et al. 2023

Proc. VLDB Endow.

View full text Add to dashboard Cite

show abstract

scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.

Contact Info

customersupport@researchsolutions.com

10624 S. Eastern Ave., Ste. A-614

Henderson, NV 89052, USA

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Blog Terms and Conditions API Terms Privacy Policy Contact Cookie Preferences Do Not Sell or Share My Personal Information

Made with 💙 for researchers

Part of the Research Solutions Family.

Transactional Python for Durable Machine Learning: Vision, Challenges, and Feasibility

Cited by 1 publication

References 21 publications

ElasticNotebook: Enabling Live Migration for Computational Notebooks

ElasticNotebook: Enabling Live Migration for Computational Notebooks

Contact Info

Product

Resources

About