DockerizeMe: Automatic Inference of Environment Dependencies for Python Code Snippets

Abstract

Platforms like Stack Overflow and GitHub's gist system promote the sharing of ideas and programming techniques via the distribution of code snippets designed to illustrate particular tasks. Python, a popular and fast-growing programming language, sees heavy use on both sites, with nearly one million questions asked on Stack Overflow and 400 thousand public gists on GitHub. Unfortunately, around 75% of the Python example code shared through these sites cannot be directly executed. When run in a clean environment, over 50% of public Python gists fail due to an import error for a missing library.We present DockerizeMe, a technique for inferring the dependencies needed to execute a Python code snippet without import error. DockerizeMe starts with offline knowledge acquisition of the resources and dependencies for popular Python packages from the Python Package Index (PyPI). It then builds Docker specifications using a graph-based inference procedure. Our inference procedure resolves import errors in 892 out of nearly 3,000 gists from the Gistable dataset for which Gistable's baseline approach could not find and install all dependencies.

V2: Fast Detection of Configuration Drift in Python

“…V2, "Version 2", is an extension of DockerizeMe [9] that adds support for validating multiple environments and reasoning over dependency versions. V2 determines if a code snippet or its dependencies is out-of-date by discovering instances of configuration drift.…”

“…This leaves 2,096 gists that we refer to as the Gist dataset. Prior work was unable to make 1,999 of these gists execute successfully after candidate environment generation using the latest versions of all dependencies [9]. Second, we collect larger Python code snippets in the form of Python notebooks.…”

Restoring reproducibility of Jupyter notebooks

“…In [3] In [2] from pandas import read_excel as re re(…) # read_excel(…) from pandas import * read_excel(…) In [6] In [4] The following code cell demonstrates how to draw a histogram using pandas's DataFrame. This is an example of Jupyter notebook.…”

Docable: evaluating the executability of software tutorials

“…Other interesting approach would involve automatic inference and construction base images for running tutorials. For example, DockerizeMe [13] is a technique that given a Python code snippet, can automatically infer system and transitive dependencies required to successfully execute a code snippet. Such a technique could be adopted to determine the dependencies necessary for running a tutorial.…”