Error Identification Strategies for Python Jupyter Notebooks

Abstract

Computational notebooks-such as Jupyter or Colab-combine text and data analysis code. They have become ubiquitous in the world of data science and exploratory data analysis. Since these notebooks present a different programming paradigm than conventional IDE-driven programming, it is plausible that debugging in computational notebooks might also be different. More specifically, since creating notebooks blends domain knowledge, statistical analysis, and programming, the ways in which notebook users find and fix errors in these different forms might be different. In this paper, we present an exploratory, observational study into how notebook users find and understand potential errors in notebooks. We presented users with notebooks pre-populated with common notebook errors-errors rooted in either the statistical data analysis, the knowledge of domain concepts, or in the programming. We then analyzed the strategies our study participants used to find these errors and determined how successful each strategy was at identifying errors. Our findings indicate that while the notebook programming environment is different from the environments used for traditional programming, debugging strategies remain quite similar. It is our hope that the insights presented in this paper will help both notebook tool designers and educators make changes to improve how data scientists discover errors more easily in the notebooks they write.

“Looks Okay to Me”: A Study of Best Practice in Data Analysis Code Review

“…The notebooks we have developed are Creative Commons licensed, and could be used by other researchers or instructors seeking to teach or assess data analytic troubleshooting skills (Bodwin et al, 2022). In fact, the Python translations of the notebooks have already been used for a replication of this work (Robinson et al, 2022). Our three-stage coding practice, starting with separate coding by multiple authors and funneling down to a final set of themes, gives credence to the set of codes we eventually identified.…”