Documentation Matters: Human-Centered AI System to Assist Data Science Code Documentation in Computational Notebooks

Abstract

Computational notebooks allow data scientists to express their ideas through a combination of code and documentation. However, data scientists often pay attention only to the code, and neglect creating or updating their documentation during quick iterations. Inspired by human documentation practices learned from 80 highly-voted Kaggle notebooks, we design and implement Themisto, an automated documentation generation system to explore how human-centered AI systems can support human data scientists in the machine learning code documentation scenario. Themisto facilitates the creation of documentation via three approaches: a deep-learning-based approach to generate documentation for source code, a query-based approach to retrieve online API documentation for source code, and a user prompt approach to nudge users to write documentation. We evaluated Themisto in a within-subjects experiment with 24 data science practitioners, and found that automated documentation generation techniques reduced the time for writing documentation, reminded participants to document code they would have ignored, and improved participants’ satisfaction with their computational notebook.

The Programmer's Assistant: Conversational Interaction with a Large Language Model for Software Development

“…Generative models based on the transformer architecture [96] have recently been applied to the domain of software engineering. Code-fluent large language models are capable of generating code from natural language descriptions [105], translating code from one language to another [75], generating unit tests [92], and even generating documentation for code [36,38,97,98]. These models are probabilistic systems, and as such, do not always produce perfect results (e.g.…”

“…Recently, models leveraging the transformer architecture [96] have been developed to perform domain-specific software engineering tasks, such as translating code between languages [75], generating documentation for code [36,38,97,98], and generating unit tests for code [92] (see Talamadupula [90] and Allamanis et al [5] for surveys). Recently developed foundation models -large language models that can be adapted to multiple tasks and which exhibit emergent behaviors for which they have not been explicitly trained [14] -have also proven to be capable with source code.…”

Improving Real-Time Collaborative Data Science Through Context-Aware Mechanisms

“…These findings motivate us to consider AI automation as a potential solution to support the tedious process of crafting documentation. Thus, we proposed Themisto, an automated code documentation generation system that integrates into the Jupyter Notebook environment [10]. We found that automated documentation generation techniques reduced the time for writing documentation, reminded participants to document code that they would have ignored, and improved their satisfaction with the final notebook.…”

“…When data scientists handle off analysis work, it is critical to understand how the analysis code is changed. Our previous interactive tools [9], [10] demonstrate two different approaches to help data scientists make sense of the code evolvement. However, explaining code changes only tackles half of the problem.…”

“…for generating natural language explanations of the data science code [10]; 3) a concept and prototype of implementing dataframe visualizations as a first-class citizen in data science programming environments to make sense of the impact of code changes [11]. Lastly, I will also present an ongoing project on improving awareness and avoiding conflict editing in real-time collaborative notebooks.…”