Collaboration Challenges in Building ML-Enabled Systems: Communication, Documentation, Engineering, and Process

Nahar, Nadia; Zhou, Shurui; Lewis, Grace A.; Kästner, Christian

doi:10.48550/arxiv.2110.10234

Cited by 5 publications

(11 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Most ML models are developed by cross-functional teams with stakeholders in technical and non-technical roles. While collaboration is essential for deciding how a model should behave and identifying potential failures, there is often limited communication between stakeholders [40]. This can lead to unrealistic expectations of model performance or results that do not match designers' expectations.…”

Section: Collaboration and Reportingmentioning

confidence: 99%

“…It requires going beyond measuring aggregate metrics, such as accuracy or F1 score, and understanding patterns of model output for subgroups, or slices, of input data. Enumerating what behaviors a model should have or what types of errors it could produce requires collaboration between stakeholders such as ML engineers, designers, and domain experts [40,54]. Behavioral evaluation is also a continuous, iterative process, as practitioners update their models to fix limitations or add features while ensuring that new failures are not introduced [10].…”

Section: Introductionmentioning

confidence: 99%

“…Despite a growing focus on the importance of behavioral evaluation, it remains a challenging task in practice. Models are often developed without practitioners having clear model requirements or a deep understanding of the products or services in which the model will be deployed [40]. Furthermore, many behavioral evaluation tools, such as fairness toolkits, often do not support the types of models, data, and behaviors that practitioners work with in the real world [17].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Zeno: An Interactive Framework for Behavioral Evaluation of Machine Learning

Cabrera,

Fu,

Bertucci

et al. 2023

Preprint

View full text Add to dashboard Cite

Figure 1: zeno is a framework for behavioral evaluation of machine learning (ML) models. It has two components, a Python API and an interactive UI. The API is used to generate information such as model outputs and metrics. Users then interact with the UI to see metrics, create slices, and write unit tests. In this toy example, a user is evaluating a cat and dog classifier. They see that the model has lower accuracy for dogs with pointy ears, and create a test expecting the slice accuracy to be higher than 70%.

show abstract

Section: Collaboration and Reportingmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Zeno: An Interactive Framework for Behavioral Evaluation of Machine Learning

Cabrera,

Fu,

Bertucci

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…The massive adoption of AI-based technologies in the modern software industry is raising new intriguing challenges, many of which concern the shift of machine learning (ML) model prototypes into production-ready software components. Often, a variety of factors contribute to rendering this shift difficult and costly to achieve; these range from technical matters -like the complexity of reproducing lab model performances in live systems -to human aspects, arising from the coexistence of varied backgrounds and perspectives in multidisciplinary teams [4].…”

Section: Introductionmentioning

confidence: 99%

Pynblint: a Static Analyzer for Python Jupyter Notebooks

Quaranta,

Calefato,

Lanubile

2022

Preprint

View full text Add to dashboard Cite

Jupyter Notebook is the tool of choice of many data scientists in the early stages of ML workflows. The notebook format, however, has been criticized for inducing bad programming practices; indeed, researchers have already shown that open-source repositories are inundated by poor-quality notebooks. Low-quality output from the prototypical stages of ML workflows constitutes a clear bottleneck towards the productization of ML models. To foster the creation of better notebooks, we developed Pynblint, a static analyzer for Jupyter notebooks written in Python. The tool checks the compliance of notebooks (and surrounding repositories) with a set of empirically validated best practices and provides targeted recommendations when violations are detected.

show abstract

“…ML-enabled systems, that is, software systems incorporating machine learning (ML) models, are receiving more and more attention from both researchers and practitioners [8,12,13].…”

Section: Introductionmentioning

confidence: 99%

A Preliminary Investigation of MLOps Practices in GitHub

Calefato

Lanubile

Quaranta

2022

Proceedings of the 16th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement

View full text Add to dashboard Cite

Background. The rapid and growing popularity of machine learning (ML) applications has led to an increasing interest in MLOps, that is, the practice of continuous integration and deployment (CI/CD) of ML-enabled systems. Aims. Since changes may affect not only the code but also the ML model parameters and the data themselves, the automation of traditional CI/CD needs to be extended to manage model retraining in production. Method. In this paper, we present an initial investigation of the MLOps practices implemented in a set of ML-enabled systems retrieved from GitHub, focusing on GitHub Actions and CML, two solutions to automate the development workflow. Results. Our preliminary results suggest that the adoption of MLOps workflows in open-source GitHub projects is currently rather limited. Conclusions. Issues are also identified, which can guide future research work.

show abstract

Collaboration Challenges in Building ML-Enabled Systems: Communication, Documentation, Engineering, and Process

Cited by 5 publications

References 0 publications

Zeno: An Interactive Framework for Behavioral Evaluation of Machine Learning

Zeno: An Interactive Framework for Behavioral Evaluation of Machine Learning

Pynblint: a Static Analyzer for Python Jupyter Notebooks

A Preliminary Investigation of MLOps Practices in GitHub

Contact Info

Product

Resources

About