Collaboration challenges in building ML-enabled systems

Opening up ChatGPT: Tracking openness, transparency, and accountability in instruction-tuned text generators

“…In the age of large language models, this is amplified by the possibility to extract human labour and repackage it in amiable conversational formats. Openness not only aligns with principles of sound and ethical scholarship [51]; it also safeguards transparent and reproducible research [40,41]. Recent work on legal datasets offers an example in responsible data curation with insights that may be more broadly applicable [21].…”

Speeding Up Transformer Training By Using Dataset Subsampling - An Exploratory Analysis

“…Some challenges can impact several phases of the ML lifecycle, such as collaboration among diverse teams and roles, including software and data engineers, data scientists, and other stakeholders (Takeuchi & Yamamoto, 2020;Nahar et al, 2022;Pei et al, 2022;. Furthermore, there are challenges of bias, fairness, and accountability in ethics (Mehrabi et al, 2021;Kim & Doshi-Velez, 2021), various regulations set by law (Marchant, 2011;Politou et al, 2018) and adversarial attacks in security (Ren et al, 2020;Rosenberg et al, 2021), among others.…”

“…Often it is necessary to update the ML model regularly after it has been deployed and is running in production, in order to keep it aligned with the most current changes in data and environment. The need for updating models is one of the most important requirements of ML production systems (Pacheco et al, 2018;Abdelkader, 2020;Lakshmanan et al, 2020;Paleyes et al, 2022;Huyen, 2022;Wu & Xie, 2022;Nahar et al, 2022). In this section, we discuss the challenges of model updating and how MTL approaches can alleviate these challenges.…”

Maintaining and Monitoring AIOps Models Against Concept Drift

“…One solution that ML practitioners adopt to handle the evolving character of data is retraining/updating ML models over time [24]. Periodical model retraining has also been studied for failure detection AIOps solutions [22], [21] and has proved that continuous model updates achieve better performance over time compared to non-updated models.…”

“…Regarding concept drift monitoring-based model retraining, this technique was not previously applied to any AIOps solution. Furthermore, previous work suggests that organizations do not have monitoring infrastructure to detect drift in production [24], [23] and they only perform periodic model retraining based on human decisions. Examining the impact of drift detection monitoring tools is the first step toward automating the maintenance of machine learning in production.…”