Evolvable SPL management with partial knowledge

Abstract

In Machine Learning (ML), the resolution of anomaly detection problems in time series presents a great diversity of practices as it can correspond to many different contexts. These practices cover both grasping the business problem and designing the solution itself. By practice, we designate explicit and implicit steps toward resolving a problem, while a solution corresponds to a combination of algorithms selected for their performance on a given problem. Two related issues arise. The first one is that the practices are individual and not explicitly mutualized. The second one is that choosing one solution over another is all the more difficult to justify because the space of solutions and the evaluation criteria are vast and evolve rapidly with the advances in ML. To solve these issues and tame the evolving diversity in ML, a Software Product Line (SPL) approach can be envisaged to represent the variable set of solutions. However, this requires characterizing an ML business problem through an explicit set of criteria and justifying one ML solution over all others. The resolution of anomaly detection problems is thus different from finding the best configuration workflow from past configurations but lies more in guiding the configuration towards a solution that may never have been studied before. This paper proposes an SPL approach that capitalizes on past practices by exploiting a variability-aware representation to detect new criteria and constraints when practices adopt different solutions to seemingly similar problems. We report on the evaluation of our approach using a set of applications from the literature and an ML software company. We show how the analysis of practices makes it possible to consolidate the knowledge contained in the SPL.

Taming the Diversity of Computational Notebooks

“…In the longer term, we aim to evolve the SPL by involving data scientists in the enrichment process. This includes using automated reasoning techniques based on past configurations [5,14,24,35] and exploring model evolution [2,6,13,16,36]. Additionally, we plan to support multiple implementations within ML components using the Multi-Level Feature Trees approach [10,31].…”

“…We use the vocabulary of Idowu et al in [17] to present the assets that enable us to find notebooks or generate a primitive version of a new one 5 . We structure the application space into two main areas: firstly, the code source and job assets, which represent possible products (notebooks), and secondly, the already realized products that we organize into variability subspaces ExperimentProducts and NotebookProducts for easy retrieval and potential cloning.…”