Capturing Provenance from Deep Learning Applications Using Keras-Prov and Colab: a Practical Approach

Abstract

Due to the exploratory nature of DNNs, DL specialists often need to modify the input dataset, change a filter when preprocessing input data, or fine-tune the models’ hyperparameters, while analyzing the evolution of the training. However, the specialist may lose track of what hyperparameter configurations have been used and tuned if these data are not properly registered. Thus, these configurations must be tracked and made available for the user’s analysis. One way of doing this is to use provenance data derivation traces to help the hyperparameter’s fine-tuning by providing a global data picture with clear dependencies. Current provenance solutions present provenance data disconnected from W3C PROV recommendation, which is difficult to reproduce and compare to other provenance data. To help with these challenges, we present Keras-Prov, an extension to the Keras deep learning library to collect provenance data compliant with PROV. To show the flexibility of Keras-Prov, we extend a previous Keras-Prov demonstration paper with larger experiments using GPUs with the help of Google Colab. Despite the challenges of running a DBMS with virtual environments, DL analysis with provenance has added trust and persistence in databases and PROV serializations. Experiments show Keras-Prov data analysis, during training execution, to support hyperparameter fine-tuning decisions, favoring the comparison, and reproducibility of such DL experiments. Keras-Prov is open source and can be downloaded from https://github.com/dbpina/keras-prov.

Capturing and querying fine-grained provenance of preprocessing pipelines in data science

“…between predictions and training data. However, this work has been used by [65] to create an entire tracking of data from pre-processing through deep learning. Future work in this area includes understanding the granularity of provenance required for users of deep learning systems.…”

Flower-PROV: Captura Distribuída de Dados de Proveniência em Experimentos de Aprendizado Federado

“…Por meio dos dados de proveniência, o usuário é capaz de consultar e analisar como os dados são produzidos em cada iterac ¸ão, consultando valores de hiperparâmetros, métricas de avaliac ¸ão, etc. Embora o uso de proveniência para compreensão de treinamento de modelos não seja algo novo [Chapman et al 2022, Pina et al 2023, ainda existem poucas propostas no contexto do AF [Peregrina et al 2022]. Entretanto, as soluc ¸ões propostas ou não capturam o histórico de derivac ¸ão completo (focando apenas em atribuic ¸ão de responsabilidades , ou seja, nos agentes que executam as atividades do workflow e não nas atividades em si ), ou possuem representac ¸ões proprietárias para os dados que não são aderentes a padrões de representac ¸ão de proveniência como o W3C PROV [Groth and Moreau 2013].…”