From Ad-Hoc Data Analytics to DataOps

Munappy, Aiswarya Raj; Mattos, David Issa; Bosch, Jan; Olsson, Helena Holmström; Dakkak, Anas

doi:10.1145/3379177.3388909

Cited by 56 publications

(19 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Furthermore, the proposed PDS structure should comply with DataSecOps: a methodology for improving data quality and analysis within the lifecycle of data use. This methodology is oriented and integrated for data collection and analysis [16] [17]. DataSecOps includes security internalization in the DataOps data lifecycle and applies security policies and governance to the stages of data collection, access, and analysis.…”

Section: Proposal Of the Mydata Platform And The Pds Modelmentioning

confidence: 99%

MyData Personal Data Store Model(PDS) to Enhance Information Security for Guarantee the Self-determination rights

2022

KSII TIIS

View full text Add to dashboard Cite

The European Union recently established the General Data Protection Regulation (GDPR) for secure data use and personal information protection. Inspired by this, South Korea revised their Personal Information Protection Act, the Act on Promotion of Information and Communications Network Utilization and Information Protection, and the Credit Information Use and Protection Act, collectively known as the "Three Data Bills," which prescribe safe personal information use based on pseudonymous data processing. Based on these bills, the personal data store (PDS) has received attention because it utilizes the MyData service, which actively manages and controls personal information based on the approval of individuals, and it practically ensures their rights to informational self-determination. Various types of PDS models have been developed by several countries (e.g., the US, Europe, and Japan) and global platform firms. The South Korean government has now initiated MyData service projects for personal information use in the financial field, focusing on personal credit information management. There is also a need to verify the efficacy of this service in diverse fields (e.g., medical). However, despite the increased attention, existing MyData models and frameworks do not satisfy security requirements of ensured traceability, transparency, and distributed authentication for personal information use. This study analyzes primary PDS models and compares them to an internationally standardized framework for personal information security with guidelines on MyData so that a proper PDS model can be proposed for South Korea.

show abstract

Section: Proposal Of the Mydata Platform And The Pds Modelmentioning

confidence: 99%

MyData Personal Data Store Model(PDS) to Enhance Information Security for Guarantee the Self-determination rights

2022

KSII TIIS

View full text Add to dashboard Cite

show abstract

“…Some bad practices in data manipulation can end up in a misleading interpretation of the achieved results. The use of tools that allow the selection of the pertinent steps in an ad-hoc designed pipeline helps to reduce programming errors [26]. The Pipeline object of the scikit-learn module allows combining several transformers and an estimator to create a combined estimator [25].…”

Section: Introductionmentioning

confidence: 99%

A Framework for the Optimization of Complex Cyber-Physical Systems via Directed Acyclic Graph

Castejón-Limas

Fernández-Robles

Aláiz‐Moretón

et al. 2022

Sensors

View full text Add to dashboard Cite

Mathematical modeling and data-driven methodologies are frequently required to optimize industrial processes in the context of Cyber-Physical Systems (CPS). This paper introduces the PipeGraph software library, an open-source python toolbox for easing the creation of machine learning models by using Directed Acyclic Graph (DAG)-like implementations that can be used for CPS. scikit-learn’s Pipeline is a very useful tool to bind a sequence of transformers and a final estimator in a single unit capable of working itself as an estimator. It sequentially assembles several steps that can be cross-validated together while setting different parameters. Steps encapsulation secures the experiment from data leakage during the training phase. The scientific goal of PipeGraph is to extend the concept of Pipeline by using a graph structure that can handle scikit-learn’s objects in DAG layouts. It allows performing diverse operations, instead of only transformations, following the topological ordering of the steps in the graph; it provides access to all the data generated along the intermediate steps; and it is compatible with GridSearchCV function to tune the hyperparameters of the steps. It is also not limited to (X,y) entries. Moreover, it has been proposed as part of the scikit-learn-contrib supported project, and is fully compatible with scikit-learn. Documentation and unitary tests are publicly available together with the source code. Two case studies are analyzed in which PipeGraph proves to be essential in improving CPS modeling and optimization: the first is about the optimization of a heat exchange management system, and the second deals with the detection of anomalies in manufacturing processes.

show abstract

“…According to reference [43], DataOps can be defined as "an approach that accelerates the delivery of high-quality results by automation and orchestration of data life cycle stages". It brings speed and agility to the end-to-end process of data pipelines, from collection to delivery.…”

Section: Dataopsmentioning

confidence: 99%

“…Much more recently, new ways of working have begun to emerge and gain traction for companies to leverage on what these new technologies aim to deliver. Practices such as DevOps [118] [119] [120], DataOps [43] and MLOps [121] [44] are being introduced with the intention of advancing product automation as well as quality in terms of development, data and ML operations. Penners and Dyck [122] propose DevOps as a collaboration of teams working in development and IT operations within a software-intensive organization to deliver faster software changes [118].…”

Section: Digitalization: New Technologies and Ways Of Workingmentioning

confidence: 99%

“…They are inadequacies in infrastructure automation, high demand for skills and knowledge, project and resource constraints and monitoring challenges. According to reference [43], DataOps can be defined as "an approach that accelerates the delivery of high-quality results by automation and orchestration of data life cycle stages. DataOps chooses the best practices, processes, tools and technologies from Agile software engineering and DevOps to regulate analytics development, optimizing code verification, building and delivering new analytics and thereby fostering a culture of collaboration and continuous improvement".…”

Section: Digitalization: New Technologies and Ways Of Workingmentioning

confidence: 99%

See 1 more Smart Citation

Design Methods and Processes for ML/DL models

John¹

View full text Add to dashboard Cite

Context: With the advent of Machine Learning (ML) and especially Deep Learning (DL) technology, companies are increasingly using Artificial Intelligence (AI) in systems, along with electronics and software. Nevertheless, the end-to-end process of developing, deploying and evolving ML and DL models in companies brings some challenges related to the design and scaling of these models. For example, access to and availability of data is often challenging, and activities such as collecting, cleaning, preprocessing, and storing data, as well as training, deploying and monitoring the model(s) are complex. Regardless of the level of expertise and/or access to data scientists, companies in all embedded systems domain struggle to build high-performing models due to a lack of established and systematic design methods and processes. Objective: The overall objective is to establish systematic and structured design methods and processes for the end-to-end process of developing, deploying and successfully evolving ML/DL models. Method: To achieve the objective, we conducted our research in close collaboration with companies in the embedded systems domain using different empirical research methods such as case study, action research and literature review. Results and Conclusions: This research provides six main results: First, it identifies the activities that companies undertake in parallel to develop, deploy and evolve ML/DL models, and the challenges associated with them. Second, it presents a conceptual framework for the continuous delivery of ML/DL models to accelerate AI-driven business in companies. Third, it presents a framework based on current literature to accelerate the end-to-end deployment process and advance knowledge on how to integrate, deploy and operationalize ML/DL models. Fourth, it develops a generic framework with five architectural alternatives for deploying ML/DL models at the edge. These architectural alternatives range from a centralized architecture that prioritizes (re)training in the cloud to a decentralized architecture that prioritizes (re)training at the edge. Fifth, it identifies key factors to help companies decide which architecture to choose for deploying ML/DL models. Finally, it explores how MLOps, as a practice that brings together data scientist teams and operations, ensures the continuous delivery and evolution of models.

show abstract

From Ad-Hoc Data Analytics to DataOps

Cited by 56 publications

References 10 publications

MyData Personal Data Store Model(PDS) to Enhance Information Security for Guarantee the Self-determination rights

MyData Personal Data Store Model(PDS) to Enhance Information Security for Guarantee the Self-determination rights

A Framework for the Optimization of Complex Cyber-Physical Systems via Directed Acyclic Graph

Design Methods and Processes for ML/DL models

Contact Info

Product

Resources

About