SDPF: A Framework for Online, Real-time Cleansing of Upstream Operating Data

Abstract

Since the quality of the raw upstream operating data can be poor in many instances (due to errors, incompleteness, and inconsistency), there is often an urgent need to cleanse the data in real time before using the data for process and business decision-making. In this paper, we present the design and development of an integrated data cleansing framework that can address a variety of upstream operating data quality issues systematically. Our proposed framework, dubbed SDPF (short for Scalable Data Processing Framework), benefits from the following features: Online/Real-time Data Cleansing, Scalability, Configurability, Reusability, and Comprehensiveness. We have tested and verified the performance of our data cleansing system with both real upstream operating data collected and synthetic data.