No abstract

No abstract

Estimating the dependency of variables is a fundamental task in data analysis. Identifying the relevant attributes in databases leads to better data understanding and also improves the performance of learning algorithms, both in terms of runtime and quality. In data streams, dependency monitoring provides key insights into the underlying process, but is challenging. In this paper, we propose Monte Carlo Dependency Estimation (MCDE), a theoretical framework to estimate multivariate dependency in static and dynamic data. MCDE quantifies dependency as the average discrepancy between marginal and conditional distributions via Monte Carlo simulations. Based on this framework, we present Mann-Whitney P (MWP), a novel dependency estimator. We show that MWP satisfies a number of desirable properties and can accommodate any kind of numerical data. We demonstrate the superiority of our estimator by comparing it to the state-ofthe-art multivariate dependency measures.

Estimating dependencies from data is a fundamental task of Knowledge Discovery. Identifying the relevant variables leads to a better understanding of data and improves both the runtime and the outcomes of downstream Data Mining tasks. Dependency estimation from static numerical data has received much attention. However, real-world data often occurs as heterogeneous data streams: On the one hand, data is collected online and is virtually infinite. On the other hand, the various components of a stream may be of different types, e.g., numerical, ordinal or categorical. For this setting, we propose Monte Carlo Dependency Estimation (MCDE), a framework that quantifies multivariate dependency as the average statistical discrepancy between marginal and conditional distributions, via Monte Carlo simulations. MCDE handles heterogeneity by leveraging three statistical tests: the Mann-Whitney U, the Kolmogorov-Smirnov and the Chi-Squared test. We demonstrate that MCDE goes beyond the state of the art regarding dependency estimation by meeting a broad set of requirements. Finally, we show with a real-world use case that MCDE can discover useful patterns in heterogeneous data streams.

The recent development of renewable energy sources (RES) challenges energy systems and opens many new research questions. Energy System Models (ESM) are important tools to study these problems. However, including RES into ESM strongly increases the model complexity, because one needs to model the fluctuant, weather-dependent electricity production from RES with a high level of granularity. This leads to long execution times. To deal with this issue, our objective is to reduce the input time series of ESM without losing their energy-related key characteristics, such as weather-dependent fluctuations in production or peak demands. This task is challenging, because of the variety and high-dimensionality of the data. We describe a carefully engineered data-processing pipeline to reduce energy time series. We use Self-Organizing Maps, a specific kind of neural network, to select “representative days”. We show that our approach outperforms the existing ones with respect to the quality of ESM results, and leads to a significant reduction of ESM execution times.