Approach for mapping data to client projects

Abstract

In today's technological era, the volume of data being processed on a daily basis is growing exponentially. The influence of data is affecting consulting firms and the expectations client companies have of consultants. Consulting firms are expected to not only use their client's data but also to leverage open source data and data from other firms that can help address the clients' needs. Manually reviewing the relevance of a dataset to a given problem is time intensive and prone to error. Additionally, it is near impossible to detect non-intuitive correlations in the data through manual review. Hence, this project focuses on building an application that enables users to automatically find and assess the applicability of datasets to a given problem. Given that the project objectives are to increase the applicability of individual datasets, to increase the applicability of clusters of correlated datasets, and to increase the usability of individual datasets, three metrics were derived respectively. They are relevance, coverage, and quality of data. The application, known as UVa Open Miner, measures the relevance of the dataset to the client problem, the degree to which a group of datasets covers different dimensions of a client problem, and the quality of the dataset. The application can be refactored into a reusable solution that consulting firms can use for their client work.