A Bias-Variance Analysis of a Real World Learning Problem: The CoIL Challenge 2000

Abstract

Abstract. The CoIL Challenge 2000 data mining competition attracted a wide variety of solutions, both in terms of approaches and performance. The goal of the competition was to predict who would be interested in buying a specific insurance product and to explain why people would buy. Unlike in most other competitions, the majority of participants provided a report describing the path to their solution. In this article we use the framework of bias-variance decomposition of error to analyze what caused the wide range of prediction performance. We characterize the challenge problem to make it comparable to other problems and evaluate why certain methods work or not. We also include an evaluation of the submitted explanations by a marketing expert. We find that variance is the key component of error for this problem. Participants use various strategies in data preparation and model development that reduce variance error, such as feature selection and the use of simple, robust and low variance learners like Naive Bayes. Adding constructed features, modeling with complex, weak bias learners and extensive fine tuning by the participants often increase the variance error.

Discovering Latent Structures: Experience with the CoIL Challenge 2000 Data Set

“…Before answering this question, we note that HLC models and observed BN models both have their pros and cons when it classification task is to identify a subset of 800 that contains as many mobile home policy owners as possible. As can be seen from the following The classification performance of M HLC ranks at Number 5 among the 43 entries to the CoIL Challenge 2000 contest [5], and it is not far from the performance of the best entry. This is impressive considering that no attempt was made to minimize classification error when learning M HLC .…”

“…The CoIL Challenge 2000 data set [5] contains information on customers of a Dutch insurance company. The data consists of 86 variables, around half of which are about ownership of various insurance products.…”

Scaling log-linear analysis to datasets with thousands of variables

“…Internet demographic information on internet users, 70 variables, 10k examples [12]. CoIL2000 insurance customer management, 86 variables, 6k examples [23]. MITFace face recognition dataset, discretized to 4 bins using equal frequency, 362 variables, 31k examples [17].…”

Lazy naive credal classifier

“…On the one hand, NB is a highbias, low variance classifier (see [8] for a theoretical analysis and [11] for an experimental one) and local learning reduces the bias; on the other hand, local learning reduces the chance of encountering strong dependencies between features [7]; there is in fact evidence that local naive Bayes can outperform global naive Bayes [15,7].…”