Understanding the merits of winning data competition solutions for varied sets of objectives

Abstract

Data competitions provide an efficient cost-effective way to obtain diverse solutions for challenging problems across a wide variety of applications. The competition leaderboard, by necessity, must combine multiple objectives into a single scoring formula to determine winners and allocate prize money. However, after the competition concludes, the host may wish to choose a best solution for a particular scenario that focuses on only a subset of all the competition objectives.Through the use of Pareto fronts and graphical summaries, we describe how top solutions for a specific scenario can be identified and compared. The strategy uses intentional tie-handling, thresholds to eliminate undesirable solutions and Pareto fronts to identify objectively superior solutions for a subset of objectives. Then the strengths and weaknesses of different alternatives can be compared to find the ideal solution for the problem. The methods are illustrated with a real Topcoder data competition hosted by Los Alamos National Laboratory that used 16 different objectives to evaluate the quality of solutions for urban radiation search.