Instance space analysis for a personnel scheduling problem

Abstract

This paper considers the Rotating Workforce Scheduling Problem, and shows how the strengths and weaknesses of various solution methods can be understood by the in-depth evaluation offered by a recently developed methodology known as Instance Space Analysis. We first present a set of features aiming to describe hardness of test instances. We create a new, more diverse set of instances based on an initial instance space analysis that reveals gaps in the instance space, and offers the opportunity to generate additional instances to add diversity to the test suite. The results of three algorithms on our extended instance set reveal insights based on this visual methodology. We observe different regions of strength and weakness in the instance space for each algorithm, as well as a phase transition from feasible to infeasible instances with more challenging instances at the phase transition boundary. This rigorous and insightful approach to analyzing algorithm performance highlights the critical role played by the choice of test instances, and the importance of ensuring diversity and unbiasedness of test instances to support valid conclusions.

The international timetabling competition on sports timetabling (ITC2021)

“…Over recent years, Smith-Miles and co-authors have developed a framework known as instancespace analysis that, among others, can be used to visualize how similar or dissimilar problem instances are with regard to each other (see e.g. Kletzander et al (2021), Smith-Miles et al (2014), Smith-Miles & Bowly (2015, Smith-Miles & Lopes ( 2012)). To be suitable for an optimization competition, we believe that problem instances should (i) challenge existing algorithms such that progression towards new solutions methods is made, (ii) be feasible and allow to nd (suboptimal) solutions with reasonable eort (at least for the majority of the instances so as to encourage participants to enter the competition), (iii) be as dissimilar as possible from each other such that algorithms generalize well outside the competition, and (iv) be as similar as possible to real-life problem instances so as to bridge the gap between theory and practice.…”

Relating instance hardness to classification performance in a dataset: a visual approach

“…The Instance Space Analysis (ISA) framework was originally proposed in (Smith-Miles & Lopes, 2011 ; Smith-Miles et al, 2014 ) for the analysis of optimization problems and algorithms, and later extended to ML and other domains (Muñoz et al, 2018 ; Kang et al, 2017 ). Since its proposal, the ISA methodology has been used and validated in the analysis of multiple problems, including: rostering (Kletzander et al, 2021 ), knapsack problems (Smith-Miles et al, 2021 ), timetabling (Smith-Miles and Lopes, 2011 ), traveling salesman problems (Smith-Miles and Tan, 2012 ), graph coloring (Smith-Miles et al, 2014 ), black-box optimization (Muñoz and Smith-Miles, 2017 ), time series forecasting (Kang et al, 2017 ), classification (Muñoz et al, 2018 ), anomaly detection (Kandanaarachchi et al, 2020 ) and regression (Muñoz et al, 2021 ).…”