The purpose of this paper is two fold. First, we present a set of benchmark problems for maintenance optimisation called VMELight. This model allows the user to define the number of components in the system to maintain and a number of customisable parameters such as the failure distribution of the components, the spare part stock level and every costs associated with the preventive and corrective maintenances, unavailability and spare parts. From this model, we create a benchmark of 175 optimisation problems across different dimensions. This benchmark allows us to test the idea of using an iterated racing algorithm called IRACE based on the Friedman statistical test, to reduce the number of simulations needed to compare solutions in the population. We assess different population size and truncation rate to show that those parameters can have a strong influence on the performance of the algorithm.