Commonly, when developing an algorithm it is necessary to define a certain number of variables that control its behavior. Optimal parameters result in better performance that could translate into profits for companies, stand out among similar applications or better ranking on algorithm competitions. However, it is not a simple task to find the combination of parameters that provides the best results. Manual tuning could be a stressful and difficult task even for expert users. In this paper we present, evaluate and compare several tools in the literature for hyper-parameter optimization. We focus on 4 tools that have been selected due to their number of citations, code availability and impact on literature: MCMC, SMAC, TPE and Spearmint. We analyze these tools in the context of Multi Object Tracking (MOT) that have not been well studied in the literature. MOT itself has been well-studied topic with multiple parameters to be tuned. We evaluate these tools using public benchmarks such as PETS09 or ETH and using the publicly available source code provided by the authors. We analyze the impact of these tools in terms of stability, performance, and usability, among others. Our results show how the use of these tools change the performance of the application and how this would affect the results of real ranked competitions. Our goal is (1) to encourage the reader to use these tools and (2) to provide a guide that helps to choose the most pertinent tool.

HAL is a multidisciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers. L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

This paper evaluates and compares different hyperparameters optimization tools that can be used in any vision applications for tuning their underlying free parameters. We focus in the problem of multiple object tracking, as it is widely studied in the literature and offers several parameters to tune. The selected tools are freely available or easy to implement. In this paper we evaluate the impact of parameter optimization tools over the tracking performances using videos from public datasets. Also, we discuss differences between the tools in term of performances, stability, documentation, etc.