Benchmarking HEp-2 Cells Classification Methods

Abstract

In this paper, we report on the first edition of the HEp-2 Cells Classification contest, held at the 2012 edition of the International Conference on Pattern Recognition, and focused on indirect immunofluorescence (IIF) image analysis. The IIF methodology is used to detect autoimmune diseases by searching for antibodies in the patient serum but, unfortunately, it is still a subjective method that depends too heavily on the experience and expertise of the physician. This has been the motivation behind the recent initial developments of computer aided diagnosis systems in this field. The contest aimed to bring together researchers interested in the performance evaluation of algorithms for IIF image analysis: 28 different recognition systems able to automatically recognize the staining pattern of cells within IIF images were tested on the same undisclosed dataset. In particular, the dataset takes into account the six staining patterns that occur most frequently in the daily diagnostic practice: centromere, nucleolar, homogeneous, fine speckled, coarse speckled, and cytoplasmic. In the paper, we briefly describe all the submitted methods, analyze the obtained results, and discuss the design choices conditioning the performance of each method.

HEp-2 fluorescence pattern classification

“…This is supported by the recently published detailed report of the contest findings [5], which shows great variability between method rankings by cell-level and by sample-level performance. Consideration of the sample as a whole, including complete measurements from all the relevant cells, allows the application of a much richer set of pattern recognition methods, and is a better match for the ultimate goal of replicating the diagnostic decision of a physician.…”

“…Therefore any possible cross-validation arrangement which could be used for algorithm selection or parameter tuning by the contestants would necessarily include cells from the same sample in both training and validation sections. The performance estimates provided by such cross-validation (error rates of around 5% were reported by a number of participants) proved very far from the final test-set performance reported in [5], as the validation task is very much easier than the hold-out test, which contains genuinely independent samples from patients not represented in the training set.…”

“…The range of features employed by the participants, which are summarised in [5] and covered in more detail in a number of papers published at ICPR ( [6,7,8,9,10,11,12,13]), is quite broad. Most algorithms included some form of texture measurement, and often a combination of several; co-occurrence matrices, Local Binary Patterns, and various extensions thereof, as well as gradients, frequency transforms and Gabor wavelets.…”

HEp-2 staining pattern recognition at cell and specimen levels: Datasets, algorithms and results

“…We deem that the future adoption of deep architectures might be a viable way for doing a step forward, due to their ability to learn the discriminant properties of the cells without requiring any features engineering. Although deep learning methods have not been widely used for this task, it is worth mentioning the methods by the team of Malon et al as reported in Foggia et al (2013) and by Gao et al (2014), both based on convolutional neural networks. On the one hand, the method by Malon reached an accuracy of 60% over a small dataset, the MIVIA HEp-2 Images Dataset (Foggia et al (2013)), comprised approximatively 720 cells images for the training and 730 for the test set.…”

“…Although deep learning methods have not been widely used for this task, it is worth mentioning the methods by the team of Malon et al as reported in Foggia et al (2013) and by Gao et al (2014), both based on convolutional neural networks. On the one hand, the method by Malon reached an accuracy of 60% over a small dataset, the MIVIA HEp-2 Images Dataset (Foggia et al (2013)), comprised approximatively 720 cells images for the training and 730 for the test set. Over the same dataset, the highest score was achieved by Nosaka (69%), while the scientist achieved an accuracy of approximatively 73%.…”