This chapter describes the annotation of the medical image data that were used in the VISCERAL project. Annotation of regions in the 3D images is nontrivial, and tools need to be chosen to limit the manual work and have semi-automated annotation available. For this, several tools that were available free of charge or with limited costs were tested and compared. The GeoS tool was finally chosen for the annotation based on the detailed analysis, allowing for efficient and effective annotations. 3D slice was chosen for smaller structures with low contrast to complement the annotations. A detailed quality control was also installed, including an automatic tool that attributes organs to annotate and volumes to specific annotators, and then compares results. This allowed to judge the confidence in specific annotators and also to iteratively refine the annotation instructions to limit the subjectivity of the task as much as possible. For several structures, some subjectivity remains and this was measured via double annotations of the structure. This allows the judgement of the quality of automatic segmentations.

In the VISCERAL project, several Gold Corpus datasets containing medical imaging data and corresponding manual expert annotations have been created. These datasets were used for training and evaluation of participant algorithms in the VISCERAL Benchmarks. In addition to Gold Corpus datasets, the architecture of VISCERAL enables the creation of Silver Corpus annotations of far larger datasets, which are generated by the collective ensemble of submitted algorithms. In this chapter, three Gold Corpus datasets created for the VISCERAL Anatomy, Detection and Retrieval Benchmarks are described. Additionally, we present two datasets that have been created as a result of the anatomy and retrieval challenge.

This chapter describes the ethical and privacy aspects of using medical data in the context of the VISCERAL project. The project had as main goals the creation of a benchmark for organ segmentation, landmark detection, lesion detection and similar case retrieval. The availability of a large amount of imaging data was extremely important for the project goals, and thus, we present an analysis of the procedures that were followed for getting access to the data from IRB (internal review board) approval to data extraction and usage. This chapter details the requirements stated by medical ethics committees in three partner countries that supplied data.