A Pathology Image Diagnosis Network with Visual Interpretability and Structured Diagnostic Report

A Survey on Deep Learning and Explainability for Automatic Report Generation from Medical Images

“…The third column in Table 2 lists the image modalities for each dataset, showing chest X-rays concentrates most of the efforts in report datasets [18,27,43,67,83], though there are also datasets with biomedical images from varied types [34,38,66,102], mammography [96] and hip X-rays [36], ultrasound images [7,150], retinal images [57], doppler echocardiographies [95], cervical images [92], and kidney [93] and bladder biopsies [155]. This adds an extra challenge, since different kinds of exams may need different solutions, as the clinical conditions will be diverse.…”

“…Used by papers DenseNet [59] [ 15,37,83,84,87,90,143,147,154] ResNet [51] [ 39,43,47,60,65,87,92,136,145,146,148] VGG [116] [7, 35, 39, 50, 66, 74, 85, 93, 95, 149, 150] Faster R-CNN [106] [74, 149] Inception V3 [124] [117] GoogLeNet [123] [114] MobileNet V2 [58] [48] SRN [158] [43] U-Net [110] [122] EcNet (*) [155] FCN + shallow CNN (*) [125] RGAN (*) [46] StackGAN [151] (slightly modified version) (*) [120] CNN (*) [120, 126] CNN (unspecified architecture) [140,142] Table 4. Summary of convolutional neural network architectures used in the literature.…”

“…We also identify a group of papers [46,48,74,92,95] following a Template based approach. The language component in these works operates programmatically by following if-then rules or other heuristics in order to retrieve, fill and/or combine templates from a database in order to generate a report.…”

“…This category encompasses the remaining optimization strategies found in the literature. The most important one is multitask learning [20], adopted by 14 papers [47,65,66,74,84,92,93,120,125,126,136,147,149,155]. The main idea is to jointly train a model in multiple complementary tasks, so that the model can learn robust parameters that perform well in all of them.…”

“…classification and localization at the same time. The model by Ma et al [92] uses a CNN to classify the severity of four different key characteristics of cervical cancer, and then uses an attention mechanism over the visual spatial features to generate heatmaps indicating the position of each relevant property. Tian et al [125] used an FCN to classify each pixel of an image with the presence of a liver or tumor, and the result is averaged with an attention map to further improve localization.…”