MDNet: A Semantically and Visually Interpretable Medical Image Diagnosis Network

Abstract

The inability to interpret the model prediction in semantically and visually meaningful ways is a well-known shortcoming of most existing computer-aided diagnosis methods. In this paper, we propose MDNet to establish a direct multimodal mapping between medical images and diagnostic reports that can read images, generate diagnostic reports, retrieve images by symptom descriptions, and visualize attention, to provide justifications of the network diagnosis process. MDNet includes an image model and a language model. The image model is proposed to enhance multi-scale feature ensembles and utilization efficiency. The language model, integrated with our improved attention mechanism, aims to read and explore discriminative image feature descriptions from reports to learn a direct mapping from sentence words to image pixels. The overall network is trained end-to-end by using our developed optimization strategy. Based on a pathology bladder cancer images and its diagnostic reports (BCIDR) dataset, we conduct sufficient experiments to demonstrate that MDNet outperforms comparative baselines. The proposed image model obtains state-ofthe-art performance on two CIFAR datasets as well.

Deep Learning in Neuroradiology

“…Recently, a framework for inputting medical images (in this case, pathology slides) and outputting diagnostic text-based reports has been reported. 42 While this technology is still very rudimentary, it is not difficult to imagine training a similar network with CT scans and their reports. While deep learning does hold much promise to automate tasks that radiologists find unpleasant, ways of checking and verifying results will be needed.…”

Permutohedral Attention Module for Efficient Non-local Neural Networks

“…To our knowledge, attention modules in deep learning either compute the entire self-attention matrix on a low dimensional input or use a local attention mechanism that can be seen as a strong approximation of the non-local self-attention formulation. Specifically in the medical imaging context, previous works [12,15,11] implicitly used a simplification of (2) with a diagonal self-attention matrix. This solution can be applied to large images since it scales linearly with the number of voxels but does not help to capture contextual information.…”

“…Both of these solutions result in a fixed receptive field, which means that all contextual information in the receptive field will be taken into account whether it is relevant or not. Attention modules have been used to prune irrelevant information in medical imaging [12,15]. Yet, these tools remain suboptimal as they do not allow to capture large scale context.…”

Towards interpretable and robust hand detection via pixel-wise prediction

“…In recent years, some researchers have begun to explore interpretable deep leaning methods. [1] focuses on network interpretability in medical image diagnosis. [2] decomposes output into contributions of its input features to interpret the image classification network.…”