CORAL8: Concurrent Object Regression for Area Localization in Medical Image Panels

Abstract

This work tackles the problem of generating a medical report for multi-image panels. We apply our solution to the Renal Direct Immunofluorescence (RDIF) assay which requires a pathologist to generate a report based on observations across the eight different WSI in concert with existing clinical features. To this end, we propose a novel attention-based multi-modal generative recurrent neural network (RNN) architecture capable of dynamically sampling image data concurrently across the RDIF panel. The proposed methodology incorporates text from the clinical notes of the requesting physician to regulate the output of the network to align with the overall clinical context. In addition, we found the importance of regularizing the attention weights for word generation processes. This is because the system can ignore the attention mechanism by assigning equal weights for all members. Thus, we propose two regularizations which force the system to utilize the attention mechanism. Experiments on our novel collection of RDIF WSIs provided by a large clinical laboratory demonstrate that our framework offers significant improvements over existing methods.

A State-of-the-art Survey of Artificial Neural Networks for Whole-slide Image Analysis:from Popular Convolutional Neural Networks to Potential Visual Transformers

“…In [82], a new attention-based multi-mode RNN architecture, called CORAL8, is proposed for direct renal immunofluorescence (RDIF) detection, which solves the problem of generating medical reports for multiple image panels. Among them, the prior encoder learns to extract the context features of doc- Table.…”

“…From 2006 till now, the number of cases in these three main applications has increased year by year. Among them, the cases applied to classification are the most, followed by detection, then segmentation, and other applications such as machine-generated text [82] and automatic labeling [107]. The growth rate of its number is accelerating year by year, which also shows the progress of technology.…”

A Survey on Deep Learning and Explainability for Automatic Report Generation from Medical Images

“…The third column in Table 2 lists the image modalities for each dataset, showing chest X-rays concentrates most of the efforts in report datasets [18,27,43,67,83], though there are also datasets with biomedical images from varied types [34,38,66,102], mammography [96] and hip X-rays [36], ultrasound images [7,150], retinal images [57], doppler echocardiographies [95], cervical images [92], and kidney [93] and bladder biopsies [155]. This adds an extra challenge, since different kinds of exams may need different solutions, as the clinical conditions will be diverse.…”

“…All report datasets include images and reports, and most of them also include labels for each report. Furthermore, INbreast [96] includes contours locating the labels in the images, the Ultrasound collection [149,150] includes bounding boxes locating organs, and IU X-ray [27] and RDIF [93] include additional text written by the physician who requested the exam. The complete detail of additional information is shown in Table 9 in appendix 9.1.…”

“…On the one hand, the labels and image localization can be used to design auxiliary tasks (see section 5.2.5), and to further evaluate the text generation process (see section 5.4). On the other hand, the indication may contain additional information not present in the image, such as a patient's previous condition, which in some cases may be essential to address the task [93].…”