Similar image search for histopathology: SMILY

Abstract

The increasing availability of large institutional and public histopathology image datasets is enabling the searching of these datasets for diagnosis, research, and education. Although these datasets typically have associated metadata such as diagnosis or clinical notes, even carefully curated datasets rarely contain annotations of the location of regions of interest on each image. As pathology images are extremely large (up to 100,000 pixels in each dimension), further laborious visual search of each image may be needed to find the feature of interest. In this paper, we introduce a deep-learning-based reverse image search tool for histopathology images: Similar Medical Images Like Yours (SMILY). We assessed SMILY’s ability to retrieve search results in two ways: using pathologist-provided annotations, and via prospective studies where pathologists evaluated the quality of SMILY search results. As a negative control in the second evaluation, pathologists were blinded to whether search results were retrieved by SMILY or randomly. In both types of assessments, SMILY was able to retrieve search results with similar histologic features, organ site, and prostate cancer Gleason grade compared with the original query. SMILY may be a useful general-purpose tool in the pathologist’s arsenal, to improve the efficiency of searching large archives of histopathology images, without the need to develop and implement specific tools for each application.

Pan-cancer computational histopathology reveals mutations, tumor composition and prognosis

“…Interestingly, tiles with greater probability for WGD status showed an increased nuclear (haematoxylin) staining ( Figure 2b ), which could be explained by the increased amount of DNA. We therefore explicitly quantified the average cell nucleus size and intensity per tile using Cell profiler 12 . Indeed, the cell nucleus size and intensity were highly correlated with the WGD probability predicted by histopathological features, but gave a lower predictive accuracy (average AUC of 0.71, range [0.56,1], Figure 2c, Supplementary Figure 3 ).…”

“…These computational histopathological features are automatically learned for the original task of classifying the entire and/or subregions of images into cancer or non-cancerous tissues. However, once learned, the feature representation may also be used to find similar images 12 and quantify associations with traits beyond tissue types 13 . This approach, known as transfer or weakly supervised learning, has been used, for example, to establish associations with genetic variants, transcriptomic alterations and also survival 14,15 .…”

SHIFT: speedy histological-to-immunofluorescent translation of whole slide images enabled by deep learning

“…Since data limitations are common to many biomedical data domains (Campanella et al, 2019;Costa et al, 2018;Udrea and Mitra, 2017;Xiao et al, 2018), we begin by exploring the possibility that the choice of training samples can be optimized by selecting the few samples that are most representative of the population of samples at our disposal. Some DL-based applications have been proposed for histological image comparison and retrieval (Hegde et al, 2019;Otálora et al, 2018), but to the best of our knowledge none have been proposed for the express purpose of training set selection in a data-limited biomedical imaging domain. We describe the use of a data-driven DL-based method to select samples that optimizes the morphological heterogeneity of the dataset and promotes SHIFT model generalizability.…”

Interpretable multimodal deep learning for real-time pan-tissue pan-disease pathology search on social media

“…Second, prior studies investigating pathology search take a variety of 458 pathology-agnostic approaches, e.g. (i) using neural networks that were not trained with 459 pathology data [18,19] or (ii) using scale-invariant feature transform (SIFT) 460 features [19,43,44] that do not represent texture or color [45]. Our inclusive approach is 461 different, building a search method for pathology data represented by thousands of Prior work has found texture and/or color to be important for tissue-related tasks in 465 computational pathology [46][47][48].…”

“…tissue type and marker mention).354 Both modalities improve search performance, as discussed in the following section. Disease state search, first pan-tissue pan-disease method356 In light of pathology-agnostic approaches to pathology search[18,19], we ask if 357 pathology-specific approaches to pathology search may perform better. Indeed, search is 358 the main purpose of our social media bot.…”