Deep learning models have demonstrated the remarkable ability to infer cancer patient prognosis from molecular and anatomic pathology information. Studies in recent years have demonstrated that leveraging information from complementary multimodal data can improve prognostication, further illustrating the potential utility of such methods. Model interpretation is crucial for facilitating the clinical adoption of deep learning methods by fostering practitioner understanding and trust in the technology. However, while prior works have presented novel multimodal neural network architectures as means to improve prognostication performance, these approaches: 1) do not comprehensively leverage biological and histomorphological relationships and 2) make use of emerging strategies to "pretrain" models (i.e., train models on a slightly orthogonal dataset/modeling objective) which may aid prognostication by reducing the amount of information required for achieving optimal performance. Here, we develop an interpretable multimodal modeling framework that combines DNA methylation, gene expression, and histopathology (i.e., tissue slides) data, and we compare the performances of crossmodal pretraining, contrastive learning, and transfer learning versus the standard procedure in this context. Our models outperform the existing state-of-the-art method (average 11.54% C-index increase), and baseline clinically driven models. Our results demonstrate that the selection of pretraining strategies is crucial for obtaining highly accurate prognostication models, even more so than devising an innovative model architecture, and further emphasize the all-important role of the tumor microenvironment on disease progression.

Background Deep learning models can infer cancer patient prognosis from molecular and anatomic pathology information. Recent studies that leveraged information from complementary multimodal data improved prognostication, further illustrating the potential utility of such methods. However, current approaches: 1) do not comprehensively leverage biological and histomorphological relationships and 2) make use of emerging strategies to “pretrain” models (i.e., train models on a slightly orthogonal dataset/modeling objective) which may aid prognostication by reducing the amount of information required for achieving optimal performance. In addition, model interpretation is crucial for facilitating the clinical adoption of deep learning methods by fostering practitioner understanding and trust in the technology. Methods Here, we develop an interpretable multimodal modeling framework that combines DNA methylation, gene expression, and histopathology (i.e., tissue slides) data, and we compare performance of crossmodal pretraining, contrastive learning, and transfer learning versus the standard procedure. Results Our models outperform the existing state-of-the-art method (average 11.54% C-index increase), and baseline clinically driven models (average 11.7% C-index increase). Model interpretations elucidate consideration of biologically meaningful factors in making prognosis predictions. Discussion Our results demonstrate that the selection of pretraining strategies is crucial for obtaining highly accurate prognostication models, even more so than devising an innovative model architecture, and further emphasize the all-important role of the tumor microenvironment on disease progression.

Importance: Intraoperative margin analysis is crucial for the successful removal of cutaneous squamous cell carcinomas (cSCC). Artificial intelligence technologies (AI) have previously demonstrated potential for facilitating rapid and complete tumor removal using intraoperative margin assessment for basal cell carcinoma. However, the varied morphologies of cSCC present challenges for AI margin assessment. Objective: To develop and evaluate the accuracy of an AI algorithm for real-time histologic margin analysis of cSCC. Design: A retrospective cohort study was conducted using frozen cSCC section slides and adjacent tissues. Setting: This study was conducted in a tertiary care academic center. Participants: Patients undergoing Mohs micrographic surgery for cSCC between January and March 2020. Exposures: Frozen section slides were scanned and annotated, delineating benign tissue structures, inflammation, and tumor to develop an AI algorithm for real-time margin analysis. Patients were stratified by tumor differentiation status. Epithelial tissues including epidermis and hair follicles were annotated for moderate-well to well differentiated cSCC tumors. A convolutional neural network workflow was used to extract histomorphological features predictive of cSCC at 50-micron resolution. Main Outcomes and Measures: The performance of the AI algorithm in identifying cSCC at 50-micron resolution was reported using the area under the receiver operating characteristic curve. Accuracy was also reported by tumor differentiation status and by delineation of cSCC from epidermis. Model performance using histomorphological features alone was compared to architectural features (i.e., tissue context) for well-differentiated tumors. Results: The AI algorithm demonstrated proof of concept for identifying cSCC with high accuracy. Accuracy differed by differentiation status, driven by challenges in separating cSCC from epidermis using histomorphological features alone for well-differentiated tumors. Consideration of broader tissue context through architectural features improved the ability to delineate tumor from epidermis. Conclusions and Relevance: Incorporating AI into the surgical workflow may improve efficiency and completeness of real-time margin assessment for cSCC removal, particularly in cases of moderately and poorly differentiated tumors/neoplasms. Further algorithmic improvement is necessary to remain sensitive to the unique epidermal landscape of well-differentiated tumors, and to map tumors to their original anatomical position/orientation. Future studies should assess the efficiency improvements and cost benefits and address other confounding pathologies such as inflammation and nuclei.

No abstract

No abstract