Spatially resolved transcriptomics or proteomics data have the potential to contribute fundamental insights into the mechanisms underlying physiologic and pathological processes. However, analysis of these data capable of relating spatial information, multiplexed markers, and their observed phenotypes remains technically challenging. To analyze these relationships, we developed SORBET, a deep learning framework that leverages recent advances in graph neural networks (GNN). We apply SORBET to predict tissue phenotypes, such as response to immunotherapy, across different disease processes and different technologies including both spatial proteomics and transcriptomics methods. Our results show that SORBET accurately learns biologically meaningful relationships across distinct tissue structures and data acquisition methods. Furthermore, we demonstrate that SORBET facilitates understanding of the spatially-resolved biological mechanisms underlying the inferred phenotypes. In sum, our method facilitates mapping between the rich spatial and marker information acquired from spatial ‘omics technologies to emergent biological phenotypes. Moreover, we provide novel techniques for identifying the biological processes that comprise the predicted phenotypes.