Cancer is a disease of gene dysregulation, where cells acquire somatic and epigenetic alterations that drive aberrant cellular signaling. These alterations adversely impact transcriptional programs and cause profound changes in gene expression. Ultimately, interpreting patient somatic alterations within context-specific regulatory programs will facilitate personalized therapeutic decisions for each individual. Towards this goal, we develop a partially interpretable neural network model with encoder-decoder architecture, called Chromatin-informed Inference of Transcriptional Regulators Using Self-attention mechanism (CITRUS), to model the impact of somatic alterations on cellular states and further onto downstream gene expression programs. The encoder module employs a self-attention mechanism to model the contextual impact of somatic alterations in a tumor-specific manner. Furthermore, the model uses a layer of hidden nodes to explicitly represent the state of transcription factors (TFs), and the decoder learns the relationships between TFs and their target genes guided by the sparse prior based on TF binding motifs in the open chromatin regions of tumor samples. We apply CITRUS to genomic, mRNA sequencing and ATAC-seq data from tumors of 17 cancer types profiled by The Cancer Genome Atlas. Our computational framework enables us to share information across tumors to learn patient-specific TF activities, revealing regulatory program similarities and differences between and within tumor types. We show that CITRUS not only outperforms the competing models in predicting RNA expression, but also yields biological insights in delineating TFs associated with somatic alterations in individual tumors. We also validate the differential activity of TFs associated with mutant PIK3CA in breast cancer cell line and xenograft models using a panel of PI3K pathway inhibitors.