RNA binding protein (RBP) binding and N6-methyladenosine (m6A) are both essential post-transcriptional regulatory layers for RNA fate decisions. However, the intricate mechanism underlying the interaction between m6A and RBP binding remains underexplored. Here, we develop TransRBP, an interpretable deep learning framework, to model the base-resolution binding of RBPs from RNA sequences and to subsequently investigate the interaction between m6A and RBPs. TransRBP achieves a median accuracy of 0.59 across 32 m6A-related RBPs, representing a 28% increase over the state-of-the-art model. Using gradient-based interpretation, we demonstrate that the binding motifs of the m6A-related RBPs strongly enrich for splicing consensus, laying a foundation for studying the RBP-dependent crosstalk between m6A and splicing. Moreover, we develop an in-silico mutagenesis assay to assess the impact of m6A on RBPs, and utilize the self-attention mechanism to elucidate the interplay between RBP binding and m6A. We further uncover 1,806 variant-RBP combinations with the in-silico mutagenesis, revealing variants that strongly alter RBP binding for genetic diseases including Parkinson's disease, autism, and cardiomyopathy. In particular, we identify m6A cis-acting variants that alter RBP binding in an m6A-proximal manner, including the binding of UPF1 that contributes to Alzheimer's disease, and the DDX3X binding to cardiomyopathy and muscular dystrophy. Together, TransRBP accurately models the binding of RBP and its interaction with m6A, shedding light on the m6A-RBP dynamics and providing multi-layer mechanistic insights for genetic diseases.