Protein-RNA interactions play an essential role in the regulation of transcription, translation, and metabolism of cellular RNA. Here, we develop Reformer, a deep learning model that predicts protein-RNA binding affinity purely from sequence. We developed Reformer with 155 RNA binding protein (RBP) targets from 3 cell lines. Reformer achieved high prediction accuracy at single-base resolution when tasking with inferring protein- and cell-type-specific binding affinity. We conducted electrophoretic mobility shift assays to validate high-impact RNA regulation mutations predicted by Reformer. In addition, Reformer learned to capture protein binding motifs that cannot be discovered by eCLIP-seq experiments. Furthermore, we demonstrated that motif signatures related to RNA processing functions are encoded within Reformer. In conclusion, Reformer will facilitate interpretation of the regulation mechanisms underlying RNA processing.