Binding prediction tools are commonly used to identify peptides presented on MHC class II molecules. Recently, a wealth of data in the form of naturally eluted ligands has become available and discrepancies between ligand elution data and binding predictions have been reported. Quantitative metrics for such comparisons are currently lacking. In this study, we assessed how efficiently MHC class II binding predictions can identify naturally eluted peptides, and investigated instances with discrepancies between the two methods in detail. We found that, in general, MHC class II eluted ligands are predicted to bind to their reported restriction element with high affinity. But, for several studies reporting an increased number of ligands that were not predicted to bind, we found that the reported MHC restriction was ambiguous. Additional analyses determined that most of the ligands predicted to not bind are either weak binders or predicted to bind other co-expressed MHC class II molecules. For selected alleles, we addressed discrepancies between elution data and binding predictions by experimental measurements, and found that predicted and measured affinities correlate well. For DQA1*05:01/DQB1*02:01 (DQ2.5) however, binding predictions did miss several peptides that were determined experimentally to be binders. For these peptides and several known DQ2.5 binders we determined key residues for conferring DQ2.5 binding capacity, which revealed that DQ2.5 utilizes two different binding motifs, of which only one is predicted effectively. These findings have important implications for the interpretation of ligand elution data and for the improvement of MHC class II binding predictions.