Polymer-based protein recognition systems have enormous potential within clinical and diagnostic fields due to their reusability, biocompatibility, ease of manufacturing, and potential specificity. Imprinted polymer matrices have been extensively studied and applied as a simple technique for creating artificial polymer-based recognition gels for a target molecule. Although this technique has been proven effective when targeting small molecules (such as drugs), imprinting of proteins have so far resulted in materials with limited selectivity due to the large molecular size of the protein and aqueous environment. Using coarse-grained molecular simulation, we investigate the relation between protein makeup, polymer properties, and the selectivity of imprinted gels. Nonspecific binding that results in poor selectivity is shown to be strongly dependent on surface chemistry of the template and competitor proteins as well as on polymer chemistry. Residence time distributions of proteins diffusing within the gels provide a transparent picture of the relation between polymer constitution, protein properties, and the nonspecific interactions with the imprinted gel. The pronounced effect of protein surface chemistry on imprinted gel specificity is demonstrated.