Coarse-grained (CG) force fields are essential for molecular dynamics simulations of biomolecules, striking a balance between computational efficiency and biological realism. These simulations employ simplified models grouping atoms into interaction sites, enabling the study of complex biomolecular systems over biologically relevant timescales. Efforts are underway to develop accurate and transferable CG force fields, guided by a bottom-up approach that matches the CG energy function with the potential of mean force (PMF) defined by the finer system. However, practical challenges arise due to many-body effects, lack of analytical expressions for the PMF, and limitations in parameterizing CG force fields. To address these challenges, a machine learning-based approach is proposed, utilizing graph neural networks (GNNs) to represent CG force fields and potential contrasting for parameterization from atomistic simulation data. We demonstrate the effectiveness of the approach by deriving a transferable GNN implicit solvent model using 600,000 atomistic configurations of six proteins obtained from explicit solvent simulations. The GNN model provides solvation free energy estimations much more accurately than state-of-the-art implicit solvent models, reproducing configurational distributions of explicit solvent simulations. We also demonstrate the reasonable transferability of the GNN model outside the training data. Our study offers valuable insights for building accurate coarse-grained models bottom-up.