We report a multiscale investigation of water inside graphene slit-like channels that extends from the detailed all-atom level (AA) to the cheaper particle-based coarse-grained (CG) level, and to the continuum-based level. Since water is a highly polar solvent, the detailed description of its structural and dielectric properties close to the interfaces is of paramount importance in many applications. For this purpose, we have systematically developed an extended dipole-based CG model using the relative entropy method that can accurately reproduce the radial distribution function (RDF), diffusion coefficient, and bulk dielectric permittivity of the underlying AA reference model. The extended model is simple yet complex enough to shed light on the role of dipolar interactions in polar liquids such as water. Using the CG potentials developed in this work, we show that the structure, parallel dielectric permittivity, and polarization profiles can be captured reasonably well compared to all-atom molecular dynamics (AAMD) simulations. Furthermore, we use the empirical potential-based quasi-continuum (EQT) framework to predict the density and polarization of water molecules inside nano slit channels of various widths. Our continuum analyses reveal that the mean-field treatment of dipolar correlations in combination with the use of CG potentials are sufficient to accurately reproduce the structural variations of water inside the confined graphene slit channels. Finally, by using coarse-grained molecular dynamics (CGMD) and EQT simulations, we comment on the applicability of dipolar-based CG models in reproducing the structure of water near charged interfaces.