In spite of their importance for understanding phonon transport phenomena in thin films and polycrystalline solids, the effects of boundary roughness scattering on phonon specularity and coherence are poorly understood because there is no general method for predicting their dependence on phonon momentum, frequency, branch and boundary morphology. Using the recently formulated atomistic S-matrix method, we develop a theory of boundary roughness scattering to determine the mode-resolved phonon coherence and specularity parameters from the scattering amplitudes. To illustrate the theory, we apply it to phonon scattering in realistic nonsymmetric graphene grain boundary (GB) models derived from atomic structure predictions. The method is validated by comparing its predictions with frequency-resolved results from lattice dynamics-based calculations. We prove that incoherent scattering is almost perfectly diffusive. We show that phonon scattering at the graphene GB is not diffuse although coherence and specularity are significantly reduced for long-wavelength flexural acoustic phonons. Our approach can be generalized to other atomistic boundary models. 42 ries while in perfectly diffuse scattering (P = 0) or the 43 so-called Casimir limit as shown in Fig. 1(b), the incom-44 ing phonon energy is redistributed uniformly over the en-45 tire spectrum of outgoing phonon channels, resulting in 46 maximum momentum loss in the direction parallel to the 47 boundary [9]. Another challenge lies in predicting the 48 effect of boundary roughness on coherent and incoher-49 ent scattering, an unresolved issue in phonon transport 50 in superlattices where the role of phonon interference in 51 thermal conductivity is still debated [18-21]. 52 In order to address these challenges, we develop in 53 this paper a theory of boundary roughness scattering, 54 based on the recently formulated atomistic S-matrix 55 method [22], to determine the mode-resolved phonon co-56 herence and specularity parameters for boundary models. 57 Unlike existing approaches [23, 24], our method is fully 58 atomistic, not restricted to long-wavelength modes, and 59 distinguishes coherent and incoherent scattering [25-27] 60 by treating boundary roughness in a statistical manner 61 analogous to the theory of multiple scattering in disor-62 dered systems [25, 28-30] and conceptually similar to the 63 approach in Ref. [31]. We apply this theory to phonon 64 scattering at the grain boundary (GB) between armchair-65 and zigzag-terminated graphene like in Fig. 1(c), using 66 realistic nonsymmetric low-energy GB models derived 67 from ab initio-based structure predictions [32]. We val-68 idate our method by comparing its predictions with the 69 less precise Zhao-Frend method [33] and analyze how the 70 coherence and specularity parameters vary with phonon 71 frequency, momentum and polarization/branch for the p LA,L over the entire frequency range in Fig. 3. The 288 agreement between P TA,L and p TA,L is also remarkably 289 good although the two quantities diverge at hig...