An energy conserving discretization of the elastic wave equation in second order formulation is developed for a composite grid, consisting of a set of structured rectangular component grids with hanging nodes on the grid refinement interface. Previously developed summationby-parts properties are generalized to devise a stable second order accurate coupling of the solution across mesh refinement interfaces. The discretization of singular source terms of point force and point moment tensor type are also studied. Based on enforcing discrete moment conditions that mimic properties of the Dirac distribution and its gradient, previous single grid formulas are generalized to work in the vicinity of grid refinement interfaces. These source discretization formulas are shown to give second order accuracy in the solution, with the error being essentially independent of the distance between the source and the grid refinement boundary. Several numerical examples are given to illustrate the properties of the proposed method.