To improve models of ground motion estimation and probabilistic seismic hazard analyses, the engineering seismology field is moving toward developing fully nonergodic ground motion models, models specific for individual source‐to‐site paths. Previous work on this topic has examined systematic variations in ground‐motion along particular paths (from either recorded or simulated earthquake data) and has not included physical properties of the path. We present here a framework to include physical path properties, by seeking correlations between ground motion amplitudes along specific paths and crustal properties, specifically seismic velocity and anelastic attenuation, along that path. Using a large data set of small‐magnitude earthquakes recorded in Southern California, we find a correlation between the gradient of seismic S wave velocity and the path term residual, after accounting for an average geometric spreading and anelastic attenuation, indicating that heterogeneity in crustal velocity primarily controls the path‐specific attenuation. Even in aseismic regions, details of path‐specific ground motion prediction equations can be developed from crustal structure and property data.