We consider the one-dimensional nonlinear Schrödinger equation with an attractive delta potential and mass-supercritical nonlinearity. This equation admits a one-parameter family of solitary wave solutions in both the focusing and defocusing cases. We establish asymptotic stability for all solitary waves satisfying a suitable spectral condition, namely, that the linearized operator around the solitary wave has a two-dimensional generalized kernel and no other eigenvalues or resonances. In particular, we extend our previous result [30] beyond the regime of small solitary waves and extend the results of [15, 25] from orbital to asymptotic stability for a suitable family of solitary waves.