Simulations of Al thin film sputter depositions rely on accurate plasma and surface interaction models. Establishing the latter commonly requires a higher level of abstraction and means to dismiss the fundamental atomic fidelity. Previous works on sputtering processes addressed this issue by establishing machine learning surrogate models, which include a basic surface state (i.e., stoichiometry) as static input. In this work, an evolving surface state and defect structure are introduced to jointly describe sputtering and growth with physics-separating artificial neural networks. The data describing the plasma-surface interactions stem from hybrid reactive molecular dynamics/time-stamped force bias Monte Carlo simulations of Al neutrals and Ar+ ions impinging onto Al(001) surfaces. It is demonstrated that the fundamental processes are comprehensively described by taking the surface state as well as defect structure into account. Hence, a machine learning plasma-surface interaction surrogate model is established that resolves the inherent kinetics with high physical fidelity. The resulting model is not restricted to input from modeling and simulation, but may similarly be applied to experimental input data.