A ductile damage theory is presented by coupling the covariant formulation of finite deformation plasticity with the phase field modeling of fracture, including kinematic hardening for the ductile response of the materials. A phase field coupled nonlinear kinematic hardening equation is proposed in the reference configuration having equivalent representation through the Lie derivative of the kinematic hardening tensor by push‐forward operation in the spatial configuration, thus ensuring the satisfaction of frame invariance. To capture the correct physical response of the material by the phase field evolution equation, the fracture driving function, that is, the difference between the sum of elastic and plastic energies and threshold energy, after the damage initiation is modified by an energy release controlling function, which is an empirical relation of equivalent plastic strain. In defining the energy release controlling function, well‐defined points with physical significance in the experimental load versus displacement curve are used. To simulate the response of the material in relatively large time steps, a modified staggered scheme is presented, evaluating the fracture driving and energy release controlling functions from the previous converged step and using the updated phase field variable in the weak form of the momentum balance equation. To quantify different material parameters from available experimental results in the literature, the developed phase field coupled elasto‐plastic model uses a neural network optimization procedure consisting of neural network training together with optimization in MATLAB and finite element model evaluation in Abaqus user element subroutine UEL. Model capabilities are demonstrated by simulating the crack propagation in complex 3D geometries such as the second and third Sandia Fracture Challenges.