Metal additive manufacturing (AM) has the potential to tailor the mechanical performance of materials. Due to the complex thermal history and unique microstructure, AM materials are reported to contain distinct dislocation networks with a high dislocation density, which affect the plastic deformation behavior and fracture. However, it is challenging to experimentally observe the formation of such dislocation structures. In this work, a multi-scale multi-physics crystal plasticity modeling framework that integrates the process-structure-property relationship in metal AM is developed. The temperature field obtained from thermal-fluid flow simulations of the AM process and the microstructure from the phase field model of grain growth are combined into thermo-mechanical crystal plasticity simulations to obtain grain-scale thermal stresses. These stresses are used as input to simulate the evolution of dislocation structures within individual grains. Taking AM 316L stainless steel as the material of interest, the effect of initial dislocation configuration on the slip plane and cross-slip mechanism on the dislocation structure formation are investigated. Furthermore, a phase field damage model is implemented to study the initiation of microscale damage and their relationship with dislocation structures, which is a main novelty of this work. This modeling framework provides comprehensive simulations of all aspects of metal AM and offers insights into the dislocation mechanisms and damage formation at microscale in AM materials, which could be used to guide the manipulation of the mechanical properties of AM materials.