Molecular dynamics (MD) simulations were performed to investigate the nanoindentation behavior of Al/Mg-layered nanocomposites with varying layer thicknesses and Mg layer orientations in this study. The aim is to understand the weakening mechanisms at low layer thicknesses and the phase transition mechanisms associated with the dislocation slip angle in the Mg layer. Results indicate that the nanoindentation strength of nanocomposites increases with the layer thickness in the range of 1–10 nm, with the strength of 9.5 × 10−7 N at 10 nm being approximately 73% higher than that at 1 nm. This strength increase is mainly attributed to high interfacial stress, the higher percentage of amorphous atoms, weakened interatomic interactions, and the transition of adjacent interfaces to fully coherent interfaces that significantly reduce their ability to hinder dislocations at the low-layer thickness range. Additionally, in the initial deformation process, the hexagonal close-packed (HCP) phase of the Mg layer firstly transforms into the body-centered cubic (BCC) phase due to its lower energy barrier, followed by the emergence of a faced-centered cubic (FCC) phase driven by 1/3<1−100> dislocations. In the late stage of deformation, new dislocations are generated in the FCC phase and move along its slip planes, altering the dislocation direction. The FCC/HCP interfacial configuration also affects the HCP phase transition mechanism in the Mg layer. When the dislocation slip angle is 0°, the primary phase transition is the BCC phase, whereas a 45° slip angle results in the FCC phase. These findings will provide a guide for the preparation and manufacturing of new high-quality layered nanocomposites.