The hypothalamic-pituitary-adrenal (HPA) axis is impacted by a multitude of pre- and postnatal factors. Developmental programming of HPA axis function by prenatal alcohol exposure (PAE) has been demonstrated in animal models and in human infants, but remains understudied in older children and adolescents. Moreover, early life adversity (ELA), which occurs at higher rates in children with PAE than in non-exposed children, may also play a role in programming the HPA or stress response system. In a cohort of children and adolescents with PAE and ELA (PAE + ELA), we evaluated HPA function through assessment of diurnal cortisol activity compared to that in typically developing controls, as well as the associations among specific ELAs, adverse outcomes, protective factors, and diurnal cortisol. Morning and evening saliva samples were taken under basal conditions from 42 children and adolescents (5–18 years) with PAE + ELA and 43 typically developing controls. High rates of ELA were shown among children with PAE, and significantly higher evening cortisol levels and a flatter diurnal slope were observed in children with PAE + ELA, compared to controls. Medication use in the PAE + ELA group was associated with lower morning cortisol levels, which were comparable to controls. Complex associations were found among diurnal cortisol patterns in the PAE + ELA group and a number of ELAs and later adverse outcomes, whereas protective factors were associated with more typical diurnal rhythms. These results complement findings from research on human infants and animal models showing dysregulated HPA function following PAE, lending weight to the suggestion that PAE and ELA may interact to sensitize the developing HPA axis. The presence of protective factors may buffer altered cortisol regulation, underscoring the importance of early assessment and interventions for children with FASD, and in particular, for the many children with FASD who also have ELA.