The current study examined patterns of event‐related potential (ERP) responses during a face processing task in groups of preschoolers uniquely impacted by autism spectrum disorder (ASD), including (1) children with ASD; (2) children with fragile X syndrome (FXS); (3) children with familial risk for ASD, but without a diagnosis (i.e., ASIBs); and (4) a low‐risk control (LRC) group. Children with FXS have a high incidence of ASD diagnoses, but there have been no studies of the ERP response to faces in children with FXS and little work focused on children with ASD who have cognitive impairment. The current study examined children's ERP responses to faces and houses in four groups: LRC (N = 28, age = 5.2 years), ASIB (N = 23, age = 5.5 years), FXS (N = 19, age = 5.82 years), and ASD (N = 23, age = 5.5 years). The FXS and ASD groups were characterized by the presence of cognitive impairment. Pictures of upright and inverted faces and houses were presented while recording EEG with a 128‐channel system. The N170 occurred at about 200 ms post stimulus onset, was largest on the posterior‐lateral electrodes, and was larger for faces than houses. The P1 and N170 ERP components were larger for the FXS group than for the other three groups. The N170 ERP amplitude for the ASD and ASIB groups was smaller than both the LRC and FXS groups, and the LRC and FXS groups had the largest N170 responses on the right side. No difference was found in N170 latency between groups. The similarity of the ASD and ASIB responses suggest a common genetic or environmental origin of the reduced response. Although children with FXS have a high incidence of ASD outcomes, they differed from ASD and ASIB children in this study. Specifically, the children with FXS were hyperresponsive to all stimulus types while the ASD and ASIB groups showed attenuated responses for specific stimuli.