Studies have shown that individuals with autism spectrum disorder (ASD) tend to perform significantly below typical developing individuals on standardized measures of memory, even when not significantly different on measures of IQ. The current study sought to examine within ASD whether anatomical correlates of memory performance differed between those with average-to-above-average IQ (AIQ Group) compared to those with low average to borderline ability (LIQ group) as well as in relations to typically-developing comparisons (TDC). Using automated volumetric analyses, we examined regional volume of classic memory areas including the hippocampus, parahippocampal gyrus, entorhinal cortex, and amygdala in an all-male sample AIQ (n = 38) and LIQ (n = 18) individuals with ASD along with 30 typically-developing comparisons (TDC). Memory performance was assessed using the Test of Memory and Learning (TOMAL) compared among the groups and then correlated with regional brain volumes. Analyses revealed group differences on almost all facets of memory and learning as assessed by the various subtests of the TOMAL. The three groups did not differ on any ROI memory-related brain volumes. However, significant size-memory function interactions were observed. Negative correlations were found between the volume of the amygdala and composite, verbal, and delayed memory indices for the LIQ ASD group indicating larger volume related to poorer performance. Implications for general memory functioning and dysfunctional neural connectivity in ASD are discussed.