BackgroundThe RTS,S/AS01E vaccine provides partial protection against malaria in African children, but immune responses have only been partially characterized and do not reliably predict protective efficacy. We aimed to evaluate comprehensively the immunogenicity of the vaccine at peak response, the factors affecting it, and the antibodies associated with protection against clinical malaria in young African children participating in the multicenter phase 3 trial for licensure.MethodsWe measured total IgM, IgG, and IgG1–4 subclass antibodies to three constructs of the Plasmodium falciparum circumsporozoite protein (CSP) and hepatitis B surface antigen (HBsAg) that are part of the RTS,S vaccine, by quantitative suspension array technology. Plasma and serum samples were analyzed in 195 infants and children from two sites in Ghana (Kintampo) and Mozambique (Manhiça) with different transmission intensities using a case-control study design. We applied regression models and machine learning techniques to analyze immunogenicity, correlates of protection, and factors affecting them.ResultsRTS,S/AS01E induced IgM and IgG, predominantly IgG1 and IgG3, but also IgG2 and IgG4, subclass responses. Age, site, previous malaria episodes, and baseline characteristics including antibodies to CSP and other antigens reflecting malaria exposure and maternal IgGs, nutritional status, and hemoglobin concentration, significantly affected vaccine immunogenicity. We identified distinct signatures of malaria protection and risk in RTS,S/AS01E but not in comparator vaccinees. IgG2 and IgG4 responses to RTS,S antigens post-vaccination, and anti-CSP and anti-P. falciparum antibody levels pre-vaccination, were associated with malaria risk over 1-year follow-up. In contrast, antibody responses to HBsAg (all isotypes, subclasses, and timepoints) and post-vaccination IgG1 and IgG3 to CSP C-terminus and NANP were associated with protection. Age and site affected the relative contribution of responses in the correlates identified.ConclusionsCytophilic IgG responses to the C-terminal and NANP repeat regions of CSP and anti-HBsAg antibodies induced by RTS,S/AS01E vaccination were associated with malaria protection. In contrast, higher malaria exposure at baseline and non-cytophilic IgG responses to CSP were associated with disease risk. Data provide new correlates of vaccine success and failure in African children and reveal key insights into the mode of action that can guide development of more efficacious next-generation vaccines.Electronic supplementary materialThe online version of this article (10.1186/s12916-018-1186-4) contains supplementary material, which is available to authorized users.