Convalescent plasma (CP) recurs as a frontline treatment in epidemics because it is available as soon as there are survivors. The COVID-19 pandemic represented the first large-scale opportunity to shed light into mechanisms of action, safety and efficacy of convalescent plasma using modern evidence-based medicine approaches. Studies ranging from observational case series to randomized controlled trials (RCT) have reported highly variable efficacy results for COVID-19 CP (CCP), resulting in more doubt than certainty. Reasons for CCP success and failure may be hidden in study details, which are usually difficult to explain to physicians and the public but provide fertile ground for designing next-generation studies. In this paper we analyzed variables associated with efficacy such as clinical settings, disease severity, CCP SARS-CoV-2 antibody levels and function, dose, timing of administration (variously defined as time from onset of symptoms, molecular diagnosis, diagnosis of pneumonia, or hospitalization, or by serostatus), outcomes (defined as hospitalization, requirement for ventilation, clinical improvement or mortality), CCP provenance and time for collection, and criteria for efficacy. Focusing only on the results from the 23 available RCT we noted that these were more likely to show signals of efficacy, including reductions in mortality, if the plasma neutralizing titer was ≥ 160 and the time to randomization was ≤ 9 days, consistent with passive antibody therapy efficacy requiring dosing with sufficient antibody. The fact that most studies revealed signals of efficacy despite variability in CCP and its use suggest robust therapeutic effects that become apparent despite the data noise.