Infections of the mammary gland remain a frequent disease of dairy ruminants that negatively impact animal welfare, milk quality, farmer serenity, farming profitability, and increases use of antimicrobials. There is a need for efficacious vaccines to alleviate the burden of mastitis in dairy farming, but despite decades of research this need has not been satisfactorily fulfilled. A careful appraisal of past and current research on mastitis vaccines reveals the peculiarities but also the commonalities among mammary gland infections associated with the major mastitis pathogens, Escherichia coli, Staphylococcus aureus, Streptococcus uberis, S. agalactiae or S. dysgalactiae. A major pitfall is that the immune mechanisms of effective protection have not been fully identified. Until now, vaccine development has been directed towards the generation of antibodies. In this review, we drew up an inventory of the main approaches used to design vaccines aiming at the major pathogens for the mammary gland, and critically appraised the current and tentative vaccines. In particular, we sought to relate efficacy to vaccine-induced defense mechanisms, in order to shed light on some possible reasons for current vaccine shortcomings. Based on the lessons learned from past attempts and the recent results of current research, the design of effective vaccines may take a new turn in the years to come.