Background Mislabeling patients as allergic to beta lactams poses an increased risk of morbidity, healthcare costs, and even mortality. This study aimed to define the accuracy of medical history, taken by a specialist, in diagnosing immediate reaction to beta lactams. Methods All patients labeled as allergic to beta lactam were interviewed by a specialist in allergy and clinical immunology and defined as suspected of having a history of immediate or non-immediate reaction. When indicated, skin tests to major and minor determinants and oral graded challenge to the culprit drug were performed. Results A total of 909 patients were evaluated. A total of 798 (87.7%) were labeled as allergic to penicillin. In 108 (11.9%) cases, the allergist suspected an immediate reaction based on clinical history. Skin test or challenge proven diagnosis of IgE-mediated allergy to beta lactam were significantly more prevalent in the group with an allergist's suspicion of an immediate allergy (23.1% vs. 5%, p < 0.01). The sensitivity and negative predictive values of an anamnesis of immediate reaction were high (0.9 and 0.95, respectively), but the specificity and positive predictive value were low (0.37 and 0.23, respectively). Conclusion Medical history taken by an allergist can exclude immediate hypersensitivity reaction, but it is not specific enough to confirm the diagnosis. Skin testing and graded challenge in suspected cases of immediate hypersensitivity reaction are indicated.