BACKGROUND: Viral laryngotracheobronchitis croup is the most common cause of acute upper airway obstruction in young children. Clinical assessment of children with croup is often performed using 'croup scores'; however, these scores have not been validated outside of the research setting. OBJECTIVE: To determine the reliability of clinical observation items in croup scores in a paediatric emergency department (ED) setting. DESIGN: Literature review identified 12 observation items (level of consciousness or mental status, inspiratory breath sounds, air entry, stridor, cough, cyanosis or colour, anxiety or air hunger, retractions and/or flaring, respiratory rate and heart rate, oxygen saturation and respiratory distress); overlapping items were combined, yielding 10 variables. In a prospective cohort study over 13 months, patients presenting with croup were observed independently, and croup scores were assigned by the triage nurse, ED nurse and the ED physician before treatment. Agreement among observers for clinical observations was analysed using Cohen's quadratic weighted kappa. SETTING: University-affiliated, paediatric hospital ED providing primary care to an urban area (population 330,000). PATIENTS: Children aged three months to five years presenting with viral croup (preceding history of at least one day of upper respiratory tract symptoms associated with barking cough and/or hoarseness and/or stridor). RESULTS: One hundred fifty-eight children meeting inclusion criteria for croup were assessed by three observers within 1 h of each other's assesments and before treatment. Interobserver agreement among the three observers using weighted kappa was greater than chance for all clinical observation items and ranged from fair to moderate (0.2 to 0.4 and 0.4 to 0.6, respectively). CONCLUSIONS: In the busy practice setting of a paediatric ED, substantial interobserver variability exists among health care providers in the measurement of respiratory signs associated with croup in young children. Based on the present study in a practice setting and two research