PurposeTo illustrate the utility of the self‐controlled study design for studies without an active comparator, we compared the results of a cohort design study with a non‐user comparator with those of a self‐controlled design study in evaluating the risk of varenicline on cardiovascular outcomes, using a Japanese medical claims database.MethodsThe participating smokers were identified from health‐screening results collected between May 2008 and April 2017. Using a non‐user‐comparator cohort study design, we estimated the hazard ratios (HRs) and 95% confidence intervals (CIs) of varenicline on initial hospitalization with cardiovascular outcomes using Cox's model adjusted for patients' sex, age, medical history, medication history, and health‐screening results. Using a self‐controlled study design, the within‐subject HR was estimated using a stratified Cox's model adjusted for medical history, medication history, and health‐screening results. The estimate from a recent meta‐analysis was considered the gold standard (risk ratio: 1.03).ResultsWe identified 460 464 smokers (398 694 males [86.6%]; mean (standard deviation) age: 42.9 [10.8] years) in the database. Of these, 11 561 had been dispensed varenicline at least once, and 4511 had experienced cardiovascular outcomes. The estimate of the non‐user‐comparator cohort study design exceeded the gold standard (HR [95% CI]: 2.04 [1.22–3.42]), whereas that of the self‐controlled study design was close to the gold standard (within‐subject HR [95% CI]: 1.12 [0.27–4.70]).ConclusionsThe self‐controlled study design is useful alternative to a non‐user‐comparator cohort design when evaluating the risk of medications relative to their non‐use, based on a medical information database.