Data on the first appearance of species in the field season are widely used in phenological studies. However, there are probabilistic arguments for bias in estimates of phenological change if sampling methods or population abundances change. We examined the importance of bias in three measures of phenological change: (1) the date of the first X appearances, (2) the date of the first Y% of all first appearances and (3) the date of the first Z% of the individuals observed during the entire flight period. These measures were tested by resampling the data of the Dutch Butterfly Monitoring Scheme and by simulations using artificial data. We compared datasets differing in the number of sampling sites, population abundance and the start of the observation period. The date of the first X appearances proved to be sensitive to the number of sampling sites. Both the date of the first X appearances and the date of the first Y% of all first appearances were sensitive to population trend. No such biases were found for estimates of the first Z% of the flight period, but all three measures were sensitive to changes in the start of the observation period. The conclusions were similar for both the study on butterfly data and the simulation study. Bias in phenology assessments based on first appearance data may be considerable and should no longer be ignored in phenological research.