Several features in the mass spectrum of merging binary black holes (BBHs) have been identified using data from the Third Gravitational Wave Transient Catalog (GWTC-3). These features are of particular interest as they may encode the uncertain mechanism of BBH formation. We assess if the features are statistically significant or the result of Poisson noise due to the finite number of observed events. We simulate realistic catalogs of BBHs whose underlying distribution does not have the features of interest, apply the analysis previously performed on GWTC-3, and determine how often such features are spuriously found. We find that two of the features found in GWTC-3, the peaks at ∼ 10 M and ∼ 35 M , cannot be explained by Poisson noise alone: peaks as significant occur in < 0.33% of catalogs generated from a featureless population. These peaks are therefore likely to be of astrophysical origin. However, additional structure beyond a power law, such as the purported dip at ∼ 14 M , can be explained by Poisson noise. We also provide a publicly-available package, GWMockCat, that creates simulated catalogs of BBH events with realistic measurement uncertainty and selection effects according to user-specified underlying distributions and detector sensitivities.