Urbanization modifies landscape structure in three major ways that impact avian diversity in remnant habitat: habitat amount is reduced and habitat configuration and matrix quality are altered. The relative effects of these three components of landscape structure are relatively well-studied in agricultural landscapes, but little is known about the relative effect of urban matrix quality. We addressed this gap by investigating the relative effects of forest amount, forest configuration, and matrix quality, indicated by degree of urbanization and agriculture amount, on the diversity of three guilds of forest birds using data from 13,763 point counts from Pennsylvania, USA. Forest amount had the largest independent effect on forest bird diversity, followed by matrix quality, then forest configuration. In particular, urbanization had strong negative effects on the relative abundance and species evenness of all forest birds and the relative abundance of forest generalist birds. To our knowledge, these are the first results of the effect of urban matrix quality on forest bird relative abundance and species evenness independent of forest amount and forest configuration. Our results imply that conservation practitioners in human-modified landscapes prioritize maximizing forest amount, then reducing the effects of disturbances originating in the matrix, and then preserving large, spatially-dispersed forest patches to most effectively conserve forest birds.