Chondrules are commonly surrounded by fine‐grained rims (FGRs) whose origin remains highly debated; both nebular and parent body settings are generally proposed. Deciphering their origin, however, is of fundamental importance as they could clarify the matrix–chondrule relationship and thus constrain the formation and transport conditions of chondrules in the circumsolar disk. Here, we report a systematic survey of FGRs in CO, CM, CV, and CR chondrites; we compare (i) the thickness of FGRs to the size of their host chondrules and (ii) the frequency of FGRs to the modal abundance of matrix in the respective host chondrites. Although FGRs show textural variations depending on the petrologic type of the considered chondrites, our data show a positive correlation between apparent rim thickness and the radius of the host chondrule in all chondrite groups. We also found a positive correlation between the evaluated percentages of rimmed chondrules and the modal abundance of matrix material in the chondrites. We show that this relationship could not result from parent body processes, whether matrix compaction or FGR fragmentation. Therefore, we propose that FGRs were accreted under warm conditions at the end of chondrule‐forming events. Our results thus support (i) a nebular origin for FGR, whose abundances are directly related to the abundance of available dust in regions of chondrite accretion; and (ii) the accretion of chondrites from locally formed chondrules and matrix, suggesting limited radial transport in the protoplanetary disk.