Vascular bundles are responsible for transporting photosynthesis products, minerals, and water from source to sink. A large vascular bundle number (VBN) in the panicle neck is related to the ability to transport assimilates from stem and leaf to reproductive organs during seed maturation. Several quantitative trait loci (QTLs) for VBN with major effects have been identified by using segregating populations derived from a cross between indica and japonica rice cultivars. However, there are few reports of QTLs for VBN with minor effects. Here, we identified three stable QTLs for VBN—qVBN5 on chromosome 5, qVBN6 on chromosome 6, and qVBN11 on chromosome 11—by using 71 recombinant inbred lines derived from a cross between indica IR24 and japonica Asominori. We confirmed their positions and characterized their effects by using chromosome segment substitution lines (CSSLs) with an IR24 genetic background. qVBN6 had the most substantial effect on VBN, followed by qVBN11 and qVBN5. We developed pyramided lines carrying two QTLs for VBN to estimate their interaction. Two lines carrying both qVBN6 and qVBN11 had a considerably lower VBN than the parental lines, each carrying a single QTL. The combination accumulated VBN negatively in the pyramided lines owing to the independent actions of each QTL. The QTLs detected for VBN will enhance our understanding of genetic mechanisms of VBN and can be used in rice breeding.