The poor grain-filling initiation often causes the poor development of inferior spikelets (IS) which limits the yield potential of large panicle rice (Oryza sativa L.). However, it remains unclear why IS often has poor grain-filling initiation. In addressing this problem, this study conducted a field experiment involving two large panicle rice varieties, namely CJ03 and W1844, in way of removing the superior spikelets (SS) during flowering to force enough photosynthate transport to the IS. The results of this study showed that the grain-filling initiation of SS was much earlier than the IS in CJ03 and W1844, whereas the grain-filling initiation of IS in W1844 was evidently more promoted compared with the IS of CJ03 by removing spikelets. The poor sucrose-unloading ability, i.e., carbohydrates contents, the expression patterns of OsSUTs, and activity of CWI, were highly improved in IS of CJ03 and W1844 by removing spikelets. However, there was a significantly higher rise in the efficiency of sucrose to starch metabolism, i.e., the expression patterns of OsSUS4 and OsAGPL1 and activities of SuSase and AGPase, for IS of W1844 than that of CJ03. Removing spikelets also led to the changes in sugar signaling of T6P and SnRK1 level. These changes might be related to the regulation of sucrose to starch metabolism. The findings of this study suggested that poor sucrose-unloading ability delays the grain-filling initiation of IS. Nonetheless, the efficiency of sucrose to starch metabolism is also strongly linked with the grain-filling initiation of IS.