Rice (Oryza sativa L.) is a staple food of Madagascar, where per capita rice consumption is among the highest worldwide. Rice in Madagascar is mainly grown on smallholder farms on soils with low fertility and in the absence of external inputs such as mineral fertilizers. Consequently, rice productivity remains low and the gap between rice production and consumption is widening at the national level. This study evaluates genetic resources imported from the IRRI rice gene bank to identify potential donors and loci associated with low soil fertility tolerance (LFT) that could be utilized in improving rice yield under local cultivation conditions. Accessions were grown on-farm without fertilizer inputs in the central highlands of Madagascar. A Genome-wide association study (GWAS) identified quantitative trait loci (QTL) for total panicle weight per plant, straw weight, total plant biomass, heading date and plant height. We detected loci at locations of known major genes for heading date (hd1) and plant height (sd1), confirming the validity of GWAS procedures. Two QTLs for total panicle weight were detected on chromosomes 5 (qLFT5) and 11 (qLFT11) and superior panicle weight was conferred by minor alleles. Further phenotyping under P and N deficiency suggested qLFT11 to be related to preferential resource allocation to root growth under nutrient deficiency. A donor (IRIS 313–11949) carrying both minor advantageous alleles was identified and crossed to a local variety (X265) lacking these alleles to initiate variety development through a combination of marker-assisted selection with selection on-farm in the target environment rather than on-station as typically practiced.