Despite their advantages, biotechnological and omic techniques have not been applied often to characterize phytotoxicity in depth. Here, we show the distribution of phytotoxicity and glycoalkaloid content in a diploid potato population and try to clarify the source of variability of phytotoxicity among plants whose leaf extracts have a high glycoalkaloid content against the test plant species, mustard. Six glycoalkaloids were recognized in the potato leaf extracts: solasonine, solamargine, α-solanine, α-chaconine, leptinine I, and leptine II. The glycoalkaloid profiles of the progeny of the group with high phytotoxicity differed from those of the progeny of the group with low phytotoxicity, which stimulated mustard growth. RNA sequencing analysis revealed that the upregulated flavonol synthase/flavonone 3-hydroxylase-like gene was expressed in the progeny of the low phytotoxicity group, stimulating plant growth. We concluded that the metabolic shift among potato progeny may be a source of different physiological responses in mustard. The composition of glycoalkaloids, rather than the total glycoalkaloid content itself, in potato leaf extracts, may be a driving force of phytotoxicity. We suggest that, in addition to glycoalkaloids, other metabolites may shape phytotoxicity, and we assume that these metabolites may be flavonoids.