Segmented thermoplastic polyurethanes (PU) were synthetized using a polycarbonatediol macrodiol as a flexible or soft segment with a molar mass of 2000 g/mol, and different diisocyanate molecules and 1,4-butanediol as a rigid or hard segment. The diisocyanate molecules employed are 3,3′-Dimethyl-4,4′-biphenyl diisocyanate (TODI), 4,4′-diphenylmethane diisocyanate (MDI), 4,4’-Methylenebis(phenyl isocyanate) 1-isocyanato-4- [(4-phenylisocyanate)methyl]benzene and 1-isocyanate-4-[(2-phenylisocyanate) methyl]benzene (ratio 1:1) (MDIi), isophorone diisocyanate (IPDI), and hexamethylene diisocyanate (HDI). The polyurethanes obtained reveal a wide variation of microphase separation degree that is correlated with mechanical properties. Different techniques, such as DSC, DMA, and FTIR, have been used to determine flexible–rigid segment phase behavior. Mechanical properties, such as tensile properties, Shore D hardness, and “compression set”, have been determined. This work reveals that the structure of the hard segment is crucial to determine the degree of phase miscibility which affects the resulting mechanical properties, such as tensile properties, hardness, and “compression set”.