Minute concentrations of big hydrophobic ions have the ability to induce substantial effects in soft matter systems, including novel phases in lipid layers, giant charge inversion in colloids and nanostructuration in polymer surfaces in contact with water. The effects are so strong that the term "soft matter disruptors" was coined to describe their deep impact on interfaces, which goes far beyond that found by using the classical ions considered in lyotropic (Hofmeister) sequences. In these effects, solvation thermodynamics plays a fundamental role. Interestingly, it is possible to obtain big hydrophobic cations and anions with an almost identical size and structure (e.g. Ph4B(-), Ph4As(+)), which only differ in their central atom. Here we employ different techniques (Molecular Dynamics (MD) simulations, electrophoretic mobility and Atomic Force Microscopy (AFM)) to demonstrate the dramatic differences in the interaction of Ph4B(-) and Ph4As(+) with poly(N-isopropylacrylamide) (PNIPAM), a thermoresponsive polymer with expanded (well hydrated) and collapsed (poorly hydrated) states. Although both ions interact strongly with neutral PNIPAM chains and cationic or anionic PNIPAM microgels in the collapsed states, the effects of Ph4B(-) on PNIPAM are always substantially stronger than the effects of Ph4As(+). MD simulations predict that ion-PNIPAM free energy of interaction is four times larger for Ph4B(-) than for Ph4As(+). Electrokinetic and AFM experiments show that, acting as counter-ions, both ions are able to invert the charge of anionic or cationic PNIPAM microgels at minute concentrations, but the charge inversion due to Ph4B(-) is much larger than that obtained with Ph4As(+). Therefore, even for big ions of identical size, shape and valence, the affinity of anions and cations for interfaces is intrinsically different.