A pH- and temperature-responsive magnetic adsorbent [poly(N-isopropylacrylamide) grafted chitosan/Fe3O4 composite particles, CN-MCP], was synthesized for the removal of the endocrine-disrupting chemical nonylphenol. According to the structural characteristics (changeable surface-charge and hydrophilic/hydrophobic properties) of the targeted contaminant, CN-MCP was designed owning special structure (pH- and temperature-responsiveness for the changeable surface-charge and adjustable hydrophilic/hydrophobic properties, respectively). Compared to chitosan magnetic composite particles without grafting modification (CS-MCP) and several other reported adsorbents, CN-MCP exhibited relatively high adsorption capacity for nonylphenol under corresponding optimal conditions (123 mg/g at pH 9 and 20 °C; 116 mg/g at pH 5 and 40 °C). Meanwhile, high selectivity of the novel adsorbent in selective adsorption of nonylphenol from bisolute solution of nonylphenol and phenol was found. Effects of grafting ratio of the grafted polymer branches and coexisting inorganic salts on the adsorption were systematically investigated. Moreover, CN-MCP demonstrated desired reusability during 20 times of adsorption-desorption recycling. The high adsorption capacity, high selectivity, and desired reusability aforementioned revealed the significant application potential of CN-MCP in the removal of NP. On the basis of the adsorption behaviors, isotherms equilibrium, thermodynamics and kinetics studies, and instrumental analyses including X-ray photoelectron spectroscopy, BET specific surface area, zeta potential, and static water contact angle measurements, distinct adsorption mechanisms were found under various conditions: charge attraction between CN-MCP and the contaminant, as well as binding between polymeric branches of CN-MCP and nonyls, contributed to the adsorption at pH 9 and 20 °C; whereas hydrophobic interaction between CN-MCP and nonylphenol played a dominant role at pH 5 and 40 °C. The current study provided a strategy for the structural design of adsorbents according to the features of targeted emerging contaminants, and the continuity of the work was discussed and proposed.