The pollution of phthalic acid esters (PAEs) in edible oils is a serious problem. In the current study, we attempt to remove dibutyl phthalate ester (DBP) from evening primrose oil (EPO) with modified activated clay. The activated clay, commonly used for de-coloration in the oil refining process, was modified by chitosan and hexadecyl trimethyl ammonium bromide (CTAB). The modifications were characterized by SEM, XRD, and FT-IR. We further tested the DBP adsorption capacity of CTAB/chitosan-clay and found that the removal rate was 27.56% which was 3.24 times higher than with pristine activated clay. In addition, the CTAB/chitosan-clay composite treatment had no significant effect on the quality of evening primrose oil. In summary, the CTAB/chitosan-clay composite has a stronger DBP adsorption capacity and can be used as a new adsorbent for removing DBP during the de-coloration process of evening primrose oil.