In this paper, a novel collector, dodecyl dimethyl betaine (BS-12), was used in the selective separation of ilmenite from titanaugite. The flotation performance and associated adsorption mechanism were studied by micro-flotation experiments, particle size analysis, Fourier-transform infrared (FT-IR) spectroscopy analysis, and X-ray photoelectron spectroscopy (XPS) analysis. The micro-flotation results indicated that BS-12 exhibited a stronger collecting ability towards ilmenite than titanaugite within an acidic pH range, and that the recovery of ilmenite was about 50% higher than that of titanaugite under the optimum flotation conditions. Particle size analysis demonstrated that BS-12 could selectively agglomerate ilmenite to a certain extent and then contribute to the flotation difference between ilmenite and titanaugite. FT-IR results showed some characteristic bands of BS-12 on treated ilmenite, and on titanaugite with BS-12. The XPS analysis further confirmed that BS-12 chemisorbed onto ilmenite and titanaugite in a similar way, but the limited active sites on titanaugite in comparison with ilmenite accounted for their differences in flotation.