Etching characteristics of tantalum (Ta) masks used in magnetoresistive random-access memory etching processes by carbon monoxide and ammonium (CO/NH3) or methanol (CH3OH) plasmas have been examined by mass-selected ion beam experiments with in-situ surface analyses. It has been suggested in earlier studies that etching of magnetic materials, i.e., Fe, Ni, Co, and their alloys, by such plasmas is mostly due to physical sputtering and etch selectivity of the process arises from etch resistance (i.e., low-sputtering yield) of the hard mask materials such as Ta. In this study, it is shown that, during Ta etching by energetic CO+ or N+ ions, suboxides or subnitrides are formed on the Ta surface, which reduces the apparent sputtering yield of Ta. It is also shown that the sputtering yield of Ta by energetic CO+ or N+ ions has a strong dependence on the angle of ion incidence, which suggests a correlation between the sputtering yield and the oxidation states of Ta in the suboxide or subnitride; the higher the oxidation state of Ta, the lower is the sputtering yield. These data account for the observed etch selectivity by CO/NH3 and CH3OH plasmas.