Recently developed single-cell profiling technologies, such as single cell sequencing (scRNA-seq) and mass cytometry (CyTOF), provide critical biological insights from a large number of cells and a large number of features. While experimental and informatic techniques around scRNA-seq are much more advanced, research around CyTOF data analysis has severely lagged behind. However, CyTOF profiles the proteomics makeup of the single cells and are more closely related to disease phenotypes than scRNA-seq. CyTOF data are also dramatically different from scRNA-seq data in many aspects. This calls for the evaluation and development of statistical methods for CyTOF data. Dimension reduction (DR) is one of the most critical first steps of analyses of scRNA-seq and CyTOF data. Here, we benchmark 20 of these methods on 110 real and 425 synthetic CyTOF datasets, including 10 Imaging CyTOF datasets, for accuracy, scalability and usability. In particular, we checked the concordance of DR for CyTOF data against scRNA-seq data that were generated from the same samples. We found that SAUCIE and scvis surprisingly perform well across all categories, whereas UMAP particularly excels in downstream analyses and MDS preserves local and global structure well. Our results highlight the complementarity of existing tools, and that the choice of method should depend on the underlying data structure of the single cells. Based on these results, we develop a set of guidelines to help users select the best DR method for their dataset. Our freely available data and evaluation pipeline will aid in the development of improved DR algorithms specifically tailored to analyze data generated from the increasingly popular CyTOF technique.