We herein examined the effects of different doses, forms, and compatibilities of selenium on a canine mammary gland tumor cell line, CTM1211, and explored the related mechanisms. Three selenium compounds, sodium selenite (SSE), methylseleninic acid (MSA), and methylselenocysteine (MSC), were selected for these experiments, and cyclophosphamide (CTX) served as a positive control. In the cell viability assay, the cell viability of each group at 48/72 h decreased significantly compared with the control group (p < 0.05), and the cell viability of the CTX + MSA group was lower than that of CTX and MSA groups (p < 0.05). Moreover, the inhibitory effect of selenium on cell proliferation was time-dependent but not concentration-dependent. In the cell apoptosis assay, the apoptosis values of each group increased significantly compared with the control group, and the apoptosis values of the CTX + MSA group increased the most significantly (p < 0.01). The protein and mRNA expression levels of vascular endothelial growth factor-alpha (VEGF-alpha), angiopoietin-2 (Ang-2), and hypoxia inducible factor-1 alpha (HIF-1 alpha) were downregulated in each group, while that of phosphatase and tensin homolog (PTEN) were upregulated (p < 0.05). In conclusion, these three selenium compounds, especially MSA, could significantly inhibit the viability and growth of the CTM1211 cell line, which is partly due to the induction of apoptosis and regulation of tumor angiogenesis.