It is well-known that sex hormones can directly and indirectly influence immune cell function. Different studies support a suppressive role of androgens on different components of the immune system by decreasing antibody production, T cell proliferation, NK cytotoxicity, and stimulating the production of anti-inflammatory cytokines. Androgen receptors have also been detected in many different cells of hematopoietic origin leading to direct effects of their ligands on the development and function of the immune system. The immunosuppressive properties of androgens could contribute to gender dimorphisms in autoimmune and infectious disease and thereby also hamper immune surveillance of tumors. Consistently, females generally are more prone to autoimmunity, while relatively less susceptible to infections, and have lower incidence and mortality of the majority of cancers compared to males. Some studies show that androgen deprivation therapy (ADT) can induce expansion of naïve T cells and increase T-cell responses. Emerging clinical data also reveal that ADT might enhance the efficacy of various immunotherapies including immune checkpoint blockade. In this review, we will discuss the potential role of androgens and their receptors in the immune responses in the context of different diseases. A particular focus will be on cancer, highlighting the effect of androgens on immune surveillance, tumor biology and on the efficacy of anti-cancer therapies including emerging immune therapies.