The impact of sex hormones on anticancer immunity deserves attention due to the importance of the immune system in cancer therapy and the recognition of sex differences in immunity. Cancer is ultimately the result of failed immune surveillance, and the diverging effects of male and female sex hormones on anticancer immunity could contribute to the higher cancer incidence and poorer outcome in men. Estrogens and androgens affect the number and function of immune cells, an effect that depends on cell type, tumor microenvironment, and the age and reproductive status of the individual. Despite the recent progress in immuno-oncology, our current understanding of the interplay between sex hormones and anticancer immune responses is in its infancy. In this review, we will focus on the impact of sex hormones on anticancer immunity and immunotherapy. We will discuss the potential role of the changing hormone levels in anticancer immunity during aging and in the context of menopausal hormone therapies and oral contraception. We will review emerging data on sex differences in PD-L1 expression and potential biomarkers predictive for the efficacy of immune checkpoint inhibitors such as the microbiome and consider ongoing clinical trials evaluating the potential impact of hormone deprivation therapies to increase response to immune checkpoint inhibitors in breast and prostate cancer. Finally, we will point to areas of future research.