Tumors are complex ecosystems of multiple cell types in addition to cancer cells. The response to therapies can be shaped by not only cancer cell vulnerabilities but also by the cells of the tumor microenvironment (TME). For cellular immune therapies, cancer cells are the direct target, but for most of the immune checkpoint blockades (ICB), the direct therapeutic targets are the immune cells. Current immune therapy approaches, especially immune checkpoint blockades, only work for select patient populations. A pre-treatment, ex vivo assay could help in selecting between immune therapy options. However, for immune therapy applications, ex vivo assays would require the co-culturing of both cancer cells and cells of the TME. New results show that it is now feasible to co-culture both cell types with organoid technologies. However, many challenges remain to both optimize organoid cultures as well as to validate ex vivo assays as predictors of therapeutic benefits from immune therapies.