Tumor organoids are three-dimensional in vitro models which can recapitulate the complex mutational landscape and tissue architecture observed in cancer patients, providing a realistic tumor microenvironment for testing novel therapies, including immunotherapies. A significant challenge in organoid research in oncology lies in developing efficient and reliable methods for segmenting organoid images, quantifying organoid growth, regression and response to treatments, as well as predicting the behavior of organoid systems. Up to now, a curated dataset of organoids co-cultured with immune cells is not available. To address this gap, we present a new public dataset, comprising both phase-contrast images of murine and patient-derived tumor organoids of one of the deadliest cancer types, the Pancreatic Ductal Adenocarcinoma, co-cultured with immune cells, and state-of-the-art algorithms for object detection and segmentation. Our dataset,OrganoIDNetData, encompassing 190 images with 33906 organoids, can be a potential common benchmark for different organoids segmentation protocols, moving beyond the current practice of training and testing these algorithms on isolated datasets.