Background: Prognostic models are of high relevance in many medical application domains. However, many common machine learning methods have not been developed for direct applicability to right-censored outcome data. Recently there have been adaptations of these methods to make predictions based on only structured data (such as clinical data). Pseudo-observations has been suggested as a data pre-processing step to address right-censoring in deep neural network. There is a theoretical backing for the use of pseudo-observations to replace the right-censored response outcome, and this allows for algorithms and loss functions designed for continuous, non-censored data to be used. Medical images have been used to predict time-to-event outcomes applying deep convolutional neural network (CNN) methods using a Cox partial likelihood loss function under the assumption of proportional hazard. We propose a method to predict the cumulative incidence from images and structured clinical data by integrating (or combining) pseudo-observations and convolutional neural networks.Results: The performance of the proposed method is assessed in simulation studies and a real data example in breast cancer from The Cancer Genome Atlas (TCGA). The results are compared to the existing convolutional neural network with Cox loss. Our simulation results show that our proposed method performs similar to or even outperforms the comparator, particularly in settings where both the dependent censoring and the survival time do not follow proportional hazards in large sample sizes. The results found in the application in the TCGA data are consistent with the results found in the simulation for small sample settings, where both methods perform similarly. Conclusions: The proposed method facilitates the application of deep CNN methods to time-to-event data and allows for the use of simple and easy to modify loss functions thus contributing to modern image-based precision medicine.