The development of prognostic time to event models is an area of active research, notably in the fields of cardiology, oncology and intensive care medicine. These models try to find a link between patient covariates and an event at a later time point, and are used for example for therapy assignment, risk stratification or interhospital quality assurance. Increasingly, interest is not only in prognostic models with one possible type of event (e.g. death), but in models that distinguish between several competing endpoints. However, research into methods for the evaluation of the prognostic potential of these models is still needed, as most proposed methods measure either discrimination or calibration of models, but do not examine both simultaneously. We adapt the prediction error proposal of Graf et al. (1999) and Gerds and Schumacher (2006) to handle models with more than one possible event type and introduce a consistent estimator. A simulation study investigating the behaviour of the estimator in small sample size situations and for different levels of censoring together with a real data application follows, highlighting the usefulness of the proposed approach for quantifying effects of model misspecification in summary models for a competing risks setting.