Abstract-In X-and gamma ray based medical systems, detector performance is a key driver for diagnostic quality. Over the last years and decades, indirect conversion scintillator detectors have become the standard for many medical applications including X-ray Radiography, Computed Tomography and SPECT. Recently, direct conversion semiconductor detectors based on CdTe and CdZnTe (CZT) have come into focus, as they might offer improved or additional performance for specific applications. In this paper we use generic physical models to compare the spatial and spectral resolution of both detector types. The spatial resolution is quantified by the Modulation Transfer Function (MTF). We find that the direct conversion of quanta leads to an approximately sinc-like MTF. In comparison, the MTF of a pixelized scintillator detector shows a mid-frequency drop due to optical signal cross-talk.The spectral resolution of both detector types is described by the Detector Response Function (DRF). It yields the probability density D(E, E ) to measure an incoming quantum of energy E as the output energy E . As a general result we find that fluorescence X-ray cross-talk on one hand and charge sharing and optical cross-talk on the other hand significantly influence the DRF in both detector types. We compare the two detector types in a Computed Tomography application. For an integrating scintillator detector, a Poisson excess noise is observed and quantified. For a counting semiconductor detector, we describe the signficant low-energy shift in D(E, E ). The weighting functions of a two bin counting detctor are analyzed and compared to tube-based approaches. The results are summarized to show the trade-off between spectral and spatial resolution in medical imaging with direct conversion detectors.