Photon-counting CT (PCCT) is an emerging x-ray spectral imaging technology. There are several PCCT prototypes available for clinical and experimental applications. Despite many published results, objective quantitative evaluation of PCCT systems remains challenging due to varied and unknown influences. In addition, CT acquisition geometry and image processing mechanisms are often proprietary, complicating the correlations between the detector's physical characteristics and the corresponding CT image quality, hindering fair comparisons between different technologies. This study aimed to provide a resource to address these challenges by developing a versatile model that accurately replicates the physics of any semiconductor-based photon-counting detector (PCD) and integrating it into a virtual imaging framework to generate detector-specific CT images. The methodology involved a Monte Carlo simulation to model x-ray photon interactions with PCDs and an analytical Gaussian charge sharing model to model charge-diffusion and -repulsion in the detector. Finally, the energy deposited post-crosstalk by source photons (1-120 keV) in the 3×3-pixelneighborhood was translated into photon counts across two energy thresholds to compute detectorspecific spatio-energetic covariance correlation matrices. This framework was validated against experimental measurements of a clinical CdTe-based PCCT, showcasing its accuracy in reproducing real-world scenarios. Additionally, to demonstrate the model's efficacy, we simulated key PCD materials (Si, CdTe, CZT) and designed standardized scanner components, developing virtual systems (DukeCounter scanners). These systems were used to "scan" a physics phantom (ACR) and anthropomorphic-computational models (XCAT). In low-energy (i.e., 5 keV for Si, 20 keV for CdTe and CZT) threshold images, for 125 mAs acquisitions, (noise magnitudes, HU for bone insert) measured in ACR phantom were (76.0 HU, 889.4 HU), (76.9 HU, 932.9 HU), (81.7 HU, 939.4 HU), and the percent difference in Pi10 (airway-based biomarker) between the ground-truth and CT images of XCAT phantoms were 33.7%±3.6%, 22.6%±15.2%, 33.0%±3.6% for CdTe-, CZT-, Si-based DukeCounter systems, respectively. In this way, we demonstrated the utility of our framework to simulate detector-specific CT images for PCCT systems.