Purpose In the era of precision medicine, patient-specific dose calculation using Monte Carlo (MC) simulations is deemed the gold standard technique for risk-benefit analysis of radiation hazards and correlation with patient outcome. Hence, we propose a novel method to perform whole-body personalized organ-level dosimetry taking into account the heterogeneity of activity distribution, non-uniformity of surrounding medium, and patient-specific anatomy using deep learning algorithms. Methods We extended the voxel-scale MIRD approach from single S-value kernel to specific S-value kernels corresponding to patient-specific anatomy to construct 3D dose maps using hybrid emission/transmission image sets. In this context, we employed a Deep Neural Network (DNN) to predict the distribution of deposited energy, representing specific S-values, from a single source in the center of a 3D kernel composed of human body geometry. The training dataset consists of density maps obtained from CT images and the reference voxelwise S-values generated using Monte Carlo simulations. Accordingly, specific S-value kernels are inferred from the trained model and whole-body dose maps constructed in a manner analogous to the voxel-based MIRD formalism, i.e., convolving specific voxel S-values with the activity map. The dose map predicted using the DNN was compared with the reference generated using MC simulations and two MIRD-based methods, including Single and Multiple S-Values (SSV and MSV) and Olinda/EXM software package. Results The predicted specific voxel S-value kernels exhibited good agreement with the MC-based kernels serving as reference with a mean relative absolute error (MRAE) of 4.5 ± 1.8 (%). Bland and Altman analysis showed the lowest dose bias (2.6%) and smallest variance (CI: − 6.6, + 1.3) for DNN. The MRAE of estimated absorbed dose between DNN, MSV, and SSV with respect to the MC simulation reference were 2.6%, 3%, and 49%, respectively. In organ-level dosimetry, the MRAE between the proposed method and MSV, SSV, and Olinda/EXM were 5.1%, 21.8%, and 23.5%, respectively. Conclusion The proposed DNN-based WB internal dosimetry exhibited comparable performance to the direct Monte Carlo approach while overcoming the limitations of conventional dosimetry techniques in nuclear medicine.