Spectral computed tomography (SCT) is an powerful imaging modality with broad applications and advantages such as contrast enhancement, artifact reduction, and material differentiation. The positive process or data collected process of SCT is a nonlinear physical process existing scatter and noise, which make it is an extremely ill-posed inverse problem in mathematics. In this paper, we propose a dual-domain iterative network combining a joint learning reconstruction method (JLRM) with a physical process. Specifically, a physical module network is constructed according to the SCT physical process to accurately describe this forward process, which makes the nonlinear use of the traditional mathematical iterative algorithm effective and stable. Additionally, we build a residual-to-residual strategy with an attention mechanism to overcome the slow speed of the traditional mathematical iterative algorithm. We have verified the feasibility of the method through our winning submission to the AAPM DL-spectral CT challenge, and demonstrated that high-accuracy also basis material decomposition results can be achieved with noisy data.