We present a new procedure rooted in deep learning to construct science images from data cubes collected by astronomical instruments using HxRG detectors in low-flux regimes. It improves on the drawbacks of the conventional algorithms to construct 2D images from multiple readouts by using the readout scheme of the detectors to reduce the impact of correlated readout noise. We train a convolutional recurrent neural network on simulated astrophysical scenes added to laboratory darks to estimate the flux on each pixel of science images. This method achieves a reduction of the noise on constructed science images when compared to standard flux-measurement schemes (correlated double sampling, up-the-ramp sampling), which results in a reduction of the error on the spectrum extracted from these science images. Over simulated data cubes created in a low signal-to-noise ratio regime where this method could have the largest impact, we find that the error on our constructed science images falls faster than a 1 / N decay, and that the spectrum extracted from the images has, averaged over a test set of three images, a standard error reduced by a factor of 1.85 in comparison to the standard up-the-ramp pixel sampling scheme. The code used in this project is publicly available on GitHub 7 7 https://github.com/GuillaumePayeur/HxRG-denoiser