In steel production, the recognition of hot-cast billet numbers suffers from low efficiency and susceptibility to misjudgment. This paper proposes a novel method for identifying hot-cast billet numbers on the basis of improved Convolutional Recurrent Neural Network (ICRNN). Although the existing CRNN has achieved acceptable results in text recognition and music symbol recognition, it is not effective in recognizing industrial characters that are blurred or low-contrast. Based on the theoretical framework of CRNN, the Grayscale Spatial Transformation Network (GSTN) is added before character recognition to rectify the skew caused by shooting angles. The backbone network for feature extraction is changed to ResNet50. Moreover, the Efficient Channel Attention (ECA) module is added to construct the Res-ECA network, which extracts more features of the billet number characters. In the sequence modeling stage, the Bidirectional Gated Recurrent Unit (BiGRU) is used to reduce the risk of overfitting and accelerate convergence. After experimental comparison on a self-made billet number dataset, the put forward ICRNN has faster recognition speed and higher accuracy, with a recognition accuracy of 99.49%, which is 4.8% higher than that of the CRNN. The result fully demonstrates that ICRNN meets the requirements of accuracy and speed for billet number recognition.