This work presents a novel multitask learning approach featuring a dual convolutional neural network (CNN) system for determining the elastic constants of orthotropic rolled steel sheets. In the proposed model, resonance frequency spectra from the impulse excitation technique are converted into 2D image data. The first CNN focuses on detecting and predicting missing peak intensities, while the second CNN utilizes features from the entire spectrum image to predict elastic constants, including E11, E22, and G12. The input features include raw pixel data alongside three key categories for enhanced analysis: image-based features (such as the mean, median, mode, and skewness of pixel intensity distributions), spatial relations (including spatial frequency, pixel intensity correlations, and local contrast), and geometric features (such as shape descriptors and pixel connectivity). The results reveal that the optimal number of peaks (14), image resolution (121 pixels), and sample size (20×20×0.3 cm) maximize the model’s efficiency. Under these conditions, the model achieves R² values of 0.952, 0.902, and 0.913, and RMSE values of 0.0583, 0.0946, and 0.0832 for E11, E22, and G12, respectively. It is suggested that the superior prediction accuracy for E11 is attributed to the stability of the Young's modulus along the rolling direction, which is less variable in orthotropic materials. Furthermore, the study finds a dependency between input weight functions—including image-based features, spatial relations, and geometric features—as the material’s anisotropy changes, underscoring the importance of accounting for process variability in predictive modeling.