To organize accurate and effective emergency responses after an earthquake, it is vital to conduct an early and precise assessment of damage to structures. The use of fragility/vulnerability curves is an advanced evaluation approach for structural damage assessments. However, the analysis based on fragility curves significantly varies depending on soil conditions, ground motion, and structural characteristics. To overcome this issue, a stacked long short‐term memory network was proposed in this research. Unlike previous studies, two input features (acceleration time history in the form of vector and the number of stories in the scalar) are utilized to generalize the results for the same plan building frames with different stories. Three different approaches are presented in this work to link the ground motion time history with the number of stories (2, 4, 8, 12, and 20 stories) in the reinforced concrete building frame, and the networks were tested for unknown ground motions. Of the three approaches, those providing good results were selected for further analysis. For the approaches chosen, the network architectures were changed to a diamond shape and an autoencoder‐like shape with more hidden units (to obtain higher accuracy), which were tested for unknown same plan layout frames. The accuracy obtained using these approaches was significantly high (80%–90%) with a low training time. The proposed model is compared with other techniques and shows significant accuracy. The suggested networks exhibited a number of scenarios for estimating the damage state for unknown ground motions, as well as for unknown frames with various stories. Moreover, the capability of the networks to handle more scalar input features is examined by adding them probabilistically; with additional input variables, the networks predicted the damage state with higher accuracy.