Autoregressive moving average (ARMA) models are important in many fields and applications, although they are most widely applied in time series analysis. Expanding the ARMA models to the case of various complex data is arguably one of the more challenging problems in time series analysis and mathematical statistics. In this study, we extended the ARMA model to the case of linguistic data that can be modeled by some symmetric fuzzy sets, and where the relations between the linguistic data of the time series can be considered as the ordinary stochastic correlation rather than fuzzy logical relations. Therefore, the concepts of set-valued or interval-valued random variables can be employed, and the notions of Aumann expectation, Fréchet variance, and covariance, as well as standardized process, were used to construct the ARMA model. We firstly determined that the estimators from the least square estimation of the ARMA (1,1) model under some L2 distance between two sets are weakly consistent. Moreover, the justified linguistic data-valued ARMA model was applied to forecast the linguistic monthly Hang Seng Index (HSI) as an empirical analysis. The obtained results from the empirical analysis indicate that the accuracy of the prediction produced from the proposed model is better than that produced from the classical one-order, two-order, three-order autoregressive (AR(1), AR(2), AR(3)) models, as well as the (1,1)-order autoregressive moving average (ARMA(1,1)) model.