Motivated by the growing availability of complex time series observed in real applications, we propose a threshold matrix‐variate factor model, which simultaneously addresses the sample‐wise and time‐wise complexities of a time series. The sample‐wise complexity is characterized by modeling matrix‐variate observations directly, while the time‐wise complexity is modeled by a threshold variable to describe the nonlinearity in time series. The estimators for loading spaces and threshold values are introduced and their asymptotic properties are investigated. Our matrix‐variate models compress data more efficiently than traditional vectorization‐based models. Furthermore, we greatly extend the scope of current research on threshold factor models by removing several restrictive assumptions, including existence of only one threshold, fixed factor dimensions across different regimes, and stationarity within regime. Under the relaxed assumptions, the proposed estimators are consistent even when the numbers of factors are overestimated. Simulated and real examples are presented to illustrate the proposed methods.