Urban Traffic Data Imputation With Detrending and Tensor Decomposition

Abstract

Due to various uncontrollable factors (such as random faulty acquisition equipment and data distortion), urban traffic flow data inevitably suffers from some form of data loss. Finding an effective filling method to estimate the missing data is of great help to the study of transportation networks. Traffic flow during a day are likely to have its regular peak period and off-peak period. For most regions of the urban road network, normally there is a certain trend in the traffic flow data. In this paper, we propose a data imputation method that employs a tensor decomposition approach, which fully considers the characteristics of the traffic flow in both time and space. The proposed method is based on high order singular value decomposition with soft thresholding core. In this method, traffic data are divided into its main trend part and the residual part, which is called detrending. And tensor decomposition is performed on these two parts separately. For each part, dynamic rank method is used to adjust the rank of tensor decomposition. With the actual 214 anonymous road segments with 10 minutes interval data in Guangdong, China, the highway data with 15 minutes interval in Madrid, Spain, and the traffic flow data from PeMS with 5 minutes interval in California, USA. The results of the different models are discussed in the case of continuous data missing and random data missing by different time intervals. In addition, by comparing with other data imputation methods, our method can fill the missing data with better performance. INDEX TERMS Detrending, incomplete data filling, singular value decomposition, tensor decomposition. CHUANFEI GONG received the B.S. degree in computer science from Tongji University, Shanghai, China, in 2017, where he is currently pursuing the master' degree with the Key Laboratory of Embedded System and Service Computing, and the Department of Computer Science. His research interests include intelligent transportation systems and data mining.

Missing Traffic Data Imputation for Artificial Intelligence in Intelligent Transportation Systems: Review of Methods, Limitations, and Challenges

“…Even papers that focus on traffic forecasts, such as [68], make use of tensor decomposition to deal with their missing data before moving on to their proposed model. Papers such as [69], [70], [71], [72], [18], [73], [74], [75], [19], [76], [77], [78] and [79] are some of the recent state-of-the-art missing data imputation methods that have been proposed in the past three years that have utilized tensor factorization as a core part of their model. These tensor-based models performed well due to their being able to extract latent features from a traffic dataset and, through decomposition and completion, can fill in the missing blanks in an accurate manner.…”

A local enhanced spatiotemporal tensor decomposition for missing travel time completion

“…As a high-dimensional space data storage model, the tensor can preserve the original spatial structure and internal latent information of traffic data. By mining the multicollinearity between the multiple dimensions of the data, the invisible knowledge discovery process is enhanced, and the accuracy is higher than the traditional data completion method under the high data missing rate (Signoretto et al , 2011; Chen et al , 2020; Gong and Zhang, 2020; Liu et al , 2009). Tan first introduced the tensor for four-dimensional modeling of traffic data and proposed an estimation method based on TDI to estimate missing traffic values (Tan et al , 2013).…”

Traffic State Estimation of Bus Line With Sparse Sampled Data

“…The interpolation methods regress the model of traffic state relationships by analyzing the spatial-temporal distribution of the traffic. There are several common interpolation methods for estimating traffic state such as Probabilistic principal component analysis (PPCA) [26], [27], tensor decomposition [28]- [31], Convolutional neural network (CNN) [32], [33], auto-encoders [34]- [36], Fuzzy neural network [37], Random forest [38]. L. Qu et al [26] employed the PPCA to estimate the traffic state by extracting the periodic spatial-temporal dependencies in traffic flow.…”