Masked Token Enabled Pre-Training: A Task-Agnostic Approach for Understanding Complex Traffic Flow

Abstract

The conventional deep learning model performs well for traffic flow analysis by training with a large number of labeled data using a one-model-for-one-task approach, leading to huge computational complexity in dynamic intelligent transportation system (ITS) applications. To overcome this limitation, this paper propose a Token-based Self-Supervised Network (TSSN), which can learn TF features in a task-agnostic way, and provide a well bootstrapped pre-training model for a variety of tasks. TSSN tokenizes TF data into segments, each of which is named as a token and comprised of numerous consecutive points. Masked Token Prediction (MTP), a pretext task, is designed to understand the TF correlations by forecasting tokens that are randomly masked. MTP enables TSSN to capture the high-level intrinsic semantics of TF, and provide general-purpose token embeddings. Therefore, TSSN can be more generalized while keeping high performance. As a result, by replacing the final fully-connected layer with a set of untrained new layers and fine-tuning with small-scale taskspecific data, TSSN can be deployed for a variety of downstream tasks. The simulation results demonstrate that the TSSN can improve overall performance on various downstream tasks when compared to state-of-the-art models.