Understanding the employment status of passengers in public transit systems is significant for transport operators in many real applications such as forecasting travel demand and providing personalized transportation service. This paper develops a deep learning approach to infer a passenger's employment status by using smart card data (SCD) with a household survey. This paper first extracts an individual passenger's weekly travel patterns in different travel modes from the raw SCD as a three-dimensional image. A deep learning architecture, called a thresholding multi-channel convolutional neural network, was developed to predict an individual's employment status. The approach proposed here solves two critical problems of using the SCD for employment status studies. First, it automatically incorporates learning temporal features in different travel modes without the need for handcrafted travel feature design. Second, it considers the class-imbalance problem by leveraging the ensemble of oversampling and thresholding techniques. By applying our approach to a real dataset collected from the metropolitan area of London, U.K., about 72% of passengers were correctly categorized into six types of employment statuses. The promising results show the tight correlation between temporal travel behavior, mode choice, and social-demographic roles. To the best of our knowledge, this is the first paper to infer employment status by using the SCD.Index Terms-Deep learning, employment status inference, travel mode choice, smart card data, temporal travel behavior.