Despite any doubts about driving safety, many stroke drivers drive again due to the absence of valid screening tools. The on-road test is considered a formal assessment, but there are safety issues in testing directly on stroke patients who are not fully capable of driving. A driving simulator is a promising tool since it provides meaningful information for identifying hazards to driving safety across different driver populations and driving conditions. Using the advantages of a driving simulator, we propose a Driving Performance Assessment System for Stroke drivers (Driving-PASS). Driving-PASS is designed not only to pre-screen invalid stroke drivers before the on-road test but also to provide problematic driving items for the use in driving rehabilitation. To design assessment classifiers, i.e., the core engine of Driving-PASS, we collect driving data from a total of twenty-seven participants in thirteen driving scenarios. Thereafter, we get subjective assessment results from ten driving evaluators in eleven assessment items. By using driving data and subjective assessment results, we construct eleven assessment classifiers for ten driving ability items and one driving suitability item. We addressed the technical challenges such as handcrafted features and imbalanced dataset by a feature extraction method using pre-trained CNN models and a resampling method. Through comprehensive performance evaluation, we build eleven accurate assessment classifiers in Driving-PASS by carefully selecting deep features in each assessment item. We envision that Driving-PASS can be used as a pre-screening tool for evaluating stroke drivers and will ultimately improve road safety.