Background: Doctor performance evaluation (DPE) is an important task in eHealth, which aims to evaluate the overall quality of online diagnosis and patient outcomes so that high customer satisfaction and loyalty can be attained. However in reality most of customers trend not to give ratings to doctor performance. Therefore it is imperative to develop a model to make DPE automatically. When making auto-evaluation of doctor performance, we expect to rate the doctor performance into a score label that is as close as possible to the true one. Objective: This study aims to do DPE automatically from online textual consultation contents between doctors and customers by a novel machine learning method. Methods: We propose a solution which models DPE as an ordinal regression problem. In doing so, a combined SVM and Ordinal Partitioning model, namely SVMOP, along with an innovative prediction function is developed to capture the ordering labels hidden in preferences over DPE. Specifically, when conducting feature engineering, in addition to the basic text features, eight handcrafted features extracted from over 70,000 medical entries, are added and then further boosted by Gradient Boosting Decision Tree. Results: Real data sets from one of the largest mobile doctor-patient communication platforms in China are used in our study. In according with the statistics, 64% of data on eHealth platforms loss the evaluation labels from customers. Experimental results reveal that our approach can well support doctor performance evaluation automatically. Specifically, compared with other auto-evaluation models, SVMOP improve the MAE by up to 0.1, MSE by up to 0.5, PAcc by up to 5%; the handcrafted features that we suggest, improve the MAE by up to 0.1, MSE by 0.2, PAcc by up to 3%. After boosting, the