Radiation therapy is an essential part of the comprehensive breast cancer treatment strategy, and radiation dermatitis is the inevitable side-effect. According to either patient-related or treatment-related factors, patients will experience different degrees of acute radiation dermatitis. This study proposes a machine learning architecture based on image and time series features. Using the skin image of the irradiated part during radiotherapy, the image feature is extracted with a gray-level co-occurrence matrix (GLCM) and color space, combined with the time series feature with gradient boosting decision trees (GBDT) to predict the severity of dermatitis after seven days of treatment. The results show that, through the combination of image and time series features, the predicted accuracy (ACC) and area under the curve (AUC) can be effectively improved to 0.8 and 0.85 respectively. The results of GBDT show higher prediction accuracy and robustness than AdaBoost algorithm. This framework can be used as an auxiliary diagnostic tool to assist doctors in making appropriate treatments before severe dermatitis occurs, in order to reduce the radiotoxicity caused by radiotherapy of patients.