Purpose The level of daily physical activity in patients with cancer is frequently assessed by questionnaires, such as the Physical Activity Scale for the Elderly (PASE). Objective assessments, with for example accelerometers, may be a good alternative. The aim of this study was to investigate the agreement between the PASE questionnaire and accelerometer-assessed physical activity in a large group of patients with different types of cancer. Methods Baseline accelerometer and PASE questionnaire data of 403 participants from the REACT (Resistance and Endurance Exercise After Chemotherapy, n = 227), the EXIST (Exercise Intervention After Stem-Cell Transplantation, n = 74), and NET-QUBIC (NEtherlands QUality of Life And Biomedical Cohort Studies In Cancer, n = 102) studies were available for the current analyses. Physical activity was assessed by the PASE questionnaire (total score) and accelerometers (total minutes per day > 100 counts). Linear mixed models regression analysis was used to assess the agreement between the PASE questionnaire and accelerometer-assessed physical activity. Results The mean (SD) PASE score was 95.9 (75.1) points and mean (SD) time in physical activity measured with the accelerometer was 256.6 (78.8) min per day. The agreement between the PASE score and the accelerometer data was significant, but poor (standardized regression coefficient (B) = 0.36, 95%CI = 0.27; 0.44, p < 0.01).