The COVID-19 pandemic potentially increases doctors’ work demands and limits their recovery opportunity; this consequently puts them at a high risk of adverse mental health impacts. This study aims to estimate the level of doctors’ fatigue, recovery, depression, anxiety, and stress, and exploring their association with work demands and recovery experiences. This was a cross-sectional study among all medical doctors working at all government health facilities in Selangor, Malaysia. Data were collected in May 2020 immediately following the COVID-19 contagion peak in Malaysia by using self-reported questionnaires through an online medium. The total participants were 1050 doctors. The majority of participants were non-resident non-specialist medical officers (55.7%) and work in the hospital setting (76.3%). The highest magnitude of work demands was mental demand (M = 7.54, SD = 1.998) while the lowest magnitude of recovery experiences was detachment (M = 9.22, SD = 5.043). Participants reported a higher acute fatigue level (M = 63.33, SD = 19.025) than chronic fatigue (M = 49.37, SD = 24.473) and intershift recovery (M = 49.97, SD = 19.480). The majority of them had no depression (69.0%), no anxiety (70.3%), and no stress (76.5%). Higher work demands and lower recovery experiences were generally associated with adverse mental health. For instance, emotional demands were positively associated with acute fatigue (adj. b = 2.73), chronic fatigue (adj. b = 3.64), depression (adj. b = 0.57), anxiety (adj. b = 0.47), and stress (adj. b = 0.64), while relaxation experiences were negatively associated with acute fatigue (adj. b = −0.53), chronic fatigue (adj. b = −0.53), depression (adj. b = −0.14), anxiety (adj. b = −0.11), and stress (adj. b = −0.15). However, higher detachment experience was associated with multiple mental health parameters in the opposite of the expected direction such as higher level of chronic fatigue (adj. b = 0.74), depression (adj. b = 0.15), anxiety (adj. b = 0.11), and stress (adj. b = 0.11), and lower level of intershift recovery (adj. b = −0.21). In conclusion, work demands generally worsen, while recovery experiences protect mental health during the COVID-19 pandemic with the caveat of the role of detachment experiences.