The classification of sleep state in preterm infants, particularly in distinguishing between active sleep (AS) and quiet sleep (QS), has been investigated using cardiorespiratory information such as electrocardiography (ECG) and respiratory signals. However, accurately differentiating between AS and wake remains challenging; therefore, there is a pressing need to include additional information to further enhance the classification performance. To address the challenge, this study explores the effectiveness of incorporating video-based actigraphy analysis alongside cardiorespiratory signals for classifying the sleep states of preterm infants. The study enrolled eight preterm infants, and a total of 91 features were extracted from ECG, respiratory signals, and video-based actigraphy. By employing an extremely randomized trees (ET) algorithm and leave-one-subject-out cross-validation, a kappa score of 0.33 was achieved for the classification of AS, QS, and wake using cardiorespiratory features only. The kappa score significantly improved to 0.39 when incorporating eight video-based actigraphy features. Furthermore, the classification performance of AS and wake also improved, showing a kappa score increase of 0.21. These suggest that combining video-based actigraphy with cardiorespiratory signals can potentially enhance the performance of sleep-state classification in preterm infants. In addition, we highlighted the distinct strengths and limitations of video-based actigraphy and cardiorespiratory data in classifying specific sleep states.