Motor activity data allows for analysis of complex behavioral patterns, including the diagnosis of mental disorders, such as depression or schizophrenia. However, the classification of actigraphy signals remains a challenge. The main reasons are small datasets and the need for sophisticated feature engineering. The recent development of AutoML approaches allows for automating feature extraction and selection. In this work, we compare automatic and manual feature engineering for applications in mental health. We also analyze classifier evaluation methods for small datasets. The automated approach results in better classification, as measured with several metrics, and in a shorter, cleaner code, providing software engineering advantages.