Electroencephalography (EEG) has a long history as a clinical tool to study brain function, and its potential to derive biomarkers for various applications is far from exhausted. Machine learning (ML) can guide future innovation by harnessing the wealth of complex EEG signals to isolate relevant brain activity. Yet, ML studies in EEG tend to ignore physiological artifacts, which may cause problems for deriving biomarkers specific to the central nervous system (CNS). We present a framework for conceptualizing machine learning from CNS versus peripheral signals measured with EEG. A common signal representation across the frequency spectrum based on Morlet wavelets allowed us to define traditional brain activity features (e.g. log power) and alternative inputs used by state-of-the-art ML approaches (covariance matrices). Using more than 2600 EEG recordings from large public databases (TUAB, TDBRAIN), we studied the impact of peripheral signals and artifact removal techniques on ML models in exemplary age and sex prediction analyses. Across benchmarks, basic artifact rejection improved model performance whereas further removal of peripheral signals using ICA decreased performance. Our analyses revealed that peripheral signals enable age and sex prediction. However, they explained only a fraction of the performance provided by brain signals. We show that brain signals and body signals, both reflected in the EEG, allow for prediction of personal characteristics. While these results may depend on specific prediction problems, our work suggests that great care is needed to separate these signals when the goal is to develop CNS-specific biomarkers using ML.