The increasingly central role of robotic agents in daily life requires effective human–robot interaction (HRI). For roboticists to optimize interaction design, it is crucial to understand the potential effects of robotic agents on human performance. Yet a systematic specification of contributing factors is lacking, and objective measures of HRI performance are still limited. In these regards, the findings of research on human–human interaction can provide valuable insights. In this review, we break down the complex effects of robotic agents on interacting humans into some basic building blocks based on human–human interaction findings, i.e., the potential effects of physical presence, motor actions, and task co-representation in HRI. For each effect, we advise on future directions regarding its implication. Furthermore, we propose that the neural correlates of these effects could support real-time evaluation and optimization of HRI with electroencephalograph (EEG)-based brain–computer interface (BCI).