Putting laypeople in an active role as direct expert contributors in the design of service robots becomes more and more prominent in the research fields of human–robot interaction (HRI) and social robotics (SR). Currently, though, HRI is caught in a dilemma of how to create meaningful service robots for human social environments, combining expectations shaped by popular media with technology readiness. We recapitulate traditional stakeholder involvement, including two cases in which new intelligent robots were conceptualized and realized for close interaction with humans. Thereby, we show how the robot narrative (impacted by science fiction, the term robot itself, and assumptions on human-like intelligence) together with aspects of power balancing stakeholders, such as hardware constraints and missing perspectives beyond primary users, and the adaptivity of robots through machine learning that creates unpredictability, pose specific challenges for participatory design processes in HRI. We conclude with thoughts on a way forward for the HRI community in developing a culture of participation that considers humans when conceptualizing, building, and using robots.