Communication among some older adults is affected by cognitive and mobility impairments. This increases isolation, particularly for those residing in care homes, and leads to accelerated cognitive decline. Previous research has leveraged assistive robots to promote recreational routines and communication among older adults, with the robot leading the interaction. However, older adults could have more agency in the interaction, as robots could extend elders’ intentions and needs. Therefore, we explored an approach whereby the robot’s agency is shifted to the older adults who lead the interaction by commanding a robot’s actions using interactive physical blocks (tangible blocks). We conducted sessions with 22 care home dwellers where they could exchange messages and objects using the robot. Based on older adults’ observed behaviors during the sessions and perspectives gathered from interviews with geriatric professionals, we reflect on the opportunities and challenges for increased user agency and the asymmetries that emerged from differing abilities and personality traits. Our qualitative results highlight the potential of robotic approaches to extend the agency and communication of older adults, anchored on human values, such as the exchange of affection, collaboration, and competition.