A growing number of domains for Human-Robot and Human-Machine Interactions (HRI/HMI) will involve fleets of autonomous machines. In these fleet environments, robots will encounter not just primary interactors in one-on-one encounters, but also secondary, and even tertiary ones who are bystanders to direct human-to-robot interactions. Thus, the interaction paradigms used by such robots may need to be reconsidered to meet a growing diversity of interactions as fleet HRI/HMI applications continue to grow. Relying on use cases from field testing in mock urban environments, the purpose of this paper is to discuss our lessons learned when supporting multiple “layers” of interactors and how they relate to needs that Human Factors and Ergonomic (HF/E) science can help to address.