Joint actions typically involve a sense of togetherness that has a distinctive phenomenological component. While it has been hypothesized that group size, hierarchical structure, division of labour, and expertise impact agents' phenomenology during joint actions, the studies conducted so far have mostly involved dyads performing simple actions. We explore in this study the complex case of collectively improvised musical performances, focusing particularly on the way group size and interactional patterns modulate the phenomenology of joint action. We recorded two expert improvisation ensembles of contrasting sizes (16 vs 4 musicians) and collected data about their musical behaviour, as well as reports about five aspects of their phenomenology (sense of agency, agentive identity, integration, dependence, reflexivity) and about their musical intentions. Our overall data enabled us to assess how those five phenomenological dimensions related to one another during jointly improvised performances. They also show how such phenomenology varied with the way improvisers dynamically related to one another throughout the performance. Finally, we observe that group size strongly altered the phenomenology of improvisers who otherwise shared many characteristics (high expertise, similar aesthetic preferences, etc.). Our study thus sheds light on the interactional and structural parameters that shape and modulate our felt experience when acting together, and thereby highlights the importance of pluralism for studying the phenomenology of joint action.