In this article, the couple-group consensus of a class of heterogeneous fractional multi-agent systems with time-varying delay (HFMASs) has been studied.Based on cooperative-competitive interaction, pinning and self-adaptive controller has been designed to realize the couple-group consensus for this system with single integrator and double integrator fractional-order dynamics. By using Lyapunov function, mode-dependent average dwell time (MDADT) technology, stability theory and delta operator method, some sufficient conditions have been obtained to guard the success of couple-consensus for this HFMASs with fixed or switching topologies. According to these sufficient conditions, the related feasible solutions can be solved by using liner matrix inequality (LMI). In addition, these results are also applicable to the system with weakly connected topology without containing a spanning tree or meeting the demand of equilibrium of degree. Finally, several simulation experiments have been presented to prove the validity of the related results.