Horizontal visibility graph (HVG) motifs have been recently introduced to analyze the dynamical information encoded by biological signals. However, the result of the analysis strongly depends on the selected window size of the motifs. Different sizes ranging from 3 to 5 have been previously used, but such small window sizes are insufficient to cope with the complexity of biological systems and often fail to extract salient features of the encoded information. It is known that larger window size increases the total number of possible motifs, and it leads to the distribution of the statistics into too many motifs, which causes each individual motif to contain too little information and make it even more difficult to reliably detect system dynamics. To resolve this problem, we group the motifs based on the number of edges. Using the grouped motifs, we propose grouped horizontal visibility entropy (GHVE) to quantify the complexity based on the probability distribution of the observations within these groups. We apply GHVE to quantify the complexity of simulated white and 1/f noise. The results reveal that the 1/f noise time series exhibits a higher complexity than white noise time series, which indicates that the 1/f noise is structurally more complex than white Gaussian noise. We apply the method for analyzing interbeat intervals time series. The results show that the proposed GHVE measure is more accurate in distinguishing healthy and pathological subjects than its non-grouped counterpart HVG. It is, therefore, better suited to detect changes in aging, disease severity, and activity levels (sleep and wake period). INDEX TERMS Complex network, HVG motifs, HRV analysis, time series data.