Efficient service composition in real time, while satisfying desirable Quality of Service (QoS) guarantees for the composite solution has always been one of the topmost research challenges in the domain of service computing. On one hand, optimal QoS aware service composition algorithms, that come with the promise of solution optimality, are inherently compute intensive, and therefore, often fail to generate the optimal solution in real time for large scale web services. On the other hand, heuristic solutions that have the ability to generate solutions fast and handle large and complex service spaces, settle for sub-optimal solution quality. The problem of balancing the trade-off between compute efficiency and optimality in service composition has alluded researchers for several decades, and several proposals for taming the scale and complexity of web service composition have been proposed in literature. In this paper, we present a new perspective towards this trade-off in service composition based on abstraction refinement, that can be seamlessly integrated on top of any off-the-shelf service composition method to tackle the space complexity, thereby, making it more time and space efficient. Instead of considering services individually during composition, we propose a set of abstractions and corresponding refinements to form service groups based on functional characteristics. The composition and QoS satisfying solution construction steps are carried out in the abstract service space. Our abstraction refinement methods give a significant speed-up compared to traditional composition techniques, since we end up exploring a substantially smaller space on average. Experimental results on benchmarks show the efficiency of our proposed mechanism in terms of time and the number of services considered for building the QoS satisfying composite solution.