Rainwater harvesting is an important step towards maximizing the water availability and land productivity in arid and semi-arid areas. The present study shows that the area of Ghazi Tehsil within Khyber Pakhtunkhwa Province, Pakistan, has great potential for rainwater harvesting due to its feasible climatic and topographic conditions. This area of 348 km2 normally receives high rainfall annually, but, due to hilly terrain, the bulk of rainwater is lost in the runoff process. In order to enhance agricultural output for such a large area, the practice of rainwater harvesting is a sustainable and decisive approach. However, the selection of appropriate sites for rainwater harvesting on a large scale presents a critical challenge. In such areas, geospatial technology has proved very decisive in the identification of potential sites. In this study, we have used the HEC-GeoHMS tool (ArcGIS 9.3) to compute a curve number to represent the effects of rainfall against the hydrological soil group and landcover. Subsequently, the curve number was used as an input parameter in the soil conservation service runoff-curve number (SCS-CN) method to estimate surface runoff potential for different combinations of landcover and hydrological soil groups. It was observed that runoff was higher in mountainous areas and relatively low in plain areas. Finally, to identify the potential sites for rainwater harvesting, weighted overlay analysis-based related thematic map layers were further reclassified, and weights were assigned according to the technical guidelines of suggested international standards and under consideration of the study area’s topographic, hydrological, and climatic factors. As a result, about 20% of the area was found suitable, 52% less suitable, and 29% as not suitable. Furthermore, relative suitability was assigned to the results of suitable sites as an input for the identification of potential sites for different rainwater harvesting storage structures. These results show that 10% of the area was suitable for farm ponds, 5.74% for check dams, 21.5% for Nigarims, and 8.9% was found to be suitable for gully plugs. The comparison of our GIS-derived and field-based results spatially affirms that the analyzed results were agreeably overlaid in the context of spatial results for check dams, gully plugs, and Nigarims.