The availability of high-resolution satellite imagery has enabled several new applications such as identification of brick kilns for the elimination of modern-day slavery. This requires automated analysis of approximately 1, 551, 997 km 2 area within the "Brick-Kiln-Belt" of South Asia. Although modern machine learning techniques have achieved high accuracy for a wide variety of applications, problems involving large-scale analysis using high-resolution satellite imagery requires both accuracy as well as computational efficiency. We propose a coarse-to-fine strategy consisting of an inexpensive classifier and a detector which work in tandem to achieve high accuracy at low computational cost. More specifically, we propose a two-stage gated neural network architecture called Kiln-Net. At the first stage, imagery is classified using the ResNet-152 model which filters out over 99% of irrelevant data. At the second stage, a YOLOv3-based object detector is applied to find the precise location of each brick kiln in the candidate regions. The dataset, named Asia14, consisting of 14, 000 Digital Globe RGB images and 14 categories is also developed to train the proposed Kiln-Net architecture. Our proposed network architecture is evaluated on approximately 3, 300 km 2 region (3, 37, 723 image patches) from 14 different cities in five different countries of South Asia. It outperforms state-of-the-art methods employed for the recognition of brick kilns and achieved an accuracy of 99.96% and average F1 score of 0.91. To the best of our knowledge, it is also 20x faster than existing methods.