Early detection and quantification of microaneurysms and hemorrhages can potentially help reduce the risk of vision loss. In this paper, an automatic algorithm for extraction of microaneurysms and hemorrhages from color fundus images is presented. The color retinal image is analyzed and segmented using a top-down segmentation after some image enhancement steps, namely shade correction and contrast enhancement. The whole segmented images create a dataset of regions. Edge detection followed by region-growing techniques are also used to refine the top-down segmentation results. Segmented regions could be classified into true lesions and artifacts using a set of prior features like size, edge strength, color and texture. These are extracted and then utilized with a local-region properties classifier. To validate the outcomes of the proposed method, clinician's references were used to calculate the average performance measures. The proposed method achieved sensitivity of 98.8%, specificity of 97.7%, accuracy of 99.3%, and positive predictive value of 83.7% per-pixel basis. Superior performance measures and high computation speed have assured that the proposed method is more efficient and reproducible compared to the manual methods. The significant novelty of this work is in the use of three refining operations, i.e. Edge detection, region growing and classification to achieve optimal performance especially for the sensitivity and positive predictive value. The performance measures of the proposed algorithm are compared against many recent automated systems and are found to outperform most of them. The proposed algorithm performance is very close to that of the specialist ground truth. Hence, premium performance and high computation speed of this algorithm make it promising approach for a computer-aided mass screening of retinal diseases.