This paper presents a novel automatic mammography recognition approach used to develop computer-aided diagnostic systems that require a robust method to assist the radiologist in identifying and recognizing speculations from a multitude of lines corresponding to the normal fibrous breast tissue. Following this rationale, this paper introduces a novel approach for detecting the speculated lesions in digital mammograms based on multi-scale SIFT (scale-invariant feature transform) orientations. The proposed method starts by estimating a set of key points that best represent the image mammography in a scale space. We then benefit from SIFT algorithm to locally characterize each key point by assigning a consistent orientation. Thereafter, a set of three features are extracted for each pixel in the image mammogram based on these orientations. The extracted features are fed into BDT (binary decision tree) in order to perform per pixel classification and decide whether the pixel is normal or abnormal. We evaluate the proposed system on BCDR (breast cancer digital repository) database and the experimental results show that our method is accurate with 97.95% recognition rate, while it is robust to illumination changes, rotation and scale variations.