This paper presents a novel spectral filtering based deep learning algorithm (SFDL) for detecting logos and stamps in a scanned document image. In a document image, textual contents are main source of high spatial frequency components. Accordingly, the high frequency filtering is used to suppress the text symbols. In the next step, segmentation process is used for localizing the candidate regions of interests such as logos and stamps. Preprocessing of these candidate regions is essential before classification. The proposed preprocessing includes steps such as region fusion, resizing and key point based pooling. Finally, the preprocessed candidate regions are classified using deep convolutional neural network. The main advantage of the SFDL is its capability to detect logos without prior information or assumption about their locations in a document. The performance of the proposed SFDL algorithm is evaluated using publicly accessible document image database StaVer. It is observed that SFDL performs satisfactorily for detecting logo and stamp. The precision and recall measures of the proposed SFDL are compared with existing techniques. Experimental results show that recall and precision of logo detection are 86.8%, 97.2%, respectively. Similarly, recall and precision for stamp detection are 85.3% and 94.8%.