Biometric authentication system uses some technique that measures the physical and biological characteristics of human to identify individuals and thus provide security to a system against fraud or intrusion. Common biometric authentication processes are vulnerable and possibility for imitation. Teeth are an important biological entity that plays a major role in forensic research to identify an individual whom cannot be identified visually. There are different algorithms used in biometric authentication. This paper proposes a unique method to recognize the human teeth by using a combination of Discrete Fourier Transform (DFT) and Discrete Cosine Transform (DCT) to extract significant features and an improved version of Binary Particle Swarm Optimization (BPSO) for feature selection is employed to search the feature vector space in order to obtain optimal feature subset to increase the performance rate. A combination of image pre-processing techniques like background removal, gamma intensity correction and Laplacian of Gaussian (LoG) filter are used to help in correct feature extraction. Using the shift invariance property of DFT, a circular feature extraction technique and the energy compaction property of DCT, a circular sector feature extraction method is presented. Experimental results on IvisionLab/dental-image standard database are shown which exhibit promising performance of the teeth recognition system.