Aflatoxin belongs to a group of fungal toxins known as mycotoxins, and is widespread in agricultural products and food. Consumption of aflatoxin contaminated peanuts causes severe health problems, like immune system suppression, cancer, and may lead to death. Therefore, quality classification of peanut using an efficient non-destructive method is very essential for food grain industries. In this paper imaging techniques such as thermal imaging, fluorescence imaging and color imaging are identified and evaluated. The results show that, thermal and fluorescence imaging techniques are not suitable for detection of contaminated peanuts. Hence, an algorithm for color imaging technique is proposed as an effective alternative method to detect contaminated peanuts based on external appearance. The main objective of the proposed algorithm is to classify peanuts into good and bad, based on color feature. The captured images are first pre-processed, and database is prepared automatically. Statistical and histogram features are then extracted for classification using Feed Forward Neural Network (FFNN), and Linear Discriminant Analysis (LDA). Proposed algorithm is developed using MATLAB 7.12, and tested on several peanut samples.