Privacy preserving data mining is a substantial research area that aims at protecting the privacy of individuals while enabling to perform data mining techniques. In this study, we propose a secure protocol that fulfils the privacy restriction by combining homomorphic encryption with differential privacy and integrate this protocol into Holte's One Rule which is a simple, but accurate and efficient classification algorithm. The proposed method allows a researcher to get the answers of his/her queries to build One Rule classifier by processing the encrypted training dataset under Paillier's cryptosystem and also applies differential privacy to minimize the privacy leakage of individuals as much as possible in this training dataset. Therefore, both of security and privacy of the individuals in the training dataset for classification are provided thanks to our proposed method; since neither the parties, nor the researcher attain any information about the individuals in the database. Besides the One Rule classifier, we apply our proposed privacy preservation model to Naïve Bayes classification algorithm for the performance comparison, and show the efficiency of the proposed method through experiments on real data from UCI repository.