Anomaly‐based detection techniques have been widely studied in recent years. Most of these efforts have focused to improve the accuracy of these techniques. The poor accuracy performance is caused by two factors: (i) the data used for the analysis is insufficient and/or unrepresentative of the application behavior, or (ii) inappropriate algorithms are used to model the behavior of the application. In this paper, we attempt to improve anomaly‐based detection techniques by examining these two factors. First, we use system call filtering and abstraction process. This process refines the system call traces. The refined traces are compact and should be more representative of the application main behavior. Second, we use machine learning classifiers to characterize the benign behavior. Generally, there are two main categories of machine learning classifiers: generative classifiers and discriminative classifiers. In their initial training phases, the classifiers build models characterizing the benign behavior. Later on, these models are used to distinguish between different classes of data. They are simply defined by their parameters. The k‐means classifier is considered as a representative of the generative classifier category, and the support vector machine classifier as a representative of the discriminative classifier category. The efficiency of these classifiers are reviewed and compared, as well as the impact of the filtering and abstraction process on their performances is evaluated. The experimental results show that the support vector machine model outperforms the k‐means model, and the filtering and abstraction process has positive impacts on the performance of both models. Copyright © 2016 John Wiley & Sons, Ltd.