Application whitelisting is a method for establishing security by restricting systems to only execute applications that are on a given list -the white list. One of the main questions when using such technology within industrial control systems is whether anti-virus software becomes obsolete, and whether security patches of the underlying system are still required. To answer these questions, we have evaluated several application whitelisting solutions and present the evaluation method and the condensed test results. In addition, we highlight the difficulties for end users to evaluate of the level of protection that is provided by such whitelisting software. As a conclusion, we have identified several benefits of application whitelisting software, but also argue why whitelisting alone is not the "silver bullet", which -once deployed -does not need attention anymore.