Today, it is common to include machine learning components in software products. These components offer specific functionalities such as image recognition, time series analysis, and forecasting but may not satisfy the non-functional constraints of the software products. It is difficult to identify suitable learning algorithms for a particular task and software product because the non-functional requirements of the product affect algorithm suitability. A particular suitability evaluation may thus require the assessment of multiple criteria to analyse trade-offs between functional and non-functional requirements. For this purpose, we present a method for APPlication-Oriented Validation and Evaluation (APPrOVE). This method comprises four sequential steps that address the stated evaluation problem. The method provides a common ground for different stakeholders and enables a multi-expert and multi-criteria evaluation of machine learning algorithms prior to inclusion in software products.