Context: Systematic Review (SR) is a well-defined and rigorous methodology used to find relevant evidence about a specific topic of interest. Depending on the number of identified primary studies, the selection activity can be very time-consuming and a strategy, like SCAS, to semi-automate this activity can be helpful. Objective: To present an experimental study carried out to evaluate the SCAS strategy. Method: We conducted an experiment to compare the efficiency and effectiveness between participants using SCAS and the manual approach. They received necessary training for applying SCAS using tool support. They were divided into five groups, which conducted SRs based on their research areas. Results: When applying SCAS, the average effort reduction was 22.33%, and the average percentage error was 3.95% with a minimal loss of 1.6 evidence per SR. In addition, results showed an overall precision of 65.49% on an overall recall of 90.24% when using SCAS. The overall Kappa showed that there is a substantial agreement level between the groups and SCAS. Conclusion: The experiment increased the confidence in the strategy, reinforcing that it can reduce the effort required to select primary studies without adversely affecting the overall results of SRs.