The literature indicates that 90% of clinical alarms in intensive care units might be false. This high percentage negatively impacts both patients and clinical staff. In patients, false alarms significantly increase stress levels, which is especially dangerous for cardiac patients. In clinical staff, alarm overload might lead to desensitization and could result in true alarms being ignored. In this work, we applied the random forest method to reduce false arrhythmia alarms and specifically explored different methods of probability and class assignment, as these affect the classification accuracy of the ensemble classifiers. Due to the complex nature of the problem, i.e., five types of arrhythmia and several methods to determine probability and the alarm class, a synthetic measure based on the ranks was proposed. The novelty of this contribution is the design of a synthetic measure that helps to leverage classification results in an ensemble model that indicates a decision path leading to the best result in terms of the area under the curve (AUC) measure or the global accuracy (score). The results of the research are promising. The best performance in terms of the AUC was 100% accuracy for extreme tachycardia, whereas the poorest results were for ventricular tachycardia at 87%. Similarly, in terms of the accuracy, the best results were observed for extreme tachycardia (91%), whereas ventricular tachycardia alarms were the most difficult to detect, with an accuracy of only 51%.