False alerts due to misconfigured or compromised intrusion detection systems (IDS) in industrial control system (ICS) networks can lead to severe economic and operational damage. To solve this problem, research has focused on leveraging deep learning techniques that would help reduce false alerts. However, a shortcoming is that these works often require or implicitly assume the physical and cyber sensor data to be trustworthy. Implicit trust of data is a major problem with using artificial intelligence or machine learning (AI/ML) for cyber-physical system (CPS) security, because the times when these solutions are needed most to detect an attack are also the times when they are more at risk, with both greater likelihood and greater impact, of also being compromised. To address this inevitable shortcoming, the problem can thus be reframed as how to make good decisions given uncertainty. Then, the decision is detection, and the uncertainty includes whether or not the data that would be used in ML-based IDS is compromised. Thus, this article presents an approach for reducing false alerts in cyber-physical power systems that addresses this critical problem of dealing with uncertainty without the knowledge of prior distribution of the alerts. Specifically, an evidence theoretic based approach leveraging Dempster Shafer (DS) combination rules and their variants is proposed for reducing false alerts. A multi-hypothesis mass function model is designed that leverages probability scores obtained from various supervised-learning classifiers. Using this model, a location-cum-domain based fusion framework is proposed to evaluate the intrusion detector's performance using Disjunctive, Conjunctive and Cautious Conjunctive rules of combinations, that fuse multiple piece of evidences from inter-domain and intradomain sensors. The approach is demonstrated in a cyber-physical power system testbed (RESLab), and the classifiers are trained with datasets from Man-In-The-Middle attack emulation in a large-scale synthetic electric grid. For evaluating the performance, we consider plausibility, belief, pignistic, general Bayesian theorem based metrics as decision functions. To improve the performance, a multi-objective based genetic algorithm is proposed for feature selection considering the decision metrics as the fitness function. Finally, we present a software application to evaluate the DS fusion approaches with different parameters and architectures.