Over the last four decades, researchers have recognized that traditional automation has many negative consequences stemming from human out‐of‐the‐loop problems. However, there is a paucity of research to optimize function allocation problems under uncertainty by considering both human and machine influential factors. This work attempts to address this standing paucity of research by designing an optimization model with the objective function of maximizing system reliability besides balancing human workload over a mission time and keeping him/her in the control loop. To maximize system reliability, the prediction of human and machine fault rates is inspired by the human‐in‐the‐loop fault tree analysis (FTA). Furthermore, the failure probabilities of human‐related events in the proposed fault tree are estimated via fuzzy logic inference systems. A typical supervisory control task is selected to demonstrate the application and feasibility of the proposed method. In conclusion, this mathematical model as a decision support method provides guidance for automation designers to improve an automated system via systematic prediction of failures in the early stage of the design phase.