With the prevalence of Internet of Things (IoT), edge computing has emerged as a novel computing model for optimizing traditional cloud computing systems by moving part of the computational tasks to the edge of the network for better performance and security. With the technique of services computing, edge computing systems can accommodate the application requirements with more agility and flexibility. In large-scale edge computing systems, service composition as one of the most important problems in services computing suffers from several new challenges, i.e., complex layered architecture, failures and recoveries always in the lifecycle, and search space explosion. In this paper, we make an attempt at addressing these challenges by designing a simulation-based optimization approach for reliability-aware service composition. Composite stochastic Petri net models are proposed for formulating the dynamics of multi-layered edge computing systems, and their corresponding quantitative analysis is conducted. To solve the state explosion problem in large-scale systems or complex service processes, time scale decomposition technique is applied to improving the efficiency of model solving. Additionally, simulation schemes are designed for performance evaluation and optimization, and ordinal optimization technique is introduced to significantly reduce the size of the search space. Finally, we conduct experiments based on real-life data, and the empirical results validate the efficacy of the approach.