In the satellite lifetime optimization, reliability is a critical issue. For the complex satellite system, Bayesian network (BN) is an important method for reliability modeling and inference. As the number of system's components increases dramatically, the memory storage requirements of the system's node probability table (NPT) will exceed the computer's random access memory (RAM). To solve this challenge, compression methods have been proposed to reduce the memory storage requirements of NPT. However, for the complex satellite system with extreme large number of components and the explosion of probable state combination, the existing methods still face big challenge in compression efficiency. Therefore, an improved encoding compression algorithm is proposed to further enhance the NPT compression effect in this paper. For the hierarchical complex satellite system that has multiple subsystems which are further composed of multiple components, the multilevel BN reliability model is first constructed based on the proposed encoding compression algorithm. By the variable elimination algorithm, a multilevel BN reliability inference algorithm is proposed to perform the inference of the multilevel BN reliability model. Based on the basis of the reliability model above, further considering system mass, power and cost requirement, the satellite lifetime is properly designed by optimizing the system component configuration, including component model/type selection and number determination for redundancy. Finally, two cases are studied to demonstrate and validate the proposed algorithms.