Fragility plays a pivotal role in performance‐based earthquake engineering, which represents the seismic performance of structural systems. To comprehensively understand the structural performance under seismic events, it is necessary to consider uncertainties in the structural model, i.e., epistemic uncertainties. However, considering such uncertainties is challenging due to computational complexity, leading most fragility analyses only to consider the chaotic behavior of ground motions on structural responses, i.e., aleatoric uncertainties. To address this challenge, this study proposes an adaptive algorithm that intertwines with the conventional fragility analysis procedures to consider both aleatoric and epistemic uncertainties. The algorithm introduces Gaussian process‐based metamodels to efficiently consider epistemic uncertainties with a small number of time history analyses. Steel moment‐resisting frame structures and a reinforced concrete building are used to demonstrate the improved efficiency and wide applicability of the proposed method. In each case, the proposed method yields fragility curves consistent with reference solutions but with substantially lower computational effort. Comprehensive discussions are provided regarding ground motion sets, structural types, and definitions of limit‐states to demonstrate the robustness of the proposed approach.