This study proposes a new damage identification method based on topology optimization, combined with visualized ultrasonic wave propagation. Although a moving diagram of traveling waves aids in damage detection, it is difficult to acquire quantitative information about the damage, for which topology optimization is suitable. In this approach, a damage parameter, varying Young’s modulus, represents the state of the damage in a finite element model. The feature of ultrasonic wave propagation (e.g., the maximum amplitude map in this study) is inversely reproduced in the model by optimizing the distribution of the damage parameters. The actual state of the damage was successfully estimated with high accuracy in numerical examples. The sensitivity of the objective function, as well as the appropriate penalization exponent for Young’s modulus, was discussed. Moreover, the proposed method was applied to experimentally measured wave propagation in an aluminum plate with an artificial crack, and the estimated damage state and the sensitivity of the objective function had the same tendency as the numerical example. These results demonstrate the feasibility of the proposed method.