The damage tolerant design philosophy is based on periodical inspections and provides safe bridge operation preventing fatigue cracks from growing up to a critical size. It is possible to optimize the management costs of a bridge throughout its life by designing the inspection frequency, which depends on the capabilities of the inspection method. However, such optimization requires knowledge about the performance of inspections that are envisioned to take place during the use of the bridge. Regular visual inspections is the most frequently applied type of inspection of bridge structures. Recent advances in computer vision technologies provide a strong basis for the development of automatic damage detection systems that can support regular visual inspection, thus increasing the reliability of the inspection. Several automatic crack detection systems have been developed in the past years. However, the performances of such systems have not been evaluated in the way as for traditional non‐destructive inspection methods, i.e. in terms of probability of detection curves and detectability limits. This restricts the applicability of automatic visual inspections for inspection planning and damage tolerant design. This paper proposes an encoder‐decoder neural network for segmentation of cracks on images of steel bridges. A probability of detection curve is calculated for this neural network.