Abstract-Improving the ultrasound inspection capability for coarse grain metals remains of longstanding interest and is expected to become increasingly important for next generation electricity power plants. Conventional ultrasonic A, B, and C-scans have been found to suffer from strong background noise due to grain scattering which can severely limit the detection of defects. However, in recent years, array probes and Full Matrix Capture (FMC) imaging algorithms have unlocked exciting possibilities for improvements. In order to progress and compare these algorithms we must rely on robust methodologies to quantify their performance. This article proposes such a methodology to evaluate the detection performance of imaging algorithms. For illustration, the methodology is applied to some example data using three FMC imaging algorithms; Total Focusing Method (TFM), Phase Coherent Imaging (PCI), and Decomposition of the Time Reversal Operator with Multiple Scattering Filter (DORT MSF). However it is important to note that this is solely to illustrate the methodology; this article does not attempt the broader investigation of different cases that would be needed to compare the performance of these algorithms in general. The methodology considers the statistics of detection, presenting the detection performance as Probability of Detection (POD) and probability of False Alarm (PFA). A test sample of coarse grained INCONEL 625, manufactured to represent materials used for future power plant components and containing some simple artificial defects, is used to illustrate the method on the candidate algorithms. The data is captured in pulse-echo mode using 64 element array probes at centrefrequencies of 1MHz and 5MHz. In this particular case, it turns out that all three algorithms are shown to perform very similarly when comparing their flaw detection capabilities.