To investigate the effect of the exposure parameters on image quality (IQ) metrics of phantom images, obtained automatically using software or from visual evaluation. Methods: Three commercial phantoms and a homemade phantom constructed according to the instructions given in the IAEA Human Health Series No. 39 publication were used, along with the respective software that estimate automatically various IQ metrics. Images with various exposure parameters were acquired in a digital radiography (DR) unit. For the commercial phantoms, visual evaluations were also performed. The IQ scores obtained were analyzed to investigate the effects of increasing incident air kerma (IAK), tube potential (kVp), additional filtration, and acquisition protocol on IQ. Results: The effects of the exposure parameters on the IQ metrics, determined with the commercial and the IAEA phantoms, were not the same. For example, clear trends of improvement of IQ scores with increased IAK and reduction of most IQ scores with increased kVp were observed mostly with the IAEA phantom, but not with the commercial phantoms (for both automatic and visual scoring methods). For all phantoms, the maximum variations in IQ scores observed for repeated identical exposures were almost always below 10% with automatic evaluation whereas, for visual evaluation, reached 17%. Conclusions: Failure to detect some expected trends with the complex commercial phantoms may be attributed to the fact that IQ in DR is more strongly affected by the post-processing procedures, which may mask the effect of other parameters on IQ, something that was not observed with the simple IAEA phantom.