This study proposes a wrapper‐based technique to improve the classification performance of chest infection (including COVID‐19) detection using X‐rays. Deep features were extracted using pretrained deep learning models. Ten optimization techniques, including poor and rich optimization, path finder algorithm, Henry gas solubility optimization, Harris hawks optimization, atom search optimization, manta‐ray foraging optimization, equilibrium optimizer, slime mold algorithm, generalized normal distribution optimization, and marine predator algorithm, were used to determine the optimal features using a support vector machine. Moreover, a network selection technique was used to select the deep learning models. An online chest infection detection X‐ray scan dataset was used to validate the proposed approach. The results suggest that the proposed wrapper‐based automatic deep learning network selection and feature optimization framework has a high classification rate of 97.7%. The comparative analysis further validates the credibility of the framework in COVID‐19 and other chest infection classifications, suggesting that the proposed approach can help doctors in clinical practice.