Pneumonia is one of the leading causes of death in both infants and elderly people, with approximately 4 million deaths each year. It may be a virus, bacterial, or fungal, depending on the contagious pathogen that damages the lung’s tiny air sacs (alveoli). Patients with underlying disorders such as asthma, a weakened immune system, hospitalized babies, and older persons on ventilators are all at risk, particularly if pneumonia is not detected early. Despite the existing approaches for its diagnosis, low accuracy and efficiency require further research for more accurate systems. This study is a similar endeavor for the detection of pneumonia by the use of X-ray images. The dataset is preprocessed to make it suitable for transfer learning tasks. Different pre-trained convolutional neural network (CNN) variants are utilized, including VGG16, Inception-v3, and ResNet50. Ensembles are made by incorporating CNN with Inception-V3, VGG-16, and ResNet50. Besides the common evaluation metrics, the performance of the pre-trained and ensemble deep learning models is measured with Cohen’s kappa as well as the area under the curve (AUC). Experimental results show that Inception-V3 with CNN attained the highest accuracy and recall score of 99.29% and 99.73%, respectively.