Objectives: To create a deep learning system for pneumonia detection that is both effective and gradually optimized. Methods: A customized CNN is used with an incremental approach for pneumonia classification and detection. Starting with a baseline model, hypertuning parameters such as four convolution layers with filters of 16, 32, 64, and 128 sizes, a dropout layer with values of 0.3, 0.5, and 0.7, four batch normalization layers, and an Adam optimizer are added. A total images of 5,863 for training, 624 for testing, and 16 for validation from the Paul Mooney dataset were used to test the suggested model. Findings: The study recorded a test accuracy of 94% for the customized CNN followed by ResNet50 at 79.9%, VGG16 at 90.14%, VGG19 at 82.21%, InceptionV3 at 74.51%, and EfficientNetB1 at 83.17%. Recall of 98.20%, accuracy of 85.55%, AUC of 93.52%, and F1_score of 92.45% obtained were all fairly excellent for the customized CNN. 15 epochs, a learning rate of 0.0001, callbacks with a patience of 3, and an early stopping feature were applied to the training model. Novelty: Five convolution blocks, two separable convolution layers, one batch normalization layer, one maxpooling layer, and a fully connected layer with an Adam optimizer were all included in the customized CNN that was developed to identify and categorize pneumonia. With Explainable AI's GradCAM technology, pneumonia-infected areas on chest X-rays were highlighted and the sickness was seen. Keywords: Customized CNN, VGG16, VGG19, ResNet50, Explainable AI