The potential of automated severity assessment of COVID-19 pneumonia is immense due to its ability to facilitate clinical decision-making. It enables efficient escalation or de-escalation of COVID-19 care. In this work, we propose an efficient pipeline based on weakly-supervised learning for severity score prediction. In the first stage, Attention feature fusion (AFF)-ResNet-101 is trained on five large Chest X-ray (CXR) datasets. In the second stage, using transfer learning on 94 posteroanterior (PA) COVID-19 CXR images, refined feature maps are extracted. These feature descriptors are then mapped to severity ratings based on Geographic Extent score and lung opacity output using the linear regression model. Experimental results show the efficacy of our proposed architecture for the severity score prediction.