Novel coronavirus, known as COVID-19, is a very dangerous virus. Initially detected in China, it has since spread all over the world causing many deaths. There are several variants of COVID-19, which have been categorized into two major groups. These groups are variants of concern and variants of interest. Variants of concern are more dangerous, and there is a need to develop a system that can detect and classify COVID-19 and its variants without touching an infected person. In this paper, we propose a dual-stage-based deep learning framework to detect and classify COVID-19 and its variants. CT scans and chest X-ray images are used. Initially, the detection is done through a convolutional neural network, and then spatial features are extracted with deep convolutional models, while handcrafted features are extracted from several handcrafted descriptors. Both spatial and handcrafted features are combined to make a feature vector. This feature vector is called the vocabulary of features (VoF), as it contains spatial and handcrafted features. This feature vector is fed as an input to the classifier to classify different variants. The proposed model is evaluated based on accuracy, F1-score, specificity, sensitivity, specificity, Cohen’s kappa, and classification error. The experimental results show that the proposed method outperforms all the existing state-of-the-art methods.