The automatic and accurate classification of medical imaging data has potential applications in computer-aided disease diagnosis, prognosis, and treatment. However, it remains a challenge to optimize recent deep learning algorithms in medical domain for the accurate classification of large-scale 3D volumetric data. To address these challenges, we propose an efficient deep volumetric classification network based on the aggregation of multilevel deep features for accurate classification of large-scale medical 2D/3D imaging data. To perform a detailed quantitative analysis of our method, 26 different datasets were fused to construct a single large-scale multimodal database that comprises a total of seventy different classes, including 151,095 data samples. Additionally, 15 different baseline methods were configured under the same experimental protocol for volumetric model genesis and extensive performance comparison with our method. The experimental results of our method exhibited promising performance as area under the curve of 93.66% and outperformed various state-of-the-art methods.