Nowadays, data is increasing exponentially, although cloud computing is an eminent approach for the organization, processing, and availability of data for organizational growth over the internet. Besides, a lot of advantages of cloud computing, yet it is suffered from security challenges, which affect big data while using cloud services on the internet. For this purpose, we conducted a detailed systematic literature review (SLR) study, to identify and capture the security challenges of big data on the cloud computing platform. Our research findings determine and develop a taxonomy, based on the prioritization of the security challenges of big data on cloud computing. We identified a total of 15 critical security challenges using the proposed SLR, with the frequency of each challenge >25%, and are further validated by the industrial specialists using a questionnaire survey study. The identified security challenges are data secrecy issue, geographical data location issue, unauthorized data access issue, lack of control, lack of data management, network‐level issues, data integrity issue, data recovery issue, lack of trust, data sharing issue, data availability, asset issues, legal amenabilities, lack of quality issues, and lack of consistency. The security challenges identified and captured through the SLR study are categorized into four levels, namely, steadiness, management, control, and eminence. Conclusively, we applied the fuzzy‐TOPSIS approach to prioritize and identify the significance of each identified security challenge for the big data usage on cloud computing. Based on our proposed approach, the “data secrecy issue” has been identified as the most prominent security challenge with the captured value of “0.765.” The fuzzy TOPSIS is an effective and innovative research approach in the field of computer science. It has been applied positively to other research areas to address and identify the fuzziness and uncertainty of multiple decision‐making glitches. The findings of the research paper will assist the software vendor organization when using the cloud platform for big data security. Also, using the proposed approach, software vendors can prioritize and analyze the uncertainty and ambiguousness among these security challenges.