Introduction: Recently, the tendency of artificial intelligence (AI) and big data use/applications is has been rapidly expanding across the globe, improving people’s lifestyles with data-driven services (i.e., recommendations, smart healthcare, etc.). The synergy between AI and big data has become imperative considering the drastic growth in personal data stemming from diverse sources (cloud computing, IoT, social networks, etc.). However, when data meet AI at some central place, it invites unimaginable privacy issues, and one of those issues is group privacy. Despite being the most significant problem, group privacy has not yet received the attention of the research community it is due. Problem Statement: We study how to preserve the privacy of particular groups (a community of people with some common attributes/properties) rather than an individual in personal data handling (i.e., sharing, aggregating, and/or performing analytics, etc.), especially when we talk about groups purposely made by two or more people (with clear group identifying markers), for whom we need to protect their privacy as a group. Aims/Objectives: With this technical letter, our aim is to introduce a new dimension of privacy (e.g., group privacy) from technical perspectives to the research community. The main objective is to advocate the possibility of group privacy breaches when big data meet AI in real-world scenarios. Methodology: We set a hypothesis that group privacy (extracting group-level information) is a genuine problem, and can likely occur when AI-based techniques meet high dimensional and large-scale datasets. To prove our hypothesis, we conducted a substantial number of experiments on two real-world benchmark datasets using AI techniques. Based on the experimental analysis, we found that the likelihood of privacy breaches occurring at the group level by using AI techniques is very high when data are sufficiently large. Apart from that, we tested the parameter effect of AI techniques and found that some parameters’ combinations can help to extract more and fine-grained data about groups. Findings: Based on experimental analysis, we found that vulnerability of group privacy can likely increase with the data size and capacity of the AI method. We found that some attributes of people can act as catalysts in compromising group privacy. We suggest that group privacy should also be given due attention as individual privacy is, and robust tools are imperative to restrict implications (i.e., biased decision making, denial of accommodation, hate speech, etc.) of group privacy. Significance of results: The obtained results are the first step towards responsible data science, and can pave the way to understanding the phenomenon of group privacy. Furthermore, the results contribute towards the protection of motives/goals/practices of minor communities in any society. Concluding statement: Due to the significant rise in digitation, privacy issues are mutating themselves. Hence, it is vital to quickly pinpoint emerging privacy threats and suggest practical remedies for them in order to mitigate their consequences on human beings.