Introduction to Big Data Technology

Abstract

Big data is no more "all just hype" but widely applied in nearly all aspects of our business, governments, and organizations with the technology stack of AI. Its influences are far beyond a simple technique innovation but involves all rears in the world. This chapter will first have historical review of big data; followed by discussion of characteristics of big data, i.e. from the 3V's to up 10V's of big data. The chapter then introduces technology stacks for an organization to build a big data application, from infrastructure/platform/ecosystem to constructional units and components. Finally, we provide some big data online resources for reference.

Investigation into the Challenges of Implementing Scalable Big Data Technologies and Techniques

“…Furthermore, Abu-Salih et al (2021) claims that expensive, highly reliable hardware isn't required; instead, readily available hardware on the market is a viable option when constructing Hadoop Ecosystems. There is no Single Point of Failure (SPOF) issue in HDFS, because one of HDFS's design goals is to handle SPOF, so when a workstation fails in a Hadoop cluster, there will be no visible obstruction to users (Abu-Salih et al, 2021). One issue to be aware of is that HDFS may not be the best choice when an application demands low-latency data access (Abu-Salih et al, 2021).…”

“…There is no Single Point of Failure (SPOF) issue in HDFS, because one of HDFS's design goals is to handle SPOF, so when a workstation fails in a Hadoop cluster, there will be no visible obstruction to users (Abu-Salih et al, 2021). One issue to be aware of is that HDFS may not be the best choice when an application demands low-latency data access (Abu-Salih et al, 2021). According to Abu-Salih et al (2021), if the data set comprises a large number of small files, it is not a good idea to utilize HDFS to store the data since HDFS stores files to a block that is often set to 128 MB (by default) or 256 MB, resulting in storage waste.…”

“…This makes Big data management and storage more difficult and therefore would undoubtedly necessitate assistance by advance data management techniques and technology. As a result, sophisticated, unique, and flexible analytics are required to deal with the challenges of collecting and analyzing a diverse set of Big data islands, which are rapidly growing as a result of the massive amounts of data generated by tracking sensors, social media, transaction records (Abu-Salih et al, 2021). According to ELE Times Research Desk (2018) it is critical to understand your systems' transactional and analytical data processing requirements and to make appropriate selections.…”

“…According to ELE Times Research Desk (2018) it is critical to understand your systems' transactional and analytical data processing requirements and to make appropriate selections. Additionally, it is essential to select the appropriate tools to process Big data because, data is essential for gaining important insights into target demographics and client preferences (Abu-Salih et al, 2021). Every connection with technology, whether active or passive, generates new data that might be used to describe us.…”

“…Velocity is further explained by Abu-Salih et al (2021) as the collection of data at high speeds, near realtime, and in real-time from diverse data sources. Moreover, Abu-Salih et al (2021) states that the velocity of data necessitates advanced solutions capable of storing, processing, managing, and analysing streams of heterogeneous data and inferring value on motion. Figure 1 shows how much data was produced in a minute from numerous sources such as Google, You Tube, Instagram and Facebook in 2019.…”