Handling big data: research challenges and future directions

Anagnostopoulos, Ioannis; Zeadally, Sherali; Expósito, Ernesto

doi:10.1007/s11227-016-1677-z

Cited by 117 publications

(56 citation statements)

References 56 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Then, we introduce the key components of Apache Spark stack in Sect. 3. Section 4 introduces data and computation abstractions in Apache Spark.…”

Section: At Databricks and Sparkmentioning

confidence: 99%

“…Mobius 62 (formerly Spark-CLR) is a cross-company open source project at Microsoft Research that aims to provide C# language bindings for Apache Spark. There is also a considerable body of research on distributed frameworks, including Apache Spark, for solving big data challenges [3,27].…”

Section: Related Researchmentioning

confidence: 99%

“…In addition, Databricks, the company founded by the creators of Apache Spark, has developed a set of reference applications 2 to demonstrate how Apache Spark can be used for different workloads. Other good sources are the official blog 3 …”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Big data analytics on Apache Spark

Salloum

Dautov

Chen

et al. 2016

Int J Data Sci Anal

327

121

View full text Add to dashboard Cite

Apache Spark has emerged as the de facto framework for big data analytics with its advanced in-memory programming model and upper-level libraries for scalable machine learning, graph analysis, streaming and structured data processing. It is a general-purpose cluster computing framework with language-integrated APIs in Scala, Java, Python and R. As a rapidly evolving open source project, with an increasing number of contributors from both academia and industry, it is difficult for researchers to comprehend the full body of development and research behind Apache Spark, especially those who are beginners in this area. In this paper, we present a technical review on big data analytics using Apache Spark. This review focuses on the key components, abstractions and features of Apache Spark. More specifically, it shows what Apache Spark has for designing and implementing big data algorithms and pipelines for machine learning, graph analysis and stream processing. In addition, we highlight some research and development directions on Apache Spark for big data analytics.

show abstract

“…Then, we introduce the key components of Apache Spark stack in Sect. 3. Section 4 introduces data and computation abstractions in Apache Spark.…”

Section: At Databricks and Sparkmentioning

confidence: 99%

Section: Related Researchmentioning

confidence: 99%

See 1 more Smart Citation

Big data analytics on Apache Spark

Salloum

Dautov

Chen

et al. 2016

Int J Data Sci Anal

327

121

View full text Add to dashboard Cite

show abstract

“…The ultimate research challenge in the field remains, of course, the efficient illustration of contextual information in such a manner that would allow for the merging of the gap between applications using contextual information and the deployment of context-aware services [45], keeping always in mind that efficient development of such applications, like the one dealing with spam detection [46], demands clearly defined models of context, as depicted in an extremely recent related research effort [47]. Table 4 summarizes the most representative context-aware approaches.…”

Section: Context-awarenessmentioning

confidence: 99%

“…Due to its inherent nature, the sub-field of context-awareness allows for great diversity in the research works tackling it. They range from the ones dealing with computing applications and systems in general [31,33,34,37], to the ones focusing on context modeling [40,41,44], human-computer interaction [39], big and social data analytics [43,[45][46][47], as well as survey papers [32,38,42]. Specific, targeted apps and systems, like infomobility [35] or archaeology [36], are also included herein.…”

Section: Context-awarenessmentioning

confidence: 99%

The Evolution of Contextual Information Processing in Informatics

Mylonas

2018

Information

View full text Add to dashboard Cite

After many decades of flourishing computer science it is now rather evident that in a world dominated by different kinds of digital information, both applications and people are forced to seek new, innovative structures and forms of data management and organization. Following this blunt observation, researchers in informatics have strived over the recent years to tackle the non-unique and rather evolving notion of context, which aids significantly the data disambiguation process. Motivated by this environment, this work attempts to summarize and organize in a researcher-friendly tabular manner important or pioneer related research works deriving from diverse computational intelligence domains: Initially, we discuss the influence of context with respect to traditional low-level multimedia content analysis and search, and retrieval tasks and then we advance to the fields of overall computational context-awareness and the so-called human-generated contextual elements. In an effort to provide meaningful information to fellow researchers, this brief survey focuses on the impact of context in modern and popular computing undertakings of our era. More specifically, we focus to the presentation of a short review of visual context modeling methods, followed by the depiction of context-awareness in modern computing. Works dealing with the interpretation of context by human-generated interactions are also discussed herein, as the particular domain gains an ever-increasing proportion of related research nowadays. We then conclude the paper by providing a short discussion on (i) the motivation behind the included context type categorization into three main pillars; (ii) the findings and conclusions of the survey for each context category; and (iii) a couple of brief advices derived from the survey for both interested developers and fellow researchers.

show abstract

Deep learning support for intelligent transportation systems

Ibáñez

Contreras-Castillo

Zeadally

2020

Trans Emerging Tel Tech

Self Cite

View full text Add to dashboard Cite

Intelligent Transportation Systems (ITS) help improve the ever-increasing vehicular flow and traffic efficiency in urban traffic to reduce the number of accidents. The generation of massive amounts of data generated by all the digital devices connected to the transportation network enables the creation of datasets to perform an in-depth analysis of the data using deep learning methods. Such methods can help predict traffic performance, automated traffic light management, lane detection, and identifying objects near vehicles to increase the safety and efficiency of ITS. We discuss some of the challenges that need to be solved to achieve seamless integration between ITS and deep learning methods to address issues such as (1) improving traffic flow/transportation logistics, (2) predicting best routes for the transportation of goods, (3) optimal fuel consumption, (4) intelligent environmental conditions perception, (5) traffic speed management, and accident prevention.

show abstract

Handling big data: research challenges and future directions

Cited by 117 publications

References 56 publications

Big data analytics on Apache Spark

Big data analytics on Apache Spark

The Evolution of Contextual Information Processing in Informatics

Deep learning support for intelligent transportation systems

Contact Info

Product

Resources

About