A smart city implies a consistent use of technology for the benefit of the community. As the city develops over time, components and subsystems such as smart grids, smart water management, smart traffic and transportation systems, smart waste management systems, smart security systems, or e-governance are added. These components ingest and generate a multitude of structured, semi-structured or unstructured data that may be processed using a variety of algorithms in batches, micro batches or in real-time. The ICT architecture must be able to handle the increased storage and processing needs. When vertical scaling is no longer a viable solution, Hadoop can offer efficient linear horizontal scaling, solving storage, processing, and data analyses problems in many ways. This enables architects and developers to choose a stack according to their needs and skill-levels. In this paper, we propose a Hadoop-based architectural stack that can provide the ICT backbone for efficiently managing a smart city. On the one hand, Hadoop, together with Spark and the plethora of NoSQL databases and accompanying Apache projects, is a mature ecosystem. This is one of the reasons why it is an attractive option for a Smart City architecture. On the other hand, it is also very dynamic; things can change very quickly, and many new frameworks, products and options continue to emerge as others decline. To construct an optimized, modern architecture, we discuss and compare various products and engines based on a process that takes into consideration how the products perform and scale, as well as the reusability of the code, innovations, features, and support and interest in online communities.

Big data frameworks enable companies from various fields to build models that allow them to increase profit margins by improving decision making at different levels (middle management, senior management, and board) or by attempting to boost sales by customizing consumers' experience based on their history and feedback. Institutions and other entities also use big data coming from all kinds of sensors, data that can be used to detect, in real time or in retrospect, possible problems (e.g., frauds, malfunctions, and supply shortages), or to identify patterns and trends. In this paper, we organize large volumes of community electricity consumption data coming from smart meters, smart plugs, and other sensors, but also data regarding consumers' preferences in order to assist them to dynamically optimize their electricity consumption. In this regard, we develop a novel optimization approach that reschedules every fifteen min the appliances for residential consumers to reduce both the consumption peaks and the payments at the community level. The consumers send their day-ahead schedule that is optimized and further implemented to some extent. Thus, we monitor the electricity consumption via sensors and smart meters and dynamically adjust the schedule in case the real consumption deviates from the optimized plan, considering appliances constraints and consumers' preferences. Every fifteen min, the algorithm evaluates the differences between the optimized schedule and the actual consumption and controls the operation of the interruptible appliances to stick with the day-ahead schedule as much as possible.

No abstract

Big Data technologies can be very useful when it comes to storing and processing using sophisticated algorithms, terabytes or petabytes of data. With the latest advancements, such as Hadoop YARN, processing can be done not only in batch but also in real time. In this paper, we detail a methodology followed by a case study that investigates the power of machine learning algorithms used in a Hadoop environment in classifying unstructured data. We also investigate how to capture geolocated messages from social networks and how kriging can be used to see if there is a strong relationship between two or more such datasets.

No abstract