Query-driven video event processing for the internet of multimedia things

Modern distributed computing infrastructure need to process vast quantities of data streams generated by a growing number of participants with information generated in multiple formats. With the Internet of Multimedia Things (IoMT) becoming a reality, new approaches are needed to process realtime multimodal event data streams. Existing approaches to event processing have limited consideration for the challenges of multimodal events, including the need for complex content extraction, increased computational and memory costs. The paper explores event processing as a basis for processing real-time IoMT data. The paper introduces the Multimodal Event Processing (MEP) paradigm, which provides a formal basis for native approaches to neural multimodal content analysis (i.e., computer vision, linguistics, and audition) with symbolic event processing rules to support real-time queries over multimodal data streams using the Multimodal Event Processing Language to express single, primitive multimodal, and complex multimodal event patterns. The content of multimodal streams is represented using Multimodal Event Knowledge Graphs to capture the semantic, spatial, and temporal content of the multimodal streams. The approach is implemented and evaluated within an MEP Engine using single and multimodal queries achieving near real-time performance with a throughput of ~30 fps and sub-second latency of 0.075-0.30 seconds for video streams of 30 fps input rate. Support for high input stream rates (45 fps) is achieved through content-aware load shedding techniques with a ~127X latency improvement resulting in only a minor decrease in accuracy.

show abstract

UrbanAccess

Salwala

Yadav

Munirathnam

et al. 2021

Proceedings of the 1st International Workshop on Multimedia Computing for Urban Data

Self Cite

View full text Add to dashboard Cite

The smart city concept has now become one of the key enablers in urban city management. The adoption and permeation of ICT and AI-driven techniques have enabled the authorities to resolve poor urban planning issues with improved delivery of citizen services. Major urban problem is addressing the accessibility issue across cities road crossing and facilitating visually impaired people via well-defined infrastructure. The research presented in this paper emphasized urban analytics that studies the road crossings and challenges one faces when accessing the footpaths of a city using the Tactile surfaces. This work demonstrates a distributed event analytics platform-GNOSIS to detect complex accessibility event patterns. GNOSIS ingest video data streams from cities infrastructure such as CCTV and detect tactile surface event patterns using an ensemble of deep learning models using a declarative query language. The work analyzes mainly three types of tactile surface -Blister, Cycleway and Directional, collected from different cities in Ireland using crowd-sourcing techniques. GNOSIS makes decisions in real-time based on the type of tactile surface, colour and the making pattern.

show abstract

Query-driven video event processing for the internet of multimedia things

Cited by 6 publications

References 17 publications

Towards Open World Event Processing: A Paradigm and Research Agenda for Event Processing for the Internet of Multimedia Things (Vision Paper)

Towards Open World Event Processing: A Paradigm and Research Agenda for Event Processing for the Internet of Multimedia Things (Vision Paper)

Multimodal Event Processing: A Neural-Symbolic Paradigm for the Internet of Multimedia Things

UrbanAccess

Contact Info

Product

Resources

About