Image Captioning for Ambient Awareness on a Sidewalk

Ahmed, Faruk; Mahmud, Sultan; Al-Fahad, Rakib; Alam, Shahinur; Yeasin, Mohammed

doi:10.1109/icdis.2018.00020

Cited by 20 publications

(8 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…QUEST is very generic and was not used widely. PYTHEIA scale is used to measure reliability and validity of assistive services (Anam et al, 2014;Alam et al, 2016;Ahmed et al, 2018;Alam et al, 2020;Alam, 2021). However, it can be customized to evaluate other applications as well.…”

Section: Subjective Evaluationmentioning

confidence: 99%

An End-To-End Pipeline for Fully Automatic Morphological Quantification of Mouse Brain Structures From MRI Imagery

Alam

Eom²,

Steinberg³

et al. 2022

Front. Bioinform.

View full text Add to dashboard Cite

Segmentation of mouse brain magnetic resonance images (MRI) based on anatomical and/or functional features is an important step towards morphogenetic brain structure characterization of murine models in neurobiological studies. State-of-the-art image segmentation methods register image volumes to standard presegmented templates or well-characterized highly detailed image atlases. Performance of these methods depends critically on the quality of skull-stripping, which is the digital removal of tissue signal exterior to the brain. This is, however, tedious to do manually and challenging to automate. Registration-based segmentation, in addition, performs poorly on small structures, low resolution images, weak signals, or faint boundaries, intrinsic to in vivo MRI scans. To address these issues, we developed an automated end-to-end pipeline called DeepBrainIPP (deep learning-based brain image processing pipeline) for 1) isolating brain volumes by stripping skull and tissue from T2w MRI images using an improved deep learning-based skull-stripping and data augmentation strategy, which enables segmentation of large brain regions by atlas or template registration, and 2) address segmentation of small brain structures, such as the paraflocculus, a small lobule of the cerebellum, for which DeepBrainIPP performs direct segmentation with a dedicated model, producing results superior to the skull-stripping/atlas-registration paradigm. We demonstrate our approach on data from both in vivo and ex vivo samples, using an in-house dataset of 172 images, expanded to 4,040 samples through data augmentation. Our skull stripping model produced an average Dice score of 0.96 and residual volume of 2.18%. This facilitated automatic registration of the skull-stripped brain to an atlas yielding an average cross-correlation of 0.98. For small brain structures, direct segmentation yielded an average Dice score of 0.89 and 5.32% residual volume error, well below the tolerance threshold for phenotype detection. Full pipeline execution is provided to non-expert users via a Web-based interface, which exposes analysis parameters, and is powered by a service that manages job submission, monitors job status and provides job history. Usability, reliability, and user experience of DeepBrainIPP was measured using the Customer Satisfaction Score (CSAT) and a modified PYTHEIA Scale, with a rating of excellent. DeepBrainIPP code, documentation and network weights are freely available to the research community.

show abstract

Section: Subjective Evaluationmentioning

confidence: 99%

An End-To-End Pipeline for Fully Automatic Morphological Quantification of Mouse Brain Structures From MRI Imagery

Alam

Eom²,

Steinberg³

et al. 2022

Front. Bioinform.

View full text Add to dashboard Cite

show abstract

“…Implementing this idea falls under "system thinking" which is the next subsection. Moreover, it is noticed that the image-based solutions suffer from high computation cost and heavily depend on visible light [6,47,48] which is another motivation for the revision of the problem.…”

Section: Image Captioningmentioning

confidence: 99%

“…For comparison, we selected the base case as 78.75% "in field obstacle detection accuracy" from [6]. We used the AG on the real sidewalk and found that the average accuracy measured is about 81.29%.…”

Section: Evaluation Of the Modelsmentioning

confidence: 99%

See 1 more Smart Citation

Virtual Experience to Real World Application: Sidewalk Obstacle Avoidance Using Reinforcement Learning for Visually Impaired

Ahmed,

Mahmud,

Moinuddin

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Finding a path free from obstacles that poses minimal risk is critical for safe navigation. People who are sighted and people who are visually impaired require navigation safety while walking on a sidewalk. In this research we developed an assistive navigation on a sidewalk by integrating sensory inputs using reinforcement learning. We trained a Sidewalk Obstacle Avoidance Agent (SOAA) through reinforcement learning in a simulated robotic environment. A Sidewalk Obstacle Conversational Agent (SOCA) is built by training a natural language conversation agent with real conversation data. The SOAA along with SOCA was integrated in a prototype device called augmented guide (AG). Empirical analysis showed that this prototype improved the obstacle avoidance experience about 5% from a base case of 81.29%.

show abstract

“…In last decade, numerous application has been developed to assist people with disabilities in navigation [24], expression detection [14] , currency recognition [9], ambient awareness [12], object recognition [13,29,33,17,16]. However, developing an automated system to identify persons and to assess incoming threats to homes did not receive considerable attention from the researchers.…”

Section: Problem Backgroundmentioning

confidence: 99%

Toward Building Safer Smart Homes for the People with Disabilities

Alam,

Mahmud,

Yeasin

2020

Preprint

Self Cite

View full text Add to dashboard Cite

Situational awareness is a critical foundation for the protection of human life/properties and is challenging to maintain for people with disabilities (i.e., visual impairments and limited mobility). In this paper, we present a dialog enabled end-to-end assistive solution called SafeAccess to build a safer smart home by providing situational awareness. The key functions of SafeAccess are: -1) monitoring homes and identifying incoming persons; 2) helping users in assessing incoming threats (e.g., burglary, robbery, gun violence); and, 3) allowing users to grant safe access to homes for friends/families. In this work, we focus on building a robust model for detecting and recognizing person, generating image descriptions, and designing a prototype for the smart door. To interact with the system, we implemented a dialog enabled smartphone app, especially for creating a personalized profile from face images or videos of friends/families. A Raspberry pi connected to the home monitoring cameras captures the video frames and performs change detection to identify frames with activities. Then, we detect human presence using Faster r-cnn and extract faces using Multi-task Cascaded Convolutional Networks (MTCNN). Subsequently, we match the detected faces using FaceNet/support vector machine (SVM) classifiers. The system notifies users with an MMS containing the name of incoming persons or as unknown, scene image, facial description, and contextual information. The users can grant access or call emergency services using the SafeAccess app based on the received notification. Our system identifies persons with an F-score 0.97 and recognizes items to generate image description with an average F-score 0.97.

show abstract

Image Captioning for Ambient Awareness on a Sidewalk

Cited by 20 publications

References 10 publications

An End-To-End Pipeline for Fully Automatic Morphological Quantification of Mouse Brain Structures From MRI Imagery

An End-To-End Pipeline for Fully Automatic Morphological Quantification of Mouse Brain Structures From MRI Imagery

Virtual Experience to Real World Application: Sidewalk Obstacle Avoidance Using Reinforcement Learning for Visually Impaired

Toward Building Safer Smart Homes for the People with Disabilities

Contact Info

Product

Resources

About