Camera-based, mobile disease surveillance using Convolutional Neural Networks

Abstract

ObjectiveAutomated syndromic surveillance using mobile devices is an emerging public health focus that has a high potential for enhanced disease tracking and prevention in areas with poor infrastructure. Pacific Northwest National Laboratory sought to develop an Android mobile application for syndromic biosurveillance that would i) use the phone camera to take images of human faces to detect individuals that are sick through a machine learning (ML) model and ii) collect image data to increase training data available for ML models. The initial prototype use case is for screening and tracking the health of soldiers for use by the Department of Defense’s Disease Threat Reduction Agency.IntroductionInfectious diseases present with multifarious factors requiring several efforts to detect, prevent, and break the chain of transmission. Recently, machine learning has shown to be promising for automated surveillance leading to rapid and early interventions, and extraction of phenotypic features of human faces [3, 5]. In addition, mobile devices have become a promising tool to provide on-the-ground surveillance, especially in remote areas and geolocation mapping [4].Pacific Northwest National Laboratory (PNNL) combines machine learning with mobile technology to provide a groundbreaking prototype of disease surveillance without the need for internet, just a camera. In this android application, VisionDx, a machine learning algorithm analyses human face images and within milliseconds notifies the user with confidence level whether or not the person is sick. VisionDx comes with two modes, photo and video, and additional features of history, map, and statistics. This application is the first of its kind and provides a new way to think about the future of syndromic surveillance.MethodsData. Human healthy (n = 1096) and non-healthy (n = 1269) facial images met the criteria for training the Machine Learning model after preprocessing them. The healthy images were obtained from the Chicago face database [6] and California Institute of Technology [2]. There are no known collections of disease facial images. Using open source image collection/curation services, images were identified by a variety of keywords, including specific infectious diseases. The criteria for image inclusion was 1. a frontal face was identified using OpenCV library [1], and 2. the image contained signs of disease through visual inspection (e.g., abnormal color, texture, swelling).Model. To identify a sick face from a healthy one, we used transfer machine learning and experimented with various pretrained Convolutional Neural Networks (CNN) from Google for mobile and embedded vision applications. Using MobileNet, we trained the final model with our data and deployed it to our prototype mobile app. Google Mobile Vision API and TensorFlow mobile were used to detect human faces and run predictions in the mobile app.Mobile Application. The Android app was built using Android Studio to provide an easily navigable interface that connects every action between tabbed features. The app features (i.e., Map, Camera, History, and Statistics) are in tab view format. The custom-made camera is the main feature of the app, and it contains face detection capability. A real-time health status detection function gives a level of confidence based the algorithm results found on detected faces in the camera image.ResultsPNNL's prototype Android application, VisionDx, was built with user-friendly tab views and functions to take camera images of human faces and classify them as sick or healthy through an inbuilt ML model. The major functions of the app are the camera, map, history, and statistics pages. The camera tab has a custom-made camera with face detection algorithm and classification model of sick or healthy. The camera has image or video mode and results of the algorithm are updated in milliseconds. The Statistics view provides a simple pie chart on sick/healthy images based on user selected time and location. The Map shows pins representing all labeled images stored, and the History displays all the labeled images. Clicking on an image in either view shows the image with metadata, i.e., model confidence levels, geolocation, and datetime.The CNN model prediction accuracy has ~98% validation accuracy and ~96% test accuracy. High model performance shows the possibility that deep learning could be a powerful tool to detect sickness. However, given the limited dataset, this high accuracy also means the model is most likely overfit to the data. The training set is limited: a. the number of training images is small compared to the variability in facial expressions and skin coloring, and b. the sick images only contained overt clinical signs. If trained on a larger, diverse set of data, this prototype app could prove extremely useful in surveillance efforts of individual to large groups of people in remote areas, e.g., to identify individuals in need of medical attention or get an overview of population health. In effort to improve the model, VisionDx was developed as a data collection tool to build a more comprehensive dataset. Within the tool, users can override the model prediction, i.e., false positive or false negative, with a simple toggle button. Lastly, the app was built to protect privacy so that other phone aps can't access the images unless shared by a user.ConclusionsDeveloped at PNNL for the Defense Threat Reduction Agency, VisionDx is a novel, camera-based mobile application for real-time biosurveillance and early warning in the field without internet dependency. The prototype mobile app takes pictures of faces and analyzes them using a state-of-the-art machine learning model to give two confidence levels of likelihood of being sick and healthy. With further development of a labeled dataset, such as by using the app as a data collection too, the results of the algorithm will quickly improve leading to a ground-breaking approach to public health surveillance.References1. Bradski G. (n.d.) The OpenCV Library. Retrieved Sept 30, 2018 at http://www.drdobbs.com/open-source/the-opencv-library/1844043192. Computational Vision: Archive. (1999). Retrieved Sept 22, 2018 at http://www.vision.caltech.edu/html-files/archive.html3. Ferry Q, Steinberg J, Webber C, et al (2014). Diagnostically relevant facial gestalt information from ordinary photos. ELife, 3, e02020.4. Fornace KM, Surendra H, Abidin TR, et al (2018). Use of mobile technology-based participatory mapping approaches to geolocate health facility attendees for disease surveillance in low resource settings. International Journal of Health Geographics, 17(1), 21. https://doi.org/10.1186/s12942-018-0141-05. Lopez DM, de Mello FL, G Dias, CM, et al (2017). Evaluating the Surveillance System for Spotted Fever in Brazil Using Machine-Learning Techniques. Frontiers in Public Health, 5. https://doi.org/10.3389/fpubh.2017.003236. Ma DS, Correll J, Wittenbrink B. (2015) The Chicago face database: A free stimulus set of faces and norming data. Behavior Research Methods, 47(4), 1122–1135. https://doi.org/10.3758/s13428-014-0532-5