Studies have shown that a dominant class of questions asked by visually impaired users on images of their surroundings involves reading text in the image. But today's VQA models can not read! Our paper takes a first step towards addressing this problem. First, we introduce a new "TextVQA" dataset to facilitate progress on this important problem. Existing datasets either have a small proportion of questions about text (e.g., the VQA dataset) or are too small (e.g., the VizWiz dataset). TextVQA contains 45,336 questions on 28,408 images that require reasoning about text to answer. Second, we introduce a novel model architecture that reads text in the image, reasons about it in the context of the image and the question, and predicts an answer which might be a deduction based on the text and the image or is composed of the strings found in the image. Consequently, we call our approach Look, Read, Reason & Answer (LoRRA) 1 . We show that LoRRA outperforms existing state-of-the-art VQA models on our TextVQA dataset. We find that the gap between human performance and machine performance is significantly larger on TextVQA than on VQA 2.0, suggesting that TextVQA is well-suited to benchmark progress along directions complementary to VQA 2.0. VQA ComponentSimilar to many VQA models [7,17], we first embed the question words w 1 , w 2 , . . . , w L of the question q with a pre-trained embedding function (e.g. GloVe [36]) and then encode the resultant word embeddings iteratively with a re-

Skin and subcutaneous conditions affect an estimated 1.9 billion people at any given time and remain the fourth leading cause of non-fatal disease burden worldwide.Access to dermatology care is limited due to a shortage of dermatologists, causing long wait times and leading patients to seek dermatologic care from general practitioners.However, the diagnostic accuracy of general practitioners has been reported to be only 0. 24-0. 70 (compared to 0. 77-0. 96 for dermatologists), resulting in over-and under-referrals, delays in care, and errors in diagnosis and treatment. In this paper, we developed a deep learning system (DLS) to provide a differential diagnosis of skin conditions for clinical cases (skin photographs and associated medical histories). The DLS distinguishes between 26 of the most common skin conditions, representing roughly 80% of the volume of skin conditions seen in a primary care setting. The DLS was developed and validated using de-identified cases from a teledermatology practice serving 17 clinical sites via a temporal split: the first 14,021 cases for development and the last 3,756 cases for validation. On the validation set, where a panel of three board-certified dermatologists defined the reference standard for every case, the DLS achieved 0.71 and 0.93 top-1 and top-3 accuracies respectively, indicating the fraction of cases where the DLS's top diagnosis and top 3 diagnoses contains the correct diagnosis. For a stratified random subset of the validation set (n=963 cases), 18 clinicians (of three different training levels) reviewed the cases for comparison. On this subset, the DLS achieved a 0.67 top-1 accuracy, non-inferior to board-certified dermatologists (0.63, p<0.001), and higher than primary care physicians (PCPs, 0.45) and nurse practitioners (NPs, 0.41). The top-3 accuracy showed a similar trend: 0.90 DLS, 0.75 dermatologists, 0.60 PCPs, and 0.55 NPs . These results highlight the potential of the DLS to augment the ability of general practitioners who did not have additional specialty training to accurately diagnose skin conditions by suggesting differential diagnoses that may not have been considered. Future work will be needed to prospectively assess the clinical impact of using this tool in actual clinical workflows.

No abstract

No abstract

This document describes Pythia v0.1, the winning entry from Facebook AI Research (FAIR)'s A-STAR team to the VQA Challenge 2018 1 .Our starting point is a modular re-implementation of the bottom-up top-down (up-down) model [1,14]. We demonstrate that by making subtle but important changes to the model architecture and the learning rate schedule, finetuning image features, and adding data augmentation, we can significantly improve the performance of the up-down model on VQA v2.0 dataset [6] -from 65.67% to 70.24%.Furthermore, by using a diverse ensemble of models trained with different features and on different datasets, we are able to significantly improve over the 'standard' way of ensembling (i.e. same model with different random seeds) by 1.31%. Overall, we achieve 72.27% on the teststd split of the VQA v2.0 dataset. Our code in its entirety (training, evaluation, data-augmentation, ensembling) and pre-trained models are publicly available at: https:// github.com/facebookresearch/pythia . * indicates equal contributions.1 and changes made after the challenge deadline.2 Agents that See, Talk, Act, and Reason.