Objective: Objective, sensitive, and meaningful disease assessments are critical to support clinical trials and clinical care. Speech changes are one of the earliest and most evident manifestations of cerebellar ataxias. The purpose of this work is to develop models that can accurately identify and quantify these abnormalities. Methods: We use deep learning models such as ResNet 18, that take the time and frequency partial derivatives of the log-mel spectrogram representations of speech as input, to learn representations that capture the motor speech phenotype of cerebellar ataxia. We train classification models to separate patients with ataxia from healthy controls as well as regression models to estimate disease severity. Results: Our model was able to accurately distinguish healthy controls from individuals with ataxia, including ataxia participants with no detectable clinical deficits in speech. Furthermore the regression models produced accurate estimates of disease severity, were able to measure subclinical signs of ataxia, and captured disease progression over time in individuals with ataxia. Conclusion: Deep learning models, trained on time and frequency partial derivatives of the speech signal, can detect sub-clinical speech changes in ataxias and sensitively measure disease change over time. Significance: Such models have the potential to assist with early detection of ataxia and to provide sensitive and low- burden assessment tools in support of clinical trials and neurological care.