Introduction: We aimed to evaluate whether a novel deep neural network (DNN) can predict cardiovascular risk factors from off-the-shelf wearables with a photoplethysmographic (PPG) heart rate sensor and accelerometer. Longitudinal heart rate variability and activity patterns have previously been associated with incident hypertension, diabetes, and sleep apnea, conditions which are frequently undiagnosed. Methods: Health eHeart, an IRB-approved UCSF study, enrolled 6,115 active users of the Cardiogram app for Apple Watch. Heart rate and step counts were collected for a period of 1 to 53 weeks (mean=8.9). Data from 70% of participants (33,628 person-weeks of data) was used to train a semi-supervised, multi-task DNN with both convolutional and recurrent layers to simultaneously predict prevalent hypertension, sleep apnea, and diabetes. Test performance characteristics were estimated using the remaining 30% of participants. Results: Mean age was 42.3 ± 12.1, 69% male. 2,230 (36.5%) of participants had hypertension, 1,016 (16.6%) had sleep apnea, and 462 (7.6%) had diabetes. In the validation set, the DNN outperformed a baseline logistic regression model incorporating age, sex, and beta blocker use, predicting prevalent hypertension with a c-statistic of 0.819 (95% CI 0.76-0.88; with an optimal operating point yielding 84.8% sensitivity and 63.6% specificity) vs a baseline c-statistic of 0.682 (95% CI 0.60-0.76), and prevalent sleep apnea with a c-statistic of 0.902 (95% CI 0.85-0.95; with an optimal operating point yielding 90.4% sensitivity and 59.8% specificity) vs a baseline c-statistic of 0.459 (95% CI 0.39-0.53). Results were not statistically significant for diabetes. Conclusions: Our DNN demonstrates surprisingly good prediction of hypertension and sleep apnea given that its only inputs are heart rate and step count. Whether such DNNs can provide durable and portable predictions for these conditions in other study samples is worth pursuing.