We report from the Do Not Disturb Challenge where 30 volunteers disabled notification alerts for 24 hours across all devices. The effect of the absence of notifications on the participants was isolated through an experimental study design: we compared self-reported feedback from the day without notifications against a baseline day. The evidence indicates that notifications have locked us in a dilemma: without notifications, participants felt less distracted and more productive. But, they also felt no longer able to be as responsive as expected, which made some participants anxious. And, they felt less connected with one's social group. In contrast to previous reports, about two third of the participants expressed the intention to change how they manage notifications. Two years later, half of the participants are still following through with their plans.