Adrenal crisis is the most severe manifestation of adrenal insufficiency (AI), but AI can present with variable signs and symptoms of gradual severity. Despite current hormone replacement strategies, adrenal crisis is still one of the leading causes of mortality in AI patients. Although underlying factors explaining differences in interindividual susceptibility are not completely understood, several subgroups are particularly vulnerable to adrenal crises, such as patients with primary AI, and patients treated for Cushing’s syndrome. Currently, the health care professional faces several challenges in the care for AI patients, including the lack of reliable biomarkers measuring tissue cortisol concentrations, absence of a universally used definition for adrenal crisis, and lack of clinical tools to identify individual patients at increased risk. Also from the patient’s perspective, there are a number of steps to be taken in order to increase and evaluate self-management skills and, finally, improve health-related quality of life (HR-QoL). In this respect, the fact that inadequate handling of AI patients during stressful situations is a direct consequence of not remembering how to act due to severe weakness and cognitive dysfunction in the context of the adrenal crisis is quite underexposed. In this narrative review, we give an overview of different clinical aspects of adrenal crisis, and discuss challenges and unmet needs in the management of AI and the adrenal crisis from both the doctor’s and patient’s perspective. For the latter, we use original focus group data. Integration of doctor’s and patient’s perspectives is key for successful improvement of HR-QoL in patients with AI.