COVID-19 has created a general state of worry and distress, especially among vulnerable groups such as those with psychiatric diagnoses. Worldwide, psychiatric care provision has drastically suffered during the pandemic, with many patients unable to access proper care, which may have implications for increased mental health consequences in patients with psychiatric disorders (e.g., relapse and suicide). This cross-sectional study used structural equation modeling to investigate COVID-19-related trauma and distress among Arab psychiatric population during COVID-19 quarantine. Patients with pre-existing psychiatric disorders (N = 168) completed an online survey that comprised the Depression Anxiety Stress Scale 21 (DASS-21), the Impact of Event Scale-Revised (IES-R), and a questionnaire on COVID-19-related attitudes/perceptions, sources of information, used protective measures, and socio-demographic information. Respondents commonly reported feeling down-hearted/blue, trouble concentrating, along with symptoms of avoidance and rumination related to the pandemic. Patients with depression and sleep disorders expressed higher COVID-19-related trauma than patients with other disorders. Perceived physical health mediated the effect of co-morbid chronic physical disorders on COVID-19 trauma, psychological distress, perceived vulnerability to COVID-19, and perceived likelihood of recovery in case of contracting COVID-19. Perceived physical health and perceived vulnerability to COVID-19 were strong direct predictors of COVID-19-related trauma and psychological distress. Staying at home negatively predicted COVID-19 trauma and exerted an indirect negative effect on psychological distress via COVID-19 trauma. COVID-19 trauma, age, and marital status directly predicted psychological distress, with COVID-19 trauma being the strongest predictor. Educational level, income, having family members working in the medical field, keeping up to date with the news on deaths/infected cases or the development of COVID-19 drugs or vaccines, satisfaction with available information on COVID-19, and using different protective measures were not associated with significant differences in COVID-19 trauma and psychological distress scores. Immuno-psychiatric interventions should be designed to target COVID-19-trauma and distress among younger single patients with perceived poor physical health, especially those diagnosed with depression and sleep disorders.