Anxiety, a condition which is commonly found in patients with cancer, has negative impacts on their quality of life and treatment outcome. This study aimed to determine the level of anxiety in patients with cancer and explore sociodemographic, disease-related, and hospital-related factors associated with anxiety in those patients. A cross-sectional study was conducted on 510 inpatients with cancer at Thanh Hoa Oncology Hospital, Vietnam. Data were collected from self-administered questionnaire forms on hospital depression anxiety-A, interviews with patients, and patient medical records. The univariate and multivariate linear regression analyses were performed using STATA ver. 14.0. Our finding that the patients’ mean anxiety score (standard deviation) was 7.22 (3.8); 27.6% of the patients had an anxiety score between 8 and 10 points, and 15.5% had an anxiety score of ≥11 points. In the multivariate model, in more advanced stages of cancer, and patients with metastasis were more likely to have higher levels of anxiety than those who presented no sign of metastasis. The longer the patients had cancer, the less anxious they became. Lower levels of anxiety were observed in patients who stated that hospital facilities were adequate or had trust in health workers. Patients with cancer need to be provided with psychological support in the early stage of cancer detection and when metastases form. A strong patient–health-care provider relationship after diagnosis may help reduce distress among patients with cancer with higher levels of medical mistrust.