Various physical and psychosocial difficulties including anxiety affect cancer patients. Patient surroundings also have psychological effects on caregiving. Assessing the current status of palliative care intervention, specifically examining anxiety and its associated factors, is important to improve palliative care unit (PCU) patient quality of life (QOL). This study retrospectively assessed 199 patients admitted to a PCU during August 2018–June 2019. Data for symptom control, anxiety level, disease insight, and communication level obtained using Support Team Assessment Schedule Japanese version (STAS-J) were evaluated on admission and after 2 weeks. Palliative Prognostic Index (PPI) and laboratory data were collected at admission. Patient anxiety was significantly severer and more frequent in groups with severer functional impairment (p = 0.003) and those requiring symptom control (p = 0.006). Nevertheless, no relation was found between dyspnea and anxiety (p = 0.135). Patients with edema more frequently experienced anxiety (p = 0.068). Patient survival was significantly shorter when family anxiety was higher after 2 weeks (p = 0.021). Symptoms, edema, and disabilities in daily living correlate with patient anxiety. Dyspnea is associated with anxiety, but its emergence might be attributable mainly to physical factors in this population. Family members might sensitize changes reflecting worsened general conditions earlier than the patients.