(1) Background: The ability to communicate with patients and their relatives is a crucial skill for a physician. Unfortunately, many physicians and medical students are not well-equipped in this area. Therefore, this study aims to better understand the views on critical components of physician-patient communication to improve their skills. (2) Methods: This qualitative study utilized focus group discussions (FGDs) and in-depth interviews (IDIs). Through a purposive sampling technique, 32 medical students and physicians from the Faculty of Medicine and Health Sciences, Universiti Putra Malaysia (FMHS UPM) and Universiti Putra Malaysia Teaching Hospital (HPUPM), as well as patients and relatives from government and private hospitals or clinics were recruited. All sessions were audio-recorded, transcribed, and analyzed thematically. (3) Results: Seven themes were identified: professionalism, content of communication, verbal, non-verbal and paraverbal communication skills, environment, and visual communication. Good eye contact, providing treatment plans, and ensuring patient privacy and confidentiality were emphasized by physicians and medical students. In comparison, patients and relatives focused on the prognosis of disease, physician’s empathy and advice, and physician’s skills in building rapport with their patients and relatives. (4) Conclusion: The critical components that were highlighted by both professionals and laymen in the study should be practiced to ensure effective communication between physician and patient. There were different expectations in terms of the content of information between both groups. Patients and relatives were more interested in the physician’s advice regarding their diet, care plans, physical activities, and daily routine. They were also focused on the prognosis of the disease, which indicates how quickly they would get better. Meanwhile, physicians and medical students were concentrating on management and treatment strategies, such as what additional procedures should be considered and what medications might work best for their patients. We also found that the patients and relatives had a lack of awareness on confidentiality issues. These findings provide an insight on the improvement of medical training and patient education to improve patient care. Patients have a right to privacy protection, and physicians should be well trained to carry out all procedures and treatment plans to ensure patients are treated with respect.