Aim Complementary and alternative medicine is increasingly being used by primary care physicians in Japan, although this usage has not been studied in detail, and the barriers in the way of its use by physicians have not been explored. The aim of this study was therefore to clarify the factors that interfere with the use of complementary and alternative medicine, from the perspective of the primary care physicians. Methods We conducted interviews with primary care physicians who use complementary and alternative medicine in their practice. Transcripts of interviews were analyzed qualitatively using the Steps for Coding and Theorization method. Results Four categories and 15 concepts were extracted. The categories generated were ‘Awareness of health‐care providers’, ‘Characteristics of complementary and alternative medicine’, ‘Medical system and organization’ and ‘Regional factors’. Conclusion Various factors interfere in the use of complementary and alternative medicine by primary care physicians. Approaches to overcome lack of education and insufficient collaboration among regional health professionals need to be developed. Future studies are needed to generalize the present findings in order to enhance integrative medicine and the utilization of complementary and alternative medicine in primary care settings.