Background: Long work hours for physicians not only harm the health of physicians, but also endanger patient safety. Compared with resident physicians, attending physicians—especially hospital-employed attending physicians—assume more responsibilities but has not gotten enough attention. The purpose of this study was to explore whether a hospital’s geographic location and emergency care responsibility might influence the number of hours worked. Methods: The respondents of 2,365 attending physicians from 152 hospitals in the 2018 survey of Taiwan physician work hours were used as the data source. The total work hour per week and its components, the regular scheduled shift and three types of on-call shifts, were used as outcome variables. Hospital geographic location and emergency care responsibility were the independent variables. The multilevel random effect model was employed to examine the study objective after adjusting for clinical specialty, hospital teaching status, and ownership. Results: The average number of total working hours was 69.09 hours per week; the regular scheduled shift was account for 75% of total work hours. The results showed the total work hours were only varied by the level of hospital’s emergency care responsibility. However, the results also demonstrated the hours of duty shifts were varied by hospital’s geographic location and emergency care responsibility. The results of the multilevel random effect model revealed that the hospital’s emergency care responsibility was the factor consistently associated with attending physician’s work hour, no matter the total work hours or its composition. Conclusion: In this study, we explored how a hospital’s location and its level of emergency care responsibility were associated with physicians’ work hours for each type of shift. Our findings offer an opportunity to review the rationality of physician workforce allocation, and financial incentives and administrative measures could be the next steps for balancing the work hours of attending physicians.