This study focuses on understanding the health impacts of hazardous air pollutant (HAP) emissions from the Kaohsiung Coastal Industrial Park and port areas in southern Taiwan on neighboring communities. Six important HAPs (formaldehyde, benzene, arsenic, vinyl chloride, 1,3-butadiene, and diesel particulate matter (DPM)) were identified in this area. By considering the impact of emissions from stationary sources, mobile sources, and port activities, the relative importance of each emission source was assessed. In addition, the AERMOD (AMS (American Meteorological Society)/EPA (U.S. Environmental Protection Agency)) diffusion model was employed to simulate the increases in target pollutant concentrations and to analyze the influence and spatial distribution of various emission sources on atmospheric HAP concentrations in nearby communities. This study further evaluated the exposure risks of composite HAP sources, to understand their impacts and to determine their control priorities. The findings revealed that emissions and carcinogenic weighting from composite sources, particularly DPM emissions from port activities, including from ocean-going vessels and heavy-duty vehicles, had a significant impact. The maximum incremental concentration for DPM in the study area occurred around the port area, whereas the maxima for formaldehyde, benzene, arsenic, vinyl chloride, and 1,3-butadiene were all observed within the industrial complex. DPM emissions from port activities, 1,3-butadiene emissions from mobile sources, and benzene emissions from stationary sources were the composite sources with the greatest potential impacts. Over 90% of health risks were due to DPM, and the remaining health risks were due to 1,3-butadiene (6%), benzene (2%), arsenic (1%), and other species (less than 1%). DPM emissions were primarily influenced by port activities (77%), 1,3-butadiene emissions by mobile sources (45%), and benzene emissions by stationary sources (41%). A total of 25% of the area had risk values greater than 10−3, and 75% of the area had risk values between 10−3 and 10−4. The risk values in the densely populated areas were all greater than 10−4. The potential risk hotspots with risk values greater than 10−3 were located on the northwest side of the port and downwind of the industrial park. The key pollutants contributing to these hotspots were, in order, DPM (up to 80% cancer risk), formaldehyde, and 1,3-butadiene, all of which were significantly influenced by port activities. This indicates that the control of, and reduction in, HAP emissions from port activities should be prioritized.