SUMMARYIn order to support user data rates of Gbps and above in the fifth generation (5G) communication systems, millimeter wave (mm-wave) communication is proposed as one of the most important enabling technologies. In this paper, we consider the spectrum bands shared by 5G cellular base stations (BS) and some existing networks, such as WiGig and proposed a method for spectrally efficient coexistence of multiple interfering BSs through adaptive self-organized beam scheduling. These BSs might use multiple radio access technologies belonging to multiple operators and are deployed in the unlicensed bands, such as 60 GHz. Different from the recently emerging coexistence scenarios in the unlicensed 5 GHz band, where the proposed methods are based on omni-directional transmission, beamforming needs to be employed in mm-wave bands to combat the high path loss problem. The proposed method is concerned with this new scenario of communication in the unlicensed bands where (a) beam-forming is mandatory to combat severe path loss, (b) without optimal scheduling of beams mutual interference could be severe due to the possibility of beam-collisions, (c) unlike LTE which users time-frequency resource blocks, a new resource, i.e., the beam direction, is used as mandatory feature. We propose in this paper a novel multi-RAT coexistence mechanism where neighbouring 5G BSs, each serving their own associated users, schedule their beam configurations in a self-organized manner such that their own utility function, e.g. spectral efficiency, is maximized. The problem is formulated as a combinatorial optimization problem and it is shown via simulations that our proposed distributed algorithms yield a comparable spectral efficiency for the entire networks as that using an exhaustive search, which requires global coordination among coexisting RATs and also has a much higher algorithmic complexity. key words: millimeter-wave, 5G, licensed and unlicensed bands, coexistence, spectrum sharing