Unmanned aerial vehicle (UAV) swarm connected to millimeter wave (mmWave) cellular networks is emerging as a new promising solution to provide ubiquitous high-speed and long distance wireless communication services for supporting various applications. To satisfy different quality of service (QoS) requirements in future large-scale applications of such networks, this paper investigates the rate performance, fairness and their tradeoff in the networks with directional antennas in terms of sum-rate maximization, fairness index maximization, max-min fair rate and proportional fairness. We first consider a more realistic mmWave 3D directional antenna array model for UAVs and base station (BS), where the antenna gain depends on the radiation angle of the antenna array. Based on this antenna array model, we formulate the performance, fairness and their tradeoff as four constrained optimization problems, and propose corresponding iterative algorithm to solve these problems by jointly optimizing elevation angle, azimuth angle and height of antenna array at BS in the downlink transmission scenario. Furthermore, we also explore them in uplink transmission scenario, where the interference issue among links is carefully considered. Finally, according to the sum rate, minimum rate and fairness index under each optimization problem, numerical results are provided to illustrate the impacts of network parameters on the performance, fairness and their tradeoff, and also to reveal new findings under both downlink and uplink transmission scenarios, respectively.