Radio frequency identification (RFID) technology attracts extensive attention for industrial applications, especially supply chain businesses and inventory management. In this paper, a passive RFID localization scheme based on unmanned aerial vehicle (UAV) or drones is established for inventory management in warehouses. In case of the major challenge of the on-board antenna tracking errors in practical scenarios, the state-of-the-art numerical methods rely on accurate antenna positions, such as synthetic aperture radar-based algorithms, which will introduce large positioning errors and become unreliable. To this end, we propose a new relative RFID localization method based on phase, received signal strength indicator (RSSI), and readability, which is little affected by antenna tracking errors. In the proposed method, the tagged items on the racks are located laterally through pinpointing the minimum of the unwrapped phases, which have been unwrapped based on the hybrid random forest model. RSSI differences and the read count of each patch antenna's beam are utilized to distinguish the rack level of tagged assets via a boosting tree classifier. The experimental results in a real warehouse show that the proposed method achieves 27.1-cm mean lateral positioning accuracy, and orders the tagged items horizontally and vertically with more than 96.7% and 98.0% accuracy, respectively.