Physical layer identification is an emerging technique that exploits physical layer features to identify wireless devices. The identification accuracy and the device quantity that can be identified at most are significant for the identification scheme. Existing works primarily focus on the feature correlation analysis for multifeature selection without investigating the least upper bound (supremum) of the performance of a single feature. The supremum indicates the limit of the performance, which is another sight for evaluating the quality of features and improving the performance of the identification scheme. Therefore, this paper first investigates the supremum of the performance of the most commonly used physical layer feature, i.e., carrier frequency offset (CFO). Specifically, we offer a rigorous mathematical analysis and derive the closed-form expression of the supremum of identification accuracy based on the max-min distance analysis (MMDA) criterion. And then, the supremum of the number of distinguishable devices is also analyzed. Finally, we conducted a simulation study to verify the theoretical analysis result.