The characteristics of the wireless physical layer can be exploited to complement and enhance traditional security. In this thesis, we study the channel-based physical layer authentication. The authentication problem is formulated as a sequence of hypothesis test problems. By exploiting the time-of-arrivals, received signal strengths, and cyclic-features of the channels, support vector machine (SVM) based authentication schemes, the linear Fisher discriminant analysis (LFDA) based authentication scheme, and the combining scheme are proposed to improve the detection probability and to reduce the false alarm probability. These schemes can reliably authenticate the sender by identifying channels from different users.In SVM based schemes, the linear and nonlinear SVMs are used to generate classifiers to solve the hypothesis test problems. Using the real channel data measured in a regular office from Utah University, simulation is performed. Simulation results demonstrate that SVM based schemes have lower misdetection probability and false alarm probability than some existing schemes at a cost of extra time complexity and space complexity due to the training stage.To reduce the space complexity and time complexity during the training stage, LFDA based authentication scheme is proposed. In LFDA based scheme, a linear combination of the channel features is used as the test statistic, which is compared with a threshold to perform authentication. LFDA is used to compute the weights based on some training data. Furthermore, an adaptive threshold scheme (ATS) is proposed to set and adjust the threshold. Simulation results demonstrate that the proposed LFDA based scheme performs better in terms of the sum of misdetection probability and false alarm probability, and the receiver operating characteristic curves, compared with several existing channel-based authentication schemes. Moreover, the analysis of time complexity and space complexity is provided, which shows that the LFDA based scheme is also better than SVM based schemes in terms of space complexity, time complexity, misdetection probability, and false alarm probability.The misdetection probability and false alarm probability can be reduced greatly by two-user cooperative authentication. The combining scheme is proposed to combine the data from another legitimate user when cooperation is available. The combining scheme is proven to have the capacity to improve the performance at cost of extra communication and computation overhead. The time complexity, space complexity, and communication overhead are analyzed.iii Acknowledgements I would like to thank all the people who made this thesis possible. They are my supervisor and my colleagues in the Broadband Communications Research (BBCR) group, and my families. Without their help and encouragement, I cannot finish my study.