In this paper, firstly, we study the linear processing of amplify-and-forward (AF) relays for the multiple relays multiple users scenario. We regard all relays as one special "relay", and then the subcarrier pairing, relay selection and channel assignment can be seen as a linear processing of the special "relay". Under fixed power allocation, the linear processing of AF relays can be regarded as a permutation matrix. Employing the partitioned matrix, we propose an optimal linear processing design for AF relays to find the optimal permutation matrix based on the sorting of the received SNR over the subcarriers from BS to relays and from relays to users, respectively. Then, we prove the optimality of the proposed linear processing scheme. Through the proposed linear processing scheme, we can obtain the optimal subcarrier paring, relay selection and channel assignment under given power allocation in polynomial time. Finally, we propose an iterative algorithm based on the proposed linear processing scheme and Lagrange dual domain method to jointly optimize the joint optimization problem involving the subcarrier paring, relay selection, channel assignment and power allocation. Simulation results illustrate that the proposed algorithm can achieve a perfect performance.