Entanglement distillation has many applications in quantum information processing and is an important tool for improving the quality and efficiency of quantum communication, cryptography, computing, and simulation. We propose an entanglement distillation scheme using only one pair of polarizationfrequency hyperentangled photons, which can be equivalently viewed as containing two pairs of entangled logical qubits: a pair of polarization-entangled qubits and a pair of frequencyentangled qubits. To perform the required CNOT operation between the two qubits we consider the use of a polarizationdependent frequency converter. Compared to past methods of entanglement distillation that relied on polarization and spatialmode/energy-time degree of freedom, the utilization of frequencyencoded qubits offers an advantage in that it is immune to bit-flip errors when the channel is linear. After distillation, the fidelity of polarization entanglement can be significantly improved by sacrificing the frequency degree of freedom. Through simulation, we show that high fidelity gains, large yield, and high distillation rate can be achieved. Our distillation scheme is simple to implement with current technologies, compatible with existing telecommunication fiber networks, and is a promising approach for achieving efficient quantum communication.