Quantum secure direct communication (QSDC) can guarantee security using the characteristics of quantum mechanics even when a message is directly transmitted through a quantum channel without using a secret key. However, the transmission rate of the QSDC is limited by the dead time of a single photon detector (SPD) as well as channel loss over the distance. To overcome this limited transmission rate, we propose a high-dimensional single photon-based QSDC protocol that applies two optical degrees of freedom: time and phase state. First, an N-dimensional time and phase state generation method that considers the dead time is proposed to minimize the measurement loss of a transmitted message. Second, among the two types of quantum states, the phase state with relatively low measurement efficiency is used only for eavesdropping detection,and the time state is used for sending messages with differential delay time bin-based encoding techniques. Lastly, we propose an efficient method for measuring N-dimensional time and phase-based quantum states and recovering classical bit information.This study performs security analysis against various attacks, and verifies the transmission rate improvement effect through simulation. The result indicates that our proposal can guarantee higher security and transmission rates compared to the conventional DL04 QSDC.