In this paper, we propose a distributed intelligent reflecting surface (IRS) assisted single-user and multi-user millimeter wave (mmWave) system. Then, we formulate the resource allocation problem as an optimization to maximize energy efficiency under individual quality of service (QoS) constraints. We first propose a centralized algorithm, and further a low-complexity distributed one where the access point (AP) and IRSs independently adjust the transmit beamforming of AP, the phase shifts, and the on-off status of IRSs in an alternating manner until the convergence is reached. In a multi-user scenario, in the first stage, the successive convex approximation (SCA) and fractional programming (FP) approaches are applied to achieve a solution for optimization subproblems of the phase-shift coefficients and element on-off status of IRSs. Then, for the beamforming subproblem, a modified nested FP approach is proposed that finds an optimal solution for the beamforming vectors of AP. Our performance analysis on a practical scenario shows that the proposed centralized and distributed approach respectively enhances the energy efficiency up to 55%, 42% for single-user and up to 984% for multi-user scenarios, in comparison to the case where the on-off status and phase-shift coefficients of IRS elements are not selected optimally.