We study the phase structure of dense hadronic matter including ∆(1232) as well as N (939) based on the parity partner structure, where the baryons have their chiral partners with a certain amount of chiral invariant masses. We show that, in symmetric matter, ∆ enters into matter in the density region of about one to four times of normal nuclear matter density, ρB ∼ 1 -4ρ0. The onset density of ∆ matter depends on the chiral invariant mass of ∆, m∆0: The lager m∆0, the bigger the onset density. The ∆ matter of ρB ∼ 1 -4ρ0 is unstable due to the existence of ∆, and the stable ∆-nucleon matter is realized at about ρB ∼ 4 ρ0, i.e., the phase transition from nuclear matter to ∆-nucleon matter is of first order for small m∆0, and it is of second order for large m∆0. We find that, associated with the phase transition, the chiral condensate changes very rapidly, i.e., the chiral symmetry restoration is accelerated by ∆ matter. As a result of the accelerations, there appear N * (1535) and ∆(1700), which are the chiral partners to N (939) and ∆(1232), in high density matter, signaling the partial chiral symmetry restoration. Furthermore, we find that complete chiral symmetry restoration itself is delayed by ∆ matter. We also calculate the effective masses, pressure and symmetry energy to study how the transition to ∆ matter affects such physical quantities. We observe that the physical quantities change drastically at the transition density.