A theory of additive Markov chains with long-range memory, proposed earlier in Phys. Rev. E 68, 06117 (2003), is developed and used to describe statistical properties of long-range correlated systems. The convenient characteristics of such systems, a memory function, and its relation to the correlation properties of the systems are for the analytically solvable model of the system with a step-wise memory function.