We consider two classical stochastic inventory control models, the periodic-review stochastic inventory control problem and the stochastic lot-sizing problem. The goal is to coordinate a sequence of orders of a single commodity, aiming to supply stochastic demands over a discrete, finite horizon with minimum expected overall ordering, holding and backlogging costs. In this paper, we address the important problem of finding computationally efficient and provably good inventory control policies for these models in the presence of correlated and non-stationary (time-dependent) stochastic demands. This problem arises in many domains and has many practical applications in supply chain management.Our approach is based on a new marginal cost accounting scheme for stochastic inventory control models combined with novel cost-balancing techniques. Specifically, in each period, we balance the expected cost of over ordering (i.e, costs incurred by excess inventory) against the expected cost of under ordering (i.e., costs incurred by not satisfying demand on time). This leads to what we believe to be the first computationally efficient policies with constant worst-case performance guarantees for a general class of important stochastic inventory models. That is, there exists a constant C such that, for any instance of the problem, the expected cost of the policy is at most C times the expected cost of an optimal policy. In particular, we provide worst-case guarantee of 2 for the periodic-review stochastic inventory control problem and a worst-case guarantee of 3 for the stochastic lot-sizing problem. Our results are valid for all of the currently known approaches in the literature to model correlation and non-stationarity of demands over time.