In communities where people lack on-demand, safely managed drinking water, stored household water often becomes contaminated by fecal bacteria, regardless of the source-water quality. The objectives of this paper are to assess and control bacterial contamination in stored household water in Toamasina, a rapidly urbanizing city in eastern coastal Madagascar. We collected samples of source water and stored household water from 10 representative households that use different water sources and different storage strategies, and we analyzed the samples for several fecal indicator bacteria. We also tested three methods that residents of Toamasina could realistically employ for cleaning their household water storage vessels, assessing the effect of the cleaning methods on measured bacterial levels in the water. Consistent with the previous literature, we found that concentrations of total coliforms in stored household water were significantly higher than in the corresponding source water (p < 0.05). In 100% of households that stored their water in 20 L polyethylene jerrycans (n = 4), biofilms on the walls of the jerrycan harbored total coliforms and Enterococcus. The use of a closed storage container was, on its own, not found to provide a meaningful protective effect against bacterial regrowth; to be protective, closed storage containers must be combined with high-quality source water and/or with adequate cleaning to prevent biofilm formation. A dilute solution of sodium hypochlorite, known locally as Sûr’Eau or Manadio Rano, was both the most effective and the least expensive method for cleaning household water storage containers. We conclude that regular and effective cleaning of storage containers is an essential component of safe water storage. Because household storage of collected water is common in many low- and middle-income countries, these results are important towards the worldwide achievement of the United Nations’ Sustainable Development Goal 6.