The effects of membranes on the early-stage aggregation of amyloid β (Aβ) have come to light as potential mechanisms by which neurotoxic species are formed in Alzheimer’s disease. We have shown that direct Aβ-membrane interactions dramatically enhance the Aβ aggregation, allowing for oligomer assembly at physiologically low concentrations of the monomer. Membrane composition is also a crucial factor in this process. Our results showed that apart from phospholipids composition, cholesterol in membranes significantly enhances the aggregation kinetics. It has been reported that free cholesterol is present in plaques. Here we report that free cholesterol, along with its presence inside the membrane, further accelerate the aggregation process by producing aggregates more rapidly and of significantly larger sizes. These aggregates, which are formed on the lipid bilayer, are able to dissociate from the surface and accumulate in the bulk solution; the presence of free cholesterol accelerates this dissociation as well. All-atom molecular dynamics simulations show that cholesterol binds Aβ monomers and significantly changes the conformational sampling of Aβ monomer; more than doubling the fraction of low-energy conformations compared to those in the absence of cholesterol, which can contribute to the aggregation process. The results indicate that Aβ-lipid interaction is an important factor in the disease prone amyloid assembly process.