In this paper, perlite mining and rice production by-products, namely run-of-mine perlite and rice husk ash, are used as fine aggregates in combination with pumice and calcareous aggregates to produce lightweight concrete. Their use is evaluated mainly in terms of the durability of the concrete, by comparing four optimized lightweight concrete mixtures of similar density and strength with a reference one of normal weight. The sorptivity due to capillary sorption, open porosity, chloride migration, penetration resistance, and freeze and thaw response were studied to evaluate the durability of the lightweight concrete. According to the experimental results, the examined mixtures developed an adequate strength in order to be classified into strength classes greater than LC25/28 and, therefore, be used in structural applications. The durability of the mixtures was also sufficient, especially as far as the chlorides’ penetration resistance is concerned, which was found to be up to 39% lower compared to the reference mixture. The sorptivity and open porosity of the LWC mixtures increased due to the porous nature of the lightweight aggregates, and the mixtures were also found to be susceptible to freeze and thaw cycles. Exceptionally, the lightweight concrete mixtures comprising pumice and perlite exhibited a lower sorptivity and resistance to chloride penetration than the standard concrete and a promising tolerance to freezing and thawing. Thus, the optimized combination of pumice and perlite is a sustainable recommendation for structural lightweight concrete production and use, promoting the wider exploitation of natural aggregates with an acceptable compromise on strength and durability.