Materials discovery lays the foundation for many technological advancements. Predicting and discovering new materials are not simple tasks. We here outline some basic principles of solid-state chemistry, which might help to advance both, and discuss pitfalls and challenges in materials discovery. Using the recent work of Szymanski et al., which reported the autonomous discovery of 43 novel materials, as an example, we discuss problems that can arise in unsupervised materials discovery, and hope that by addressing these, autonomous materials discovery can be brought closer to reality. We discuss all 43 synthetic products and point out four common shortfalls in the analysis. These errors unfortunately lead to the conclusion that no new materials have been discovered in that work. We conclude that there are two important points of improvement that require future work from the community: (i) automated Rietveld analysis of powder x-ray diffraction data is not yet reliable. Future improvement of such, and the development of a reliable artificial intelligence-based tool for Rietveld fitting, would be very helpful, not only to autonomous materials discovery, but also the community in general. (ii) We find that disorder in materials is often neglected in predictions. The predicted compounds investigated herein have all their elemental components located on distinct crystallographic positions, but in reality, elements can share crystallographic sites, resulting in higher symmetry space groups and - very often - known alloys or solid solutions. This error might be related to the difficulty of modeling disorder in a computationally economical way, and needs to be addressed both by computational and experimental material scientists. We find that two-thirds of the claimed successful materials in Szymanski et al are likely to be known, compositionally disordered versions of the predicted, ordered compounds. We highlight important issues in materials discovery, computational chemistry, and autonomous interpretation of x-ray diffraction. We discuss concepts of materials discovery from an experimentalist point of view, which we hope will be helpful for the community to further advance this important new aspect of our field.