Predicting crystal structure from the chemical composition is one of the most challenging and long-standing problems in condensed matter physics. This problem resides at the interface between materials sciences and physics. With reliable data and proper physics-guided modeling, machine learning (ML) can provide an alternative venue to undertake and reduce the problem's complexity. In this work, very robust ML classifiers for crystallographic symmetry groups were developed and applied for ternary (A l B m C n ) and binary (A l B m ) materials starting only from the chemical formula. This is the first essential step toward predicting full geometry. Such a problem is highly multi-label and multi-class from an ML perspective and requires careful preprocessing due to the size imbalance of the data. The resulting predictive models are highly accurate for all symmetry groups, including crystal systems, point groups, Bravais lattices, and space groups, with weighted balanced accuracies exceeding 95%. The models were developed with only a small set of ionic and compositional features, namely, stoichiometry, ionic radii, ionization energies, and oxidation states for each element in the ternary and binary compounds. Considering such minimal feature space, the obtained high accuracies ascertain that the physics is well captured. This is even further confirmed as we demonstrate that the accuracy of our approach is limited only by the size of data by comparing the size of ternary and binary materials with the accuracy of developed models. The presented work could effectively contribute to accelerating new materials discovery and development.