Extant baleen whales (Cetacea, Mysticeti) are all large filter-feeding marine mammals that lack teeth as adults, instead possessing baleen, and feed on small marine animals in bulk. The early evolution of these superlative mammals, and their unique feeding method, has hitherto remained enigmatic. Here, I report a new toothed mysticete from the Late Oligocene of Australia that is more archaic than any previously described. Unlike all other mysticetes, this new whale was small, had enormous eyes and lacked derived adaptations for bulk filter-feeding. Several morphological features suggest that this mysticete was a macrophagous predator, being convergent on some Mesozoic marine reptiles and the extant leopard seal (Hydrurga leptonyx). It thus refutes the notions that all stem mysticetes were filter-feeders, and that the origins and initial radiation of mysticetes was linked to the evolution of filter-feeding. Mysticetes evidently radiated into a variety of disparate forms and feeding ecologies before the evolution of baleen or filterfeeding. The phylogenetic context of the new whale indicates that basal mysticetes were macrophagous predators that did not employ filter-feeding or echolocation, and that the evolution of characters associated with bulk filter-feeding was gradual.

Mammalodon colliveri is an unusual toothed archaic mysticete (Cetacea) from the Upper Oligocene Jan Juc Formation of south-east Australia. The morphology of the holotype skull and postcrania are described in detail. Superimposed on the generally plesiomorphic archaeocete-like morphology of Mammalodon are subtle mysticete synapomorphies. Derived features of Mammalodon include a short and bluntly rounded rostrum, reduced premaxillae, and anterodorsally directed orbits. Within Mysticeti, this suite of features is unique. The aberrant rostral morphology of Mammalodon suggests specialization for suction feeding. Janjucetus hunderi is placed in an expanded family Mammalodontidae. Phylogenetic analysis corroborates the monophyly of Basilosauridae, Neoceti, Odontoceti, and Mysticeti, and yields a novel hypothesis of toothed mysticete relationships: a basal clade of undescribed toothed mysticetes from South Carolina, a Llanocetidae + Mammalodontidae clade, and monophyletic Aetiocetidae are posited as successive sister taxa to edentulous baleen whales (Chaeomysticeti). Toothed archaic mysticetes clearly employed diverse prey capture strategies, with exaptations for filter feeding evolving sequentially in stem group Mysticeti. A stratigraphically calibrated phylogeny implies that the initial diversification of Mysticeti occurred during the Late Eocene.