To infer the processes that gave rise to past speciation and extinction rates across taxa, space and time, we often formulate hypotheses in the form of stochastic diversification models and estimate their parameters from extant phylogenies using Maximum Likelihood or Bayesian inference. Unfortunately, however, likelihoods can easily become intractable, limiting our ability to consider more complicated diversification processes. Recently, it has been proposed that deep learning (DL) could be used in this case as a likelihood-free inference technique. Here, we explore this idea in more detail, with a particular focus on understanding the ideal network architecture and data representation for using DL in phylogenetic inference. We evaluate the performance of different neural network architectures (DNN, CNN, RNN, GNN) and phylogeny representations (summary statistics, Lineage Through Time or LTT, phylogeny encoding and phylogeny graph) for inferring rates of the Constant Rate Birth-Death (CRBD) and the Binary State Speciation and Extinction (BISSE) models. We find that deep learning methods can reach similar or even higher accuracy than Maximum Likelihood Estimation, provided that network architectures and phylogeny representations are appropriately tuned to the respective model. For example, for the CRBD model we find that CNNs and RNNs fed with LTTs outperform other combinations of network architecture and phylogeny representation, presumably because the LTT is a sufficient and therefore less redundant statistic for homogenous BD models. For the more complex BiSSE model, however, it was necessary to feed the network with both topology and tip states information to reach acceptable performance. Overall, our results suggest that deep learning provides a promising alternative for phylogenetic inference, but that data representation and architecture have strong effects on the inferential performance.