Standard maximum likelihood or Bayesian approaches to parameter estimation of stochastic differential equations are not robust to perturbations in the continuous-in-time data. In this note, we give a rather elementary explanation of this observation in the context of continuous-time parameter estimation using an ensemble Kalman filter. We employ the frequentist perspective to shed new light on two robust estimation techniques; namely subsampling the data and rough path corrections. We illustrate our findings through a simple numerical experiment.