Many interplanetary mission concepts can benefit from autonomous orbit estimation, particularly during critical mission phases. Previous studies have examined the feasibility of optical navigation using nanosatellite class instruments. While promising, these techniques are not without drawbacks. Convergence of the navigation estimates are often sensitive to errors in initial state estimates. This thesis compares various methods to perform nonlinear estimation for autonomous optical navigation. These methods include an extended Kalman filter (EKF), an unscented Kalman filter (UKF), a particle filter (PF), a fixed-lag smoother (FLS), and moving horizon estimation (MHE). The EKF, UKF, and PF can be implemented in real time, while the FLS and MHE implement a delay into the estimation process. To compare the performance of each state estimator three initial reference scenarios around Mars were considered: a hyperbolic flyby, an elliptic orbit and a orbital maneuver using observations of Mars and its moons. Parameter estimation was also explored, where the mass of Mars was to be estimated as a reference parameter in both the hyperbolic and elliptical trajectories. One last reference scenario included a low Earth orbit (LEO) using observations of satellites in a geosynchronous equatorial orbit. In each case, the FLS and MHE showed similar or better performance over each state estimator but at the cost of an increased computation time with respect to the reference EKF. Similarly the UKF was able to provide improved results withe respect to the EKF. While, the PF provided poor estimates in the Mars trajectories but improvements were seen from the UKF and EKF in the LEO scenario.