Atomic force microscopy (AFM) was invented by G. Binnig and his collaborators in 1986 after the invention of scanning tunnelling microscopy (STM) in the early 1980s. The AFM system deploys a tiny probe to interact with the sample such that topography information of the sample can be obtained. The resolutions of the AFM topography images are in the nanometer scale and they are limited only by the probe diameters. Besides, in AFM imaging, there is no requirement on sample preparation and conductivity. Thus, the AFM provides an easy platform to observe, measure and manipulate atomic interactions. Since its invention, AFM has become an indispensable tool in the research areas of material sciences and biological sciences. The AFM is also widely used in many industry applications in microelectronics and data storage. Although many research areas and industry applications have benefited from the success of AFM, there is still an urgent need to improve the measurement resolution and accuracy of the AFM in order to further exploit the atomic characteristics at the scales of atoms or even smaller. In addition, AFM has been modified as a Casimir oscillator to study quantum fluctuations where highly accurate and precise measurement is also desired. Driven by this motivation, this thesis will focus on the improvement of accuracy ix and resolution of the AFM and its modified system, the Casimir oscillator. Our approach to accuracy and resolution enhancement of the AFM system is by tackling the measurement errors that arise from its components such as the position sensitive detector (PSD) and the sensing principle in the AFM sensing system. In our work, we have proposed a new concept of linearity indices to quantitatively measure the measurement inaccuracies of various PSDs. Based on these linearity indices, new design formulae are proposed for 2-D tetra-lateral PSD and quadrant detector. Their measurement accuracy are significantly improved via our new formulae as well as robustness to the incident beam size and parameter variation. Besides the AFM sensing system, inaccuracies also come from the probe-sample separation estimation. To address this issue, a novel estimation algorithm has been proposed and accurate estimates can be obtained regardless of different nonlinear dynamics. Several methods have also been derived to improve the separation gap estimation in Casimir oscillator which is a modified AFM system. x