It is extremely important to have clear understanding of resonant beam instability for a better design of a modern particle accelerator. For this purpose, we have developed "Simulator of Particle Orbit Dynamics" (S-POD) that enables us to clarify various beam-dynamics issues without relying on large-scale machines. This unique tabletop experiment is based on an isomorphism between non-neutral plasmas in a compact Paul trap and charged-particle beams in a linear focusing channel. S-POD is particularly useful in exploring collective effects in intense hadron beams. This thesis addresses systematic Particle-In-Cell (PIC) simulations performed to explain experimental data from S-POD. A possible design of a novel multipole ion trap is also proposed for a future experiment study of nonlinear beam dynamics. The contents of the present work include the following three main subjects.(1) Collective resonance instabilities and its lattice-structure dependence [Chapter 4]. Almost all modern particle accelerator systems exploit the principle of strong focusing. Each accelerator has a unique lattice structure optimized for a certain experimental purpose. We here focus on several standard alternating-gradient (AG) lattices such as doublet, triplet, FDDF, etc. These AG focusing potentials can readily be reproduced in S-POD. We employ the PIC code Warp to support S-POD experiments. A number of systematic multi-particle simulations are carried out to explain the experimentally observed collective instabilities induced by the external AG driving forces. The excitation of extra resonance bands due to lattice symmetry breaking is also studied in detail. We confirm that PIC simulation results are consistent to experimental observations as well as theoretical predictions from the linearized Vlasov analysis.(2) Theoretical and simulation study of resonance crossing [Chapter 5]. Considerable theoretical and experimental efforts have recently been devoted to design studies of nonscaling fixed-field alternating gradient (ns-FFAG) accelerators are for various purposes including hadron therapy, accelerator-driven reactor systems, a muon collider, and a neutrino factory. In this type of machines, the bare betatron tunes keep decreasing rapidly while the beam is accelerated by radio-frequency cavities. It is almost inevitable for the operating point to cross resonance stop bands, some of which may be quite dangerous.In this chapter, we first investigate fundamental features of collective resonance crossing with the Warp code and compare simulation results with experimental observations in S-POD. A simple scaling law is derived for a quick estimate of the emittance growth caused by crossing of an in-