The primary computational bottle-neck in implicit structural dynamics is the repeated inversion of the underlying stiffness matrix. In this paper, a fast inversion technique is proposed by merging four distinct but complementary concepts: (1) voxelization with adaptive local refinement, (2) assembly-free (a.k.a. matrix-free or element-by-element) finite element analysis (FEA), (3) assembly-free deflated conjugate gradient (AF-DCG), and (4) multicore parallelization. In particular, we apply these concepts to the well-known Newmark-beta method, and the resulting AF-DCG is well-suited for large-scale problems. It can be easily ported to many-core central processing unit (CPU) and multicore graphics-programmable unit (GPU) architectures, as demonstrated through numerical experiments.