We present an algorithm to parallelize the inverse fast multipole method (IFMM), which is an approximate direct solver for dense linear systems. The parallel scheme is based on a greedy coloring algorithm, where two nodes in the hierarchy with the same color are separated by at least σ nodes. We proved that when σ ≥ 6, the workload associated with one color is embarrassingly parallel. However, the number of nodes in a group (color) may be small when σ = 6. Therefore, we also explored σ = 3, where a small fraction of the algorithm needs to be serialized, and the overall parallel efficiency was improved. We implemented the parallel IFMM using OpenMP for shared-memory machines. Successively, we applied it to a fast-multipole accelerated boundary element method (FMBEM) as a preconditioner, and compared its efficiency with (a) the original IFMM parallelized by linking a multi-threaded linear algebra library and (b) the commonly used parallel block-diagonal preconditioner. Our results showed that our parallel IFMM achieved at most 4× and 11× speedups over the reference method (a) and (b), respectively, in realistic examples involving more than one million variables.