For efficiently solving large dense complex linear systems that arise in electric field integral equations (EFIE) of electromagnetic scattering problems, the fast multipole method (FMM) is used to accelerate the matrix‐vector product operations, and the sparse approximate inverse (SAI) preconditioning technique is employed to speed up the convergence rate of the Krylov iterations. A good quality SAI preconditioner in the FMM context is constructed based on the near‐field matrix of the EFIE. The main purpose of this paper is to show that the quality of the SAI preconditioner can be greatly improved by use of information from FMM implementation. Numerical experiments indicate that the resulted SAI preconditioner is very effective with FMM and can reduce the overall simulation time significantly. © 2007 Wiley Periodicals, Inc. Microwave Opt Technol Lett 49: 1746–1750, 2007; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/mop.22538