Incoherent digital holography is a three-dimensional (3D) imaging technique that uses a natural light source. Through computer-based diffraction calculations, 3D information can be reconstructed from self-interferometer-captured holograms. As practical applications require a computer capable of high-speed diffraction calculations, an investigation was conducted into a special-purpose computer that uses a fieldprogrammable gate array (FPGA). This approach had not been used for incoherent digital holography. In this study, the computational performance of a proposed monochromatic diffraction circuit was improved. By using direct memory access transfer, the data transfer time bottleneck, which is a problem in conventional circuits, was eliminated. The calculation was also parallelized, and the processing speed was 150 times faster than that of conventional circuits. In addition, a special-purpose computer for 3D color imaging was developed by implementing the proposed circuits for the three wavelengths on a single FPGA board. As a result of the verification, a speedup of approximately nine times over a 10-core CPU was achieved. The results of this research strongly promote the realization of a holographic portable camera with real-time 3D color imaging capability.INDEX TERMS Field-programmable gate array, image reconstruction, incoherent digital holography, special-purpose computer