Several Field-Programmable Gate Array (FPGA) based Reconfigurable High-Performance computing systems have recently become available. In this paper we describe our experience developing an application accelerator for a traditional HPC application, a Finite-Difference Time-Domain (FDTD) electromagnetics simulation, for two such systems. We describe the design of our FDTD calculation engine and demonstrate that FPGA can accelerate a single precision floating point implementation. We discuss the current limitations of our design and describe design improvements to overcome them.