ICE: A Scalable, Low-Cost FPGA-Based Telescope Signal Processing and Networking System

Abstract

Received (to be inserted by publisher); Revised (to be inserted by publisher); Accepted (to be inserted by publisher);We present an overview of the 'ICE' hardware and software framework that implements large arrays of interconnected field programmable gate array (FPGA)-based data acquisition, signal processing and networking nodes economically. The system was conceived for application to radio, millimeter and sub-millimeter telescope readout systems that have requirements beyond typical off-the-shelf processing systems, such as careful control of interference signals produced by the digital electronics, and clocking of all elements in the system from a single precise observatory-derived oscillator. A new generation of telescopes operating at these frequency bands and designed with a vastly increased emphasis on digital signal processing to support their detector multiplexing technology or high-bandwidth correlators-data rates exceeding a terabyte per second-are becoming common. The ICE system is built around a custom FPGA motherboard that makes use of an Xilinx Kintex-7 FPGA and ARMbased co-processor. The system is specialized for specific applications through software, firmware, and custom mezzanine daughter boards that interface to the FPGA through the industry-standard FPGA mezzanine card (FMC) specifications. For high density applications, the motherboards are packaged in 16-slot crates with ICE backplanes that implement a low-cost passive full-mesh network between the motherboards in a crate, allow high bandwidth interconnection between crates, and enable data offload to a computer cluster. A Python-based control software library automatically detects and operates the hardware in the array. Examples of specific telescope applications of the ICE framework are presented, namely the frequency-multiplexed bolometer readout systems used for the South Pole Telescope (SPT) and Simons Array and the digitizer, F-engine, and networking engine for the Canadian Hydrogen Intensity Mapping Experiment (CHIME) and Hydrogen Intensity and Real-time Analysis eXperiment (HIRAX) radio interferometers.

HIRAX: a probe of dark energy and radio transients

“…The signal is then sent through a customized poly-phase filter bank (PFB) and FFT algorithm, performed at 18+18-bit precision to channelize the signal into 1024 frequency channels between 400-800 MHz. 42 The X-engine of the correlator will correlate all sky inputs for each of the 1024 frequency channels, and is implemented in an array of HPC GPU nodes. To network the data between the FPGA F-engine and the GPU X-engine, the data is sent via a custom backplane and point-to-point digital connections into the GPU nodes, during which it undergoes a corner-turn operation.…”

Quantization Bias for Digital Correlators

“…The CHIME correlator is based on an FX design, where the F-engine digitizes (samples at 800 MSPS and quantizes to 8 bits) and channelizes (i.e., divide the 400 MHz input bandwidth into thousands of frequency channels) the analog signals from the 2048 receivers of the interferometer (see Bandura et al, 2016a, for details of the CHIME F-engine). The complex-valued data from each frequency channel is then quantized to 4 bits (4 bits real + 4 bits imaginary) before being reorganized by a corner-turn system (see Bandura et al, 2016b, for details of the corner-turn network) and fed into the X-engine that computes the full N 2 correlation matrix (see Denman et al, 2015;Recnik et al, 2015, for details of the CHIME X-engine).…”

CHIME FRB: An application of FFT beamforming for a radio telescope

“…Input voltage data are collected from 256 dual-polarization receivers on each of the 4 cylinders, i.e., a total of 2048 distinct inputs. Custom FPGA boards (F-engine) [3,4] digitize and channelize these data to 1024 frequency channels via a 4-tap polyphase filter bank (PFB), then scale this voltage data to 4+4-bit complex numbers, at a 2.56 µs cadence (see Table 1). Spatial correlation (X-engine) is performed in a GPU cluster that consists of 256 processing nodes each with 4 AMD Fiji GPUs, similar to that presented in [10,6,7].…”