The Square Kilometre Array Science Data Processor. Preliminary compute platform design

Broekema, P. Chris; Nieuwpoort, Rob V. van; Bal, Henri E.

doi:10.1088/1748-0221/10/07/c07004

Cited by 32 publications

(24 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Empirical data describe a trend for increasing survey data capture rates over time [62]. This is expected to continue in to the Square Kilometre Array (SKA) era [80,30,17,61]. The projected data rates are large enough to make the storage of all raw observational data impossible (e.g.…”

Section: Feasibility Of Real-time Searchmentioning

confidence: 99%

A processing pipeline for high volume pulsar candidate data streams

Lyon

Stappers

Levin

et al. 2019

Astronomy and Computing

View full text Add to dashboard Cite

Pulsar data analysis pipelines have historically been comprised of bespoke software systems, supporting the off-line analysis of data. However modern data acquisition systems are making off-line analyses impractical. They often output multiple simultaneous high volume data streams, significantly increasing data capture rates. This leads to the accumulation of large data volumes, which are prohibitively expensive to retain. To maintain processing capabilities when off-line analysis becomes infeasible due to cost, requires a shift to on-line data processing. This paper makes four contributions facilitating this shift with respect to the search for radio pulsars: i) it characterises for the modern era, the key components of a pulsar search science (not signal processing) pipeline, ii) it examines the feasibility of implementing on-line pulsar search via existing tools, iii) problems preventing an easy transition to on-line search are identified and explained, and finally iv) it provides the design for a new prototype pipeline capable of overcoming such problems. Realised using Commercial off-the-shelf (COTS) software components, the deployable system is open source, simple, scalable, and cheap to produce. It has the potential to achieve pulsar search design requirements for the Square Kilometre Array (SKA), illustrated via testing under simulated SKA loads. streams (i.e. 'tuple-at-a-time' or 'click-stream'). This represents an advancement over the current state-of-the-art, and a departure from traditional off-line batch methods.

show abstract

Section: Feasibility Of Real-time Searchmentioning

confidence: 99%

A processing pipeline for high volume pulsar candidate data streams

Lyon

Stappers

Levin

et al. 2019

Astronomy and Computing

View full text Add to dashboard Cite

show abstract

“…We assume that we can tune the number of computing cores in the node. 8 We set the number of cores in the node such as to accommodate the desired λ threads.…”

Section: Extrapolating For Exascale Designmentioning

confidence: 99%

“…Additionally, in this work two case studies demonstrate how the methodology can analyze parallel applications. The first case study is a large-scale graph-analytics workload while the second case study is a part of a real-life radio-astronomy application whose implementation will require the deployment of a dedicated exascale system [8,23].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Scaling Properties of Parallel Applications to Exascale

Mariani

Anghel

Jongerius

et al. 2016

Int J Parallel Prog

View full text Add to dashboard Cite

A detailed profile of exascale applications helps to understand the computation, communication and memory requirements for exascale systems and provides the insight necessary for fine-tuning the computing architecture. Obtaining such a profile is challenging as exascale systems will process unprecedented amounts of data. Profiling applications at the target scale would require the exascale machine itself. In this work we propose a methodology to extrapolate the exascale profile from experimental observations over datasets feasible for today's machines. Extrapolation models are carefully selected by means of statistical techniques and a high-level complexity analysis is included in the selection process to speed up the learning phase and to improve the accuracy of the final model. We extrapolate run-time properties of the target applications including information about the instruction mix, memory access pattern, instruction-level parallelism, and communication requirements. Compared to state-of-the-art techniques, the proposed methodology reduces the prediction error by an order of magnitude on the instruction count and improves the accuracy by up to 1.3× for the memory access pattern, and by more than 2× for the communication requirements.

show abstract

“…Approaches implementing sparse reconstruction for radio interferometry continue to be investigated, and report consistently increasing reconstruction performance, as recent work described in Li et al (2011);Carrillo et al (2012Carrillo et al ( , 2014; Garsden et al (2015); Dabbech et al (2015); Ferrari et al (2015) demonstrates. eration are around five terabits per second (Broekema et al 2015). The resulting images are designed to be in the gigapixels range, and with a high dynamic range of six/seven orders of magnitude (Cornwell et al 2012;Wijnholds et al 2014).…”

Section: Introductionmentioning

confidence: 99%

A Fourier dimensionality reduction model for big data interferometric imaging

Kartik

Carrillo

Thiran

et al. 2017

Monthly Notices of the Royal Astronomical Society

View full text Add to dashboard Cite

Data dimensionality reduction in radio interferometry can provide savings of computational resources for image reconstruction through reduced memory footprints and lighter computations per iteration, which is important for the scalability of imaging methods to the big data setting of the next-generation telescopes. This article sheds new light on dimensionality reduction from the perspective of the compressed sensing theory and studies its interplay with imaging algorithms designed in the context of convex optimization. We propose a post-gridding linear data embedding to the space spanned by the left singular vectors of the measurement operator, providing a dimensionality reduction below image size. This embedding preserves the null space of the measurement operator and hence its sampling properties are also preserved in light of the compressed sensing theory. We show that this can be approximated by first computing the dirty image and then applying a weighted subsampled discrete Fourier transform to obtain the final reduced data vector. This Fourier dimensionality reduction model ensures a fast implementation of the full measurement operator, essential for any iterative image reconstruction method. The proposed reduction also preserves the i.i.d. Gaussian properties of the original measurement noise. For convex optimization-based imaging algorithms, this is key to justify the use of the standard 2 -norm as the data fidelity term. Our simulations confirm that this dimensionality reduction approach can be leveraged by convex optimization algorithms with no loss in imaging quality relative to reconstructing the image from the complete visibility data set. Reconstruction results in simulation settings with no direction dependent effects or calibration errors show promising performance of the proposed dimensionality reduction. Further tests on real data are planned as an extension of the current work. matlab code implementing the proposed reduction method is available on GitHub.

show abstract

The Square Kilometre Array Science Data Processor. Preliminary compute platform design

Cited by 32 publications

References 4 publications

A processing pipeline for high volume pulsar candidate data streams

A processing pipeline for high volume pulsar candidate data streams

Scaling Properties of Parallel Applications to Exascale

A Fourier dimensionality reduction model for big data interferometric imaging

Contact Info

Product

Resources

About