Full-wave nonlinear ultrasound simulation on distributed clusters with applications in high-intensity focused ultrasound

Abstract

Model-based treatment planning and exposimetry for high-intensity focused ultrasound (HIFU) requires the numerical simulation of nonlinear ultrasound propagation through heterogeneous and absorbing media. This is a computationally demanding problem due to the large distances travelled by the ultrasound waves relative to the wavelength of the highest frequency harmonic. Here, the kspace pseudospectral method is used to solve a set of coupled partial differential equations equivalent to a generalised Westervelt equation. The model is implemented in C++ and parallelised using the message passing interface (MPI) for solving large-scale problems on distributed clusters. The domain is partitioned using a 1D slab decomposition, and global communication is performed using a sparse communication pattern. Operations in the spatial frequency domain are performed in transposed space to reduce the communication burden imposed by the 3D fast Fourier transform. The performance of the model is evaluated using grid sizes up to 4096 × 2048 × 2048 grid points distributed over a cluster using up to 1024 compute cores. Given the global nature of the gradient calculation, the model shows good strong scaling behaviour, with a speed-up of 1.7x whenever the number of cores is doubled. This means large-scale simulations can be distributed across high numbers of cores on a cluster to minimise execution times with a relatively small computational overhead. The efficacy of the model is demonstrated by simulating the ultrasound beam pattern for a HIFU sonication of the kidney.

Sensitivity of simulated transcranial ultrasound fields to acoustic medium property maps

“…The spatial and temporal discretisations used conform to previously established criteria to ensure numerical accuracy for transcranial simulation [29,30], with a minimum spatial sampling step of ten spatial points per wavelength (PPW) at 1 MHz for fluid simulations, and 25 PPW for elastic simulations. Fluid simulations were run using the MPI version of k-Wave on the IT4I Salomon supercomputing cluster [26]. Each 3D simulation was carried out on a compute node with an 8-core Intel Xeon E5-4627v2 3.3 GHz CPU, and 256 GB of RAM.…”

“…The simulations were performed using both the fluid and elastic codes within the k-Wave toolbox [26,27]. The fluid code was used for most simulations, with the elastic code used to confirm that shear waves could be neglected.…”

Spectral Domain Decomposition Using Local Fourier Basis: Application to Ultrasound Simulation on a Cluster of GPUs

“…However, ultrasound simulation for these applications is computationally demanding due to the length scales involved, where the propagation length can be hundreds or thousands of times longer than the acoustic wavelength. In turn, this leads to very large domain sizes, in some cases with more than 100 billion grid points [2]. This puts the simulation times beyond clinically useful time-limits, even when using significant computational resources [4].…”

“…This approach was first proposed in [6] and further developed in [7][8][9]. The parallel implementation of the k-space pseudospectral method has previously been described by several groups [2,[10][11][12][13]. Aside from a number of element-wise matrix operations, the most significant operations performed at each time step are multiple real-to-complex and complex-toreal 3D fast Fourier transforms (FFTs).…”

“…Accurately simulating the propagation of ultrasound waves through biological tissue has a large number of practical applications, including the physics-based simulation of diagnostic ultrasound images [1] and treatment planning for ultrasound therapy [2] (a more comprehensive list is provided in [3]). However, ultrasound simulation for these applications is computationally demanding due to the length scales involved, where the propagation length can be hundreds or thousands of times longer than the acoustic wavelength.…”