k-Wave User Forum » Topic: Multiple GPUs, k-Wave 1.1

k-Wave User Forum » Topic: Multiple GPUs, k-Wave 1.1 http://www.k-wave.org/forum/topic/multiple-gpus-k-wave-11 Support for the k-Wave MATLAB toolbox en-US Tue, 12 May 2026 22:28:57 +0000 http://bbpress.org/?v=1.0.2 <![CDATA[Search]]> q http://www.k-wave.org/forum/search.php Bradley Treeby on "Multiple GPUs, k-Wave 1.1" http://www.k-wave.org/forum/topic/multiple-gpus-k-wave-11#post-7864 Mon, 05 Oct 2020 09:24:14 +0000 Bradley Treeby 7864@http://www.k-wave.org/forum/ Hi qzshi, The best way with the current version of k-Wave is to use the native GPU code by calling <code>kspaceFirstOrder3DG</code> with the optional input <code>'DeviceNum'</code> set to 0 or 1 (etc). You don't need to call <code>gpuDevice</code> from within MATLAB. Brad qzshi on "Multiple GPUs, k-Wave 1.1" http://www.k-wave.org/forum/topic/multiple-gpus-k-wave-11#post-7863 Fri, 02 Oct 2020 04:21:36 +0000 qzshi 7863@http://www.k-wave.org/forum/ hi, Brad I use the function gpuDevice(1) and gpuDevice(2) on two matlab sessions, but when I run the function kspaceFirstOrder2DG respectively, only one GPU(GPU 0 of 8 GPU) is selected on both two matlab sessions. So, is it right for this approach to call multiple GPUs? best regards qzshi Bradley Treeby on "Multiple GPUs, k-Wave 1.1" http://www.k-wave.org/forum/topic/multiple-gpus-k-wave-11#post-4949 Thu, 22 Jan 2015 22:00:58 +0000 Bradley Treeby 4949@http://www.k-wave.org/forum/ Hi Andy, In the <a href="http://www.k-wave.org/documentation/example_us_bmode_linear_transducer.php">Simulating B-mode Ultrasound Images Example</a>, the simulated ultrasound image is formed by performing multiple k-Wave simulations. These can be performed separately (with separate calls to <code>kspaceFirstOrder3D</code>), and thus can be run on different GPUs. If you want to do the same thing running multiple 2D simulations, this can be done in the same way. However, it is not possible to split a single simulation over multiple GPUs. The main reason is that the simulations rely on the FFT, which requires an all-to-all communication. This becomes very slow when data has to pass along the PCIe links between the GPUs (if you're interested, you can read more about this <a href="http://www.homepages.ucl.ac.uk/~rmapbtr/papers/CONF_11_2012_Nandapalan_PDCAT_MultiGPU.pdf">here</a>). Hope that helps, Brad. acarlson on "Multiple GPUs, k-Wave 1.1" http://www.k-wave.org/forum/topic/multiple-gpus-k-wave-11#post-4939 Fri, 16 Jan 2015 20:38:55 +0000 acarlson 4939@http://www.k-wave.org/forum/ Johan and Brad, I was wondering if there is a way to "split" the example file kspaceFirstOrder2D? I think I followed the original post of splitting into scan lines, but it is not as clear if is possible to do this with firstorder2d problem I would appreciate any help, I currently have the following hardware, and was hoping to utilize multipe GPU's, if possible. (2X) 12-core Xeon E5-2680v3 CPUs (2x) NVIDIA Tesla K40's 128GB DDR4 2133MHz Memory The system has Scientific Linux 6.5 with NVIDIA CUDA and MATLAB R2014b Andy Bradley Treeby on "Multiple GPUs, k-Wave 1.1" http://www.k-wave.org/forum/topic/multiple-gpus-k-wave-11#post-4811 Sat, 01 Nov 2014 11:32:44 +0000 Bradley Treeby 4811@http://www.k-wave.org/forum/ Hi Johan, Thanks for the update, very interesting. Glad to see you got it working! Brad. ceke on "Multiple GPUs, k-Wave 1.1" http://www.k-wave.org/forum/topic/multiple-gpus-k-wave-11#post-4788 Thu, 23 Oct 2014 19:07:45 +0000 ceke 4788@http://www.k-wave.org/forum/ Oh, it might be possible to run this in one MATLAB session if you can execute the two scripts with directing the to two different MATLAB workers using something like: parpool(2); parfor gpu = 1:2, if gpu == 1, gpuDevice(1) else gpuDevice(2) end run_the_simulation_script end parpool close; I have had some problems, though, since the parfor construction doesn't work so well with struct variables. The solution was to write a wrapper function for the kspace... with all individual struct fields as separate arguments, and then call this wrapper from parfor. Then, inside the wrapper, re-pack the struct and call the k-wave function. Haven't tried the trick above, but I can't see why it shouldn't work. Cheers, /Johan ceke on "Multiple GPUs, k-Wave 1.1" http://www.k-wave.org/forum/topic/multiple-gpus-k-wave-11#post-4787 Thu, 23 Oct 2014 18:55:37 +0000 ceke 4787@http://www.k-wave.org/forum/ Thanks for the quick response, almost too quick for me :-) So, below is what I tried, which appears to be working fine: 1) I took the script "example_us_bmode_linear_transducer.m" and split it into two parts, where the for loop for the simulation is split so that the scripts take half of the scan lines each and then save the results in separate .mat files. 2) I started two MATLAB sessions and called gpuDevice(1) in the first and gpuDevice(2) in the other. 3) In a terminal windows I typed nvidia-smi a few times to see what happened to memory usage and it seems like it uses about 750-780 MB of GPU RAM on each, and that the GPU temperature slowly ramps up on both. When starting the first MATLAB session the estimated computation time was about 1.5 minutes/scan line. When starting the second session (on the other GPU) it first increased to 2.5 minurts/scan line, but after a minute or so they both dropped back down to 1.5. All in all, it appears that the speedup of this scheme is about two times when adding the extra GPU. For your reference, I'm running a system with: 1) Intel Xeon CPU E5-1650 v2 @ 3.5 GHz (6 cores in HyperThreading -> 12 threads) 2) 64 GB of RAM 3) GeForce GTX Titan Z with dual GPUs with 6 GB of GPU RAM each 4) Ubuntu 14.04.1 LTS (x64) 5) MATLAB 2014a with the Parallel Computing Toolbox 6) NVIDIA Driver version 343.22 (v. 6.5) 7) NVIDIA Toolkit v. 5.5 Many thanks! /Johan Bradley Treeby on "Multiple GPUs, k-Wave 1.1" http://www.k-wave.org/forum/topic/multiple-gpus-k-wave-11#post-4776 Sun, 19 Oct 2014 21:58:00 +0000 Bradley Treeby 4776@http://www.k-wave.org/forum/ Hi Johan, You can use the MATLAB function <code>gpuDevice</code> with no inputs to query which device is currently selected, and then <code>gpuDevice(1)</code> or <code>gpuDevice(2)</code> (etc) to select which one you'd like to use. I've not tried using multiple GPU devices with multiple MATLAB sessions, so I'd be interested to know if you can get it to work. Hope that helps, Brad. ceke on "Multiple GPUs, k-Wave 1.1" http://www.k-wave.org/forum/topic/multiple-gpus-k-wave-11#post-4775 Sun, 19 Oct 2014 20:23:46 +0000 ceke 4775@http://www.k-wave.org/forum/ So, I just installed k-Wave 1.1 on my Ubuntu 14.04 (x64) system. It features a GTX Titan Z dual GPU card, with 6 GB of GPU memory per GPU. Running, for example 'example_us_bmode_phased_array.m' with data_cast = 'gpuArray-single' works, out of the box. It does, however, only use one of the GPUs. I tried starting two MATLAB sessions, and then launching two separate MATLAB scripts like the example above, hoping that the OS would schedule the second job to the other GPU (since the first one was already working at full load). This was not the case. So, now to the question: How can I direct my scripts to a specific GPU in the case I have more than one? I run the Parallel Computing Toolbox, but none of the other GPU solutions (except, of course, for NVIDIAs drivers and toolkit). Best regards, /Johan