k-Wave

1. ceke
   

   Member
   Joined: Aug '14
   Posts: 3

   So, I just installed k-Wave 1.1 on my Ubuntu 14.04 (x64) system. It features a GTX Titan Z dual GPU card, with 6 GB of GPU memory per GPU.

   Running, for example 'example_us_bmode_phased_array.m' with data_cast = 'gpuArray-single' works, out of the box. It does, however, only use one of the GPUs.

   I tried starting two MATLAB sessions, and then launching two separate MATLAB scripts like the example above, hoping that the OS would schedule the second job to the other GPU (since the first one was already working at full load). This was not the case.

   So, now to the question: How can I direct my scripts to a specific GPU in the case I have more than one?

   I run the Parallel Computing Toolbox, but none of the other GPU solutions (except, of course, for NVIDIAs drivers and toolkit).

   Best regards,
   /Johan

   Posted 9 years ago #

2. Bradley Treeby
   

   Developer
   Joined: Mar '10
   Posts: 1,640

   Hi Johan,

   You can use the MATLAB function gpuDevice with no inputs to query which device is currently selected, and then gpuDevice(1) or gpuDevice(2) (etc) to select which one you'd like to use. I've not tried using multiple GPU devices with multiple MATLAB sessions, so I'd be interested to know if you can get it to work.

   Hope that helps,

   Brad.

   Posted 9 years ago #

3. ceke
   

   Member
   Joined: Aug '14
   Posts: 3

   Thanks for the quick response, almost too quick for me :-)

   So, below is what I tried, which appears to be working fine:

   1) I took the script "example_us_bmode_linear_transducer.m" and split it into two parts, where the for loop for the simulation is split so that the scripts take half of the scan lines each and then save the results in separate .mat files.

   2) I started two MATLAB sessions and called gpuDevice(1) in the first and gpuDevice(2) in the other.

   3) In a terminal windows I typed nvidia-smi a few times to see what happened to memory usage and it seems like it uses about 750-780 MB of GPU RAM on each, and that the GPU temperature slowly ramps up on both.

   When starting the first MATLAB session the estimated computation time was about 1.5 minutes/scan line. When starting the second session (on the other GPU) it first increased to 2.5 minurts/scan line, but after a minute or so they both dropped back down to 1.5.

   All in all, it appears that the speedup of this scheme is about two times when adding the extra GPU.

   For your reference, I'm running a system with:
   1) Intel Xeon CPU E5-1650 v2 @ 3.5 GHz (6 cores in HyperThreading -> 12 threads)
   2) 64 GB of RAM
   3) GeForce GTX Titan Z with dual GPUs with 6 GB of GPU RAM each
   4) Ubuntu 14.04.1 LTS (x64)
   5) MATLAB 2014a with the Parallel Computing Toolbox
   6) NVIDIA Driver version 343.22 (v. 6.5)
   7) NVIDIA Toolkit v. 5.5

   Many thanks!
   /Johan

   Posted 9 years ago #

4. ceke
   

   Member
   Joined: Aug '14
   Posts: 3

   Oh, it might be possible to run this in one MATLAB session if you can execute the two scripts with directing the to two different MATLAB workers using something like:

   parpool(2);
   parfor gpu = 1:2,
   if gpu == 1,
   gpuDevice(1)
   else
   gpuDevice(2)
   end
   run_the_simulation_script
   end
   parpool close;

   I have had some problems, though, since the parfor construction doesn't work so well with struct variables. The solution was to write a wrapper function for the kspace... with all individual struct fields as separate arguments, and then call this wrapper from parfor. Then, inside the wrapper, re-pack the struct and call the k-wave function.

   Haven't tried the trick above, but I can't see why it shouldn't work.

   Cheers,
   /Johan

   Posted 9 years ago #

5. Bradley Treeby
   

   Developer
   Joined: Mar '10
   Posts: 1,640

   Hi Johan,

   Thanks for the update, very interesting. Glad to see you got it working!

   Brad.

   Posted 9 years ago #

6. acarlson
   

   Member
   Joined: Jan '15
   Posts: 1

   Johan and Brad,

   I was wondering if there is a way to "split" the example file kspaceFirstOrder2D? I think I followed the original post of splitting into scan lines, but it is not as clear if is possible to do this with firstorder2d problem

   I would appreciate any help, I currently have the following hardware, and was hoping to utilize multipe GPU's, if possible.

   (2X) 12-core Xeon E5-2680v3 CPUs
   (2x) NVIDIA Tesla K40's
   128GB DDR4 2133MHz Memory

   The system has Scientific Linux 6.5 with NVIDIA CUDA and MATLAB R2014b

   Andy

   Posted 9 years ago #

7. Bradley Treeby
   

   Developer
   Joined: Mar '10
   Posts: 1,640

   Hi Andy,

   In the Simulating B-mode Ultrasound Images Example, the simulated ultrasound image is formed by performing multiple k-Wave simulations. These can be performed separately (with separate calls to kspaceFirstOrder3D), and thus can be run on different GPUs.

   If you want to do the same thing running multiple 2D simulations, this can be done in the same way. However, it is not possible to split a single simulation over multiple GPUs. The main reason is that the simulations rely on the FFT, which requires an all-to-all communication. This becomes very slow when data has to pass along the PCIe links between the GPUs (if you're interested, you can read more about this here).

   Hope that helps,

   Brad.

   Posted 9 years ago #

8. qzshi
   

   Member
   Joined: Oct '19
   Posts: 8

   hi, Brad

   I use the function gpuDevice(1) and gpuDevice(2) on two matlab sessions, but when I run the function kspaceFirstOrder2DG respectively, only one GPU(GPU 0 of 8 GPU) is selected on both two matlab sessions.

   So, is it right for this approach to call multiple GPUs?

   best regards
   qzshi

   Posted 3 years ago #

9. Bradley Treeby
   

   Developer
   Joined: Mar '10
   Posts: 1,640

   Hi qzshi,

   The best way with the current version of k-Wave is to use the native GPU code by calling kspaceFirstOrder3DG with the optional input 'DeviceNum' set to 0 or 1 (etc). You don't need to call gpuDevice from within MATLAB.

   Brad

   Posted 3 years ago #

RSS feed for this topic

Reply

You must log in to post.