k-Wave

1. Cyrill
   

   Member
   Joined: Nov '18
   Posts: 6

   We have been successfully running sound-field simulations on an HPC using kspaceFirstOrder3D-OMP for some time now. Currently I am trying to implement an ultrasound imaging simulation and am looking for advice on how to best parallelize the approach. Based on my experience using the code so far, I noticed diminishing returns on parallelization across nodes and have thus maximized thread counts while restricting to a single node. For the imaging purposes I would have to now re-run the code for each scan-line i.e.:

   for (( i=1; i<=$NLines; i++ ))
   mpirun -np $SLURM_NTASKS kspaceFirstOrder3D-OMP ….

   This however would sequentially do one scan-line after the other not being able to capitalize on multiple nodes. I could alternatively create multiple jobs but this seems like a poor approach.
   Question: How would I best go about parallelizing lines across nodes while parallelizing simulations across threads on a single node?

   Is adapting the source code to include and manage the MPI allocations my only option?
   Any guidance is appreciated!

   Cheers, Cyrill

   Posted 6 years ago #

2. Jiri Jaros
   

   Developer
   Joined: Feb '12
   Posts: 118

   Hi Cyrill,
   Take a look at SLURM job arrays.
   https://slurm.schedmd.com/job_array.html

   You can specify there, I've got 512 scan lines (simulations), each needs one node and 12 threads. The job array system will automatically distribute the workload over all free nodes.

   Some scaling issues to consider (need some performance measurement, though):

   - Download k-Wave 1.3, it has several performance improvements.

   - You can specify the number of threads, a single simulation uses, by -t (e.g. -t 1 for a single thread), or the OMP_NUM_THREADS environmental variable.

   - Its an open question whether it is better to run multiple simulations on a single node or not. It depends on the simulation size, CPU specs, etc. Sometimes, it might be useful to use 4 threads per a simulation and run 3 simulations on a single node (considering you have a 12 core CPU). For small simulations, the best could be to run as many single-threaded simulations as there are cores, for big ones the run only a single simulation per CPU (NUMA domain - but make sure you use proper thread binding) or the whole node. Here you need to run a few experiments.

   Best
   Jiri

   Posted 6 years ago #

3. Cyrill
   

   Member
   Joined: Nov '18
   Posts: 6

   Thanks Jiri for the detailed feedback, this makes for a very simple solution while maintaining a lot of node/thread allocation! This however brings me to a new challenge. As far as defining the input .h5 files for kspaceFirstOrder3D-OMP, the only variation between jobs is the transducer.steering_angle. Is it possible to parsed this parameter as and input flag or will I have to create individual (potentially very large) files for each angle? Or am I just approaching this problem wrong entirely?

   Best,

   Cyrill

   Posted 6 years ago #

4. Jiri Jaros
   

   Developer
   Joined: Feb '12
   Posts: 118

   Hi Cyrill,
   Unfortunately, you'll have to create an independent file for every simulation. The OMP code has no logic for the steering angle, this is left to Matlab.

   Best
   Jiri

   Posted 6 years ago #

RSS feed for this topic

Reply

You must log in to post.