k-Wave

1. benji
   

   Member
   Joined: Dec '15
   Posts: 14

   Hi again,
   as mentioned in my previous post - I am simulating acoustic propagation in a humen body phantom. the desired max frequancy is around f=3MHz, and the 3D dimensions should be at minimum 12cmX12cmX12cm.

   in order to simulate body surface returns, the computational grid includs air with a sound speed of around 300m/s.

   That makes the greed spacing: dx=dy=dz= Cmin/[(num. of greed points per wave length)*fmax]
   I saw on the k-wave guide it is recomended to have num. of greed points per wave length at least 3-4 in a heteroginious medium. Cmin is 300m/s as mentioned above.
   that makes dx=dy=dz=~0.03mm.
   that meens the grid size shouldvbe about 3000X3000X3000 big.

   What would the memory requirments be to handle such a grid?
   i saw somwhere on the forum here that a 256X256X256 would require 6GB, so linearly interpulating i guess I would need some 12,000GB... :-o
   1. am i right?
   2. any practical ideas how to run such a simulation given a powerful university cluster with up to some 200GB or so ?

   thank u,
   Benji

   Posted 10 years ago #

2. Jiri Jaros
   

   Developer
   Joined: Feb '12
   Posts: 118

   Hi Benji,
   It may be possible to run such a simulation with the MPI code, however, it would be extremely expensive!

   To simulate the wave propagating over 3000^3, you would need about 3TB of RAM. The simulation would have to run over at least 50k time steps. Using 3000 compute cores, and predicting that a time step would last 20s, you would have to wait for 277 hours for the simulation to complete. Taking into account 1 core-hours is for $0.1, you’d have to pay $83,000, which is, you’ll agree, quite impractical.

   We’re working on the faster GPU code, we hope could get the simulation cost below $10k, but still it’s really expensive for routine runs.

   I’d recommend you to start with a 2D slice. That would be much cheaper and faster.

   Posted 9 years ago #

3. benji
   

   Member
   Joined: Dec '15
   Posts: 14

   hi,
   Jiri - although u havent made me any more optimistic, thank u very much for your reply.

   a 2D simulation won't be helpful in that case. It must be a 3D one.
   as i said, i have about 200GB RAM memory available (includes both physical and virtual memory) at max. i will have to do the best under those contrains.

   how can i calculte memory requirments as a function of grid size? is their some thumb rule or simlpe equation to evluate that?

   Posted 9 years ago #

4. marlenep
   

   Member
   Joined: Dec '15
   Posts: 14

   Hi Benji,
   I have your same problem...have you found any solution? thank you very much for any suggestion

   Posted 9 years ago #

5. Bradley Treeby
   

   Developer
   Joined: Mar '10
   Posts: 1,643

   Hi benji,

   If you are using the C++ code, there is a formulae for the memory requirements given in Eq. 4.1 in the user manual (section 4.9). It neglects a few small matrices, but it should give you a reasonable estimate.

   Brad.

   Posted 9 years ago #

6. benji
   

   Member
   Joined: Dec '15
   Posts: 14

   hi,
   Bradly - thank u. is the equation relevant also to the MATLAB case?
   is their a difference in memory requirments between the matlab and C codes?
   I tried running a hetrogenious grid of 768X768X768 with the MATLAB code using 45GB of physical memory and 45GB of virtual memory (90gb in total)
   it crushes because of memory issues, while according to equation 4.1 it requires only 25-30GB.

   marlen - sorry, but i have no magical solution, and as it seems so does Brad:). got to customize your simulation to meet practical limitations... I for instance will probably have to compromise on grid size and work with lower frequency.

   Posted 9 years ago #

7. Bradley Treeby
   

   Developer
   Joined: Mar '10
   Posts: 1,643

   Hi Benji,

   This equation only applies to the C++ code. There is a significant difference between the two codes, with the C++ code being heavily optimised. I would definitely use this for larger problems if at all possible. Also, if you're trying to conserve memory, it's a good idea to run the C++ code outside of MATLAB using the SaveToDisk option rather than the inbuilt kspaceFirstOrder3DC function. This avoids the variables being in memory twice (once in MATLAB and once in C++).

   Hope that helps,

   Brad.

   Posted 9 years ago #

8. umitarabul
   

   Member
   Joined: Feb '13
   Posts: 9

   Hi,

   Is it possible to divide the volume in pieces and run simulations separately and superpose the solutions? Is there a way to define the output of one block as input to the neighbor block? If we neglect non-linearity for example?

   Posted 9 years ago #

9. Bradley Treeby
   

   Developer
   Joined: Mar '10
   Posts: 1,643

   Hi umitarabul,

   Anthony Grisey did something similar to this in this paper. There are a few related forum posts, for example, here.

   Brad.

   Posted 9 years ago #

10. umitarabul
    

    Member
    Joined: Feb '13
    Posts: 9

    Thanks for the answer Dr. Treeby. I will look at those two studies.

    Umit

    Posted 9 years ago #

RSS feed for this topic

Reply

You must log in to post.