Hello,
My name is Dogu and I am a PhD student from the mechanical engineering department of the University of Bath. I have been using k-Wave for a while now to simulate ultrasonic wave propagation into human soft tissue, and I am very thankful for this product that you have put forward.
I'm at a stage where the simulation I'm running is too large for my machine to handle (6000 by 6000 2D simulation), hence I am getting the task computed on the University's supercomputer system. I have been attempting to run a C++ version of my simulation on the supercomputer, and to do this, I would produce a .h5 file by using the 'SaveToDisk' input of the function kspaceFirstOrder2D, upload it to my allocated space on the uni supercomputer memory and call it in as an input (-i) of the function kspaceFirstOrder-OMP. I am able to successfully generate the .h5 file on my machine with no problems and upload it to the uni server, yet when the uni supercomputer starts running the task, I get the following error message within seconds:
"K-Wave experienced a fatal error. Error = invalid format of the input file = iostream error"
I have tried a number of things yet I do not understand where this is going wrong. My best guess if that maybe I have a Matlab code line that is not C++ optimised. Below I have attached the the binbash script I use to remotely submit jobs, and the Matlab script that I use to generate the .h5 files. Any help or advice would be much, much appreciated, as time is ticking on my PhD. Many thanks in advance!
Best, Dogu
===========================================================
Job submission script:
[11/08 11:40] James Grant
Hi Dogu apologies for the delay here especially as it's quite a trivial issue that I should have spotted previously. Linux file systems are case sensitive so Position2.h5 is different from position2.d5. The copy command cp ../input.h5 . is only needed if the input file is in the directory above, you have it already in the same directory so your jobscript could be:
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=24
#SBATCH --mem=60000
#SBATCH --time=6:00:00
#SBATCH --job-name=position2
#SBATCH --account=free
#SBATCH --partition=batch-sky
#SBATCH --mail-user=dz322@bath.ac.uk
#SBATCH --output=StdOut.o.%j
#SBATCH --error=StdErr.e.%j
module purge
module load slurm
module load matlab
module load hdf5/intel
module load intel/compiler/64/18.5.274
module load intel/mkl/64/18.5.274
module load fftw3/intel/avx/3.3.4
module load gcc/9.2.0
export OMP_NUM_THREADS=24
export OMP_PLACES=cores
export OMP_PROC_BIND=true
../k-Wave/kspaceFirstOrder-OMP/skylake/kspaceFirstOrder-OMP -i position1.h5 -o pos2_out.h5 --checkpoint_file check_pos2 --checkpoint_interval 20000
===========================================================
Matlab script:
%4/6/2020
%This is the script file to use to run scans at 5MHz into the human limb
%model.
%Script originates from the kWave example titled:
%Monopole Point Source In A Homogeneous Propagation Medium Example
%%
% =========================================================================
% GETTING READY
% =========================================================================
clearvars;
close all;
%Number of measurements to take.
m = 1;
%Create the computational grid
%Number of grid points in x and y directions.
Nx = 6000; Ny = 6000;
%Grid point spacings and side lengths in x and y directions [m].
dx = 2.3438e-05/4; dy = dx;
X_SideLength = dx*Nx; Y_SideLength = dy*Ny;
%Kwave function to create the medium.
kgrid = kWaveGrid(Nx, dx, Ny, dy);
% Space between consecutive measurements.
Gap = round(0.0005/dx);
%Set medium dimensions.
medium.sound_speed = zeros(Ny,Nx);
medium.density = zeros(Ny,Nx);
%Set how long the simulation will go for.
EndTime = 100e-6;
%%
%Define position number.
a = 1;
%Calculate amount of gap too move by, in gridpoints.
Shift = (a-1)*Gap;
%Run Medium Generator Function and shift muscle layer accordingly. Then,
%assign sound speed and density values.
Medium5MHz = MediumGeneratorFctn(Shift);
medium.sound_speed = Medium5MHz.SS;
medium.density = Medium5MHz.Den;
DM_OG = Medium5MHz.DM;
%Create the time array (kgrid.makeTime function).
kgrid.makeTime(medium.sound_speed);
%Set number of timesteps to measure.
kgrid.Nt = round(EndTime/kgrid.dt);
% =========================================================================
% SOURCE & SENSOR
% Define points needed for source size. For 13 mm,
half_of_probe = round((0.013/2)/dx);
source_freq = 5e6; % [Hz]
source_mag = 2/(1500*1000)/65; % [Pa]
[display_mask,sensor,source] = SourcenSensor60mm5MHz(a,Nx,Ny,dx,Gap,half_of_probe,source_mag,source_freq,kgrid,DM_OG,medium);
% =========================================================================
% EXECUTION OF 2D SIMULATION
%Define the acoustic parameters to record
sensor.record = {'p', 'p_final'};
%Display Mask as input argument (PML Input can be adjusted here)
input_args = {'DisplayMask',display_mask};
% run the simulation
filename = 'position1';
sensor_data = kspaceFirstOrder2D(kgrid, medium, source, sensor, input_args{:},'SaveToDisk',filename); %,input_args{:}
%Summing SensorData
sensor_data_sum = sum(sensor_data.p);
ScanData(a,:,:) = sensor_data.p;