http://www.k-wave.org/forum/topic/gpu-error-an-illegal-memory-access-was-encountered-routine Support for the k-Wave MATLAB toolbox en-US Wed, 13 May 2026 00:26:51 +0000 http://bbpress.org/?v=1.0.2 q http://www.k-wave.org/forum/search.php http://www.k-wave.org/forum/topic/gpu-error-an-illegal-memory-access-was-encountered-routine#post-8915 Thu, 21 Sep 2023 14:58:24 +0000 BIlly 8915@http://www.k-wave.org/forum/ <p>After 3 straight days trying to debug this the error seems to have gone away on its own, that's computers for you :) Hopefully I don't encounter it again... </p> http://www.k-wave.org/forum/topic/gpu-error-an-illegal-memory-access-was-encountered-routine#post-8912 Fri, 15 Sep 2023 17:27:01 +0000 BIlly 8912@http://www.k-wave.org/forum/ <p>I'm very suspicious the error is caused by the GPU not releasing enough resources after the first process is complete, which would explain why the 2D reconstructions (which barely use any resources) are running fine. My forward simulations are on a 1024*128*1024 grid at 32 bit floating point precision, the A100s have 80GB of GPU RAM which should be more than enough.</p> <p>Thanks,<br /> Billy. </p> http://www.k-wave.org/forum/topic/gpu-error-an-illegal-memory-access-was-encountered-routine#post-8911 Fri, 15 Sep 2023 15:31:41 +0000 BIlly 8911@http://www.k-wave.org/forum/ <p>Greetings all, </p> <p>First off let me say K-Wave has been an invaluable tool for me and it's been running brilliantly on my workstation for months. Since I intend to run bigger simulations (and lots of them) I'm now trying to run kspaceFirstOrder-CUDA on my department's HPC using A100s. Unfortunately it doesn't seem to get very far and I receive the following:<br /> '<br /> INFO:root:k-wave forward, cycle: 1, wavelength_index: 1, pulse: 1<br /> INFO:root:p0 loaded in 0.09530607238411903 seconds<br /> INFO:root:kwave forward run in 229.18575995601714 seconds<br /> INFO:root:sensor data saved in 0.010119317099452019 seconds<br /> INFO:root:k-wave forward, cycle: 1, wavelength_index: 2, pulse: 1<br /> INFO:root:p0 loaded in 0.021063480526208878 seconds<br /> ┌───────────────────────────────────────────────────────────────┐<br /> │ !!! K-Wave experienced a fatal error !!! │<br /> ├───────────────────────────────────────────────────────────────┤<br /> │ GPU error: an illegal memory access was encountered routine │<br /> │ name: cudaGetLastError() in file OutputStreams/ │<br /> │ OutputStreamsCudaKernels.cu, line 130. │<br /> ├───────────────────────────────────────────────────────────────┤<br /> │ Execution terminated │<br /> └───────────────────────────────────────────────────────────────┘<br /> '<br /> Now my script is simulating time-series images, so it runs k-wave multiple times. For the first run ("cycle: 1, wavelength_index: 1, pulse: 1"), it works fine! The output also seems completely sensible. But as soon as it reaches pulse number two it encounters the error. It's most probably related to the HPC since I haven't encountered this on my workstation, just wondering if anybody has seen anything similar or might have any idea as to what's causing this? The time reversal reconstructions are also set to run after the forward simulations and these are completely fine - although they are 2D and thus only require a fraction of the resources. </p>