http://www.k-wave.org/forum/topic/linux-c-binary-sees-less-nodes-than-job-submission-system-assigns Support for the k-Wave MATLAB toolbox en-US Tue, 12 May 2026 23:29:03 +0000 http://bbpress.org/?v=1.0.2 q http://www.k-wave.org/forum/search.php http://www.k-wave.org/forum/topic/linux-c-binary-sees-less-nodes-than-job-submission-system-assigns#post-8533 Fri, 13 May 2022 15:12:23 +0000 ahmedsallam 8533@http://www.k-wave.org/forum/ <p>Hello Jiri,</p> <p>Can you also please send me the OpenMPI code? I have sent you an email. </p> <p>Thanks, </p> http://www.k-wave.org/forum/topic/linux-c-binary-sees-less-nodes-than-job-submission-system-assigns#post-6793 Wed, 20 Mar 2019 08:46:12 +0000 Jiri Jaros 6793@http://www.k-wave.org/forum/ <p>Hi DLam,<br /> yes, the released codes are targeted at multi-socket multi-core systems, but with shared memory (NUMA nodes). </p> <p>There is a distributed version, that can run across the whole cluster using OpenMPI. It is a release candidate 1 version. I plan to release it as soon as I've finished the current k-Wave release with 2D and axisymmetric codes for NUMA nodes. (I had to put it aside for two months as this releases has the priority). </p> <p>If you want to test the prototype, send me an email (jarosjir[ at ]fit.vutbr.cz) and I could become a beta tester.</p> <p>Best<br /> Jiri </p> http://www.k-wave.org/forum/topic/linux-c-binary-sees-less-nodes-than-job-submission-system-assigns#post-6790 Mon, 18 Mar 2019 16:57:07 +0000 DLam 6790@http://www.k-wave.org/forum/ <p>Dear Jiri,</p> <p>Thanks for the quick reply!<br /> Yes, we want to run the code across multiple host servers (each server with multiple sockets for CPUs, each CPU with multiple cores). If I read you correctly, the current C++ binary is incompatible with doing this message-passing between different servers?</p> <p>Best regards,</p> <p>Daniel </p> http://www.k-wave.org/forum/topic/linux-c-binary-sees-less-nodes-than-job-submission-system-assigns#post-6771 Fri, 15 Mar 2019 14:09:01 +0000 Jiri Jaros 6771@http://www.k-wave.org/forum/ <p>Hi DLam,</p> <p>the default behavior is to use as many cores as the system has. If it helps, I call routine omp_get_num_procs() to find out this number.</p> <p>I guess the cluster schedules sets the variable OMP_NUM_THREADS to inform OpenMP codes about how many threads to use. This is not implemented directly because it does not reliably work on Windows. </p> <p>You can to use -t parameter to set up the number of threads manually, e.g.<br /> kspaceFirstOrder3D -t ${OMP_NUM_THREADS}</p> <p>If you want to check, the cluster scheduler sets up the OpenMP environment correctly try:<br /> OMP_DISPLAY_ENV=true kspaceFirstOrder3DC -t XXX</p> <p>Reading the post again, you may ask about something else. Do you want to run the code across multiple nodes? If so, this is not possible. It requires a completely different implementation using message passing. If this is the case, send me an email.</p> <p>Best wishes<br /> Jiri </p> http://www.k-wave.org/forum/topic/linux-c-binary-sees-less-nodes-than-job-submission-system-assigns#post-6768 Wed, 13 Mar 2019 16:27:21 +0000 DLam 6768@http://www.k-wave.org/forum/ <p>Dear K-wave team,</p> <p>Our team has been running the K-wave C++ binary on a Linux cluster, submitting jobs within a bash script. It works great if we ensure that all workers are on a single node -- when the job is assigned to workers across multiple nodes, I notice in the console output, the 'Number of CPU Threads' heading shows a smaller number than the ones assigned when submitting the job.</p> <p>We were wondering if this is simply a display bug, or if something else is going on? We've been looking at increasing our worker assignments but the 'Time to Go' estimation doesn't change much when we double core count.</p> <p>Cluster:<br /> OS: Linux CentOS 7<br /> Cluster architecture: Multiple hosts each with multiple workers, running openMPI<br /> Job submission system: LSF </p>