{2023.06}[foss/2023a] PyTorch v2.1.2 w/ CUDA 12.1.1 by trz42 · Pull Request #825 · EESSI/software-layer

trz42 · 2024-11-21T20:26:38Z

Builds

magma/2.7.2-foss-2023a-CUDA-12.1.1
PyTorch/2.1.2-foss-2023a-CUDA-12.1.1

Superseedes #718

eessi-bot · 2024-11-21T20:26:42Z

Instance eessi-bot-mc-aws is configured to build for:

architectures: x86_64/generic, x86_64/intel/haswell, x86_64/intel/skylake_avx512, x86_64/amd/zen2, x86_64/amd/zen3, aarch64/generic, aarch64/neoverse_n1, aarch64/neoverse_v1
repositories: eessi.io-2023.06-compat, eessi-hpc.org-2023.06-software, eessi-hpc.org-2023.06-compat, eessi.io-2023.06-software

riscv-eessi-io-bot · 2024-11-21T20:26:42Z

Instance eessi-bot-riscv is configured to build for:

architectures: riscv64/generic
repositories: riscv.eessi.io-20240402

riscv-eessi-io-bot · 2024-11-21T20:26:42Z

Instance eessi-bot-riscv is configured to build for:

architectures: riscv64/generic
repositories: riscv.eessi.io-20240402

eessi-bot · 2024-11-21T20:26:43Z

Instance eessi-bot-mc-azure is configured to build for:

architectures: x86_64/amd/zen4
repositories: eessi.io-2023.06-compat, eessi.io-2023.06-software

trz42 · 2024-11-21T20:27:02Z

bot: build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 accel:nvidia/cc80

riscv-eessi-io-bot · 2024-11-21T20:27:05Z

Updates by the bot instance eessi-bot-riscv (click for details)

account trz42 has NO permission to send commands to the bot

riscv-eessi-io-bot · 2024-11-21T20:27:05Z

Updates by the bot instance eessi-bot-riscv (click for details)

account trz42 has NO permission to send commands to the bot

eessi-bot · 2024-11-21T20:27:05Z

Updates by the bot instance eessi-bot-mc-aws (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 accel:nvidia/cc80 from trz42
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80 resulted in:
- submitted job 30162, for details & status see {2023.06}[foss/2023a] PyTorch v2.1.2 w/ CUDA 12.1.1 #825 (comment)

eessi-bot · 2024-11-21T20:27:05Z

Updates by the bot instance eessi-bot-mc-azure (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 accel:nvidia/cc80 from trz42
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80 resulted in:
- no jobs were submitted

eessi-bot · 2024-11-21T20:27:08Z

New job on instance eessi-bot-mc-aws for CPU micro-architecture x86_64-amd-zen2 and accelerator nvidia/cc80 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.11/pr_825/30162

date	job status	comment
Nov 21 20:27:08 UTC 2024	submitted	job id `30162` awaits release by job manager
Nov 21 20:27:13 UTC 2024	released	job awaits launch by Slurm scheduler
Nov 21 20:28:17 UTC 2024	running	job `30162` is running
Nov 21 22:32:31 UTC 2024	finished	😢 FAILURE (click triangle for details) Details ✅ job output file `slurm-30162.out` ✅ no message matching `FATAL:` ❌ found message matching `ERROR:` ❌ found message matching `FAILED:` ❌ found message matching `required modules missing:` ❌ no message matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-amd-zen2-1732226990.tar.gz` size: 302 MiB (316730812 bytes) entries: 113 modules under 2023.06/software/linux/x86_64/amd/zen2/accel/nvidia/cc80/modules/all `magma/2.7.2-foss-2023a-CUDA-12.1.1.lua` software under 2023.06/software/linux/x86_64/amd/zen2/accel/nvidia/cc80/software `magma/2.7.2-foss-2023a-CUDA-12.1.1` other under 2023.06/software/linux/x86_64/amd/zen2/accel/nvidia/cc80 no other files in tarball
Nov 21 22:32:31 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ OK ] ( 1/10) EESSI_LAMMPS_lj %scale=1_node %device_type=cpu %module_name=LAMMPS/29Aug2024-foss-2023b-kokkos /aeb2d9df @BotBuildTests:x86-64-amd-zen2-node+default P: perf: 433.113 timesteps/s (r:0, l:None, u:None) [ OK ] ( 2/10) EESSI_LAMMPS_lj %scale=1_node %device_type=cpu %module_name=LAMMPS/2Aug2023_update2-foss-2023a-kokkos /04ff9ece @BotBuildTests:x86-64-amd-zen2-node+default P: perf: 443.086 timesteps/s (r:0, l:None, u:None) [ OK ] ( 3/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_allreduce %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /31ac6ab9 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 4.85 us (r:0, l:None, u:None) [ OK ] ( 4/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_allreduce %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /f3be40a2 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 4.52 us (r:0, l:None, u:None) [ OK ] ( 5/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_alltoall %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /10e66fba @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 9.31 us (r:0, l:None, u:None) [ OK ] ( 6/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_alltoall %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /5be57ae7 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 8.44 us (r:0, l:None, u:None) [ OK ] ( 7/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_latency %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /c8c9aff5 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 0.33 us (r:0, l:None, u:None) [ OK ] ( 8/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_latency %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /9795e491 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 0.31 us (r:0, l:None, u:None) [ OK ] ( 9/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_bw %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /48da21c5 @BotBuildTests:x86-64-amd-zen2-node+default P: bandwidth: 7756.73 MB/s (r:0, l:None, u:None) [ OK ] (10/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_bw %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /1b8c1ca2 @BotBuildTests:x86-64-amd-zen2-node+default P: bandwidth: 7744.15 MB/s (r:0, l:None, u:None) [ PASSED ] Ran 10/10 test case(s) from 10 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-30162.out` ❌ found message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

some /tmp/eb-m470fsqz/eb-r9zeygi0/tmpb1cr2ofk/rpath_wrappers/gxx_wrapper/g++ run failed with

/cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/usr/bin/ld: warning: libcupti.so.12, needed by lib/libtorch_cpu.so, not found (try using -rpath or -rpath-link)
/cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/usr/bin/ld: lib/libtorch_cpu.so: undefined reference to `cuptiGetResultString@libcupti.so.12'
/cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/usr/bin/ld: lib/libtorch_cpu.so: undefined reference to `cuptiMetricEnumEvents@libcupti.so.12'
...
/cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/usr/bin/ld: lib/libtorch_cpu.so: undefined reference to `cuptiSubscribe@libcupti.so.12'
collect2: error: ld returned 1 exit status

we should be able to fix this by adding the directory that contains libcupti to $LIBRARY_PATH in a pre_configure hook (see {2023.06}[foss/2023a] PyTorch v2.1.2 with CUDA/12.1.1 NorESSI/software-layer#369)

…PATH

trz42 · 2024-11-22T08:13:13Z

Build again after applying fix to find libcupti...

bot: build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 accel:nvidia/cc80

riscv-eessi-io-bot · 2024-11-22T08:13:16Z

Updates by the bot instance eessi-bot-riscv (click for details)

account trz42 has NO permission to send commands to the bot

eessi-bot · 2024-11-22T08:13:16Z

Updates by the bot instance eessi-bot-mc-aws (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 accel:nvidia/cc80 from trz42
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80 resulted in:
- submitted job 30339, for details & status see {2023.06}[foss/2023a] PyTorch v2.1.2 w/ CUDA 12.1.1 #825 (comment)

eessi-bot · 2024-11-22T08:13:16Z

Updates by the bot instance eessi-bot-mc-azure (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 accel:nvidia/cc80 from trz42
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80 resulted in:
- no jobs were submitted

eessi-bot · 2024-11-22T08:13:19Z

New job on instance eessi-bot-mc-aws for CPU micro-architecture x86_64-amd-zen2 and accelerator nvidia/cc80 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.11/pr_825/30339

date	job status	comment
Nov 22 08:13:18 UTC 2024	submitted	job id `30339` awaits release by job manager
Nov 22 08:13:44 UTC 2024	released	job awaits launch by Slurm scheduler
Nov 22 08:19:50 UTC 2024	running	job `30339` is running
Nov 22 18:13:32 UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-30339.out` ✅ no message matching `FATAL:` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-amd-zen2-1732297886.tar.gz` size: 509 MiB (534700273 bytes) entries: 12854 modules under 2023.06/software/linux/x86_64/amd/zen2/accel/nvidia/cc80/modules/all `magma/2.7.2-foss-2023a-CUDA-12.1.1.lua` `PyTorch/2.1.2-foss-2023a-CUDA-12.1.1.lua` software under 2023.06/software/linux/x86_64/amd/zen2/accel/nvidia/cc80/software `magma/2.7.2-foss-2023a-CUDA-12.1.1` `PyTorch/2.1.2-foss-2023a-CUDA-12.1.1` other under 2023.06/software/linux/x86_64/amd/zen2/accel/nvidia/cc80 `2023.06/init/easybuild/eb_hooks.py`
Nov 22 18:13:32 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ OK ] ( 1/10) EESSI_LAMMPS_lj %scale=1_node %device_type=cpu %module_name=LAMMPS/29Aug2024-foss-2023b-kokkos /aeb2d9df @BotBuildTests:x86-64-amd-zen2-node+default P: perf: 436.522 timesteps/s (r:0, l:None, u:None) [ OK ] ( 2/10) EESSI_LAMMPS_lj %scale=1_node %device_type=cpu %module_name=LAMMPS/2Aug2023_update2-foss-2023a-kokkos /04ff9ece @BotBuildTests:x86-64-amd-zen2-node+default P: perf: 446.803 timesteps/s (r:0, l:None, u:None) [ OK ] ( 3/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_allreduce %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /31ac6ab9 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 4.57 us (r:0, l:None, u:None) [ OK ] ( 4/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_allreduce %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /f3be40a2 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 4.69 us (r:0, l:None, u:None) [ OK ] ( 5/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_alltoall %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /10e66fba @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 8.67 us (r:0, l:None, u:None) [ OK ] ( 6/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_alltoall %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /5be57ae7 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 9.33 us (r:0, l:None, u:None) [ OK ] ( 7/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_latency %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /c8c9aff5 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 0.28 us (r:0, l:None, u:None) [ OK ] ( 8/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_latency %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /9795e491 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 0.3 us (r:0, l:None, u:None) [ OK ] ( 9/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_bw %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /48da21c5 @BotBuildTests:x86-64-amd-zen2-node+default P: bandwidth: 7868.44 MB/s (r:0, l:None, u:None) [ OK ] (10/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_bw %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /1b8c1ca2 @BotBuildTests:x86-64-amd-zen2-node+default P: bandwidth: 7743.76 MB/s (r:0, l:None, u:None) [ PASSED ] Ran 10/10 test case(s) from 10 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-30339.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

trz42 · 2024-11-22T18:21:18Z

Also build for zen3…

bot: build repo:eessi.io-2023.06-software arch:x86_64/amd/zen3 accel:nvidia/cc80

riscv-eessi-io-bot · 2024-11-22T18:21:21Z

Updates by the bot instance eessi-bot-riscv (click for details)

account trz42 has NO permission to send commands to the bot

eessi-bot · 2024-11-22T18:21:21Z

Updates by the bot instance eessi-bot-mc-aws (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen3 accel:nvidia/cc80 from trz42
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen3 accelerator:nvidia/cc80
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen3 accelerator:nvidia/cc80 resulted in:
- submitted job 30343, for details & status see {2023.06}[foss/2023a] PyTorch v2.1.2 w/ CUDA 12.1.1 #825 (comment)

riscv-eessi-io-bot · 2024-11-22T18:21:21Z

Updates by the bot instance eessi-bot-riscv (click for details)

account trz42 has NO permission to send commands to the bot

eessi-bot · 2024-11-22T18:21:21Z

Updates by the bot instance eessi-bot-mc-azure (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen3 accel:nvidia/cc80 from trz42
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen3 accelerator:nvidia/cc80
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen3 accelerator:nvidia/cc80 resulted in:
- no jobs were submitted

eessi-bot · 2024-11-22T18:21:25Z

New job on instance eessi-bot-mc-aws for CPU micro-architecture x86_64-amd-zen3 and accelerator nvidia/cc80 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.11/pr_825/30343

date	job status	comment
Nov 22 18:21:24 UTC 2024	submitted	job id `30343` awaits release by job manager
Nov 22 18:21:35 UTC 2024	released	job awaits launch by Slurm scheduler
Nov 22 18:27:37 UTC 2024	running	job `30343` is running
Nov 23 02:24:10 UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-30343.out` ✅ no message matching `FATAL:` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-amd-zen3-1732327591.tar.gz` size: 509 MiB (534708669 bytes) entries: 12854 modules under 2023.06/software/linux/x86_64/amd/zen3/accel/nvidia/cc80/modules/all `magma/2.7.2-foss-2023a-CUDA-12.1.1.lua` `PyTorch/2.1.2-foss-2023a-CUDA-12.1.1.lua` software under 2023.06/software/linux/x86_64/amd/zen3/accel/nvidia/cc80/software `magma/2.7.2-foss-2023a-CUDA-12.1.1` `PyTorch/2.1.2-foss-2023a-CUDA-12.1.1` other under 2023.06/software/linux/x86_64/amd/zen3/accel/nvidia/cc80 `2023.06/init/easybuild/eb_hooks.py`
Nov 23 02:24:10 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ OK ] ( 1/10) EESSI_LAMMPS_lj %scale=1_node %device_type=cpu %module_name=LAMMPS/29Aug2024-foss-2023b-kokkos /aeb2d9df @BotBuildTests:x86-64-amd-zen3-node+default P: perf: 522.848 timesteps/s (r:0, l:None, u:None) [ OK ] ( 2/10) EESSI_LAMMPS_lj %scale=1_node %device_type=cpu %module_name=LAMMPS/2Aug2023_update2-foss-2023a-kokkos /04ff9ece @BotBuildTests:x86-64-amd-zen3-node+default P: perf: 532.462 timesteps/s (r:0, l:None, u:None) [ OK ] ( 3/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_allreduce %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /31ac6ab9 @BotBuildTests:x86-64-amd-zen3-node+default P: latency: 2.42 us (r:0, l:None, u:None) [ OK ] ( 4/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_allreduce %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /f3be40a2 @BotBuildTests:x86-64-amd-zen3-node+default P: latency: 2.32 us (r:0, l:None, u:None) [ OK ] ( 5/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_alltoall %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /10e66fba @BotBuildTests:x86-64-amd-zen3-node+default P: latency: 5.5 us (r:0, l:None, u:None) [ OK ] ( 6/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_alltoall %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /5be57ae7 @BotBuildTests:x86-64-amd-zen3-node+default P: latency: 5.43 us (r:0, l:None, u:None) [ OK ] ( 7/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_latency %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /c8c9aff5 @BotBuildTests:x86-64-amd-zen3-node+default P: latency: 0.24 us (r:0, l:None, u:None) [ OK ] ( 8/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_latency %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /9795e491 @BotBuildTests:x86-64-amd-zen3-node+default P: latency: 0.22 us (r:0, l:None, u:None) [ OK ] ( 9/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_bw %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /48da21c5 @BotBuildTests:x86-64-amd-zen3-node+default P: bandwidth: 14283.69 MB/s (r:0, l:None, u:None) [ OK ] (10/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_bw %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /1b8c1ca2 @BotBuildTests:x86-64-amd-zen3-node+default P: bandwidth: 14294.01 MB/s (r:0, l:None, u:None) [ PASSED ] Ran 10/10 test case(s) from 10 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-30343.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

boegel · 2024-11-22T20:23:07Z

eb_hooks.py

+    if self.name == 'PyTorch' and self.version == '2.1.2':
+        if 'cudaver' in self.cfg.template_values and self.cfg.template_values['cudaver'] == '12.1.1':


I don't see a reason to make this specific to a particular PyTorch version or CUDA version?

We only know that the failure happens in this specific case. If we apply it to other cases, we will not know whether it was necessary or not.

boegel · 2024-11-22T20:24:08Z

eb_hooks.py

+            _cupti_lib_dir = os.path.join(_eessi_software_path, 'software', 'CUDA', _cudaver, 'extras', 'CUPTI', 'lib64')
+            print_msg("pre_configure_hook_pytorch_add_cupti_libdir: cupti_lib_dir: '%s'", _cupti_lib_dir)
+            if _library_path:
+                env.setvar('LIBRARY_PATH', ':'.join([_library_path, _cupti_lib_dir]))


This seems like a bug in our CUDA installation/module, no?

I'm fine with proceeding like this for now, even if we also fix it somewhere else this won't cause trouble, but there's probably a more general fix for this?

Right, we might find a better solution by changing the CUDA module, eg, by adding the directory to LIBRARY_PATH through the module.

It could be a worthwhile effort to try.

@boegel could it be that lib_path (at least parts of it) is missing in https://github.com/easybuilders/easybuild-easyblocks/blob/57c0eaed8dc29e223fe68a75f7bf195cca0c2d04/easybuild/easyblocks/c/cuda.py#L362

A little before that line lib_path is constructed as list ['lib64', 'extras/CUPTI/lib64', 'nvvm/lib64'], but in line 362 only ['lib64', 'stubs/lib64'] is used.

trz42 · 2024-11-23T10:37:09Z

Try a different approach where we rebuild the CUDA module such that it prepends the directory containing the libcupti library to LIBRARY_PATH and then not using the hook used in the previous builds...

bot: build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 accel:nvidia/cc80

riscv-eessi-io-bot · 2024-11-25T13:58:54Z

Updates by the bot instance eessi-bot-riscv (click for details)

account trz42 has NO permission to send commands to the bot

riscv-eessi-io-bot · 2024-11-25T13:58:54Z

Updates by the bot instance eessi-bot-riscv (click for details)

account trz42 has NO permission to send commands to the bot

eessi-bot · 2024-11-25T13:58:55Z

Updates by the bot instance eessi-bot-mc-azure (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 accel:nvidia/cc80 from trz42
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80 resulted in:
- no jobs were submitted

eessi-bot · 2024-11-25T13:58:58Z

New job on instance eessi-bot-mc-aws for CPU micro-architecture x86_64-amd-zen2 and accelerator nvidia/cc80 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.11/pr_825/30882

date	job status	comment
Nov 25 13:58:57 UTC 2024	submitted	job id `30882` awaits release by job manager
Nov 25 13:59:46 UTC 2024	released	job awaits launch by Slurm scheduler
Nov 25 14:05:49 UTC 2024	running	job `30882` is running
Nov 26 00:14:40 UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-30882.out` ✅ no message matching `FATAL:` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-amd-zen2-1732578090.tar.gz` size: 2577 MiB (2702396428 bytes) entries: 18370 modules under 2023.06/software/linux/x86_64/amd/zen2/accel/nvidia/cc80/modules/all `CUDA/12.1.1.lua` `magma/2.7.2-foss-2023a-CUDA-12.1.1.lua` `PyTorch/2.1.2-foss-2023a-CUDA-12.1.1.lua` software under 2023.06/software/linux/x86_64/amd/zen2/accel/nvidia/cc80/software `CUDA/12.1.1` `magma/2.7.2-foss-2023a-CUDA-12.1.1` `PyTorch/2.1.2-foss-2023a-CUDA-12.1.1` other under 2023.06/software/linux/x86_64/amd/zen2/accel/nvidia/cc80 `2023.06/init/easybuild/eb_hooks.py`
Nov 26 00:14:40 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ OK ] ( 1/10) EESSI_LAMMPS_lj %scale=1_node %device_type=cpu %module_name=LAMMPS/29Aug2024-foss-2023b-kokkos /aeb2d9df @BotBuildTests:x86-64-amd-zen2-node+default P: perf: 441.298 timesteps/s (r:0, l:None, u:None) [ OK ] ( 2/10) EESSI_LAMMPS_lj %scale=1_node %device_type=cpu %module_name=LAMMPS/2Aug2023_update2-foss-2023a-kokkos /04ff9ece @BotBuildTests:x86-64-amd-zen2-node+default P: perf: 438.418 timesteps/s (r:0, l:None, u:None) [ OK ] ( 3/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_allreduce %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /31ac6ab9 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 4.69 us (r:0, l:None, u:None) [ OK ] ( 4/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_allreduce %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /f3be40a2 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 4.63 us (r:0, l:None, u:None) [ OK ] ( 5/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_alltoall %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /10e66fba @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 8.83 us (r:0, l:None, u:None) [ OK ] ( 6/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_alltoall %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /5be57ae7 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 8.44 us (r:0, l:None, u:None) [ OK ] ( 7/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_latency %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /c8c9aff5 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 0.33 us (r:0, l:None, u:None) [ OK ] ( 8/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_latency %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /9795e491 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 0.31 us (r:0, l:None, u:None) [ OK ] ( 9/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_bw %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /48da21c5 @BotBuildTests:x86-64-amd-zen2-node+default P: bandwidth: 7899.95 MB/s (r:0, l:None, u:None) [ OK ] (10/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_bw %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /1b8c1ca2 @BotBuildTests:x86-64-amd-zen2-node+default P: bandwidth: 7899.8 MB/s (r:0, l:None, u:None) [ PASSED ] Ran 10/10 test case(s) from 10 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-30882.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

laraPPr · 2024-12-17T09:06:02Z

bot: help

eessi-bot · 2024-12-17T09:06:07Z

Updates by the bot instance eessi-bot-mc-aws (click for details)

received bot command help from laraPPr
- expanded format: help
handling command help resulted in:
How to send commands to bot instances
- Commands must be sent with a new comment (edits of existing comments are ignored).
- A comment may contain multiple commands, one per line.
- Every command begins at the start of a line and has the syntax bot: COMMAND [ARGUMENTS]*
- Currently supported COMMANDs are: help, build, show_config, status
For more information, see https://www.eessi.io/docs/bot

eessi-bot · 2024-12-17T09:06:07Z

Updates by the bot instance eessi-bot-mc-azure (click for details)

received bot command help from laraPPr
- expanded format: help
handling command help resulted in:
How to send commands to bot instances
- Commands must be sent with a new comment (edits of existing comments are ignored).
- A comment may contain multiple commands, one per line.
- Every command begins at the start of a line and has the syntax bot: COMMAND [ARGUMENTS]*
- Currently supported COMMANDs are: help, build, show_config, status
For more information, see https://www.eessi.io/docs/bot

riscv-eessi-io-bot · 2024-12-17T09:06:07Z

Updates by the bot instance eessi-bot-riscv (click for details)

account laraPPr has NO permission to send commands to the bot

trz42 · 2025-01-17T13:02:49Z

bot: show_config

eessi-bot · 2025-01-17T13:02:52Z

Updates by the bot instance eessi-bot-mc-aws (click for details)

received bot command show_config from trz42
- expanded format: show_config
handling command show_config resulted in:
- added comment {2023.06}[foss/2023a] PyTorch v2.1.2 w/ CUDA 12.1.1 #825 (comment) to show configuration

eessi-bot · 2025-01-17T13:02:52Z

Updates by the bot instance eessi-bot-mc-azure (click for details)

received bot command show_config from trz42
- expanded format: show_config
handling command show_config resulted in:
- added comment {2023.06}[foss/2023a] PyTorch v2.1.2 w/ CUDA 12.1.1 #825 (comment) to show configuration

eessi-bot · 2025-01-17T13:02:53Z

Instance eessi-bot-mc-aws is configured to build for:

architectures: x86_64/generic, x86_64/intel/haswell, x86_64/intel/sapphire_rapids, x86_64/intel/skylake_avx512, x86_64/amd/zen2, x86_64/amd/zen3, aarch64/generic, aarch64/neoverse_n1, aarch64/neoverse_v1
repositories: eessi.io-2023.06-compat, eessi.io-2023.06-software, eessi-hpc.org-2023.06-software, eessi-hpc.org-2023.06-compat

eessi-bot · 2025-01-17T13:02:55Z

Instance eessi-bot-mc-azure is configured to build for:

architectures: x86_64/amd/zen4
repositories: eessi.io-2023.06-compat, eessi.io-2023.06-software

trz42 · 2025-01-17T13:05:45Z

bot: show_config

eessi-bot · 2025-01-17T13:05:49Z

Updates by the bot instance eessi-bot-mc-aws (click for details)

received bot command show_config from trz42
- expanded format: show_config
handling command show_config resulted in:
- added comment {2023.06}[foss/2023a] PyTorch v2.1.2 w/ CUDA 12.1.1 #825 (comment) to show configuration

eessi-bot · 2025-01-17T13:05:49Z

Updates by the bot instance eessi-bot-mc-azure (click for details)

received bot command show_config from trz42
- expanded format: show_config
handling command show_config resulted in:
- added comment {2023.06}[foss/2023a] PyTorch v2.1.2 w/ CUDA 12.1.1 #825 (comment) to show configuration

eessi-bot · 2025-01-17T13:05:51Z

Instance eessi-bot-mc-aws is configured to build for:

architectures: x86_64/generic, x86_64/intel/haswell, x86_64/intel/sapphire_rapids, x86_64/intel/skylake_avx512, x86_64/amd/zen2, x86_64/amd/zen3, aarch64/generic, aarch64/neoverse_n1, aarch64/neoverse_v1
repositories: eessi.io-2023.06-software, eessi.io-2023.06-compat

eessi-bot · 2025-01-17T13:05:52Z

Instance eessi-bot-mc-azure is configured to build for:

architectures: x86_64/amd/zen4
repositories: eessi.io-2023.06-software, eessi.io-2023.06-compat

laraPPr · 2025-02-17T09:33:48Z

bot: build instance:eessi-bot-vsc-ugent repo:eessi.io-2023.06-software accel:nvidia/cc80

eessi-bot · 2025-02-17T09:33:51Z

Updates by the bot instance eessi-bot-mc-aws (click for details)

received bot command build instance:eessi-bot-vsc-ugent repo:eessi.io-2023.06-software accel:nvidia/cc80 from laraPPr
- expanded format: build instance:eessi-bot-vsc-ugent repository:eessi.io-2023.06-software accelerator:nvidia/cc80
handling command build instance:eessi-bot-vsc-ugent repository:eessi.io-2023.06-software accelerator:nvidia/cc80 resulted in:
- no jobs were submitted

eessi-bot · 2025-02-17T09:33:52Z

Updates by the bot instance eessi-bot-mc-azure (click for details)

received bot command build instance:eessi-bot-vsc-ugent repo:eessi.io-2023.06-software accel:nvidia/cc80 from laraPPr
- expanded format: build instance:eessi-bot-vsc-ugent repository:eessi.io-2023.06-software accelerator:nvidia/cc80
handling command build instance:eessi-bot-vsc-ugent repository:eessi.io-2023.06-software accelerator:nvidia/cc80 resulted in:
- no jobs were submitted

riscv-eessi-io-bot · 2025-02-17T09:33:52Z

Updates by the bot instance eessi-bot-riscv (click for details)

account laraPPr has NO permission to send commands to the bot

gpu-bot-ugent · 2025-02-17T09:33:52Z

Updates by the bot instance eessi-bot-vsc-ugent (click for details)

received bot command build instance:eessi-bot-vsc-ugent repo:eessi.io-2023.06-software accel:nvidia/cc80 from laraPPr
- expanded format: build instance:eessi-bot-vsc-ugent repository:eessi.io-2023.06-software accelerator:nvidia/cc80

eessi-bot-surf · 2025-02-17T09:33:52Z

Updates by the bot instance eessi-bot-surf (click for details)

account laraPPr has NO permission to send commands to the bot

eessi-bot-toprichard · 2025-02-17T09:33:52Z

Updates by the bot instance rt-Grace-jr (click for details)

account laraPPr has NO permission to send commands to the bot

gpu-bot-ugent · 2025-02-17T09:33:56Z

error: easystacks/software.eessi.io/2023.06/accel/nvidia/eessi-2023.06-eb-4.9.4-2023a-CUDA.yml: patch does not apply

Unable to download or merge changes between the source branch and the destination branch.
Tip: This can usually be resolved by syncing your branch and resolving any merge conflicts.

laraPPr · 2025-06-27T14:22:11Z

@trz42 Can you split up and retarget this pr?

trz42 · 2025-08-13T09:58:13Z

Superseded by #973

DEBUG {2023.06}[foss/2023a] PyTorch v2.1.2 w/ CUDA 12.1.1

e4f37fe

trz42 added 2023.06-software.eessi.io 2023.06 version of software.eessi.io accel:nvidia labels Nov 21, 2024

add pre_configure hook for PyTorch to add CUPTI lib64 dir to LIBRARY_…

4504675

…PATH

trz42 changed the title ~~DEBUG {2023.06}[foss/2023a] PyTorch v2.1.2 w/ CUDA 12.1.1~~ {2023.06}[foss/2023a] PyTorch v2.1.2 w/ CUDA 12.1.1 Nov 22, 2024

boegel requested changes Nov 22, 2024

View reviewed changes

truib added 2 commits November 23, 2024 11:30

add cupti lib dir to LIBRARY_PATH in CUDA module

6140430

rebuild CUDA with added CUPTI lib dir

eff0de6

trz42 mentioned this pull request Mar 20, 2025

{2023.06}[2023a] PyTorch v2.1.2 with CUDA/12.1.1 #973

Open

trz42 closed this Aug 13, 2025

		if self.name == 'PyTorch' and self.version == '2.1.2':
		if 'cudaver' in self.cfg.template_values and self.cfg.template_values['cudaver'] == '12.1.1':

Conversation

trz42 commented Nov 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eessi-bot bot commented Nov 21, 2024

Uh oh!

riscv-eessi-io-bot bot commented Nov 21, 2024

Uh oh!

riscv-eessi-io-bot bot commented Nov 21, 2024

Uh oh!

eessi-bot bot commented Nov 21, 2024

Uh oh!

trz42 commented Nov 21, 2024

Uh oh!

riscv-eessi-io-bot bot commented Nov 21, 2024

Uh oh!

riscv-eessi-io-bot bot commented Nov 21, 2024

Uh oh!

eessi-bot bot commented Nov 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eessi-bot bot commented Nov 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eessi-bot bot commented Nov 21, 2024 • edited by trz42 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

trz42 commented Nov 22, 2024

Uh oh!

riscv-eessi-io-bot bot commented Nov 22, 2024

Uh oh!

eessi-bot bot commented Nov 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eessi-bot bot commented Nov 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eessi-bot bot commented Nov 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

trz42 commented Nov 22, 2024

Uh oh!

riscv-eessi-io-bot bot commented Nov 22, 2024

Uh oh!

eessi-bot bot commented Nov 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

riscv-eessi-io-bot bot commented Nov 22, 2024

Uh oh!

eessi-bot bot commented Nov 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eessi-bot bot commented Nov 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

boegel Nov 22, 2024

Choose a reason for hiding this comment

Uh oh!

trz42 Nov 22, 2024

Choose a reason for hiding this comment

Uh oh!

boegel Nov 22, 2024

Choose a reason for hiding this comment

Uh oh!

trz42 Nov 22, 2024

Choose a reason for hiding this comment

Uh oh!

trz42 Nov 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

trz42 commented Nov 23, 2024

Uh oh!

riscv-eessi-io-bot bot commented Nov 25, 2024

Uh oh!

riscv-eessi-io-bot bot commented Nov 25, 2024

Uh oh!

eessi-bot bot commented Nov 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

trz42 commented Nov 21, 2024 •

edited

Loading

eessi-bot bot commented Nov 21, 2024 •

edited

Loading

eessi-bot bot commented Nov 21, 2024 •

edited

Loading

eessi-bot bot commented Nov 21, 2024 •

edited by trz42

Loading

eessi-bot bot commented Nov 22, 2024 •

edited

Loading

eessi-bot bot commented Nov 22, 2024 •

edited

Loading

eessi-bot bot commented Nov 22, 2024 •

edited

Loading

eessi-bot bot commented Nov 22, 2024 •

edited

Loading

eessi-bot bot commented Nov 22, 2024 •

edited

Loading

eessi-bot bot commented Nov 22, 2024 •

edited

Loading

trz42 Nov 23, 2024 •

edited

Loading

eessi-bot bot commented Nov 25, 2024 •

edited

Loading

eessi-bot bot commented Nov 25, 2024 •

edited

Loading

eessi-bot bot commented Dec 17, 2024 •

edited

Loading

eessi-bot bot commented Dec 17, 2024 •

edited

Loading

eessi-bot bot commented Jan 17, 2025 •

edited

Loading

eessi-bot bot commented Jan 17, 2025 •

edited

Loading

eessi-bot bot commented Jan 17, 2025 •

edited

Loading

eessi-bot bot commented Jan 17, 2025 •

edited

Loading

eessi-bot bot commented Feb 17, 2025 •

edited

Loading

eessi-bot bot commented Feb 17, 2025 •

edited

Loading