Build Issues with Cray Compilers for CUDA backend

[abagusetty]

Describe the bug

Hi,

I have had success in building the CUDA and HIP backend with Cray compilers in the past on SLES 15-SP5 OS. Using the following build script lead me to some build issues on ALCF Polaris. It shouldn't matter though but tried with both Ninja and gmake builds and both pointed me to the same error. I am not sure if this has anything to do with GLIBC versions mismatch or GCC-toolchain path etc.

CUDA_LIB_PATH=/soft/compilers/cudatoolkit/cuda-12.6.1 python3 $DPCPP_HOME/llvm/buildbot/configure.py -o /soft/compilers/oneapi/upstream/install-${cuda_version}_${NOW} --cmake-opt="-DCMAKE_C_COMPILER=cc" --cmake-opt="-DCMAKE_CXX_COMPILER=CC" --native_cpu --cuda --llvm-external-projects openmp
cd /soft/compilers/oneapi/upstream/install-${cuda_version}_${NOW}
ninja sycl-toolchain

Error:

/usr/bin/ld: /usr/lib64/gcc/x86_64-suse-linux/12/../../../../lib64/Scrt1.o: in function `_start':
/home/abuild/rpmbuild/BUILD/glibc-2.31/csu/../sysdeps/x86_64/start.S:104:(.text+0x20): undefined reference to `main'

Verbose output:

: && /opt/cray/pe/craype/2.7.30/bin/CC -fPIC -fPIC -fno-semantic-interposition -fvisibility-inlines-hidden -Werror=date-time -fno-lifetime-dse -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wno-missing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wno-maybe-uninitialized -Wno-nonnull -Wno-class-memaccess -Wno-redundant-move -Wno-pessimizing-move -Wno-noexcept-type -Wdelete-non-virtual-dtor -Wsuggest-override -Wno-comment -Wno-misleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -ffunction-sections -fdata-sections -Wno-error -O3 -DNDEBUG  -Wl,-z,defs -Wl,-z,nodelete   -Wl,-z,relro,-z,now,-z,noexecstack -pie -Wl,--version-script=/soft/compilers/oneapi/upstream/install-12.6.1_01162025/_deps/unified-runtime-build/source/adapters/opencl/ur_adapter_opencl.map -shared -Wl,-soname,libur_adapter_opencl.so.0 -o lib/libur_adapter_opencl.so.0.12.0 _deps/unified-runtime-build/source/adapters/opencl/CMakeFiles/ur_adapter_opencl.dir/ur_interface_loader.cpp.o _deps/unified-runtime-build/source/adapters/opencl/CMakeFiles/ur_adapter_opencl.dir/adapter.cpp.o _deps/unified-runtime-build/source/adapters/opencl/CMakeFiles/ur_adapter_opencl.dir/command_buffer.cpp.o _deps/unified-runtime-build/source/adapters/opencl/CMakeFiles/ur_adapter_opencl.dir/common.cpp.o _deps/unified-runtime-build/source/adapters/opencl/CMakeFiles/ur_adapter_opencl.dir/context.cpp.o _deps/unified-runtime-build/source/adapters/opencl/CMakeFiles/ur_adapter_opencl.dir/device.cpp.o _deps/unified-runtime-build/source/adapters/opencl/CMakeFiles/ur_adapter_opencl.dir/enqueue.cpp.o _deps/unified-runtime-build/source/adapters/opencl/CMakeFiles/ur_adapter_opencl.dir/enqueue_native.cpp.o _deps/unified-runtime-build/source/adapters/opencl/CMakeFiles/ur_adapter_opencl.dir/event.cpp.o _deps/unified-runtime-build/source/adapters/opencl/CMakeFiles/ur_adapter_opencl.dir/image.cpp.o _deps/unified-runtime-build/source/adapters/opencl/CMakeFiles/ur_adapter_opencl.dir/kernel.cpp.o _deps/unified-runtime-build/source/adapters/opencl/CMakeFiles/ur_adapter_opencl.dir/memory.cpp.o _deps/unified-runtime-build/source/adapters/opencl/CMakeFiles/ur_adapter_opencl.dir/physical_mem.cpp.o _deps/unified-runtime-build/source/adapters/opencl/CMakeFiles/ur_adapter_opencl.dir/platform.cpp.o _deps/unified-runtime-build/source/adapters/opencl/CMakeFiles/ur_adapter_opencl.dir/program.cpp.o _deps/unified-runtime-build/source/adapters/opencl/CMakeFiles/ur_adapter_opencl.dir/queue.cpp.o _deps/unified-runtime-build/source/adapters/opencl/CMakeFiles/ur_adapter_opencl.dir/sampler.cpp.o _deps/unified-runtime-build/source/adapters/opencl/CMakeFiles/ur_adapter_opencl.dir/tensor_map.cpp.o _deps/unified-runtime-build/source/adapters/opencl/CMakeFiles/ur_adapter_opencl.dir/usm.cpp.o _deps/unified-runtime-build/source/adapters/opencl/CMakeFiles/ur_adapter_opencl.dir/usm_p2p.cpp.o _deps/unified-runtime-build/source/adapters/opencl/CMakeFiles/ur_adapter_opencl.dir/virtual_mem.cpp.o _deps/unified-runtime-build/source/adapters/opencl/CMakeFiles/ur_adapter_opencl.dir/__/__/__/__/unified-runtime-src/source/ur/ur.cpp.o _deps/unified-runtime-build/source/adapters/opencl/CMakeFiles/ur_adapter_opencl.dir/__/__/__/__/unified-runtime-src/source/common/umf_pools/disjoint_pool_config_parser.cpp.o  -Wl,-rpath,/soft/compilers/oneapi/upstream/install-12.6.1_01162025/lib:  -lstdc++fs  lib/libur_common.a  lib/libOpenCL.so.1.0.0  lib/libdisjoint_pool.a  lib/libumf.so.0.10.1  -lstdc++fs  -lrt  -ldl && :
/usr/bin/ld: /usr/lib64/gcc/x86_64-suse-linux/12/../../../../lib64/Scrt1.o: in function `_start':
/home/abuild/rpmbuild/BUILD/glibc-2.31/csu/../sysdeps/x86_64/start.S:104:(.text+0x20): undefined reference to `main'

Compiler versions:

/opt/cray/pe/craype/2.7.30/bin/CC --version
g++-12 (SUSE Linux) 12.3.0
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

/opt/cray/pe/craype/2.7.30/bin/cc --version
gcc-12 (SUSE Linux) 12.3.0
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

To reproduce

No response

Environment

• OS: SLES 15-SP5
• Target device and vendor: Nvidia A100, CUDA 12.6.1
• DPC++ version: [e.g. commit hash or output of clang++ --version]
• Dependencies version: [e.g. the output of sycl-ls --verbose]

Additional context

No response

[aelovikov-intel]

-o lib/libur_adapter_opencl.so.0.12.0 suggests it's coming from the UR. + @intel/unified-runtime-reviewers

[kbenzie]

libur_adapter_opencl.so.0.12.0 is a shared object so it should not be expected to contain the symbol main.

What linker is being used in this configuration?

[abagusetty]

@kbenzie It's a c++ linker /opt/cray/pe/craype/2.7.30/bin/CC pointing to g++ v12.3

[kbenzie]

That's the compiler not the linker, the compiler invokes the linker to link objects together. The linker exists at /usr/bin/ld and may be a symlink pointing to another executable, could you share that please?

[DuncanMcBain]

Hi @abagusetty, we think we've found the issue here: in the UR CMake, the flag -pie is set unconditionally and should probably not be set for shared libraries. If you'd like to verify this, the following patch should work:

diff --git a/cmake/helpers.cmake b/cmake/helpers.cmake
index bdd4a6d1..78b7df51 100644
--- a/cmake/helpers.cmake
+++ b/cmake/helpers.cmake
@@ -158,9 +158,9 @@ function(add_ur_target_link_options name)
                 target_link_options(${name} PRIVATE -Werror -Wextra)
             endif()
             if (CMAKE_BUILD_TYPE STREQUAL "Release")
-                target_link_options(${name} PRIVATE
-                    $<$<CXX_COMPILER_ID:GNU>:-pie>
-                )
+                   #target_link_options(${name} PRIVATE
+                    #    $<$<CXX_COMPILER_ID:GNU>:-pie>
+                    #)
             endif()
         endif()
     elseif(MSVC)

From your build directory, you can apply this in the folder _deps/unified-runtime-src/

@kbenzie is going to work on a proper fix in the Unified Runtime source.