Skip to content

fatal error: cuda_runtime.h: No such file or directory #3

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
Daiver opened this issue May 26, 2018 · 8 comments
Closed

fatal error: cuda_runtime.h: No such file or directory #3

Daiver opened this issue May 26, 2018 · 8 comments

Comments

@Daiver
Copy link

Daiver commented May 26, 2018 •
edited
Loading

Hello!

First of all i'm sorry for my poor English.
I just tried to install torch-unique and received following message:

➜  ~ sudo pip3 install --upgrade torch-unique

The directory '/home/daiver/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/daiver/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting torch-unique
  Downloading https://files.pythonhosted.org/packages/ad/8c/3ea7b6242c9c6a30e812fe172b2ef631520c24184565b51b17ab2c3cbdab/torch_unique-1.0.1.tar.gz
Collecting numpy (from torch-unique)
  Downloading https://files.pythonhosted.org/packages/71/90/ca61e203e0080a8cef7ac21eca199829fa8d997f7c4da3e985b49d0a107d/numpy-1.14.3-cp36-cp36m-manylinux1_x86_64.whl (12.2MB)
    100% |████████████████████████████████| 12.2MB 164kB/s 
Installing collected packages: numpy, torch-unique
  Found existing installation: numpy 1.14.2
    Uninstalling numpy-1.14.2:
      Successfully uninstalled numpy-1.14.2
  Running setup.py install for torch-unique ... error
    Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-_t98woba/torch-unique/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-4xwidqu4-record/install-record.txt --single-version-externally-managed --compile:
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.6
    creating build/lib.linux-x86_64-3.6/test
    copying test/utils.py -> build/lib.linux-x86_64-3.6/test
    copying test/test_unique.py -> build/lib.linux-x86_64-3.6/test
    copying test/__init__.py -> build/lib.linux-x86_64-3.6/test
    creating build/lib.linux-x86_64-3.6/torch_unique
    copying torch_unique/unique.py -> build/lib.linux-x86_64-3.6/torch_unique
    copying torch_unique/__init__.py -> build/lib.linux-x86_64-3.6/torch_unique
    running build_ext
    building 'unique_cuda' extension
    creating build/temp.linux-x86_64-3.6
    creating build/temp.linux-x86_64-3.6/cuda
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fdebug-prefix-map=/build/python3.6-sXpGnM/python3.6-3.6.3=. -specs=/usr/share/dpkg/no-pie-compile.specs -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/local/lib/python3.6/dist-packages/torch/lib/include -I/usr/local/lib/python3.6/dist-packages/torch/lib/include/TH -I/usr/local/lib/python3.6/dist-packages/torch/lib/include/THC -I/usr/local/cuda/include -I/usr/include/python3.6m -c cuda/unique.cpp -o build/temp.linux-x86_64-3.6/cuda/unique.o -DTORCH_EXTENSION_NAME=unique_cuda -std=c++11
    cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
    /usr/local/cuda/bin/nvcc -I/usr/local/lib/python3.6/dist-packages/torch/lib/include -I/usr/local/lib/python3.6/dist-packages/torch/lib/include/TH -I/usr/local/lib/python3.6/dist-packages/torch/lib/include/THC -I/usr/local/cuda/include -I/usr/include/python3.6m -c cuda/unique_kernel.cu -o build/temp.linux-x86_64-3.6/cuda/unique_kernel.o -DTORCH_EXTENSION_NAME=unique_cuda --compiler-options '-fPIC' -std=c++11
    cc1plus: fatal error: cuda_runtime.h: No such file or directory
    compilation terminated.
    error: command '/usr/local/cuda/bin/nvcc' failed with exit status 1
    
    ----------------------------------------
Command "/usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-_t98woba/torch-unique/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-4xwidqu4-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-_t98woba/torch-unique/

I understand that i have some problem with proper cuda paths but i just cannot figure out how to solve it. Can you help me?

➜  ~ cat /etc/issue
Ubuntu 17.10 \n \l

➜  ~ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2017 NVIDIA Corporation
Built on Fri_Nov__3_21:07:56_CDT_2017
Cuda compilation tools, release 9.1, V9.1.85
➜  ~ 
➜  ~ gcc --version
gcc (Ubuntu 7.2.0-8ubuntu3.2) 7.2.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

I have following lines in my .zshrc:

export PATH=$PATH:/usr/local/cuda-9.1/bin
export CUDA_HOME=/usr/local/cuda-9.1

Also i made ln -s cuda-9.1 cuda inside my /usr/local directory, so i have a valid /usr/local/cuda/ path.

As i can see Torch/Tensorflow stuff works well so at least some of my paths are ok.

I can provide any additional information if it is needed. Sorry for disturbing you.
@rusty1s
Copy link
Owner

rusty1s commented May 26, 2018 •
edited
Loading

Hello Klimov,

your CUDA setup looks good to me. There are a number of people who cannot install torch-unique via pip. I will investigate this soon!

In the meantime: Can you clone the repository and run python setup.py install? This should work :)

@Daiver
Copy link
Author

Daiver commented May 26, 2018 

➜  pytorch_unique git:(master) sudo python3 setup.py install
[sudo] password for daiver: 
running install
running bdist_egg
running egg_info
creating torch_unique.egg-info
writing torch_unique.egg-info/PKG-INFO
writing dependency_links to torch_unique.egg-info/dependency_links.txt
writing requirements to torch_unique.egg-info/requires.txt
writing top-level names to torch_unique.egg-info/top_level.txt
writing manifest file 'torch_unique.egg-info/SOURCES.txt'
reading manifest file 'torch_unique.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'torch_unique.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build
creating build/lib.linux-x86_64-3.6
creating build/lib.linux-x86_64-3.6/test
copying test/utils.py -> build/lib.linux-x86_64-3.6/test
copying test/test_unique.py -> build/lib.linux-x86_64-3.6/test
copying test/__init__.py -> build/lib.linux-x86_64-3.6/test
creating build/lib.linux-x86_64-3.6/torch_unique
copying torch_unique/unique.py -> build/lib.linux-x86_64-3.6/torch_unique
copying torch_unique/__init__.py -> build/lib.linux-x86_64-3.6/torch_unique
running build_ext
building 'unique_cuda' extension
creating build/temp.linux-x86_64-3.6
creating build/temp.linux-x86_64-3.6/cuda
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fdebug-prefix-map=/build/python3.6-sXpGnM/python3.6-3.6.3=. -specs=/usr/share/dpkg/no-pie-compile.specs -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/local/lib/python3.6/dist-packages/torch/lib/include -I/usr/local/lib/python3.6/dist-packages/torch/lib/include/TH -I/usr/local/lib/python3.6/dist-packages/torch/lib/include/THC -I/usr/local/cuda/include -I/usr/include/python3.6m -c cuda/unique.cpp -o build/temp.linux-x86_64-3.6/cuda/unique.o -DTORCH_EXTENSION_NAME=unique_cuda -std=c++11
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
/usr/local/cuda/bin/nvcc -I/usr/local/lib/python3.6/dist-packages/torch/lib/include -I/usr/local/lib/python3.6/dist-packages/torch/lib/include/TH -I/usr/local/lib/python3.6/dist-packages/torch/lib/include/THC -I/usr/local/cuda/include -I/usr/include/python3.6m -c cuda/unique_kernel.cu -o build/temp.linux-x86_64-3.6/cuda/unique_kernel.o -DTORCH_EXTENSION_NAME=unique_cuda --compiler-options '-fPIC' -std=c++11
cc1plus: fatal error: cuda_runtime.h: No such file or directory
compilation terminated.
error: command '/usr/local/cuda/bin/nvcc' failed with exit status 1
➜  pytorch_unique git:(master)

(Not sure about sudo but setup.py needs access for /usr/local/lib/python3.6/dist-packages/test-easy-install-11925.write-test)

Offtop - i need to leave my computer for some time so it looks like I can only answer tomorrow
Offtop2: My first name is Kirill, Klimov is last name =)

@rusty1s
Copy link
Owner

rusty1s commented May 26, 2018

Two possible solutions:

  • Do not use your system python which requires sudo (look about Miniconda)
  • Run CPATH=/usr/local/cuda/include python setup.py install

@Daiver
Copy link
Author

Daiver commented May 28, 2018 •
edited
Loading

Hi! I'm sorry for delay. It's looks like it's completely my fault. I have two cuda folders - cuda-9.1 and cuda-9.0. I created symlink to 9.1 folder by hands. And there is no cuda_runtime.h inside cuda-9.1 folder! But cuda-9.0 has it. I failed to compile torch-unique with cuda-9.0 because of gcc7 but it's temporary, i hope.
Currently i trying to properly install cuda 9.1 (because i don't want switch to gcc6). I will close issue if compilation of unique will be ok.
Sorry for disturbing you.

@Daiver
Copy link
Author

Daiver commented May 28, 2018 •
edited
Loading

I changed default gcc version to gcc-6 and installed cuda-9.1 properly. Also i reinstall pytorch via pip3 with cuda9.1.
But i still have a problem =( (is it good idea to keep long template instantiation errors in gist?)
https://gist.github.com/Daiver/7d26ea3f9825e461aa581c80b2010cbf

➜  pytorch_unique git:(master) gcc --version
gcc (Ubuntu 6.4.0-8ubuntu1) 6.4.0 20171010
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Maybe my torch version too high?

>>> torch.__version__
'0.4.0'

@rusty1s rusty1s mentioned this issue May 28, 2018
Closed
@rusty1s
Copy link
Owner

rusty1s commented May 28, 2018 •
edited
Loading

torch-unique uses the new Extension API introduced in Pytorch 0.4.0.
Can you verify that the examples in https://github.com/pytorch/extension-cpp are running on your machine?

@Daiver
Copy link
Author

Daiver commented May 28, 2018

I cloned repository and tried to compile some stuff in cpp and cuda folders

Cpp looks ok:

➜  cpp git:(master) sudo python3 setup.py install
running install
running bdist_egg
running egg_info
writing lltm_cpp.egg-info/PKG-INFO
writing dependency_links to lltm_cpp.egg-info/dependency_links.txt
writing top-level names to lltm_cpp.egg-info/top_level.txt
reading manifest file 'lltm_cpp.egg-info/SOURCES.txt'
writing manifest file 'lltm_cpp.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_ext
creating build/bdist.linux-x86_64/egg
copying build/lib.linux-x86_64-3.6/lltm_cpp.cpython-36m-x86_64-linux-gnu.so -> build/bdist.linux-x86_64/egg
creating stub loader for lltm_cpp.cpython-36m-x86_64-linux-gnu.so
byte-compiling build/bdist.linux-x86_64/egg/lltm_cpp.py to lltm_cpp.cpython-36.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying lltm_cpp.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying lltm_cpp.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying lltm_cpp.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying lltm_cpp.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
writing build/bdist.linux-x86_64/egg/EGG-INFO/native_libs.txt
zip_safe flag not set; analyzing archive contents...
__pycache__.lltm_cpp.cpython-36: module references __file__
creating 'dist/lltm_cpp-0.0.0-py3.6-linux-x86_64.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing lltm_cpp-0.0.0-py3.6-linux-x86_64.egg
removing '/usr/local/lib/python3.6/dist-packages/lltm_cpp-0.0.0-py3.6-linux-x86_64.egg' (and everything under it)
creating /usr/local/lib/python3.6/dist-packages/lltm_cpp-0.0.0-py3.6-linux-x86_64.egg
Extracting lltm_cpp-0.0.0-py3.6-linux-x86_64.egg to /usr/local/lib/python3.6/dist-packages
lltm-cpp 0.0.0 is already the active version in easy-install.pth

Installed /usr/local/lib/python3.6/dist-packages/lltm_cpp-0.0.0-py3.6-linux-x86_64.egg
Processing dependencies for lltm-cpp==0.0.0
Finished processing dependencies for lltm-cpp==0.0.0
➜  cpp git:(master) ..
➜  extension-cpp git:(master) python3 check.py backward
Backward: Baseline (Python) vs. C++ ... Ok

CUDA fails

➜  extension-cpp git:(master) cd cuda 
➜  cuda git:(master) sudo python3 setup.py install
running install
running bdist_egg
running egg_info
writing lltm_cuda.egg-info/PKG-INFO
writing dependency_links to lltm_cuda.egg-info/dependency_links.txt
writing top-level names to lltm_cuda.egg-info/top_level.txt
reading manifest file 'lltm_cuda.egg-info/SOURCES.txt'
writing manifest file 'lltm_cuda.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_ext
building 'lltm_cuda' extension
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fdebug-prefix-map=/build/python3.6-sXpGnM/python3.6-3.6.3=. -specs=/usr/share/dpkg/no-pie-compile.specs -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/local/lib/python3.6/dist-packages/torch/lib/include -I/usr/local/lib/python3.6/dist-packages/torch/lib/include/TH -I/usr/local/lib/python3.6/dist-packages/torch/lib/include/THC -I/usr/local/cuda/include -I/usr/include/python3.6m -c lltm_cuda.cpp -o build/temp.linux-x86_64-3.6/lltm_cuda.o -DTORCH_EXTENSION_NAME=lltm_cuda -std=c++11
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
/usr/local/cuda/bin/nvcc -I/usr/local/lib/python3.6/dist-packages/torch/lib/include -I/usr/local/lib/python3.6/dist-packages/torch/lib/include/TH -I/usr/local/lib/python3.6/dist-packages/torch/lib/include/THC -I/usr/local/cuda/include -I/usr/include/python3.6m -c lltm_cuda_kernel.cu -o build/temp.linux-x86_64-3.6/lltm_cuda_kernel.o -DTORCH_EXTENSION_NAME=lltm_cuda --compiler-options '-fPIC' -std=c++11
lltm_cuda_kernel.cu(54): error: calling a __host__ function("std::fmax<double, float> ") from a __global__ function("_NV_ANON_NAMESPACE::lltm_cuda_forward_kernel<float> ") is not allowed

lltm_cuda_kernel.cu(54): error: identifier "std::fmax<double, float> " is undefined in device code

2 errors detected in the compilation of "/tmp/tmpxft_00005f63_00000000-6_lltm_cuda_kernel.cpp1.ii".
error: command '/usr/local/cuda/bin/nvcc' failed with exit status 1

@rusty1s
Copy link
Owner

rusty1s commented Aug 21, 2018

This may be related: pytorch/extension-cpp#14

@rusty1s rusty1s closed this as completed Aug 21, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Daiver @rusty1s