Custom CUDA path not supported in Makefile

[Nowam]

Tried installing the evo2 package and it failed when trying to build vortex, seems the issue is here:

vortex/Makefile

Line 10 in f243e8e

CUDA_PATH := /usr/local/cuda

Since my cuda path is different.
Is there a reason this doesn't default to the CUDA_HOME \ CUDA_PATH set in the environment?

Thanks!

[bio-punk]

I am really impressed that the project can be published in "Quantum Bit", "Machine Heart" and "New Intelligence" even if it is written like this makefile.

[NilaBlueshirt]

I spent a lot of time trying to make the installation work, here are some changes and findings I collected along the way:

For the Makefile, I have these edits:

PYTHON_VERSION := $(shell $(PYTHON) -c 'import sys; print(sys.version_info[0])')
TARGET_PYTHON_VERSION := 3.11

_detect_cuda_path:
ifndef CUDA_PATH
    NVCC_BIN := $(shell which nvcc 2>/dev/null)
    ifneq ($(NVCC_BIN),)
        # Derive CUDA_PATH by going one directory up from nvcc's bin directory
        CUDA_PATH := $(abspath $(dir $(NVCC_BIN))/..)
    else
        CUDA_PATH := /usr/local/cuda
    endif
endif

CUDA_HOME := $(CUDA_PATH)
CUDA_INCLUDE_PATH := $(CUDA_PATH)/include
CUDA_LIB_PATH := $(CUDA_PATH)/lib64
CPATH := $(CUDA_INCLUDE_PATH)
CUDACXX := $(CUDA_PATH)/bin/nvcc

However I also needed a CUDNN_PATH.

According to this "common errors and fixes" from transformer_engine: NVIDIA/TransformerEngine#355 (comment)

and the transformer_engine doc: https://docs.nvidia.com/deeplearning/transformer-engine/user-guide/installation.html

to install transformer_engine, I have these variables set manually:

export MAX_JOBS=1
export NVTE_BUILD_THREADS_PER_JOB=1
export NVTE_FRAMEWORK=pytorch 

I found that manually pip install the transformer_engine and flash_attn is better than installing them from vortex. Doing the installation on an H100 GPU is a must, due to the transformer_engine FP8 format.

I hope these help and good luck folks!

[bio-punk]

export MAX_JOBS=1
export NVTE_BUILD_THREADS_PER_JOB=1
export NVTE_FRAMEWORK=pytorch 

It works in transformer_engine, in file "transformer_engine/common/CMakeLists.txt" builder will check enviroment variable MAX_JOB to control ninja parallel compliance.
And I also change setup.py to support RTX4090

    # archs = cuda_archs()
    archs="89"

transformer_engine can be built by "python setup.py bdist_wheel -d ../"
In 4090, 1 gpu and 7b model works well, but multi gpu cannot run
but I cannot build evo2 binary wheel

[lazyy11]

Here's how I resolved it:

Step 1: Fixed the CUDA paths in Makefile

I identified that the default paths in the Makefile were pointing to /usr/local/cuda, while my Ubuntu system had CUDA installed in different locations:

PYTHON_VERSION := $(shell $(PYTHON) -c 'import sys; print(sys.version_info[0])')
TARGET_PYTHON_VERSION := 3.11
- CUDA_PATH := /usr/local/cuda
+ CUDA_PATH := /usr/lib/cuda
CUDA_INCLUDE_PATH := $(CUDA_PATH)/include
- CUDA_LIB_PATH := $(CUDA_PATH)/lib64
- CPATH := $(CUDA_INCLUDE_PATH):/usr/local/cuda/include
- CUDACXX := /usr/local/cuda/bin/nvcc
+ CUDA_LIB_PATH := /usr/lib/x86_64-linux-gnu
+ CPATH := $(CUDA_INCLUDE_PATH):/usr/include/cuda
+ CUDACXX := /usr/lib/nvidia-cuda-toolkit/bin/nvcc
CUDA_HOME := $(CUDA_PATH)

Step 2: Added required environment variables

I added environment variables needed for transformer_engine compilation to the Makefile:

# Environment variables for transformer_engine
export MAX_JOBS := 1
export NVTE_BUILD_THREADS_PER_JOB := 1
export NVTE_FRAMEWORK := pytorch

Step 3: Removed hardcoded MAX_JOBS parameter in Makefile

I modified the Makefile to remove the hardcoded MAX_JOBS=32 parameter from the pip install command, allowing it to use the globally exported MAX_JOBS variable that I set earlier:

setup-full: submodules
	pip install ninja cmake pybind11 numpy psutil
	pip install -e . --config-settings=build-script=local_setup.py
	pip install transformer_engine[pytorch] --no-build-isolation 
-	cd vortex/ops/attn && MAX_JOBS=32 pip install -v -e  . --no-build-isolation
+	cd vortex/ops/attn && pip install -v -e  . --no-build-isolation

This change ensures that the previously set environment variable export MAX_JOBS := 1 will be used during compilation.

Step 4: Encountered libcudnn missing error

When I tried to run make setup-full, I encountered an error that cudnn.h was missing:

fatal error: cudnn.h: No such file or directory

Step 5: Installed CUDNN library

I installed the CUDNN library using:

sudo apt-get install -y libcudnn-dev nvidia-cudnn

And verified the location of cudnn.h:

find /usr -name cudnn.h
/usr/include/x86_64-linux-gnu/cudnn.h

Step 6: Encountered GLIBCXX_3.4.32 missing error

Even after installing cudnn, I encountered a new error when importing transformer_engine. The error indicated that my conda environment's libstdc++ was missing GLIBCXX_3.4.32:

ImportError: /anaconda3/envs/evo2/lib/python3.11/site-packages/transformer_engine/lib/transformer_engine_torch.so: undefined symbol: _ZNSt15__exception_ptr13exception_ptrC1EOS0_, version GLIBCXX_3.4.32

I verified the available GLIBCXX versions:

strings $CONDA_PREFIX/lib/libstdc++.so.6 | grep GLIBCXX

And discovered my conda environment only had up to GLIBCXX_3.4.29, while the system library had GLIBCXX_3.4.33.

Step 7: Linked system libstdc++ to conda environment

I created a symbolic link to use the system's libstdc++ instead of conda's:

mv $CONDA_PREFIX/lib/libstdc++.so.6 $CONDA_PREFIX/lib/libstdc++.so.6.backup
ln -s /usr/lib/x86_64-linux-gnu/libstdc++.so.6 $CONDA_PREFIX/lib/libstdc++.so.6

Step 8: Verified the solution

I confirmed that transformer_engine could now be imported successfully:

python -c "import transformer_engine; print('Successfully imported transformer_engine')"

[ElderMedic]

Has anyone in the dev team tried running this on WINDOWS?