blog.devgenius.io Open in urlscan Pro
162.159.153.4  Public Scan

Form analysis
0 forms found in the DOM

Text Content

Open in app

Sign up

Sign in

Write


Sign up

Sign in




HOW TO RESOLVE UNDEFINED SYMBOL: NCCLREDOPDESTROY IN LIBTORCH_CUDA.SO?

Michael Humor

·

Follow

Published in

Dev Genius

·
1 min read
·
Dec 28, 2023

31



Listen

Share

If you are running PyTorch 2.1 or above with CUDA NCCL, you may encounter the
following error:

libtorch_cuda.so: undefined symbol: ncclRedOpDestroy

Traceback (most recent call last):
  File ".../train.py", line 26, in <module>
    import torch
  File ".../python3.9/site-packages/torch/__init__.py", line 235, in <module>
    from torch._C import *  # noqa: F403
ImportError: .../python3.9/site-packages/torch/lib/libtorch_cuda.so: undefined symbol: ncclRedOpDestroy

The problem is that torch (v2.1+) requires nvidia-nccl v2.18+, butpip install
nvidia-nccl only gets v2.9.8. To resolve this issue, follow two steps:


1. BUILD THE NEWEST NCCL

git clone https://github.com/NVIDIA/nccl.git
cd nccl

make -j src.build

In the above, make sure CUDA is on the default PATH /usr/local/cuda.

If not, you can define the CUDA path with:

make src.build CUDA_HOME=<path to cuda install>

After building successfully, the nccl header files and dynamic libraries will
appear in nccl/build, which contains three sub folders:

include
lib
obj

For example, libnccl.so.2.19.4 is placed under the lib folder:

libnccl.so -> libnccl.so.2
libnccl.so.2 -> libnccl.so.2.19.4
libnccl.so.2.19.4
libnccl_static.a
pkgconfig


2. LINK THE NCCL LIBRARIES

add a soft link to nccl/build in site-packages/nvidia:

cd your_path_to/python3.9/site-packages/nvidia/
ln -s nccl/build nccl

After that, you should be able to run torch successfully.

>>> import torch
>>> torch.cuda.nccl.version()
(2, 18, 1)

python -c "import torch;print(torch.cuda.nccl.version())"
(2, 18, 1)





SIGN UP TO DISCOVER HUMAN STORIES THAT DEEPEN YOUR UNDERSTANDING OF THE WORLD.


FREE



Distraction-free reading. No ads.

Organize your knowledge with lists and highlights.

Tell your story. Find your audience.


Sign up for free


MEMBERSHIP



Access the best member-only stories.

Support independent authors.

Listen to audio narrations.

Read offline.

Join the Partner Program and earn for your writing.


Try for $5/month
Cuda
Nccl
Torch
Pytorch
Large Language Models


31

31



Follow



WRITTEN BY MICHAEL HUMOR

84 Followers
·Writer for

Dev Genius

tengenius.ai

Follow




MORE FROM MICHAEL HUMOR AND DEV GENIUS

Michael Humor

in

Dev Genius


HOW TO FINE-TUNE MIXTRAL-8X7B-INSTRUCT ON YOUR OWN DATA?


IT TAKES JUST A FEW MINUTES OVER THREE STEPS:

8 min read·Jan 20, 2024

332

3




Elshad Karimov



in

Dev Genius


WANNA CODE LIKE A FAANG ENGINEER? TOGETHER, LET’S DIVE INTO ADVANCED PYTHON!


LEARN ABOUT MAGICAL LIBRARIES, EFFECTIVE LOOPING, SYNTAX, AND MORE. YOU
ABSOLUTELY MUST READ THIS IF YOU WANT TO IMPROVE YOUR TALENTS!


·12 min read·Jan 4, 2024

1.3K

14




Anusha SP

in

Dev Genius


JAVA 8 CODING AND PROGRAMMING INTERVIEW QUESTIONS AND ANSWERS


IT HAS BEEN 8 YEARS SINCE JAVA 8 WAS RELEASED. I HAVE ALREADY SHARED THE JAVA 8
INTERVIEW QUESTIONS AND ANSWERS AND ALSO JAVA 8 STREAM API…

6 min read·Jan 30, 2023

567

8




Michael Humor

in

Dev Genius


UNDERSTANDING “TOKENS” AND TOKENIZATION IN LARGE LANGUAGE MODELS


WHAT’S A TOKEN?

6 min read·Sep 10, 2023

127




See all from Michael Humor
See all from Dev Genius



RECOMMENDED FROM MEDIUM

Cristian Leo

in

Towards Data Science


THE MATH BEHIND ADAM OPTIMIZER


WHY IS ADAM THE MOST POPULAR OPTIMIZER IN DEEP LEARNING? LET’S UNDERSTAND IT BY
DIVING INTO ITS MATH, AND RECREATING THE ALGORITHM.

16 min read·Jan 30, 2024

1.4K

9




Florian June

in

Towards AI


ADVANCED RAG 02: UNVEILING PDF PARSING


INCLUDING KEY POINTS, DIAGRAMS, AND CODE


·13 min read·Feb 2, 2024

592

8





LISTS


NATURAL LANGUAGE PROCESSING

1187 stories·660 saves


AI REGULATION

6 stories·314 saves


CHATGPT PROMPTS

38 stories·1112 saves


GENERATIVE AI RECOMMENDED READING

52 stories·713 saves


Suman Das


FINE TUNE LARGE LANGUAGE MODEL (LLM) ON A CUSTOM DATASET WITH QLORA


THE FIELD OF NATURAL LANGUAGE PROCESSING HAS BEEN REVOLUTIONIZED BY LARGE
LANGUAGE MODELS (LLMS), WHICH SHOWCASE ADVANCED CAPABILITIES AND…

15 min read·Jan 24, 2024

382

4




Rohan Balkondekar


THE FULL TRAINING RUN OF GPT-5 HAS GONE LIVE


WE CAN EXPECT IT TO BE RELEASED IN NOVEMBER, MAYBE ON THE 2ND ANNIVERSARY OF THE
LEGENDARY CHATGPT LAUNCH

4 min read·Jan 27, 2024

187

3




Artturi Jalli


I BUILT AN APP IN 6 HOURS THAT MAKES $1,500/MO


COPY MY STRATEGY!


·3 min read·Jan 23, 2024

5.4K

82




Yahia Berashish

in

Stackademic


REVIEWING ZED: AN IDE OF THE FUTURE


WILL THIS HYPED, NEW, MODERN, FAST, AND -RECENTLY- OPEN-SOURCE IDE BE THE VSCODE
KILLER?

5 min read·Jan 31, 2024

1K

23



See more recommendations

Help

Status

About

Careers

Blog

Privacy

Terms

Text to speech

Teams