Note
This documentation is for a prior release of Kinetica. For the latest documentation, click here.
This section will provide instructions on installing Nvidia drivers in a Debian/Ubuntu environment, if the target servers have Nvidia GPUs.
Ensure the GPUs are Installed
Ensure that the lspci command is installed (which lists the PCI devices
connected to the server):
sudo apt-get -y install pciutils
Perform a quick check to determine what Nvidia cards have been installed:
lspci | grep VGA
The output of the lspci command above should be something similar to:
00:02.0 VGA compatible controller: Intel Corporation 4th Gen ... 01:00.0 VGA compatible controller: Nvidia Corporation ...
If you do not see a line that includes Nvidia, then the GPU is not properly installed. Otherwise, you should see the make and model of the GPU devices that are installed.
Disable Nouveau
Blacklist Nouveau in Modprobe
The nouveau driver is an alternative to the Nvidia drivers generally
installed on the server. It does not work with CUDA and must be disabled.
The first step is to edit the file at
/etc/modprobe.d/blacklist-nouveau.conf. Something like:
cat <<EOF | sudo tee /etc/modprobe.d/blacklist-nouveau.conf blacklist nouveau blacklist lbm-nouveau options nouveau modeset=0 alias nouveau off alias lbm-nouveau off EOF
Then, run the following commands:
echo options nouveau modeset=0 | sudo tee -a /etc/modprobe.d/nouveau-kms.conf sudo update-initramfs -u
Update Grub to Blacklist Nouveau
Backup your grub config template:
sudo cp /etc/default/grub /etc/default/grub.bak
Then, update your grub config template at /etc/default/grub. Add
rd.driver.blacklist=grub.nouveau and rcutree.rcu_idle_gp_delay=1 to the
GRUB_CMDLINE_LINUX variable. For example, change:
GRUB_CMDLINE_LINUX="quiet"
to:
GRUB_CMDLINE_LINUX="quiet rd.driver.blacklist=grub.nouveau rcutree.rcu_idle_gp_delay=1"
Then, rebuild your grub config:
sudo grub2-mkconfig -o /boot/grub/grub.cfg
Prerequisites
The following prerequisites should be installed before installing the Nvidia drivers:
sudo apt-get -y install linux-headers-$(uname -r) make gcc-4.8 sudo apt-get -y install acpid dkms
Close X Server
Before running the install, you should exit out of any X environment, such as Gnome, KDE or XFCE. To exit the X session, switch to a TTY console using Ctrl-Alt-F1 and then determine whether you are running lightdm or gdm by running:
sudo ps aux | grep "lightdm|gdm|kdm"
Depending on which is running, stop the service, running the following commands (substitute gdm or kdm for lightdm as appropriate):
sudo service lightdm stop sudo init 3
Install Drivers Only
Important
To accommodate GL-accelerated rendering, OpenGL and GL Vendor Neutral Dispatch (GLVND) are now required and should be installed with the Nvidia drivers. OpenGL is an installation option in the *.run type of drivers. In other types of the drivers, OpenGL is enabled by default in most modern versions (dated 2016 and later). GLVND can be installed using the installer menus or via the --glvnd-glx-client command line flag.
This section deals with installing the drivers via the *.run executables provided by Nvidia.
To download only the drivers, navigate to http://www.nvidia.com/object/unix.html and click the Latest Long Lived Branch version under the appropriate CPU architecture. On the ensuing page, click Download and then click Agree and Download on the page that follows.
Note
The Unix drivers found in the link above are also compatible with all Nvidia Tesla models.
If you'd prefer to download the full driver repository, Nvidia provides a tool to recommend the most recent available driver for your graphics card at http://www.Nvidia.com/Download/index.aspx?lang=en-us.
If you are unsure which Nvidia devices are installed, the lspci command should give you that information:
lspci | grep -i "nvidia"
Download the recommended driver executable. Change the file permissions to allow execution:
chmod +x ./NVIDIA-Linux-$(uname -m)-*.run
Run the install. If you are prompted about cryptographic signatures on the kernel module, answer Sign the Kernel Module and then Generate a new key pair. At the end, DO NOT update your X config if it asks. Note that the following attempts to diagnose a common problem where the installer fails to correctly detect and deal with the situation where the kernel has been signed, but signed kernel modules are not required.
grep CONFIG_MODULE_SIG=y /boot/config-$(uname -r) && \ grep "CONFIG_MODULE_SIG_FORCE is not set" /boot/config-$(uname -r) && \ sudo ./NVIDIA-Linux-$(uname -m)-*.run -e || \ sudo ./NVIDIA-Linux-$(uname -m)-*.run
If there are any issues with the installation, the installer should notify you where the log is kept; the default location is usually:
/var/log/nvidia-installer.log
Troubleshoot the Nvidia Installer
One common issue with installing the Nvidia driver is that it will fail out because the Nvidia driver taints the kernel. The issue is that the driver is not signed and the default install does not attempt to sign it, but the kernel is expecting a signed driver. If you encounter this error, you should re-run the install in expert mode:
sudo ./nvidia-Linux-<arch>-<version>.run -e
When prompted about cryptographic signatures on the kernel module, answer Sign the Kernel Module and then Generate a new key pair. Again, at the end, make sure to answer No when asked if you want the installer to update your X configuration.
This situation is usually detected during the above install step, but if there are issues, you can run this command separately.
Another issue that may arise is that if the kernel development version and the system kernel version don't match up, the Nvidia driver install will not proceed after accepting the license. To fix this issue:
sudo apt-get update && sudo apt-get install linux-headers-$(uname -r) sudo reboot now
Other Reference Material
Nvidia has a large readme online at:
http://us.download.nvidia.com/XFree86/Linux-<arch>/<version>/README/index.html
For example, on x86 for version 375.26, the readme is online at:
http://us.download.nvidia.com/XFree86/Linux-x86_64/375.26/README/index.html.
Test the Nvidia Installation
After the Nvidia drivers are installed, you can test the installation by running the command:
nvidia-smi
Which should return something similar to:
+------------------------------------------------------+ | NVIDIA-SMI 361.42 Driver Version: 361.42 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===============================+======================+======================| | 0 Quadro K1100M Off | 0000:01:00.0 Off | N/A | | N/A 44C P0 N/A / N/A | 8MiB / 2047MiB | 0% Default | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |=============================================================================| | No running processes found | +-----------------------------------------------------------------------------+
Multiple Driver Failure
If an error is returned, stating:
Failed to initialize NVML: GPU access blocked by the operating system
there may be multiple versions of the Nvidia drivers on the system. Try running:
dpkg --list | grep -E "cuda|nvidia"
Review any versions listed and remove them as needed. Also run:
locate libnvidia | grep ".so."
Confirm that the files all end with either a 1 or the version of the Nvidia driver that you installed, for example .375.21.
Restart X Server
If you had to stop the X Server to install your Nvidia driver, the simplest way to get back into X is to reboot the server:
sudo reboot now
