Install Nvidia Drivers on RHEL

Blacklist Nouveau in Modprobe

The nouveau driver is an alternative to the Nvidia drivers generally installed on the server. It does not work with CUDA and must be disabled. The first step is to edit the file at /etc/modprobe.d/blacklist-nouveau.conf. Something like:

cat <<EOF | sudo tee /etc/modprobe.d/blacklist-nouveau.conf
blacklist nouveau
options nouveau modeset=0
EOF

Update Grub to Blacklist Nouveau

• On RHEL 6

  Backup your grub config:

  sudo cp /boot/grub/grub.conf /boot/grub/grub.conf.bak
  

  Edit your grub config and add rdblacklist=nouveau to the end of any lines starting with kernel. For example:

  kernel /vmlinuz-... quiet rdblacklist=nouveau
  

• On RHEL 7

  Backup your grub config templates:

  sudo cp /etc/sysconfig/grub /etc/sysconfig/grub.bak
  

  Then, update your grub config template at /etc/sysconfig/grub. Add rd.driver.blacklist=grub.nouveau to the GRUB_CMDLINE_LINUX variable. For example, change:

  GRUB_CMDLINE_LINUX="crashkernel=auto ... quiet"
  

  to:

  GRUB_CMDLINE_LINUX="crashkernel=auto ... quiet rd.driver.blacklist=grub.nouveau"
  

  Then, rebuild your grub config:

  sudo grub2-mkconfig -o /boot/grub2/grub.cfg
  

Regenerate the Initramfs Image

Backup the old initramfs image, generate a new initramfs image, disable any graphical logins and reboot the server:

sudo mv /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r)-nouveau.img
sudo dracut /boot/initramfs-$(uname -r).img $(uname -r)

Exiting X

The Nvidia driver will not allow you to install a new driver while X is open, so if X is enabled, it must first be exited. The simplest way to exit X is to switch to a TTY console using Ctrl-Alt-F1 , login, and run:

sudo init 3

After that has completed, X may be disabled so that the system does not attempt to start X in the case where the system has rebooted, but the driver has not finished installing. First, determine which graphical login your server uses:

ps aux | grep -v 'grep' | grep 'lightdm|gdm|kdm'

• On RHEL 6

  Disable the graphical login and reboot as follows (adjust for the login manager that is running):

  echo  "manual" | sudo tee -a /etc/init/lightdm.override
  sudo reboot now
  

• On RHEL 7

  Disable the graphical login as follows (adjust for the login manager that is running):

  sudo systemctl disable lightdm
  sudo reboot now
  

After the system reboots, it should no longer start up with a graphical login. The graphical login will be re-enabled after completing the Nvidia driver installation.

Ensure the Nouveau Driver is Disabled

After the reboot has completed, check to ensure that the nouveau driver has been disabled:

lsmod | grep "nouveau" > /dev/null && echo "WARNING: nouveau still active" || echo "Success"

If nouveau is still active, then run the following command and repeat the above check to ensure that Nouveau has been removed:

sudo rmmod nouveau

Check if nouveau is installed as an RPM:

rpm -qa | grep xorg-x11-drv-nouveau

If the RPM is installed, then run the following command to uninstall it:

sudo yum remove xorg-x11-drv-nouveau

Troubleshoot the Nvidia Installer

One common issue with installing the Nvidia driver is that it will fail out because the Nvidia driver taints the kernel. The issue is that the driver is not signed and the default install does not attempt to sign it, but the kernel is expecting a signed driver. If you encounter this error, you should re-run the install in expert mode:

sudo ./nvidia-Linux-<arch>-<version>.run -e

When prompted about cryptographic signatures on the kernel module, answer Sign the Kernel Module and then Generate a new key pair. Again, at the end, make sure to answer No when asked if you want the installer to update your X configuration.

This situation is usually detected during the above install step, but if there are issues, you can run this command separately.

Another issue that may arise is that if the kernel-devel version and the system kernel version don't match up, the Nvidia driver install will not proceed after accepting the license. To fix this issue:

yum update
sudo reboot now

Other Reference Material

Nvidia has a large readme online at:

http://us.download.nvidia.com/XFree86/Linux-<arch>/<version>/README/index.html

For example, on x86 for version 375.26, the readme is online at:

http://us.download.nvidia.com/XFree86/Linux-x86_64/375.26/README/index.html.

Multiple Driver Failure

If an error is returned, stating:

Failed to initialize NVML: GPU access blocked by the operating system

there may be multiple versions of the Nvidia drivers on the system. Try running:

rpm -qa | grep -E "cuda|nvidia"

Review any versions listed and remove them as needed. Also run:

locate libnvidia | grep ".so."

Confirm that the files all end with either a 1 or the version of the Nvidia driver that you installed, for example .375.21.

Enable X

If you disabled the X Server to install your Nvidia driver, enable it now. First, check which service is responsible for the X Server:

ps aux | grep -v 'grep' | grep 'lightdm|gdm|kdm'

The following will enable the lightdm service, for the case where lightdm is responsible for the X Server . Adjust for the particular service running on your server from the above command.

• On RHEL 6:

  sudo rm -f /etc/init/lightdm.override
  

• On RHEL 7:

  sudo systemctl enable lightdm
  

Reboot

Then, the simplest way to get back into X is to reboot the server:

sudo reboot now