It has been a while since I haven't written a blog post, but today I wanted to share some recent experience with my public cloud of heart and their GPU instances offering. I know that, many people probably did it and did it way better than I did, especially on Ubuntu. But as I am not Ubuntu #1 fan, and much prefer CentOS I wanted to share with you my steps and results using FFMPEG and the NVIDIA codecs to leverage GPU for video encoding and manipulations.
In a near future, I will take a look at the transcoder service in AWS, but as it doesn't meet the requirements for the entire pipe of my video's lifecycle, I am yet to determine how to leverage that service.
So, historically I wanted to use the Amazon Linux image with the drivers installed (the one with the nice NVIDIA icon). But I faced much more problems with it than I thought I'd have. Therefore, I decided to go on a basic minimal install of CentOS 7 and take it from here. And here we go !
As I just mentionned before, I use for this tutorial the official CentOS 7 image available on the AWS Marketplace. This is a minimal install, so at first I recommend to install your favorite packages as well as some of the packages coming from the Base group.
Also, to install the NVIDIA drivers, you will have to remove the "nouveau" driver / module from the machine.
On the GRUB_CMDLINE_LINUX line, add rd.driver.blacklist=nouveau nouveau.modeset=0
GRUB_TIMEOUT=1 GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)" GRUB_DEFAULT=saved GRUB_DISABLE_SUBMENU=true GRUB_TERMINAL="serial console" GRUB_SERIAL_COMMAND="serial --speed=115200" GRUB_CMDLINE_LINUX="console=tty0 crashkernel=auto console=ttyS0,115200 rd.driver.blacklist=nouveau nouveau.modeset=0" GRUB_DISABLE_RECOVERY="true"
Now, generate the new grub configuration
Now, reboot the machine.
CUDA is not required to do the encode and use the latest h264_nvenc encoder available in FFMPEG. However, CUDA seems required to leverage the NVIDIA api "NVResize" which redimensions videos and uses the GPU for that. Thought, as I am not an expert, there might already be an option to do so with the nvenc encoder in the latest FFMPEG version (to be continued research). IT IS IMPORTANT THAT YOU START WITH CUDA BEFORE INSTALLING THE NVIDIA DRIVERS !!!!
So, let's get our hands on the latest CUDA SDK : https://developer.nvidia.com/cuda-downloads
There, select Linux -> x86_64 -> CentOS -> 7 -> run file (I strongly advice to go for the run file. Much more problem with the repos).
From your AWS EC2 Instance, this won't take long. Grab a tea :) Now we have it, execute the run file as root
Accept the terms, then the options I took were :
Install CUDA Install the Libraries Do not install the samples The CUDA installer will install some NVIDIA drivers. At the end of the process (if successful), you should be able to enable the nvidia kernel modules :
So far so good, with lsmod you should see those enabled.
The CUDA utils
Thanks to this guide, I realized there would be potential steps to have some CUDA library to help FFMPEG to communicate with CUDA. Pretty straight forward step.
wget http://developer.download.nvidia.com/compute/redist/ffmpeg/1511-patch/cudautils.zip unzip cudautils.zip cd cudautils make
All set for now, moving on.
The Right NVIDIA drivers
To save you lots of troubles, let's just say that after 24h of different annoying non-verbose errors, I figured something was wrong with the drivers delivered by the CUDA installer (v.352.79). So, now, let's get the NVIDIA latest drivers. You can always get the latest from NVIDIA website : http://www.nvidia.com/download/driverResults.aspx/106780/en-us Those are the latest (on Aug. 2016, v.367.44). Make sure you download the Linux x64 version for the GRID K520. On your instance :
wget http://us.download.nvidia.com/XFree86/Linux-x86_64/367.44/NVIDIA-Linux-x86_64-367.44.run chmod +x NVIDIA-Linux-x86_64-367.44.run sudo ./NVIDIA-Linux-x86_64-367.44.run
Accept the terms, acknowledge that this installer will install new drivers and uninstall the old ones. I did not pick yes for the compatibility drivers for 32bits, and went for the DKMS. Once the driver install is finished, I strongly suggest to reboot, and then, as before, check on the kernel modules to verify those are enabled and working (use lsmod).
You will need this header file in your library to compile FFMPEG with the --enable-nvenc and use the encoder. To get this one, you will need to subscribe to the developer program of NVIDIA and get the Video_Codec_SDK_7.0.1. I could have made this available for all, but, I will leave you to accept the terms and conditions and get your hands on it yourself. Once you have it, upload it (via SFTP mostlikely) to your instance. Once you have it, unzip the file, and locate the nvEncodeAPI.h file. Keep it in your back-pocket, we will need it soon.
Now arriving on the final step. As the guide referenced earlier mentions, a few steps are required prior to compile FFMPEG : get the right ffmpeg version get the patch to enable the nvresize For my own FFMPEG, I needed some additional plugins. Here is the script I used to install those. Note the exit 1 just before FFMPEG. This was a fail safe to avoid forgetting some of the little details that follow. Use the script for the non-in repos packages (ie: for x264). Prepare your compilation folders
I personally like to put extra, self-compiled packages in /opt as they are easy to find. But feel free to do as you prefer. For the following steps, I will be doing all the work as root (I know, I know ..) in /opt (If you used my script so far, skip the folders creation). mkdir -p /opt/ffmpeg_sources mkdir -p /opt/ffmpeg_build
Now, we can go ahead with building all the dependencies. The shell script I have done will cover those parts.
Download the right FFMPEG
wget http://developer.download.nvidia.com/compute/redist/ffmpeg/1511-patch/ffmpeg_NVIDIA_gpu_acceleration.patch git clone git://source.ffmpeg.org/ffmpeg.git git reset --hard b83c849e8797fbb972ebd7f2919e0f085061f37f git apply --reject --whitespace=fix ../ffmpeg_NVIDIA_gpu_acceleration.patch
From this point, you can't use ./configure with --enable-nvresize and --enable-nvenc: we are missing the libraries.
Simply copy the headers file in /opt/ffmpeg_build/include cudautils
Go back to your cuda utils folder. I did the quick and dirty, yet working, cp * /opt/ffmpeg_build/include and cp * /opt/ffmpeg_build/lib. Now, you could just put the .a in the lib and the .h in the include folders. Configure
cd /opt/ffmpeg_sources/ffmpeg/ export PKG_CONFIG_PATH="/opt/ffmpeg_build/lib/pkgconfig" ./configure --prefix="/opt/ffmpeg_build" \ --extra-cflags="-I/opt/ffmpeg_build/include" \ --extra-ldflags="-L/opt/ffmpeg_build/lib" \ --bindir="/opt/bin" \ --pkg-config-flags="--static" \ --enable-gpl \ --enable-libfaac \ --enable-libfreetype\ --enable-libmp3lame \ --enable-libtheora \ --enable-libvorbis \ --enable-libvpx \ --enable-libx264 \ --enable-nonfree \ --enable-nvenc \ --enable-nvresize \ --enable-libfribidi \ --enable-libfontconfig
If the configure succeeded, let's compile
Now, the sneaky library missing to run it 100% requires you to deploy it to /usr/lib64 :
sudo ln -s $BUILD_DIR/ffmpeg_build/lib/libfaac.so.0 /usr/lib64/ export PATH=$PATH:/opt/bin ffmpeg -version ffmpeg -encoders | grep nv
If something doesn't work to run FFMPEG at this point, something went wrong before. Test FFMPEG with NVENC and NVResize
Well for this part, I have followed the basic demo and test commands that this PDF guide suggested. So see the CPU and GPU usage, I had side-by-side in my TMUX sessions running htop and nvidia-smi (watch -d -n1 nvidia-smi). Now in a 3 part of my TMUX, I ran the different commands, such as :
ffmpeg -y -i Eucalyptus-Ansible-Deploy.mp4 -filter_complex nvresize=5:s=hd1080\|hd720\|hd480\|wvga\|cif:readback=0[out0][out1][out2][out3][out4] -map [out0] -acodec copy -vcodec nvenc -b:v 5M out0nv.mkv -map [out1] -acodec copy -vcodec nvenc -b:v 4M out1nv.mkv -map [out2] -acodec copy -vcodec nvenc -b:v 3M out2nv.mkv -map [out3] -acodec copy -vcodec nvenc -b:v 2M out3nv.mkv -map [out4] -acodec copy -vcodec nvenc -b:v 1M out4nv.mkv ffmpeg -y -i Eucalyptus-Ansible-Deploy.mp4 -acodec copy -vcodec nvenc -b:v 2M /var/tmp/test_parallel.mp4
Tada ! I hope this guide will have helped you guys in your different experimentation with AWS and will enjoy playing around with it.
Do not forget to stop/terminate those instances. A potential 600USD bill will wait for your per GPU instance running. Still less than traditional IT but still .. :P