Introduction

I’m working on a new personal project that involves recognizing and tracking vehicles in a video feed from a camera overlooking an all-way stop intersection. An early prototype of the system running on my MacBook Pro (more in a later blog post) promised technological feasibility but using Convolutional Neural Networks (CNN) for object detection and tracking in each frame turned out too processing intensive for real time.

The hardware I’m using is an old (Sandy Bridge) i7 desktop computer that was fitted with 32GB of DDR3 RAM and an NVIDIA GeForce GTX1070 graphics card. The goal is to use the GPU for Neural Network Intensive learning tasks.

Ubuntu 18.04 (Bionic Beaver)

For the installation of Ubuntu 18.04 I’ve added a blank new SSD drive to my desktop machine. I’ve used Etcher (https://etcher.io) to create a bootable USB of the latest Ubuntu ISO (http://releases.ubuntu.com/18.04/). Booted from the USB, selected Install Ubuntu, followed through the install steps guide. During hard drive partitioning, I selected LLVM and encryption.

Install NVIDIA Drivers

Once booted into the fresh Ubuntu install, the first thing I did was to install the NVIDIA drivers from the ‘official’ NVIDIA package repository. At the time of writing this was version nvidia-396.

sudo add-apt-repository ppa:graphics-drivers/ppa
sudo apt update
sudo apt install nvidia-396
reboot

After rebooting, running the nvidia-smi command should print out details about the graphics card and driver. I also installed the settings pane.

nvidia-smi
sudo apt install nvidia-settings

Install Dependencies

After the NVIDIA driver installation, I proceeded with installing a number of graphics and video libraries that are required for building OpenCV.

sudo apt install git
sudo apt install freeglut3 freeglut3-dev libxi-dev libxmu-dev
sudo apt install build-essential cmake unzip pkg-config
sudo apt install libjpeg-dev libpng-dev libtiff-dev
sudo apt install libavcodec-dev libavformat-dev libswscale-dev libv4l-dev
sudo apt install libxvidcore-dev libx264-dev
sudo apt install libgtk-3-dev
sudo apt install libatlas-base-dev gfortran
sudo apt install libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev

CUDA 9.2

The NVIDIA CUDA Toolkit provides a development environment for creating high performance GPU-accelerated applications. With the CUDA Toolkit, you can develop, optimize and deploy your applications on GPU-accelerated embedded systems, desktop workstations, enterprise data centers, cloud-based platforms and HPC supercomputers. The toolkit includes GPU-accelerated libraries, debugging and optimization tools, a C/C++ compiler and a runtime library to deploy your application. GPU-accelerated CUDA libraries enable drop-in acceleration across multiple domains such as linear algebra, image and video processing, deep learning and graph analytics.

First, we install CUDA as this is a compile time dependency for OpenCV. Download the runfile installer from https://developer.nvidia.com/deep-learning

It will complain about an “incompatible setup”, which we can safely ignore. Also, we need to say ‘NO’ to driver, and yes to everything else.

I chose /opt/cuda/9.2 as install dir and say yes to symlink to /usr/local/cuda. The following instructions assume that installation directory.

sudo sh cuda_9.2.148_396.37_linux.run
sudo sh cuda_9.2.148.1_linux.run

After installing CUDA 9.2 and the patch update, we can add the CUDA paths to our profile.

vim ~/.bashrc

export PATH=$PATH:/usr/local/cuda/bin
export CUDADIR=/usr/local/cuda
export PATH=/usr/local/cuda/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

and also update ldconfig like so:

echo "/usr/local/cuda/lib64" >> /etc/ld.so.conf.d/cuda.conf
sudo ldconfig

cuDNN 7.2

The NVIDIA CUDA Deep Neural Network library (cuDNN) is a GPU-accelerated library of primitives for deep neural networks. cuDNN provides highly tuned implementations for standard routines such as forward and backward convolution, pooling, normalization, and activation layers. cuDNN is part of the NVIDIA Deep Learning SDK.

Like CUDA, we can download cuDNN from https://developer.nvidia.com/cudnn. Installation is a straightforward unpacking fo the archive, and copying files to our CUDA installation directory.

cd ~/Downloads
tar xjf cudnn-9.2-linux-x64-v7.2.1.38.tgz
sudo cp cudnn-9.2-linux-x64-v7.2.1.38/cuda/lib64/libcudnn* /opt/cuda/9.2/lib64
sudo cp cudnn-9.2-linux-x64-v7.2.1.38/cuda/include/cudnn.h /opt/cuda/9.2/include/

Python 3

We first install the dev libraries for Python 3, as well as the pip tool.

sudo apt install python3-dev
wget https://bootstrap.pypa.io/get-pip.py
sudo python3 get-pip.py
sudo rm -rf get-pip.py

Next we setup virtualenv to work in Python virtual environments, which is best practice for Python development.

sudo pip install virtualenv virtualenvwrapper
vim ~/.bashrc

# virtualenv and virtualenvwrapper
export WORKON_HOME=$HOME/.virtualenvs
export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3
source /usr/local/bin/virtualenvwrapper.sh

Finally, we create a virtual environment for our Python 3, OpenCV 4 development:

mkvirtualenv py3cv4 -p python3
workon py3cv4

OpenCV 4

At the time of writing, OpenCV 4 is still in development, with a release date in fall. I wanted to take advantage of the many improvements added to the deep learning (dnn) module in the upcoming release. It’s straightforward to compile OpenCV 4 from the GitHub sources.

Clone Git Sources

First we clone opencv and opencv-contrib:

cd ~ && mkdir -p Repositories/github.com/opencv
cd Repositories/github.com/opencv
git clone https://github.com/opencv/opencv.git
git clone https://github.com/opencv/opencv_contrib.git

Configure and Build

The following steps will configure the OpenCV 4 build. The instructions assume the paths for the git repositories and CUDA libraries are like described above.

cd opencv && mkdir build
cd build
cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local -D INSTALL_PYTHON_EXAMPLES=ON -D INSTALL_C_EXAMPLES=OFF -D OPENCV_EXTRA_MODULES_PATH=/home/nicbet/Repositories/github.com/opencv/opencv_contrib/modules -D PYTHON_EXECUTABLE=~/.virtualenvs/py3cv4/bin/python -D BUILD_EXAMPLES=ON -D WITH_CUDA=ON -D CUDA_TOOLKIT_ROOT_DIR=/opt/cuda/9.2 ..

Compile

Compilation of the code can take a little while - about 35 minutes on my desktop machine.

make -j4

Install

When running make install I initially hit an error as we were building from ./build directory. We need to kick off header generation like so, then make install will complete. It’s an open bug tracked here: https://github.com/opencv/opencv/issues/10771

python ../modules/python/src2/gen2.py ../build/modules/python_bindings_generator ../build/modules/python_bindings_generator/headers.txt
sudo make install
sudo ldconfig

Link to Virtualenv

Finally, we will link the OpenCV module to our Python virtual environment.

cd ~/.virtualenvs/py3cv4/lib/python3.6/site-packages/
ln -s /usr/local/lib/python3.6/site-packages/cv2.cpython-36m-x86_64-linux-gnu.so cv2.so

Test OpenCV 4 Installation

We can test our installation by invoking a Python REPL:

python

>>> import cv2
>>> cv2.__version__
'4.0.0-pre'
>>> quit()

Tensorflow GPU

To install Tensorflow with GPU acceleration, I followed the instructions from the official website. Even though we installed CUDA 9.2 earlier, the Tensorflow packages require at the time of writing CUDA 9.0. That should change with the upcoming version of the Tensorflow and CUDA libraries: next Tensorflow will target CUDA 9.2 (they are skipping 9.1), and CUDA 10 will be fully backward compatible to CUDA 9.2, so we will finally have a way out of the dependency chicken game.

sudo apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/7fa2af80.pub
wget http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/cuda-repo-ubuntu1604_9.1.85-1_amd64.deb
wget http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64/nvidia-machine-learning-repo-ubuntu1604_1.0.0-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu1604_9.1.85-1_amd64.deb
sudo dpkg -i nvidia-machine-learning-repo-ubuntu1604_1.0.0-1_amd64.deb
sudo apt-get update

sudo apt-get install cuda9.0 cuda-cublas-9-0 cuda-cufft-9-0 cuda-curand-9-0 \
  cuda-cusolver-9-0 cuda-cusparse-9-0 libcudnn7=7.1.4.18-1+cuda9.0 \
   libnccl2=2.2.13-1+cuda9.0 cuda-command-line-tools-9-0

sudo apt-get update
sudo apt-get install libnvinfer4=4.1.2-1+cuda9.0

Later when Tensorflow 1.9 has CUDA 9.2 Support

See this Github issue for a discussion around Tensorflow and CUDA 9.2. I’ve decided to take the easy way out and just have CUDA 9.0 and CUDA 9.2 installed in parallel, with OpenCV 4 using CUDA 9.2 and Tensorflow using CUDA 9.0 libs.

sudo apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1710/x86_64/7fa2af80.pub
echo "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1710/x86_64 /" | sudo tee /etc/apt/sources.list.d/cuda.list
sudo apt update
sudo apt -o Dpkg::Options::="--force-overwrite" install cuda-9-2 cuda-drivers
sudo reboot

Test Tensorflow GPU

Again, we can test the install by invoking a Python REPL:

python

>> from tensorflow.python.client import device_lib
>> device_lib.list_local_devices()

And there you have it. The next few of blog posts will describe the vehicle recognition and tracking system in more detail.