Initial Setup: Install Packages Offline, Create Virtual Environment, Set Up GPU for Tensorflow

Photo by Thomas Q on Unsplash

When continuing the python work on a new computer, it usually requires an appropriate initial setup to ensure a smooth transition, e.g. package installation, virtual environment management, and GPU setup. I will discuss the most common tasks in the initial setup below, and hopefully, it can help a smooth transition for your work. All the tasks are done in the command window in Windows 10, python 3.7.

1. Install Python Packages (Offline)
2. Create Virtual Environment
3. Edit Default Setting of Jupyter Notebook
4. Install Jupyter Notebook Extension (Offline)
5. Set Up GPU for Tensorflow

Install Python Package (Offline)

To install the right version of python packages is critical for a smooth transition of your work and coding sharing. For example, if we want to install Tensorflow 1.14.0, just run:

pip install tensorflow ==1.14.0

However, I will discuss here how to install all the python packages from your favorite (virtual) environment on your new computer automatically.

First, generate a list of python packages in your favorite (virtual) environment. Activate your favorite environment, and then run the following will generate requirements.txt in the working directory.

pip freeze > requirements.txt

requirements.txt contains a list of python packages with their corresponding versions (Figure 1). Save requirement.txt properly, we will use it later.

Figure 1. Content of Requirement.txt

Second, download all the packages on requirement.txt. I save these package files in the folder: “D:\PYTHON_PKGS”. Attention: No space is allowed in the name of the directory.

pip download \
-r requirements.txt \
-d D:/PYTHON_PKGS \
--platform win_amd64 \
--python-version 37 \
--no-deps

Some parameters are helpful to specify the details of the download:

• -d: the folder where the downloaded files are saved
• --platform: operating system, e.g. win_amd64, linux_x86_64. The default is your running system
• -- python-version: the version of python. The default is your running python
• -- no-deps: do not install package dependencies

Attention: --only-binary=:all: or --no-deps is required when using any of these options: --platform, --python-version, --implementation, and --abi. More details are available at https://pip.pypa.io/en/stable/reference/pip_download/.

Sometimes, this command may fail because some python packages with specific versions cannot be found online. This will interrupt the whole command, and you need to make edits on requirements.txt and re-download everything again. In order to solve this issue, can use the following method (in jupyter notebook or python script): it will run over the whole list, even there is an error. For those failing to be downloaded, you can search ERROR in the message and download them manually.

import sys
from pip._internal import main as pip_main

def install(package):
pip_main([‘download’, package,
‘-d’, ‘D:/PYTHON_PKGS’,
‘--platform’, ‘win_amd64’,
‘--python-version’, ‘37’,
‘--no-deps’,
‘--disable-pip-version-check’
])

with open(‘requirements.txt’, ‘r’) as f:
lines = f.readlines()
index = 1
for line in lines:
print(str(index) + ‘: ‘ + line)
install(line)
index += 1

Third, install all the packages on requirement.txt on your new computer. Save what you have downloaded above and requirement.txt on your new computer, under the folder: “D:\INSTALL_PKGS”, and run:

pip install --no-index \
-f D:/INSTALL_PKGS \
-r D:/INSTALL_PKGS/requirements.txt

Some parameters are helpful to specify the details of the installation:

• --no-index: do not download online
• -f: the folder in which to find the python packages

We can also specify and install some instead of all the packages. For example, if we want to install numpy and have its .whl file:

pip install --no-index -f D:/INSTALL_PKGS -U numpy

If we don’t have .whl file but its .zip file, then we can unzip it as a subfolder “numpy-1.18.2”:

pip install --no-index D:/INSTALL_PKGS/numpy-1.18.2

Attention: Some packages have dependencies, and they will report ERROR if their dependencies cannot be found in the folder D:/INSTALL_PKGS. You need to download all their dependencies and put them in the folder in order to install them properly.

Create Virtual Environment

If you need to use different versions of packages in your work, then the virtual environment will make your work easy. For example, previously I wrote code using Tensorflow 1.14.0, but now I am starting to use Tensorflow 2.0.0. Therefore, I created a virtual environment for Tensorflow 2.0.0. See the process below.

First, install virtualenv package. It is to create a virtual environment.

pip install virtualenv

Second, create a virtual environment. I name it as .TF2.

virtualenv .TF2

Attention: If it reports an ERROR: “virtualenv is not recognized as an internal or external command”, then run:

python -m virtualenv .TF2

Third, activate the virtual environment and install necessary packages in the environment. I save the list of packages in requirements_TF2.txt (This step is similar to the section of Install Python Packages Offline).

.TF2/Scripts/activate
pip install -r requirements_TF2.txt

Attention: For a different operating system, e.g. MacOS, use different codes to activate:

source .TF2/bin/activate

Fourth*, there is a convenient way to activate a virtual environment in Jupyter Notebook: Kernel. To use this function, activate the virtual environment and install ipykernel package, and then create a new kernel to link the virtual environment. I name this new kernel as .TF2 too:

pip install ipykernel
python -m ipykernel install --user --name .TF2

To activate this virtual environment in Jupyter Notebook, go to “Kernel” > “Change kernel”, and select the kernel listed there (Figure 2).

Figure 2. Kernel Selection

Lastly, do not forget to deactivate the virtual environment after finishing your work. No need if you use the kernel.

deactivate

Edit Default Setting of Jupyter Notebook

I have a personal preference of the browser and the location of the default directory when using Jupyter Notebook, so I do change the default settings of Jupyter Notebook to fit my preference. To do so:

First, create a config file of Jupyter Notebook:

jupyter notebook --generate-config

It will generate a file called jupyter_notebook_config.py, located at C:\User\YOURUSERNAME\.jupyter.

Second, open jupyter_notebook_config.py as a .txt file, and change the settings to fit your preference. Two common options in settings,

• Default browser: edit “c.NotebookApp.browser”, and remove # in front. For example, use Chrome as default browser: c.NotebookApp.browser = u’C:/Program Files (x86)/Google/Chrome/Application/chrome.exe %s’. Attention: it is ‘chrome.exe[space]%s’
• Default directory: edit “c.NotebookApp.notebook_dir”, and remove # in front. For example, set the default directory to D:\: c.NotebookApp.notebook_dir = u’D:’

Install Jupyter Notebook Extension (Offline)

I use “Table of Contents” and “Code Prettify” a lot when coding in Jupyter Notebook. They are just a very small portion of Jupyter Notebook Extension. Many functions in Jupyter Notebook Extension can improve your efficiency.

With access to the internet, it is easy to install Jupyter Notebook Extension. However, sometimes access to the internet is limited due to security issues. I will discuss how to install Jupyter Notebook Extension in this scenario.

First, download the related packages to Jupyter Notebook Extension. I save these package files in the folder: “D:\EXT_PKGS”.

pip download jupyter_contrib_nbextensions\
-d D:/EXT_PKGS \
--platform win_amd64 \
--python-version 37

Second, install Jupyter Notebook Extension on your new computer. Save what you have downloaded above on your new computer, under the folder: “D:\INSTALL_PKGS”, and run:

pip install --no-index -f D:/INSTALL_PKGS -U jupyter_contrib_nbextensions

Third, change the settings of Jupyter Notebook Extension.

jupyter contrib nbextension install --user
jupyter nbextension enable <nbextension require path>

Attention: Sometimes jupyter contrib nbextension may not work properly (e.g. the commend is not recognized), then you need to go to the folder of Python, to check whether there is a file called jupyter-contrib-nbextension.exe in the folder of Python\Scripts. If you installed Anaconda, this may in the the folder of Anaconda\Scripts. Replace the above commands and run:

Python/Scripts/jupyter-contrib-nbextension install — user
Python/Scripts/jupyter-nbextension enable <nbextension require path>

Set Up GPU for Tensorflow

I spent 2 weeks figuring out how to set up GPU properly for Tensorflow. There are 3 important lessons I learned after numerous trials and errors, which may save you a lot of time:

• GPU cannot be properly set up with Tensorflow 2.1.0. I spent 90% of the time trying to figure it out but failed. In the end, I gave up and downgraded my Tensorflow to 2.0.0.
• MacOS does not have GPU support since Tensorflow 1.1.0. Refer to https://www.tensorflow.org/install/source
• To avoid complicated compatibility issues between your driver and CUDA, better to uninstall all old NVIDIA software and reinstall them properly.

I refer to this instruction in this section: https://blog.quantinsti.com/install-tensorflow-gpu/#uninstall-nvidia.

First, uninstall all NVIDIA software. It is to avoid complicated compatibility issues.

Figure 3. Uninstall NVIDIA

Second, install the NVIDIA driver. Download the corresponding driver from the website https://www.geforce.com/drivers. If unclear about which driver to download, key in “dxdiag” in start menu. It will pop up a window, click the “Display” tab in the new window which shows the details of NVIDIA driver (Figure 4).

Figure 4. Information on NVIDIA Driver

Third, install Visual Studio. Download Visual Studio from the website https://visualstudio.microsoft.com/downloads/.

Fourth, install CUDA. Different versions of Tensorflow requires a different version of CUDA. Since I want to install Tensorflow-GPU 2.0.0 on the new computer, I need CUDA 10.0 and cuDNN 7.4 (See Figure 5). For other Tensorflow versions, refer to the website for details: https://www.tensorflow.org/install/source#linux.

Download CUDA from the website: https://developer.nvidia.com/cuda-downloads. You need to register on the website first. I got the “NVIDIA Installer failed” Error during installation. It is caused by inappropriate installation of the NVIDIA driver before, so I reinstall NVIDIA driver to solve the problem.

Figure 5. CUDA and cuDNN Versions for Different Tensorflow Versions

Fifth, download and unzip cuDNN. From above, the version of cuDNN I need here is 7.4. Download the corresponding cuDNN from the website: https://developer.nvidia.com/cudnn.

This will download a zip file. Unzip the file, you will see three folders shown in Figure 6.

Figure 6. Unzip cuDNN

Extract these three folders and replace the same folders in “C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0” (Figure 7).

Figure 7. Folder of “NVIDIA GPU Computing Toolkit\CUDA\v10.0”

Sixth, add “C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0” to the System Variables. Go to the start menu and search for “edit the environment variables”, add the path of NVIDIA GPU Computing Toolkit to System Variables.

Seventh, the last step, install TensorFlow-GPU. If you installed tensorflow-cpu before, you need to pip uninstall it first and then install tensorflow-gpu.

pip install tensorflow-gpu==2.0.0

You can open Jupyter Notebook to check whether tensorflow-gpu is properly installed or not.

import tensorflow as tf
tf.test.is_gpu_available()

If the result is TRUE, congratulations! You have successfully installed GPU support for Tensorflow.

Reference:

• How To Install TensorFlow GPU (With Detailed Steps): https://blog.quantinsti.com/install-tensorflow-gpu/#uninstall-nvidia