Solution
I would suggest you to use conda
(Ananconda/Miniconda) to create a separate environment and install tensorflow-gpu
, cudnn
and cudatoolkit
. Miniconda has a much smaller footprint than Anaconda. I would suggest you to install Miniconda if you do not have conda
already.
Quick Installation
# Quick and dirty: with channel specification
conda create -n tf_gpu_env python=3.8
conda activate tf_gpu_env
conda install tensorflow-gpu -c anaconda
conda install cudnn -c conda-forge
conda install cudatoolkit -c anaconda
Check that Tensorflow is using the GPU
For more details see this: https://www.tensorflow.org/guide/gpu
# Sanity check for validating
# visibility of the GPUs to TF
import tensorflow as tf
print("Num GPUs Available: ", len(tf.config.list_physical_devices('GPU')))
Easily Reproducible Installation with Environment File: environment.yml
Although you can quickly create a conda
(Ananconda/Miniconda) environment as shown earlier, it is much more desirable to make the installation process as reproducible as possible: enters the environment.yml
file.
Save the environment file at the root of your repository (or project) and run the following command to install all the packages in an isolated conda environment (here I have named it tfgpu_env
) using the environment file. The top part of the environment file contains some useful commands.
conda env create -f environment.yml
Save the following as environment.yml
under your repository. And consider pinning
the following three libraries:
An example:
tensorflow-gpu=2.4
cudnn=8
cudatoolkit=11
## filename: environment.yml
## Environment File Definition
name: tfgpu_env # tensorflow-gpu environment
channels:
- conda-forge
- anaconda
- default
dependencies:
- python=3.8
## Core Necessities
- numpy # -c conda-forge, anaconda
- pandas # -c conda-forge, anaconda
- tabulate # -c conda-forge, anaconda # necessary for df.to_markdown() in pandas
- scipy # -c conda-forge, anaconda
- matplotlib # -c conda-forge, anaconda
## Jupyter Related
- jupyter # -c anaconda, conda-forge
- jupyterlab # -c anaconda, conda-forge
- jupyter_dashboards # -c conda-forge (see: https://medium.com/plotly/introducing-jupyterdash-811f1f57c02e)
- jupyter_contrib_nbextensions # -c conda-forge
## Progressbar
- tqdm # -c conda-forge, anaconda
## Machine Learning
- tensorflow-gpu # -c anaconda | version: 2.4.1 (linux), 2.3.0 (windows)
# - tensorflow # -c anaconda | version: 2.2.0 (linux), 2.1.0 (windows)
- cudnn # -c conda-forge | version: 8.1.0.77 (linux/windows)
# # -c anaconda | version: 7.6.5 (linux/windows)
- cudatoolkit # -c conda-forge | version: 8.1.0.77 (linux/windows)
# # -c anaconda | version: 11.0.221 (linux/windows)
- scikit-learn # -c conda-forge, anaconda
## Hyperparameter Optimization
- optuna # -c conda-forge # works for pytorch, tf/keras, mxnet, scikit-learn, xgboost, lightgbm
- keras-tuner # -c conda-forge
## Image Processing
- opencv # -c conda-forge, anaconda
- imageio # -c anaconda, conda-forge
## Image Augmentation
- albumentations # -c conda-forge
- imgaug # -c conda-forge
## Code Linting
- pylint # -c conda-forge, anaconda
- autopep8 # -c conda-forge, anaconda
## Installations with pip
- pip:
## Web App Framework
# - Flask-Testing
- streamlit # https://docs.streamlit.io/en/stable/troubleshooting/clean-install.html
Useful Instructions
You might as well copy and paste the following instructions in the environment file itself, to keep them handy.
# Instruction:
#-----------------------------------------------------------
#
## For an environment installed locally (under: ./.venv)
# mkdir -p .venv && cd .venv
# conda env create --prefix . -f ../environment.yml
## For Updating local environment
# cd .venv
# conda env update --prefix . -f ../environment.yml --prune
#
## For an environment installed globally
## with a name: fav_env
# NOTE: The env-name is stored inside the
# environment.yml file.
# conda env create -f environment.yml
## For Updating global environment from env-file
# conda env update -f ./environment.yml --prune
#
## Update conda itself
# conda update -n base -c defaults conda
#
## Creating a global environment in one-line: py37, py38
# conda create -n py37 python=3.7
# conda create -n py38 python=3.8
#
### In each of the envs: base, py37, py38
# conda install jupyter jupyterlab numpy scipy pandas matplotlib scikit-learn scikit-image tqdm plotly imageio requests pylint autopep8 tabulate opencv
#
## Export a platform independent copy of an environment
# conda env export --from-history > path/to/environment.yml
### Make exports directory (if not present already) and export
# $targetDir = conda_exports
# mkdir ./$targetDir
# conda env export --from-history > ./$targetDir/exported_environment.yml