Deep/Machine Learning
If you want to do some machine/deep learning and you wish to make use of departmental GPU services, then you are probably going to be using python and modules such as Tensorflow, Keras, pytorch, etc.
All of these modules can be installed either in your home directory or your scratch directory by creating a python virtual environment.
If python support for GPU's is required you will need to add some environment variables to your ~/.cshrc file.
You can find the CUDA toolkit and CUDANN libraries at /apps/cuda, and you must use the version of cudnn that is the same version number as the cuda version
To find out what shell you are using, login to a linux server and type the command:
echo $SHELL
######################## # Entries for cuda/cudnn ######################## setenv CUDA_HOME /apps/cuda/cuda-12.4.1 setenv CUDNN_HOME /apps/cuda/cudnn-linux-x86_64-9.1.1.17_cuda12 setenv PATH {$CUDA_HOME}/bin:{$PATH} setenv LD_LIBRARY_PATH {$CUDA_HOME}/lib64:{$CUDNN_HOME}/lib:{$LD_LIBRARY_PATH}
A handy thing is to add the end of your .cshrc:
## activate python virtual env alias activate 'source /home/$USER/\!*/bin/activate.csh'
This will allow you to activate different virtualenv's by just typing activate name-of-virtual-env (The virtual env name will be set below).
######################## # Entries for cuda/cudnn ######################## export CUDA_HOME=/apps/cuda/cuda-12.4.1 export CUDNN_HOME=/apps/cuda/cudnn-linux-x86_64-9.1.1.17_cuda12 export PATH=$CUDA_HOME/bin:$PATH
export LD_LIBRARY_PATH=$CUDA_HOME/lib64:$CUDNN_HOME/lib:$LD_LIBRARY_PATH
whichever shell you are using, you should logout and login again after making these changes to make sure that you can login with no errors and these environment variables are active in your session
You can choose to use a standard python3 virtual environment, or one created using the Anaconda python distribution.
Using Python 3
The default version of python on CentOS 7 is still python 2, but any new activities on python should be conducted with python3.
Versions of python available
Hostname | OS | Default "python3" | Alternative pythons |
---|---|---|---|
cork.ee.ucl.ac.uk | CentOS 7 | python3.6 | |
athens.ee.ucl.ac.uk | Rocky 9 | python3.9 | python3.11 |
geneva.ee.ucl.ac.uk |
Rocky 9 | python3.9 |
python3.11 |
london.ee.ucl.ac.uk |
Rocky 9 | python3.9 | python3.11 |
turin.ee.ucl.ac.uk |
Rocky 9 |
python3.9 | python3.11 |
medusa.ee.ucl.ac.uk | Rocky 9 | python3.9 | python3.11 |
To create a python3 virtual environment:
$ python3 -m venv ~/p3-env
The "p3-env" used above is the name of the newly created virtual environment. You are welcome to change this to whatever you want.
You can create a python3 virtual environment that uses an alternative version of python, by replacing the "python3" command with one of the python commands in the table above.
i.e. If you wanted to create a python3.11 environment, you would use the command:
$ python3.11 -m venv ~/p3-env
$ source ~/p3-env/bin/activate.csh
$ source ~/p3-env/bin/activate
The source commands should add the following prefix to your prompt:
(p3-env)$
You will need to update the version of pip installed in your virtualenv:
(p3-env)$ pip3 install -U pip
You will also need to update the setuptools installed in your virtualenv:
(p3-env)$ pip3 install --upgrade setuptools
You should be able to install packages (i.e tensorflow) from the pip repositories by using the command:
(p3-env)$ pip3 install tensorflow
When you have finished using your virtual environment, you can deactivate it by invoking the deactivate function as follows:
(p3-env)$ deactivate
Using Anaconda
Some people prefer to use the anaconda python distribution. This is also installed on all of the GPU servers.
You can create a python3 anaconda virtual environment by activating the system version:
source /opt/anaconda3/etc/profile.d/conda.csh
. /opt/anaconda3/etc/profile.d/conda.sh
Then you can use the conda command to create the virtual environment
conda create -y -p ~/p3-env anaconda
Then you need to activate your environment:
conda activate ~/p3-env
You can then use the conda command to install additional modules. i.e.
conda install -y keras matplotlib pandas spyder tensorflow pillow
When you have finished using your virtual environment you must deactivate it, to return to using the system default python:
conda deactivate
Removing the virtual environment
When you have finished using the virtual environment there is no uninstall process. To remove a python3 virtual environment you can just remove the directory:
rm -rf ~/p3-env
To remove an anaconda virtual environment:
conda remove --name p3-env