The keras, tensorflow, and theano packages are not natively installed on the Deepthought2 cluster for various technical reasons. What is provided instead are Singularity containers which have versions of both python2 and python3 installed with these packages.
To use these python packages, you must load the appropriate environmental
module (e.g. module load keras) and then launch the python
interpretter inside the Singularity container. To help with this,
the following helper/wrapper scripts have been provided:
keras or keras-python2 will invoke a Keras-enabled
python2 interpretter within the container. The backend will be defaulted
as per standard Keras rules. Any arguments given will be passed to the
python command, so you can do something like keras myscript.pl
keras-python3 will behave as above, but invoke a Keras-enabled
python3 interpretter within the container.
keras-tensorflow, keras-python2-tensorflow will
behave similarly to the keras command, but will force the use
of tensorflow as the backend.
keras-theano, keras-python2-theano will
behave similarly to the keras command, but will force the use
of theano as the backend.
keras-python3-tensorflow, keras-python3-theano
behave like keras-python2-tensorflow and
keras-python2-theano, resp., but invoke a python3 interpretter.
In all cases, any arguments given to the wrapper scripts are passed directly to the python interpretter running within the container. E.g., you can provide the name of a python script, and that script will run in the python interpretter running inside your container. Your home and lustre directories are accessible from within the container, so you can read and write to files in those directories as usual.
Note that if you load the keras environmental module and then issue the
python command, you will start up a natively installed python
interpretter which does NOT have the keras, etc. python modules
installed. You need to start one of the python interpretters inside the
container to get these modules --- you can either do that using the correct
singularity command, or use the friendlier wrapper scripts
described above.
It is hoped that for most users, the "containerization" of these packages should not cause any real issues, and hopefully not even really be noticed. However, there are some limitations to the use of containers:
foo is installed natively on Deepthought2, it is likely not
accessible from within the container (unless it was also installed inside that
container). virtualenv scripts
to install new python modules for use within the container, as that will be
installing packages natively, which would not then be available inside the
container.
However, you are permitted to create your own Singularity containers and use them on the Deepthought2 cluster. You will need to have root access on some system (e.g. a workstation or desktop) to create your own Singularity containers (we cannot provide you root access on the Deepthought2 login or compute nodes), but if you have such you can build your own containers. You can also copy the system provided containers and edit them. More details can be found under the software page for Singularity.