ALERT! Warning: your browser isn't supported. Please install a modern one, like Firefox, Opera, Safari, Chrome or the latest Internet Explorer. Thank you!
Startseite » ... » Zentrale Einrichtungen  » ZIH  » Wiki
phone prefix: +49 351 463.....

HPC Support

Operation Status

Ulf Markwardt: 33640
Claudia Schmidt: 39833

hpcsupport@zih.tu-dresden.de

Login and project application

Phone: 40000
Fax: 42328
servicedesk@tu-dresden.de

You are here: Compendium » JupyterHub

JupyterHub

With our JupyterHub service we offer you now a quick and easy way to work with jupyter notebooks on Taurus.

Disclaimer

This service is provided "as-is", use at your own discretion. Please understand that JupyterHub is a complex software system of which we are not the developers and don't have any downstream support contracts for, so we merely offer an installation of it but cannot give extensive support in every case.

Access

NOTE: This service is only available for users with an active HPC project. See here how to apply for an HPC project.

JupyterHub is available here:
https://taurus.hrsk.tu-dresden.de/jupyter

Start a session

Start a new session by clicking on the button.

A form opens up where you can customize your session. Our simple form offers you the most important settings to start quickly.

session form

For advanced users we have an extended form where you can change many settings. You can:
  • modify Slurm parameters to your needs ( more about Slurm)
  • assign your session to a project or reservation
  • load modules from the LMOD module system
  • choose a different standard environment (in preparation for future software updates or testing additional features)

session form

You can save your own configurations as additional presets. Those are saved in your browser and are lost if you delete your browsing data. Use the import/export feature (available through the иии button) to save your presets in text files.

Applications

You can choose between JupyterLab or the classic notebook app.

JupyterLab

jupyterlab app

The main workspace is used for multiple notebooks, consoles or terminals. Those documents are organized with tabs and a very versatile split screen feature. On the left side of the screen you can open several views:
  • file manager
  • controller for running kernels and terminals
  • overview of commands and settings
  • details about selected notebook cell
  • list of open tabs

Classic notebook

filebrowser in jupyter notebook server

jupyter_notebook_example_matplotlib

Initially you will get a list of your home directory. You can open existing notebooks or files by clicking on them.

Above the table on the right side is the "New ⏷" button which lets you create new notebooks, files, directories or terminals.

The notebook

In JupyterHub you can create scripts in notebooks.
Notebooks are programs which are split in multiple logical code blocks. In between those code blocks you can insert text blocks for documentation and each block can be executed individually. Each notebook is paired with a kernel which runs the code. We currently offer one for Python and one for R.

Stop a session

It's good practise to stop your session once your work is done. This releases resources for other users and your quota is less charged. If you just log out or close the window your server continues running and will not stop until the Slurm job runtime hits the limit (usually 8 hours).

At first you have to open the JupyterHub control panel.

JupyterLab: Open the file menu and then click on Logout. You can also click on "Hub Control Panel" which opens the control panel in a new tab instead.

Classic notebook: Click on the control panel button on the top right of your screen.

Now you are back on the JupyterHub page and you can stop your server by clicking on .

Error handling

We want to explain some errors that you might face sooner or later. If you need help open a ticket at HPC support.

Error while starting a session

This message often appears instantly if your Slurm parameters are not valid. Please check those settings against the available hardware. Useful pages for valid Slurm parameters:

Error message in JupyterLab

If the connection to your notebook server unexpectedly breaks you maybe will get this error message.
Sometimes your notebook server might hit a Slurm or hardware limit and gets killed. Then usually the logfile of the corresponding Slurm job might contain useful information. These logfiles are located in your home directory and have the name "jupyter-session-.log".


Advanced tips

Standard environments

The default python kernel uses conda environments based on the Watson Machine Learning Community Edition (formerly PowerAI) package suite. You can open a list with all included packages through our spawner form:

environment_package_list.png

This list shows all packages of the currently selected conda environment. This depends on your settings for partition (cpu architecture) and standard environment.

Here's a short list of some included software:
  generic* ml
Python 3.6.9 3.6.9
R** 3.5.1 3.6
WML CE 1.6.1 1.6.1
PyTorch 1.1 1.1
TensorFlow 1.14 1.14
Keras 2.2.4 2.2.4
numpy 1.14.2 1.14.5
matplotlib 3.1.1 3.1.1
* generic = all partitions except ml

** R is loaded from the module system

Creating and using your own environment

Interactive code interpreters which are used by Jupyter Notebooks are called kernels.
Creating and using your own kernel has the benefit that you can install your own preferred python packages and use them in your notebooks.

We currently have two different architectures at Taurus. Build your kernel environment on the same architecture that you want to use later on with the kernel. In the examples below we use the name "my-kernel" for our user kernel. We recommend to prefix your kernels with keywords like "intel", "ibm", "ml", "venv", "conda". This way you can later recognize easier how you built the kernel and on which hardware it will work.

Intel nodes (e.g. haswell, gpu2):

srun --pty -n 1 -c 2 --mem-per-cpu 2583 -t 08:00:00 bash -l

If you don't need Sandy Bridge support for your kernel you can create your kernel on partition 'haswell'.

Power nodes (ml partition):

srun --pty -p ml -n 1 -c 2 --mem-per-cpu 5772 -t 08:00:00 bash -l

Create a virtual environment in your home directory. You can decide between python virtualenvs or conda environments.

Python virtualenv

$ module load Python/3.6.6-fosscuda-2018b

$ mkdir user-kernel

$ cd user-kernel

$ virtualenv --system-site-packages my-kernel
Using base prefix '/sw/installed/Python/3.6.6-fosscuda-2018b'
New python executable in .../user-kernel/my-kernel/bin/python
Installing setuptools, pip, wheel...done.

$ source my-kernel/bin/activate

(my-kernel) $ pip install ipykernel
Collecting ipykernel
...
Successfully installed ... ipykernel-5.1.0 ipython-7.5.0 ...

(my-kernel) $ python -m ipykernel install --user --name my-kernel --display-name="my kernel"
Installed kernelspec my-kernel in .../.local/share/jupyter/kernels/my-kernel

[now install additional packages for your notebooks]

(my-kernel) $ deactivate

Conda environment

Load the needed module for Intel nodes

$ module load Anaconda3

... or for IBM nodes (ml partition):

$ module load PythonAnaconda

Continue with environment creation, package installation and kernel registration:

$ mkdir user-kernel

$ conda create --prefix /home/<USER>/user-kernel/my-kernel python=3.6
Collecting package metadata: done
Solving environment: done
[...]

$ source activate /home/<USER>/user-kernel/my-kernel

$ conda install ipykernel
Collecting package metadata: done
Solving environment: done
[...]

$ python -m ipykernel install --user --name my-kernel --display-name="my kernel"
Installed kernelspec my-kernel in [...]

[now install additional packages for your notebooks]

$ conda deactivate

Now you can start a new session and your kernel should be available.

In JupyterLab:

Your kernels are listed on the launcher page:

jupyterlab_app.png

You can switch kernels of existing notebooks in the menu:

jupyterlab_app.png

In classic notebook app:

Your kernel is listed in the New menu:

jupyterlab_app.png

You can switch kernels of existing notebooks in the kernel menu:

jupyterlab_app.png

Loading modules

You have now the option to preload modules from the LMOD module system.
Select multiple modules that will be preloaded before your notebook server starts. The list of available modules depends on the module environment you want to start the session in (scs5 or ml). The right module environment will be chosen by your selected partition.