{"id":4018,"date":"2017-07-12T17:17:35","date_gmt":"2017-07-12T17:17:35","guid":{"rendered":"http:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/?page_id=4018"},"modified":"2017-07-12T17:26:58","modified_gmt":"2017-07-12T17:26:58","slug":"jupyter-notebook","status":"publish","type":"page","link":"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/software\/applications\/jupyter-notebook\/","title":{"rendered":"Jupyter Notebook"},"content":{"rendered":"

Overview<\/h2>\n

Jupyter Notebook<\/a> is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and explanatory text.<\/p>\n

It is installed as part of Anaconda python. The modulefile described here provides a helper script to submit Jupyter Notebook jobs correctly on the CSF. The version number is the version of our helper script. You must also <\/em> load an Anaconda Python modulefile to access your required version of Jupyter Notebook.<\/p>\n

Please read carefully the instructions below when connecting a web-browser to your jupyter notebook on the CSF – there are some extra steps needed that you would not normally do if you were running a jupyter notebook on your local desktop\/laptop.<\/p>\n

Restrictions on use<\/h2>\n

There are no restrictions on accessing this software.<\/p>\n

Set up procedure<\/h2>\n

To access the software you must first load the modulefile:<\/p>\n

\r\n# The version number of the version of our helper script, not jupyter nodebook.\r\nmodule load apps\/binapps\/jupyter-notebook\/1.0\r\n<\/pre>\n
You must<\/strong> also load the version of Anaconda Python you require. To see available version run:<\/p>\n
\r\nmodule avail apps\/binapps\/anaconda\r\n<\/pre>\n
This will list the available modulefiles. Use the module load<\/code> to load your required version.<\/p>\n
Initial Setup (required!)<\/h2>\n
Before running Jupyter Notebook for the first time, a configuration file needs to be generated. You only need to do this once and it can be done on the login node. <\/p>\n
After loading the modulefile, run the following:<\/p>\n
\r\njupyter-notebook --generate-config\r\n\r\n# It will output:\r\n[I 10:03:01.010 NotebookApp] Writing notebook server cookie secret to \/run\/...\/notebook_cookie_secret\r\nWriting default config to: \/mnt\/iusers01\/xy01<\/em>\/mabcxyz1<\/em>\/.jupyter\/jupyter_notebook_config.py\r\n   #\r\n   # The xy01<\/em> group and mabcxyz1<\/em> username will differ\r\n<\/pre>\n
The command has reported where it has generated the configuration file (within your home area). The file is at (where ~<\/code> means your home<\/em> directory):<\/p>\n
\r\n~\/.jupyter\/jupyter_notebook_config.py\r\n<\/pre>\n
You should now create a password to protect access to your notebooks when the server is running. NOTE<\/strong>: please do NOT<\/strong> use your central IT password. This is not a terribly secure method of protecting your notebooks but will offer some protection. Please do NOT use a password that is used on other systems.<\/p>\n
Run the following on the login node:<\/p>\n
\r\npython -c \"from notebook.auth import passwd; print(passwd())\"\r\n\r\n# It will display:\r\nEnter password: Enter your new notebook password and press [return]<\/strong>\r\nVerify password: Enter the same password again and press [return]<\/strong>\r\nsha1:bf8810b0e5a2:bac141e3590ce339c955f21dd6b7de1b20<\/em>\r\n  #\r\n  # This line is the result (yours will be different).\r\n  # We need to copy this in to the config file generated earlier.\r\n<\/pre>\n
Take a copy of the above line. Now edit the config file generated earlier:<\/p>\n
\r\ngedit ~\/.jupyter\/jupyter_notebook_config.py\r\n  #\r\n  # Could also use nano or vi or emacs editors\r\n<\/pre>\n
Search the file for a line that looks like:<\/p>\n
\r\n# c.NotebookApp.password = ''\r\n<\/pre>\n
and change it to be:<\/p>\n
\r\nc.NotebookApp.password = 'sha1:bf8810b0e5a2:bac141e3590ce339c955f21dd6b7de1b20<\/em>'\r\n#\r\n# Notice we have remove the '#' at the start\r\n<\/pre>\n
where the sha1:....<\/code> is copied from the output generated earlier.<\/p>\n
Now search the config file for a line that looks like:<\/p>\n
\r\n# c.NotebookApp.ip = 'localhost'\r\n<\/pre>\n
Change this to be:<\/p>\n
\r\nc.NotebookApp.ip = '*'\r\n#\r\n# Notice we have remove the '#' at the start\r\n<\/pre>\n
When you connect to a jupyter notebook server with a web-browser you will be asked for the password you specified earlier. Note that this does NOT use https (at the moment). So you should NOT use a password that is of value to you.<\/p>\n
That is all of the setup we need to do. You can now start a jupyter-notebook job as described below. <\/p>\n
Running the application<\/h2>\n
Your jupyter notebook will run in the batch system on a compute node. The basic procedure is as follows:<\/p>\n
    \n
  1. Submit a jupyter notebook job on the CSF and wait for it to run. We do this using a helper script named jupyter-notebook-csf<\/code> run on the login node – no need<\/strong> to write a jobscript yourself!<\/li>\n
  2. On your local desktop\/laptop create an SSH tunnel to the CSF compute node running your jupyter notebook.<\/li>\n
  3. On your local desktop\/laptop connect a web-browser to the SSH tunnel – it will then connect to the CSF job.<\/li>\n<\/ol>\n
    The above procedure is shown in detail below.<\/p>\n
    jupyter-notebook-csf<\/code> helper script<\/h3>\n
    The jupyter-notebook-csf<\/code> helper script mentioned above (used to submit a batch job from the login node) accepts the following flags:<\/p>\n
    \r\njupyter-notebook-csf -h\r\n\r\njupyter-notebook-csf [-p NUMCORES] [-s] [-j] [-k] [-P PORTNUM]\r\n\r\nStarts a Jupyter Notebook server on a compute node.\r\n\r\n-p NUMCORES<\/em>  -- Run in smp.pe with NUMCORES<\/em> cores. Default is 1, max 24.\r\n                Note: -p 1 will run as a serial job, not in smp.pe.\r\n-s           -- Run in the short area (shorter max runtime allowed).\r\n-j           -- Generate the jobscript but DO NOT submit it. Debugging only.\r\n-k           -- Keep the jobscript file after it has been submitted. Deleted by default.\r\n                The autogenerated jobscript is not very useful to you.\r\n-P PORTNUM   -- Port number from which to start searching for a free port for the\r\n                the notebook server to listen on. Default starting port is 8888.\r\n-h           -- Display this help.\r\n\r\nCheck the .o file for instructions on how to connect\r\na web-browser to the server once the job runs.\r\n<\/pre>\n
    But you only need to use the flags if you want more than one core and want to run on a particular type of compute node to get a certain amount of memory. Some examples:<\/p>\n
    \r\njupyter-notebook-csf             # runs a serial (1-core) job\r\njupyter-notebook-csf -p 4        # runs a parallel 4-core job\r\njupyter-notebook-csf -p 8 -s     # runs a parallel 8-core job in the short area (max 12 cores)\r\n<\/pre>\n
    We now go through the complete steps to using a Jupyter Notebook.<\/p>\n
    Complete Steps to Running a Jupyter Notebook<\/h3>\n
      \n
    1. Submit a jupyter notebook job using the helper script run on the login node:\n
      \r\n# Run in scratch area\r\ncd ~\/scratch\r\n\r\n# Choose your version of anaconda python\r\nmodule load apps\/binapps\/anaconda\/3\/4.2.0\r\n\r\n# Load the CSF helper script modulfile\r\nmodule load apps\/binapps\/jupyter-notebook\/1.0\r\n\r\n# Run one<\/em> of the following examples on the login node:\r\n\r\njupyter-notebook-csf             # Submit a serial (1-core) job\r\njupyter-notebook-csf -p 4        # Submit a parallel 4-core job\r\njupyter-notebook-csf -p 4 -s     # As above but runs in the 'short' area\r\n<\/pre>\n
      The output from the helper script can be one of two things:<\/p>\n
        \n
      1. If the job runs straight away in the batch system you’ll see:\n
        \r\nYour job 83873 (\"jnotebook\") has been submitted\r\nChecking if job has already started...\r\n\r\nStarting jupyter-notebook on node012 port 8888\r\n\r\nYou must now ssh in to the CSF from your local machine and tunnel\r\nto the backend compute node. Use the following command on *your*\r\ncomputer (not on the CSF):\r\n\r\n  ssh -L 8888<\/strong>:node012<\/strong>:8888<\/strong> mabcxyz<\/em>@csf2.itservices.manchester.ac.uk\r\n\r\nYou should then start a web-browser on *your* computer and browse to:\r\n\r\n  http:\/\/localhost:8888<\/strong>\r\n\r\nWhen you have logged out of your notebook in the web-browser you *must*\r\nrun the following command on the CSF to stop the Jupyter server:\r\n\r\n  qdel 83873<\/strong>\r\n\r\nThen log out of your tunnelled ssh session.\r\n<\/pre>\n
        This means that the jupyter-notebook job started immediately on a compute node so you are able to proceed with setting up the SSH tunnel from your local computer to the CSF and then connecting your web-browser to it by following the instructions in the above message (see below for an example).\n<\/li>\n
      2. If the job doesn’t run immediately but waits in the batch queue:\n
        \r\nThe job is probably still waiting to run.\r\n\r\nYou *cannot* connect a web-browser to the CSF until the job runs.\r\nPlease be patient.\r\n\r\nTo check your job's status, run:\r\n\r\n   jnotebook-status 83873<\/strong>\r\n\r\nWhen the job eventually runs you should then run the following command\r\nto get further instructions on how to connect your web-browser to the\r\nJupyter notebook:\r\n\r\n   cat jnotebook.o83873<\/strong>\r\n\r\nFollow the instructions given in that file.\r\n<\/pre>\n
        Note that the job-ID 83873<\/code><\/strong> will be different for your job. The above message means that you must check on your job using the jnotebook-status JOBID<\/em><\/code> command to see when it is running. When it is running you can proceed with the instructions given in the file: jnotebook.oJOBID<\/em><\/code> where JOBID<\/em> is the job-ID of your own job.\n<\/li>\n<\/ol>\n<\/li>\n
      3. Once the notebook job is running you can now set up the SSH tunnel on your<\/strong> computer (not the CSF). You’ll need to open a terminal window. For example, if you are on Windows use MobaXterm, if on Mac use the Terminal<\/em> application and if on Linux use an Xterm or GNOME Terminal. Then run:\n
        \r\nssh -L 8888<\/strong>:node012<\/strong>:8888<\/strong> username<\/em>@csf.itservices.manchester.ac.uk\r\n         ^     ^      ^      ^\r\n         |     |      |      |\r\n         |     |      |      +----- use your own central IT username here\r\n         +-----+------+\r\n               |\r\n               |\r\n               +---------- These values (port numbers and compute node name) are reported in\r\n                           the above output when you submitted the notebook job. You should\r\n                           use the values appropriate to your job (the port will likely be\r\n                           8888 but could be different. The node name will very likely be\r\n                           a different node number).\r\n\r\n<\/pre>\n<\/li>\n
      4. Now start a web-browser on your PC<\/strong> and browse to:\n
        \r\nhttp:\/\/localhost:8888<\/strong>\r\n                   #\r\n                   # Use the same port number as used above\r\n<\/pre>\n
        You should be taken to the Jupyter Notebook password page. Enter the password your generated earlier when you did the one-time setup to create the config file.<\/li>\n<\/ol>\n
        You can now use the Jupyter Notebook through your own web-browser. Any code you enter is executed on the CSF compute node on which the Jupyter Notebook server is running. Hence you should ensure your code uses the correct number of cores you reserved when you submitted the notebook job.<\/p>\n
        Stopping a Notebook<\/h3>\n
        When you have finished with the jupyter notebook you must log out of the notebook in your web-browser and remove the job that is running on the CSF compute node. Do the following:<\/p>\n
          \n
        1. Log out of the server through your web-browser (hit the logout button)<\/li>\n
        2. Stop the CSF job that is running the server:\n
          \r\n# On the CSF login node:\r\nqdel 83873<\/strong>\r\n      #\r\n      # Change the job id to be that of your notebook job.\r\n      # If unsure, run 'qstat' to see a list of your jobs.\r\n<\/pre>\n<\/li>\n
        3. Log out of the window running the ssh tunnel created earlier.<\/li>\n<\/ol>\n
          Please do remember to stop your CSF server job via the qdel<\/code> command. Logging out of the notebook in your web-browser does NOT stop the CSF job. You MUST do this using qdel<\/code> to free up the compute node for other users. <\/p>\n
          Further info<\/h2>\n