UoM::ITS::RI::FTM

Group

Feed The Monster


Syncing Teams and SharePoint Files to/from Linux

The Problem
So, we have multiple OneDrive clients for Linux which will sync files to/from OneDrive@365, making files available locally (in addition to the Web interface which can be used for sometimes missing functionality, like right-click-share). But what about Teams/SharePoint files?
Using OneDrive (ppa.launchpad.net/yann1c)
The project docs are available at:
Teams Files and Sharepoint
Teams files are available from the underlying SharePoint site, for example:
    Team           :  UOM-RIT-Linux-Guest-Testing  
    SharePoint Site:  https://livemanchesterac.sharepoint.com/sites/UOM-RIT-Linux-Guest-Testing
Install the Client from the ABraunegg Repo
You need an up-to-date client. Using the old client version results in something like
    onedrive.OneDriveException@src/onedrive.d(874): HTTP request returned status code 400 (Bad Request)
    {
        "correlation_id": "79e89d68-714b-4df9-96f9-d6701cd309e9",
        "error": "invalid_grant",
        "error_codes": [
            9002313
        ],
        "error_description": "AADSTS9002313: Invalid request. Request is malformed or invalid"

         ...blah...
         ...blah...
Follow the instructions at the ABraunegg Github site.
Recipe
  1.         mkdir ~/SharePoint-UoM-Research-IT-RI-Leads
            mkdir ~/.config/onedrive-SharePoint-UoM-Research-IT-RI-Leads
    
            onedrive --get-O365-drive-id UoM-Research-IT-RI-Leads
    
                Configuration file successfully loaded
                Configuring Global Azure AD Endpoints
                Initializing the Synchronization Engine ...
                Office 365 Library Name Query: UoM-Research-IT-RI-Leads
                -----------------------------------------------
                Site Name:    UoM-Research-IT-RI-Leads
                Library Name: Documents
                drive_id:     b!WPy...blah...
                Library URL:  https://livemanchesterac.sharepoint.com/sites/UoM-Research-IT-RI-Leads/Shared%20Documents
                -----------------------------------------------
                Site Name:    UoM-Research-IT-RI-Leads
                Library Name: Teams Wiki Data
                drive_id:     b!WPy...blah...
                Library URL:  https://livemanchesterac.sharepoint.com/sites/UoM-Research-IT-RI-Leads/Teams%20Wiki%20Data
    	
            wget https://raw.githubusercontent.com/abraunegg/onedrive/master/config -O ~/.config/onedrive-UoM-Research-IT-RI-Leads/config
    
            emacs -nw .config/onedrive-UoM-Research-IT-RI-Leads/config
    
                sync_dir = "SharePoint-UoM-Research-IT-RI-Leads"
                drive_id = "b!WPy...blah..."
  2. Usual cut-n-past authentication steps with URLS...
            onedrive --confdir="~/.config/onedrive-UoM-Research-IT-RI-Leads"
     
                Configuration file successfully loaded
                Configuring Global Azure AD Endpoints
                Authorize this app visiting:
    
                 ...usual thing...
    
                Application has been successfully authorised, however no additional command switches were provided.
  3.      onedrive --confdir="~/.config/onedrive-UoM-Research-IT-RI-Leads" --synchronize --verbose --dry-run
     
             Using 'user' Config Dir: /home/simonh/.config/onedrive-UoM-Research-IT-RI-Leads
              ...blah...
             All operations will be performed in: SharePoint-UoM-Research-IT-RI-Leads
             Application version: v2.4.13-0~202107190106~ubuntu20.04.1
             Account Type: documentLibrary
             Default Drive ID: b!WPy...blah...
              ...blah...
             Downloading file General/2011-10-28-wuss-guarding-taragon.jpeg ... done.
             Uploading differences of SharePoint-UiM-Research-IT-RI-Leads
              ...blah...
  4. Remains to add this to systemd:
          prompt> cat Dropbox/UoM/Systemd/sharepoint-ri-leads.service
          
          [Unit]
          Description=OneDrive Free Client
          Documentation=https://github.com/abraunegg/onedrive
          After=network-online.target
          Wants=network-online.target
    
          [Service]
          ExecStart=/usr/bin/onedrive --confdir="~/.config/onedrive-UoM-Research-IT-RI-Leads" --monitor
    
          Restart=on-failure
          RestartSec=3
          RestartPreventExitStatus=3
    
          [Install]
          WantedBy=default.target
    
          prompt> systemctl --user enable /home/simonh/Dropbox/UoM/Systemd/sharepoint-ri-leads.service
          # ...full path to service file required...
          
          Created symlink /home/simonh/.config/systemd/user/default.target.wants/sharepoint-ri-leads.service
              → /home/simonh/Dropbox/UoM/Systemd/sharepoint-ri-leads.service.
    
          prompt> systemctl --user start sharepoint-ri-leads.service
          prompt> systemctl --user status sharepoint-ri-leads.service
          sharepoint-ri-leads.service - OneDrive Free Client
          Loaded: loaded (/home/simonh/.config/systemd/user/sharepoint-ri-leads.service; enabled; vendor preset: enabled)
          Active: active (running) since Wed 2022-06-08 13:02:37 BST; 4s ago
          Docs: https://github.com/abraunegg/onedrive
          Main PID: 977611 (onedrive)
          CGroup: /user.slice/user-1000.slice/user@1000.service/sharepoint-ri-leads.service
                  └─977611 /usr/bin/onedrive --confdir=~/.config/onedrive-SharePoint-UoM-Research-IT-RI-Leads --monitor
    
          Jun 08 13:02:37 talby onedrive[977611]: Configuring Global Azure AD Endpoints
          Jun 08 13:02:38 talby onedrive[977611]: Initializing the Synchronization Engine ...
          Jun 08 13:02:38 talby onedrive[977611]: Initializing monitor ...
          Jun 08 13:02:38 talby onedrive[977611]: OneDrive monitor interval (seconds): 300
          Jun 08 13:02:38 talby onedrive[977611]: Uploading new file ./Directory/File.txt ...
          Jun 08 13:02:40 talby onedrive[977611]: [470B blob data]


Using Mbsync with DavMail and O365 (2020 August)

The Problem
Below we show how to set up DavMail with Mutt to access O365-based email over IMAP. It works, but its slow; the user experience is wait, wait wait. . . Can we use Mbsync to mirror remote 0365 to local Mail_Dir?
Config: ~/.mbsyncrc
# ------------------------------------------------------------------------------
# ------------------------------------------------------------------------------

#  mbsync -c ~/.mbsyncrc Personal-365

# ------------------------------------------------------------------------------
# -- IMAP for Personal 365 via DavMail :
# ------------------------------------------------------------------------------

IMAPAccount Personal-365
#
Host 127.0.0.1
Port 1143
SSLType None
    # ...for DavMail...
#
User "simonhtest666@outlook.com"
Pass "mypassword"
####PassCmd "pass work/email"
AuthMechs LOGIN

# ------------------------------------------------------------------------------
# -- Remote and local "stores" :
# ------------------------------------------------------------------------------

IMAPStore Personal-365-remote
#
Account Personal-365


MaildirStore Personal-365-local
#
Subfolders Verbatim
    # ...Copy folder hierarchy...
#
Path ~/Mbsync-Personal-365/
    # ...The trailing "/" is important...
#
Inbox ~/Mbsync-Personal-365/Inbox


# ------------------------------------------------------------------------------
# -- Sync "channel" :
# ------------------------------------------------------------------------------

Channel Personal-365
#
Master :Personal-365-remote:
Slave :Personal-365-local:
#
Patterns *
    # ...Include everything...
#
Create Both
    # ....Automatically create missing mailboxes, both locally and on the server....
#
SyncState *
    # ...Save the synchronization state files in the relevant directory...

# ------------------------------------------------------------------------------
# END.
# ------------------------------------------------------------------------------
Run it:
    mbsync -c ~/.mbsyncrc Personal-365
Configure Mutt to talk to local Mail_Dir
...


Accessing OLE Embedded PDFs in a PPTX on Linux (2020 July)

The Problem
I received a .pptx file with embedded PDFs. Libreoffice can't see them; neither can Google Docs; neither can Microsoft 365!!!!!
Solution
Unzip the .pptx and cd to ppt/embedded to find the .bin files which are the PDFs — but with some header crap which upsets PDF readers. Remove the header crap using bbe:
    #!/bin/bash

    for i in *.bin; do bbe -b ':/%PDF/' -e 'D 1' $i > $i.nearly; done

    for i in *.nearly; do echo -n %PDF | cat - $i > $i.pdf; done
If you are more familiar with bbe that I am you might be able to tidy that up.


Accessing O365 2FA Email using DavMail and Mutt (2020 May)

Background
UoM is migrating away from using on-site Exchange, with IMAP enabled to those what want it, to using Microsoft Office 365. This causes issues:
  • On-site, 2FA is done at the network level, i.e., to access IMAP you need to be connected to the Uni VPN, even if on campus. That's fine. Once connected to the VPN, IMAP works (with SFA) as normal.
  • Off-site/365 UoM 2FA is at the application level; not a good design but true. UoM's 365 is linked to UoM AD and hence Duo for 2FA. Alternatively one can use IMAP, with OAuth2.
Issue
2FA support from Linux email clients is patchy and in development. Evolution is ok; bleeding edge Thunderbird will do IMAP/OAuth2. But. . .
Mutt
I love Mutt. I want to use Mutt. Google suggests trying an IMAP proxy: DavMail.
DavMail
Quoting the Web site: Ever wanted to get rid of Outlook? DavMail is a POP/IMAP/SMTP/Caldav/Carddav/LDAP Exchange gateway allowing users to use any mail/calendar client with an Exchange server. More at http://davmail.sourceforge.net.
An Aside on Testing IMAP Connectivity and Authentication
Telnet:
  • Bad idea, as does not escape special characters, which can lead to a great deal of frustration until you realise, use Curl instead, below...
What about IMAPS?
  • use openssl _sclient as a test client :-)
More:
  • https://tewarid.github.io/2011/05/10/access-imap-server-from-the-command-line-using-openssl.html
So, what do we do with Telnet/openssl? Basics of the protocol:
    openssl s_client -crlf -connect imap.gmail.com:993
    ...blah, blah...
    ---
    * OK The Microsoft Exchange IMAP4 service is ready. [QgBOADYAUA...AbwBtAA==]
    01 tag login simonhtest666@outlook.com mypasswordwithspecialchars
    01 OK LOGIN completed.
    02 LIST "" "*"
    * LIST (\HasNoChildren) "/" Archive
    * LIST (\HasChildren \Trash) "/" Deleted
    * LIST (\HasChildren) "/" Deleted/imaps:
    .    .
    .    .
    * LIST (\HasNoChildren) "/" Test_Folder
    02 OK LIST completed.
    03 SELECT INBOX
    * 3 EXISTS
    .    .
    .    .
    * OK [UIDNEXT 50] The next unique identifier value
    03 OK [READ-WRITE] SELECT completed.
    04 STATUS INBOX (MESSAGES)
    * STATUS "INBOX" (MESSAGES 27)
    04 OK STATUS completed.
    05 FETCH 22:27 (BODY[HEADER])          # ...fetch headers of last few messages...
    * 22 FETCH (BODY[HEADER] {321}
    Date: Sat, 15 Feb 2020 10:09:06 +0000
    From: Simon Hood <simonhtest666@outlook.com>
    .    .
    FLAGS (\Seen))
    * 3 FETCH (BODY[HEADER] {698}
    .    .
    Sender: Simon Hood <simonhtest666@outlook.com>
    Date: Sat, 8 Feb 2020 09:08:18 +0000
    .    .   
    FLAGS (\Seen))
    05 OK FETCH completed.
    06 LOGOUT
    * BYE Microsoft Exchange Server IMAP4 server signing off.
    06 OK LOGOUT completed.
  • You must enter a lable (e.g., 01, 02...) at the start of each command, which prepends the last line of the output
  • More: https://tewarid.github.io/2011/05/10/access-imap-server-from-the-command-line-using-openssl.html
And Curl does the messy stuff for you:
  • Example
            curl --url "imap://localhost:1143" \
                  --user "simonhtest666@outlook.com:mypasswordwithspecialcharacters"
            # ...lists existing folders...
  • More on Curl and IMAP: https://debian-administration.org/article/726/Performing_IMAP_queries_via_curl
DavMail Version
The version in Mint 19.3 it tooooo old and does not have 0365Interactive which is essential.
  1. Get latest from DavMail site: davmail_5.5.1-3299-1_all.deb Version 5.1, for example, does not seem to store the oauth token (in .davmail.properties) so manual 2FA is required everytime, not every N days.
  2. Add dependencies, .deb packages from distro: libopenjfx-java, openjfx.
An Aside on Mutt Configuration Debugging
    mutt -D
        # ...defaults to compiled in defaults, with overrides from .muttrc...
SFA Testing with Personal Office365 Account
Start by solving half the problem:
  • I have both a UoM account and personal account at O365, the latter mostly so I can play (test things) without causing a disaster in a "production" setting.
  • This is a SFA, but good to get the basics tested. (Microsoft plan to disable SFA at some point.)
  • DavMail config — pertinent bits:
          Exhange Protocol:  EWS
          URL             :  https://outlook.office365.com/EWS/Exchange.asmx
  • Mutt config:
          source "~/.mutt/user-365-simonhtest-davmail-localhost-settings"
          source "~/.mutt/user-davmail-localhost-encryption-settings"
          source "~/.mutt/user-davmail-localhost-smtp-settings"
        
          set imap_user="$my_username@$my_domain"
          set imap_pass="$my_password"
    
          set header_cache     = "~/.mutt/user-simon.hood.uom-caches/headers"
          set message_cachedir = "~/.mutt/user-simon.hood.uom-caches/bodies"
    
          set folder="imap://localhost:1143/"
          set spoolfile ='imap://localhost:1143/'
          set trash = 'imap://localhost:1143/Trash'
    where user-365-simonhtest-davmail-localhost-settings
        
        set my_username = "simonhtestusername"
        set my_domain = "outlook.com"
        set my_password="mypassword"
    
        # -- Save sent messages:
        set copy=yes
        set record="imap://localhost:1143/Sent_Mutt"
            # ...davmail...
    and user-davmail-localhost-encryption-settings
        # ...turn it all off for use with DavMail on localhost...
        set ssl_force_tls=no
        set ssl_starttls=no
        # set ssl_use_tlsv1=yes
        # set ssl_use_tlsv1_1=no
    
        #### set ssl_use_sslv3=yes
        #### ...deprecated, no longer works...
    and user-davmail-localhost-smtp-settings
          set realname = "Simon Hood"
          set from = "$my_username@$my_domain"
          set smtp_pass = "$my_password"
          set smtp_url = "smtp://$my_username@$my_domain@localhost:1025/"
          set smtp_authenticators = "login"
UoM 365 Account with 2FA
DavMail settings:
    Exchange Protocol:  O365Interactive
Mutt settings, as above bar
    ####source "~/.mutt/user-365-simonhtest666-davmail-localhost-settings"
    source "~/.mutt/user-365-simon.hood.uom-davmail-localhost-settings"
where user-365-simon.hood.uom-davmail-localhost-settings
    set   my_username = "simon.hood"
    set   my_domain = "manchester.ac.uk"
    unset my_password

    set copy=yes
    set record="imap://localhost:1143/Sent_Mutt"
        # ...davmail...

    # set record="+imaps://outlook.office365.com/Sent_Mutt"
        # ...not via davmail...
        # ...creates nested:  imaps/outlook/office365/com/Sent_Mutt !!!
  • First time, DavMail pops up a GUI <--- the usual Duo "push": accept/save
  • Second time: just works. Duo only required every N days.
An Aside: Using a Perl Script as a Client
Perl: Mail::IMAPClient:
  1. Point the script at DavMail, i.e., localhost:1143.
  2. Crucial lines of code:
            my $yes = $imap->move("Logwatch", $uid);
            if ($yes) {  $imap->delete_message($uid);  }
    N.B. delete_message actually just marks a message for deletion and to actually delete messages we need to call expunge. At least via DavMail, this is a time-consuming call: DavMail works its way through all messages in the folder to get an update. Hence:
  3. Loop through say a few thousand messages marking some for deletion, then expunge; then loop through the next few thousand... And
  4. Turn OFF auto-expunge within DavMail otherwise IMAP move operations are veeeerrrryyyy sloooooowwwww. . ..
Remaining Issue
Performance is very poor. Potential solution, use Isync/MBsymc?


OneDrive on Linux — Singularity and Systemd (2020 April)

Background
  • Microsoft do not offer or support a OneDrive client for Linux. Neither do they have plans to develop one (private communication, 2020, April).
  • Skilion is now abandoned, but there is an active fork, simply called OneDrive.
  • Some excellent intro notes to be found at Linux Uprising, including on integration with systemd, i.e., running the client as a service with continuous updates.
  • Other FOSS clients exist, such as rclone and CloudCross.
But Mint 19.3 (based on Ubuntu 18.x) is too old. . .
  • apt-get install onedrive yields v1, i.e., Skilion. Reject.
  • Newer Debian/Ubuntu-based distros include the v2, maintained version: Ubuntu 19.x, Debian Unstable/Sid and Debian 10/Bullseye, e.g., https://packages.debian.org/sid/net/onedrive. Mint 20.x, based on Ubuntu 20.x, is not due out until June.
Build from source on my Mint 19.3?
  • apt-get install ldc, the required D compiler is too old a version to compile v2.
  • How about a singularity container-based install?
    • apt-get install Singularity container on Mint 19.3 yields v2.4, which is too old for current Singularity images — need v3.x.
    • A manual install of Debian Sid packages does the trick:
          singularity-container_3.5.2+ds1-1_amd64.deb
          containernetworking-plugins_0.8.5-1_amd64.deb
    Bootstrap an appropriate Singularity image:
    cat bootstrap.debian-testing.docker:
      Bootstrap: docker
      From: debian:testing
    
      %post
      apt-get -y update
      apt-get -y install onedrive libcurl3-gnutls libcurl4
    
      %environment
      export LC_ALL=C
    
      %runscript
      onedrive
    
      root@host> singularity build onedrive.sif bootstrap.debian-testing.docker
    
    Initial Testing:
      prompt> singularity shell onedrive.sif
        #
        # -- enter the container and get a shell
        
      Singularity>  ondrive --logout
      Singularity>  Authorize this app [by] visiting:
      https://login.microsoftonline.com/common/oauth2/v2.0/authorize?client_id...
        #
        # -- Enter the above in a browser and ensure you point it to the right
        #    Office 365 account (e.g., personal vs work).  Once you click "yes,
        #    permission granted", you get a blank page --- that's fine, copy this
        #    blank page's URI back to your Singularity promp...
    
      Enter the response uri:
        # ...copy the blank page URI into the commandline...
    
      Application has been successfully authorised, however no additional command
      switches were provided.
    
      Singularity> onedrive --synchronize --verbose --dry-run
        #
        # ...try it out --- files go to ~/OneDrive...
    
    More Testing
      prompt> singularity exec test.sif onedrive --synchronize --verbose
        #
        # ...Run onedrive inside the container from outside the container...
    
    Integrate with Systemd
    We want a service — in particular a user service (not a system service). So we want a systemd unit — a user unit.
    • Great into to Singularity under Systemd at Sylabs but we don't need singularity instances in our case, nor forking and bear in mind that we are not running this systemd instance as a (root) service but a user service.
    /usr/lib/systemd/user/onedrive.service:
      cat onedrive.service
      [Unit]
      Description=OneDrive Free Client for %i
      Documentation=https://github.com/abraunegg/onedrive
      After=network-online.target
      Wants=network-online.target
    
      [Service]
      #
      # Type=forking
      # Restart=always
      # ...what we might want for a singularity instance, e.g., a Web server...
      #
      Restart=on-failure
      RestartSec=3
      RestartPreventExitStatus=3
      #
      # User=%i
      # Group=users
      # ...we are starting our systemd instance as a user (not root) with "--user" so
      #    we don't want these:
      #     ...Failed to determine supplementary groups: Operation not permitted
      #        Failed at step GROUP...
      #
      ExecStart=/usr/bin/singularity exec /home/simonh/Downloads/test.sif onedrive --monitor --confdi\
      r=/home/%u/.config/onedrive
    
      [Install]
      WantedBy=multi-user.target
    
    Test it under systemd!
      host> systemctl --user enable onedrive
    	Created symlink /home/simonh/.config/systemd/user/multi-user.target.wants/onedrive.service --> /usr/lib/systemd/user/onedrive.service.
    			
        host>  systemctl --user start onedrive
        
        host>  systemctl --user status onedrive.service
        
        onedrive.service - OneDrive Free Client for
        Loaded: loaded (/usr/lib/systemd/user/onedrive.service; enabled; vendor preset: enabled)
        Active: active (running) since Wed 2020-04-22 09:07:46 BST; 2s ago
        Docs: https://github.com/abraunegg/onedrive
        Main PID: 2652 (starter-suid)
        CGroup: /user.slice/user-1000.slice/user@1000.service/onedrive.service
        --2652 Singularity runtime parent
        --2695 onedrive --monitor --confdir=/home/simonh/.config/onedrive
    
        host> journalctl --user-unit onedrive -f
        
        -- Logs begin at Thu 2019-01-17 10:46:42 GMT. --
        Apr 22 09:07:46 darkstar systemd[1358]: Started OneDrive Free Client for .
        Apr 22 09:07:46 darkstar singularity[2652]: Notification (dbus) server not available, disabling
        Apr 22 09:07:49 darkstar singularity[2652]: Initializing the Synchronization Engine ...
        Apr 22 09:07:49 darkstar singularity[2652]: Initializing monitor ...
        Apr 22 09:07:49 darkstar singularity[2652]: OneDrive monitor interval (seconds): 45
        Apr 22 09:10:50 darkstar singularity[2652]: Moving xmodmap to xmodmap-copy-simonh-test
        Apr 22 09:11:35 darkstar singularity[2652]: Syncing changes from OneDrive ...
        Apr 22 09:12:17 darkstar singularity[2652]: Uploading new file ./a_test_file_with_a_long_name .\
        ..
        Apr 22 09:12:19 darkstar singularity[2652]: [314B blob data]
        Apr 22 09:12:19 darkstar singularity[2652]: done.
        Apr 22 09:15:22 darkstar singularity[2652]: Syncing changes from OneDrive ...
        Apr 22 09:19:10 darkstar singularity[2652]: Syncing changes from OneDrive ...
    
    Yippee!


    WFH Cat-on-Keyboard Problem (2020 March)

    Background
    WFH because of the Covid-19 pandemic. Issue: working with laptop connected to external mouse, keyboard and monitor. Cat likes to curl and and snooze on laptop keyboard. Which is lovely. . .

    . . . but uneven pressure results in random keys getting pushed downnnnnnnnnnnnnnnnnnnnnnnnnnn and havoc ensures!
    Solution 1
    Place some padding in a folder on the keyboard. See photo. Limited success.
    Solution 2
    Disable laptop keyboard:
      prompt> xinput list
        Virtual core pointer                    id=2[master pointer  (3)]
          Virtual core XTEST pointer            id=4[slave  pointer  (2)]
          .   .
          .   .
        AT Translated Set 2 keyboard            id=17[slave  keyboard (3)]
        DELL Wireless hotkeys                   id=19[slave  keyboard (3)]
        Logitech USB Keyboard                   id=12[slave  keyboard (3)]
      prompt>
      prompt> xinput float 17
          # ...soft-detach laptop kbd...
          #    should I need to soft re-attach it: xinput reattach 17 3
    Complete success. Happy cat, happy me.
    Phase Two
    So, I swapped from my old Dell laptop (see pic) to a newish Lenovo Thinkpad. Mungo (cat) found new ways to cause trouble:
    1. Turning the trackpad off via xinput was not enough; needed to explicitly turn off the buttons too.
    2. And there is also the function keys, which I don't seem to be able to turn off via xinput
    Complete recipe:
      xinput float 10
          # ...disable kbd...
    
      xinput float 13
          # ...extra buttons --- but what buttons?...
      
      xinput float 11
          # ...disable trackpad...
    
      xinput --set-button-map 12 0 0 0 0 0 0 0
          # ...disable trackpad buttons...
          #     -- "xinput list --long" to get info on buttons
    
      xinput float 6
      xinput float 8
          # ...disable power and sleep buttons (sneaky cat powered off my laptop
          #    at one point...
    
      # Q/ How to disable the function buttons??
      # Q/ How to disable the function buttons??
      #
      # !! Press Func+Esc !!
      # !! Works!
    


    Secure Public Job Submission Engines (JSEs) via Shared Filesystems — Drag-n-Drop (2016 April 30)

    Background
    Researchers' data is now commonly sensitive in some way (e.g., pseudo-anonymised patient data, or commercially-valuable data). So the consequences of a research computing platform being compromised ("hacked") and the data "leaked" are increasingly severe.

    So our compute platforms live in private IP space (i.e., connections to/from the platforms are not routed on/off campus); firewalls are default-deny (often in both directions); that private IP space is being structured into unrestricted, restricted and highly-restricted zones, with sensitive data being located in the latter.
    User Requirement — The Problem to be Addressed
    A publically-accessible Web interface (e.g., something Galaxy-based) which can submit computational jobs to our research computing platforms.

    This is not hard to set up — The Web site could use SGE's DRMAA API (e.g, Galaxy talks to SGE more or less out of the box). But that means we expose our compute platforms to the public internet via the Web site. Ain't Never Gonna Happen.

    It would be a bonus if the solution adopted is flexible enough to work with other JSEs, for example, a drag-n-drop submission system, or submission via SFTP.
    Solution Overview
    Keep the Web site and the compute platform walled-off from eachother by router ACLs and on-host firewalls, so network traffic cannot move from one to the other. Instead, have the Web site and the compute platform's SGE instance exchange messages and information via an dedicated, shared filesystem.
    1. The Web site "drops" a standard SGE .qsub script into the filesystem, e.g., my_job.qsub.
    2. A cron job on the compute platform's submit node polls this spool area, parses the script for errors: if no errors are found the job is submitted — with a job name based on that which SGE would normally give it, but with a long, random string appended to ensure a unique name; if the submission is successful the name is written into a .qsubmit file, in this case my_job.qsubmit; if the submission is unsuccessful, a .qfail file is written containing the exit code and contents of STDOUT and STDERR.
    3. Each time the cron job runs, SGE is interrogated for the status of the job, by means of the unique name (qstat -j <jobname) and the status information written to a .qstat file, assuming the job is still running.
    4. If the job is found to have finished, a .qacct file is written containing information obtained from SGE's accounting system about the job.
    The client (e.g. Galaxy) can watch for, and interrogate, the .qsubmit for the job name, the .qstat for job monitoring information and .qexit files for summary info about the completed (or possibly failed) job. And of course the client can access the usual job output files (both those from SGE) and any others generated by the job, by using the job name obtained from the .qsubmit file.
    Example Files
    N.B. If the job name that SGE would by default generate is less than 10 characters long, then it is padded with spaces to make it look normal to command-line users of qstat on the SGE submit node.

    
    




    NAT as a Service! (2016 April 09)

    Background
    All Research IT computational platforms (e.g., the CSF and DPSF (nee Hydra)) are directly accessible from on-campus only, for very good security-related reasons. Users who wish to access the platforms from off-campus must use either the University VPN, or our SSH gateways.

    In fact, no network traffic at all is possible, directly, from our computational platforms to/from off-campus.
    User Requirement
    Increasingly, platform users which to access data-sets which are located off-campus as part of their computational work. This is commonly the case in bioinformatics work.
    Solution: Network Address Transation as a Service!
    We provide a Linux server which offers Network-Address-Translation-as-a-Service. Example:
    • Connections come from campus-only hosts to a given port on the NATaas host;
    • are DNATed to the required IP:port,e.g., 3306 at EnsembleDB for there SQL service;
    • are SNATed to a public IP address and directed off-campus to the appropriate destination;
    • return traffic also goes through the NATaaS host.
    More. . .




    INteractive Computational LINux Environment — INCLINE aka iCSF (2013 May)

    Background
    So, we have been running batch computational platforms for many years. Yet many powerful workstations are procured each year for research groups and individuals all over campus. Such workstations:
    • are idle much of the time and therefore wasteful of financial resources;
    • make user/researcher support by Research IT very innefficient as each is set up slightly differently from the others, they are geographically dispersed and each requires installation of software applications.
    User Requirement
    Researchers:
    • often need to run GUI-based applications — use of qrsh on batch/queue platforms offers a far from delightful experience;
    • sometimes need to develop and debug code with frequent and short code runs;
    • may need to pre-process or post-process data the main processing of which is performed on our batch compute systems.
    We therefore need to provide an interactive computational resource, which can support use by many researchers simultaneously, and which mounts the same home an shared filesystems as are used on our batch platforms such as Redqueen and The CSF.
    Solution Overview
    Build a load-balanced SSH Linux Virtual Server!
    • Procure compute nodes with the same specification as those used in the CSF together with a LVS node for performing the routing and load-balancing.
    • Using dedicated network infrastructure, make the same home and shared storage areas as are mounted on Redqueen and the CSF available on all compute nodes.
    • Use mon to dynamically tweak the LVS config so as to perform load-balancing.
    More. . .




    RI on 10.xyz.pqr.0/24, SSH Gateways and Virtual Desktops (201. . .)

    Background
    User Requirement
    Solution:

    
    




    Redqueen and The CSF (201. . .)

    Background
    User Requirement
    Solution:

    
    





    Coming R-S-N. . .

    Future:

     -- DPSF + RVMI + JSE + NAT
     -- vlan tagging
    




    Millions of Small Files — SQL as a Service for HPC (2016: )

    Background
    User Requirement
    Solution:

    
    




    HPC, Storage and VMs with Dedicated Networking — A Research Computing Ecosystem (2015. . .)

    Background
    User Requirement
    Solution:

    
    




    INCLINE Part Two: Adding High-Memory Nodes and Virtual Desktop Servers (20. . .)

    Add high-mem nodes:

    PRE>
    
       ** added alias
    
        em1       Link encap:Ethernet  HWaddr B8:CA:3A:5F:D7:28
                  inet addr:10.99.203.43  Bcast:10.99.203.255  Mask:255.255.255.0
    
        em1:0     Link encap:Ethernet  HWaddr B8:CA:3A:5F:D7:28
                  inet addr:10.99.203.45  Bcast:10.255.255.255  Mask:255.0.0.0
    
      --------------------------------------
    
        ipvsadm -A -t 10.99.203.45:22 -p 86400
        ipvsadm -a -t 10.99.203.45:22 -r 10.7.7.9:22 -m
        ipvsadm -a -t 10.99.203.45:22 -r 10.7.7.10:22 -m
    
       ...so now have...
    
      ipvsadm -L -n --persistent-conn
      IP Virtual Server version 1.2.1 (size=4096)
      Prot LocalAddress:Port            Weight    PersistConn ActiveConn InActConn
        -> RemoteAddress:Port
    
      TCP  10.99.203.43:22 wlc persistent 86400
        -> 10.7.7.1:22                  1         1           0          0
        -> 10.7.7.2:22                  1         0           0          0
           .         .
           .         .
        -> 10.7.7.8:22                  1         2           0          0
    
      TCP  10.99.203.45:22 wlc persistent 86400
        -> 10.7.7.9:22                  1         1           0          0
        -> 10.7.7.10:22                 1         2           0          0
    

    Add directly-connected virtual desktop server:

    nyx3:/etc/hosts

      10.5.5.250          incline.itservices.manchester.ac.uk incline
      10.5.5.249          jabberwock
    

    nyx4:/etc/hosts:

      10.5.5.248          incline.itservices.manchester.ac.uk incline
      10.5.5.249          jabberwock
    

     ipvsadm -L
    
    IP Virtual Server version 1.2.1 (size=4096)
    Prot LocalAddress:Port Scheduler Flags
      -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
    
    TCP  10.5.5.249:ssh wlc
      -> jabberwock:ssh               Masq    1      0          0
    
    TCP  10.5.5.250:ssh wlc
      -> incline01:ssh                Masq    1      2          0
      -> incline02:ssh                Masq    1      3          0
      -> incline03:ssh                Masq    1      3          0
      -> incline04:ssh                Masq    1      2          0
    
    TCP  incline:ssh wlc
      -> incline01:ssh                Masq    1      4          0
      -> incline02:ssh                Masq    1      4          0
      -> incline03:ssh                Masq    1      4          0
      -> incline04:ssh                Masq    1      4          0
      -> incline05:ssh                Masq    1      5          0
      -> incline06:ssh                Masq    1      5          0
      -> incline07:ssh                Masq    1      5          0
      -> incline08:ssh                Masq    1      5          0
    
    TCP  incline256.itservices.manche wlc
      -> incline09:ssh                Masq    1      0          0
      -> incline10:ssh                Masq    1      0          0