Feed The Monster
Driverless Printing on Linux
lpadmin -p <myprintername> -E -v ipp://<ipaddress>/ipp/print -m everywhereThe everywhere model causes the printer referred to by the -v option to be queried. A list of printer capabilities is returned and a PPD is automatically generated by CUPS for use by command line programs and applications.
Syncing Teams and SharePoint Files to/from Linux
- The Problem
- So, we have multiple OneDrive clients for Linux which will sync files to/from OneDrive@365, making files available locally (in addition to the Web interface which can be used for sometimes missing functionality, like right-click-share). But what about Teams/SharePoint files?
- Using OneDrive (ppa.launchpad.net/yann1c)
- The project docs are available at:
- Teams Files and Sharepoint
-
Teams files are available from the underlying SharePoint site, for example:
Team : UOM-RIT-Linux-Guest-Testing SharePoint Site: https://livemanchesterac.sharepoint.com/sites/UOM-RIT-Linux-Guest-Testing - Install the Client from the ABraunegg Repo
-
You need an up-to-date client. Using the old client version results in something like
onedrive.OneDriveException@src/onedrive.d(874): HTTP request returned status code 400 (Bad Request) { "correlation_id": "79e89d68-714b-4df9-96f9-d6701cd309e9", "error": "invalid_grant", "error_codes": [ 9002313 ], "error_description": "AADSTS9002313: Invalid request. Request is malformed or invalid" ...blah... ...blah...Follow the instructions at the ABraunegg Github site. - Recipe
-
mkdir ~/SharePoint-UoM-Research-IT-RI-Leads mkdir ~/.config/onedrive-SharePoint-UoM-Research-IT-RI-Leads onedrive --get-O365-drive-id UoM-Research-IT-RI-Leads Configuration file successfully loaded Configuring Global Azure AD Endpoints Initializing the Synchronization Engine ... Office 365 Library Name Query: UoM-Research-IT-RI-Leads ----------------------------------------------- Site Name: UoM-Research-IT-RI-Leads Library Name: Documents drive_id: b!WPy...blah... Library URL: https://livemanchesterac.sharepoint.com/sites/UoM-Research-IT-RI-Leads/Shared%20Documents ----------------------------------------------- Site Name: UoM-Research-IT-RI-Leads Library Name: Teams Wiki Data drive_id: b!WPy...blah... Library URL: https://livemanchesterac.sharepoint.com/sites/UoM-Research-IT-RI-Leads/Teams%20Wiki%20Data wget https://raw.githubusercontent.com/abraunegg/onedrive/master/config -O ~/.config/onedrive-UoM-Research-IT-RI-Leads/config emacs -nw .config/onedrive-UoM-Research-IT-RI-Leads/config sync_dir = "SharePoint-UoM-Research-IT-RI-Leads" drive_id = "b!WPy...blah..."- Usual cut-n-past authentication steps with URLS...
onedrive --confdir="~/.config/onedrive-UoM-Research-IT-RI-Leads" Configuration file successfully loaded Configuring Global Azure AD Endpoints Authorize this app visiting: ...usual thing... Application has been successfully authorised, however no additional command switches were provided. onedrive --confdir="~/.config/onedrive-UoM-Research-IT-RI-Leads" --synchronize --verbose --dry-run Using 'user' Config Dir: /home/simonh/.config/onedrive-UoM-Research-IT-RI-Leads ...blah... All operations will be performed in: SharePoint-UoM-Research-IT-RI-Leads Application version: v2.4.13-0~202107190106~ubuntu20.04.1 Account Type: documentLibrary Default Drive ID: b!WPy...blah... ...blah... Downloading file General/2011-10-28-wuss-guarding-taragon.jpeg ... done. Uploading differences of SharePoint-UiM-Research-IT-RI-Leads ...blah...- Remains to add this to systemd:
prompt> cat Dropbox/UoM/Systemd/sharepoint-ri-leads.service [Unit] Description=OneDrive Free Client Documentation=https://github.com/abraunegg/onedrive After=network-online.target Wants=network-online.target [Service] ExecStart=/usr/bin/onedrive --confdir="~/.config/onedrive-UoM-Research-IT-RI-Leads" --monitor Restart=on-failure RestartSec=3 RestartPreventExitStatus=3 [Install] WantedBy=default.target prompt> systemctl --user enable /home/simonh/Dropbox/UoM/Systemd/sharepoint-ri-leads.service # ...full path to service file required... Created symlink /home/simonh/.config/systemd/user/default.target.wants/sharepoint-ri-leads.service → /home/simonh/Dropbox/UoM/Systemd/sharepoint-ri-leads.service. prompt> systemctl --user start sharepoint-ri-leads.service prompt> systemctl --user status sharepoint-ri-leads.service sharepoint-ri-leads.service - OneDrive Free Client Loaded: loaded (/home/simonh/.config/systemd/user/sharepoint-ri-leads.service; enabled; vendor preset: enabled) Active: active (running) since Wed 2022-06-08 13:02:37 BST; 4s ago Docs: https://github.com/abraunegg/onedrive Main PID: 977611 (onedrive) CGroup: /user.slice/user-1000.slice/user@1000.service/sharepoint-ri-leads.service └─977611 /usr/bin/onedrive --confdir=~/.config/onedrive-SharePoint-UoM-Research-IT-RI-Leads --monitor Jun 08 13:02:37 talby onedrive[977611]: Configuring Global Azure AD Endpoints Jun 08 13:02:38 talby onedrive[977611]: Initializing the Synchronization Engine ... Jun 08 13:02:38 talby onedrive[977611]: Initializing monitor ... Jun 08 13:02:38 talby onedrive[977611]: OneDrive monitor interval (seconds): 300 Jun 08 13:02:38 talby onedrive[977611]: Uploading new file ./Directory/File.txt ... Jun 08 13:02:40 talby onedrive[977611]: [470B blob data]
Using Mbsync with DavMail and O365 (2020 August)
- The Problem
- Below we show how to set up DavMail with Mutt to access O365-based email over IMAP. It works, but its slow; the user experience is wait, wait wait. . . Can we use Mbsync to mirror remote 0365 to local Mail_Dir?
- Config: ~/.mbsyncrc
-
# ------------------------------------------------------------------------------ # ------------------------------------------------------------------------------ # mbsync -c ~/.mbsyncrc Personal-365 # ------------------------------------------------------------------------------ # -- IMAP for Personal 365 via DavMail : # ------------------------------------------------------------------------------ IMAPAccount Personal-365 # Host 127.0.0.1 Port 1143 SSLType None # ...for DavMail... # User "simonhtest666@outlook.com" Pass "mypassword" ####PassCmd "pass work/email" AuthMechs LOGIN # ------------------------------------------------------------------------------ # -- Remote and local "stores" : # ------------------------------------------------------------------------------ IMAPStore Personal-365-remote # Account Personal-365 MaildirStore Personal-365-local # Subfolders Verbatim # ...Copy folder hierarchy... # Path ~/Mbsync-Personal-365/ # ...The trailing "/" is important... # Inbox ~/Mbsync-Personal-365/Inbox # ------------------------------------------------------------------------------ # -- Sync "channel" : # ------------------------------------------------------------------------------ Channel Personal-365 # Master :Personal-365-remote: Slave :Personal-365-local: # Patterns * # ...Include everything... # Create Both # ....Automatically create missing mailboxes, both locally and on the server.... # SyncState * # ...Save the synchronization state files in the relevant directory... # ------------------------------------------------------------------------------ # END. # ------------------------------------------------------------------------------ - Run it:
-
mbsync -c ~/.mbsyncrc Personal-365 - Configure Mutt to talk to local Mail_Dir
- ...
Accessing OLE Embedded PDFs in a PPTX on Linux (2020 July)
- The Problem
- I received a .pptx file with embedded PDFs. Libreoffice can't see them; neither can Google Docs; neither can Microsoft 365!!!!!
- Solution
-
Unzip the .pptx and cd to ppt/embedded to find the
.bin files which are the PDFs — but with some header crap which
upsets PDF readers. Remove the header crap using bbe:
#!/bin/bash for i in *.bin; do bbe -b ':/%PDF/' -e 'D 1' $i > $i.nearly; done for i in *.nearly; do echo -n %PDF | cat - $i > $i.pdf; doneIf you are more familiar with bbe that I am you might be able to tidy that up.
Accessing O365 2FA Email using DavMail and Mutt (2020 May)
- Background
-
UoM is migrating away from using on-site Exchange, with IMAP enabled to those
what want it, to using Microsoft Office 365. This causes issues:
- On-site, 2FA is done at the network level, i.e., to access IMAP you need to be connected to the Uni VPN, even if on campus. That's fine. Once connected to the VPN, IMAP works (with SFA) as normal.
- Off-site/365 UoM 2FA is at the application level; not a good design but true. UoM's 365 is linked to UoM AD and hence Duo for 2FA. Alternatively one can use IMAP, with OAuth2.
- Issue
- 2FA support from Linux email clients is patchy and in development. Evolution is ok; bleeding edge Thunderbird will do IMAP/OAuth2. But. . .
- Mutt
- I love Mutt. I want to use Mutt. Google suggests trying an IMAP proxy: DavMail.
- DavMail
- Quoting the Web site: Ever wanted to get rid of Outlook? DavMail is a POP/IMAP/SMTP/Caldav/Carddav/LDAP Exchange gateway allowing users to use any mail/calendar client with an Exchange server. More at http://davmail.sourceforge.net.
- An Aside on Testing IMAP Connectivity and Authentication
-
Telnet:
- Bad idea, as does not escape special characters, which can lead to a great deal of frustration until you realise, use Curl instead, below...
- use openssl _sclient as a test client :-)
- https://tewarid.github.io/2011/05/10/access-imap-server-from-the-command-line-using-openssl.html
openssl s_client -crlf -connect imap.gmail.com:993 ...blah, blah... --- * OK The Microsoft Exchange IMAP4 service is ready. [QgBOADYAUA...AbwBtAA==] 01 tag login simonhtest666@outlook.com mypasswordwithspecialchars 01 OK LOGIN completed. 02 LIST "" "*" * LIST (\HasNoChildren) "/" Archive * LIST (\HasChildren \Trash) "/" Deleted * LIST (\HasChildren) "/" Deleted/imaps: . . . . * LIST (\HasNoChildren) "/" Test_Folder 02 OK LIST completed. 03 SELECT INBOX * 3 EXISTS . . . . * OK [UIDNEXT 50] The next unique identifier value 03 OK [READ-WRITE] SELECT completed. 04 STATUS INBOX (MESSAGES) * STATUS "INBOX" (MESSAGES 27) 04 OK STATUS completed. 05 FETCH 22:27 (BODY[HEADER]) # ...fetch headers of last few messages... * 22 FETCH (BODY[HEADER] {321} Date: Sat, 15 Feb 2020 10:09:06 +0000 From: Simon Hood <simonhtest666@outlook.com> . . FLAGS (\Seen)) * 3 FETCH (BODY[HEADER] {698} . . Sender: Simon Hood <simonhtest666@outlook.com> Date: Sat, 8 Feb 2020 09:08:18 +0000 . . FLAGS (\Seen)) 05 OK FETCH completed. 06 LOGOUT * BYE Microsoft Exchange Server IMAP4 server signing off. 06 OK LOGOUT completed.- You must enter a lable (e.g., 01, 02...) at the start of each command, which prepends the last line of the output
- More: https://tewarid.github.io/2011/05/10/access-imap-server-from-the-command-line-using-openssl.html
- Example
curl --url "imap://localhost:1143" \ --user "simonhtest666@outlook.com:mypasswordwithspecialcharacters" # ...lists existing folders... - More on Curl and IMAP: https://debian-administration.org/article/726/Performing_IMAP_queries_via_curl
- DavMail Version
-
The version in Mint 19.3 it tooooo old and does not have 0365Interactive
which is essential.
- Get latest from DavMail site: davmail_5.5.1-3299-1_all.deb Version 5.1, for example, does not seem to store the oauth token (in .davmail.properties) so manual 2FA is required everytime, not every N days.
- Add dependencies, .deb packages from distro: libopenjfx-java, openjfx.
- An Aside on Mutt Configuration Debugging
-
mutt -D # ...defaults to compiled in defaults, with overrides from .muttrc... - SFA Testing with Personal Office365 Account
-
Start by solving half the problem:
- I have both a UoM account and personal account at O365, the latter mostly so I can play (test things) without causing a disaster in a "production" setting.
- This is a SFA, but good to get the basics tested. (Microsoft plan to disable SFA at some point.)
- DavMail config — pertinent bits:
Exhange Protocol: EWS URL : https://outlook.office365.com/EWS/Exchange.asmx - Mutt config:
source "~/.mutt/user-365-simonhtest-davmail-localhost-settings" source "~/.mutt/user-davmail-localhost-encryption-settings" source "~/.mutt/user-davmail-localhost-smtp-settings" set imap_user="$my_username@$my_domain" set imap_pass="$my_password" set header_cache = "~/.mutt/user-simon.hood.uom-caches/headers" set message_cachedir = "~/.mutt/user-simon.hood.uom-caches/bodies" set folder="imap://localhost:1143/" set spoolfile ='imap://localhost:1143/' set trash = 'imap://localhost:1143/Trash'where user-365-simonhtest-davmail-localhost-settingsset my_username = "simonhtestusername" set my_domain = "outlook.com" set my_password="mypassword" # -- Save sent messages: set copy=yes set record="imap://localhost:1143/Sent_Mutt" # ...davmail...and user-davmail-localhost-encryption-settings# ...turn it all off for use with DavMail on localhost... set ssl_force_tls=no set ssl_starttls=no # set ssl_use_tlsv1=yes # set ssl_use_tlsv1_1=no #### set ssl_use_sslv3=yes #### ...deprecated, no longer works...and user-davmail-localhost-smtp-settingsset realname = "Simon Hood" set from = "$my_username@$my_domain" set smtp_pass = "$my_password" set smtp_url = "smtp://$my_username@$my_domain@localhost:1025/" set smtp_authenticators = "login"
- UoM 365 Account with 2FA
-
DavMail settings:
Exchange Protocol: O365InteractiveMutt settings, as above bar####source "~/.mutt/user-365-simonhtest666-davmail-localhost-settings" source "~/.mutt/user-365-simon.hood.uom-davmail-localhost-settings"where user-365-simon.hood.uom-davmail-localhost-settingsset my_username = "simon.hood" set my_domain = "manchester.ac.uk" unset my_password set copy=yes set record="imap://localhost:1143/Sent_Mutt" # ...davmail... # set record="+imaps://outlook.office365.com/Sent_Mutt" # ...not via davmail... # ...creates nested: imaps/outlook/office365/com/Sent_Mutt !!!- First time, DavMail pops up a GUI <--- the usual Duo "push": accept/save
- Second time: just works. Duo only required every N days.
- An Aside: Using a Perl Script as a Client
-
Perl: Mail::IMAPClient:
- Point the script at DavMail, i.e., localhost:1143.
- Crucial lines of code:
my $yes = $imap->move("Logwatch", $uid); if ($yes) { $imap->delete_message($uid); }N.B. delete_message actually just marks a message for deletion and to actually delete messages we need to call expunge. At least via DavMail, this is a time-consuming call: DavMail works its way through all messages in the folder to get an update. Hence: - Loop through say a few thousand messages marking some for deletion, then expunge; then loop through the next few thousand... And
- Turn OFF auto-expunge within DavMail otherwise IMAP move operations are veeeerrrryyyy sloooooowwwww. . ..
- Remaining Issue
- Performance is very poor. Potential solution, use Isync/MBsymc?
OneDrive on Linux — Singularity and Systemd (2020 April)
- Background
-
- Microsoft do not offer or support a OneDrive client for Linux. Neither do they have plans to develop one (private communication, 2020, April).
- Skilion is now abandoned, but there is an active fork, simply called OneDrive.
- Some excellent intro notes to be found at Linux Uprising, including on integration with systemd, i.e., running the client as a service with continuous updates.
- Other FOSS clients exist, such as rclone and CloudCross.
- But Mint 19.3 (based on Ubuntu 18.x) is too old. . .
-
- apt-get install onedrive yields v1, i.e., Skilion. Reject.
- Newer Debian/Ubuntu-based distros include the v2, maintained version: Ubuntu 19.x, Debian Unstable/Sid and Debian 10/Bullseye, e.g., https://packages.debian.org/sid/net/onedrive. Mint 20.x, based on Ubuntu 20.x, is not due out until June.
- Build from source on my Mint 19.3?
-
- apt-get install ldc, the required D compiler is too old a version to compile v2.
- How about a singularity container-based install?
-
- apt-get install Singularity container on Mint 19.3 yields v2.4, which is too old for current Singularity images — need v3.x.
- A manual install of Debian Sid packages does the trick:
singularity-container_3.5.2+ds1-1_amd64.deb containernetworking-plugins_0.8.5-1_amd64.deb
- Bootstrap an appropriate Singularity image:
-
cat bootstrap.debian-testing.docker:
Bootstrap: docker From: debian:testing %post apt-get -y update apt-get -y install onedrive libcurl3-gnutls libcurl4 %environment export LC_ALL=C %runscript onedrive root@host> singularity build onedrive.sif bootstrap.debian-testing.docker
- Initial Testing:
-
prompt> singularity shell onedrive.sif # # -- enter the container and get a shell Singularity> ondrive --logout Singularity> Authorize this app [by] visiting: https://login.microsoftonline.com/common/oauth2/v2.0/authorize?client_id... # # -- Enter the above in a browser and ensure you point it to the right # Office 365 account (e.g., personal vs work). Once you click "yes, # permission granted", you get a blank page --- that's fine, copy this # blank page's URI back to your Singularity promp... Enter the response uri: # ...copy the blank page URI into the commandline... Application has been successfully authorised, however no additional command switches were provided. Singularity> onedrive --synchronize --verbose --dry-run # # ...try it out --- files go to ~/OneDrive... - More Testing
-
prompt> singularity exec test.sif onedrive --synchronize --verbose # # ...Run onedrive inside the container from outside the container... - Integrate with Systemd
-
We want a service — in particular a user service (not a system service).
So we want a systemd unit — a user unit.
- Great into to Singularity under Systemd at Sylabs but we don't need singularity instances in our case, nor forking and bear in mind that we are not running this systemd instance as a (root) service but a user service.
cat onedrive.service [Unit] Description=OneDrive Free Client for %i Documentation=https://github.com/abraunegg/onedrive After=network-online.target Wants=network-online.target [Service] # # Type=forking # Restart=always # ...what we might want for a singularity instance, e.g., a Web server... # Restart=on-failure RestartSec=3 RestartPreventExitStatus=3 # # User=%i # Group=users # ...we are starting our systemd instance as a user (not root) with "--user" so # we don't want these: # ...Failed to determine supplementary groups: Operation not permitted # Failed at step GROUP... # ExecStart=/usr/bin/singularity exec /home/simonh/Downloads/test.sif onedrive --monitor --confdi\ r=/home/%u/.config/onedrive [Install] WantedBy=multi-user.target
- Test it under systemd!
-
host> systemctl --user enable onedrive Created symlink /home/simonh/.config/systemd/user/multi-user.target.wants/onedrive.service --> /usr/lib/systemd/user/onedrive.service. host> systemctl --user start onedrive host> systemctl --user status onedrive.service onedrive.service - OneDrive Free Client for Loaded: loaded (/usr/lib/systemd/user/onedrive.service; enabled; vendor preset: enabled) Active: active (running) since Wed 2020-04-22 09:07:46 BST; 2s ago Docs: https://github.com/abraunegg/onedrive Main PID: 2652 (starter-suid) CGroup: /user.slice/user-1000.slice/user@1000.service/onedrive.service --2652 Singularity runtime parent --2695 onedrive --monitor --confdir=/home/simonh/.config/onedrive host> journalctl --user-unit onedrive -f -- Logs begin at Thu 2019-01-17 10:46:42 GMT. -- Apr 22 09:07:46 darkstar systemd[1358]: Started OneDrive Free Client for . Apr 22 09:07:46 darkstar singularity[2652]: Notification (dbus) server not available, disabling Apr 22 09:07:49 darkstar singularity[2652]: Initializing the Synchronization Engine ... Apr 22 09:07:49 darkstar singularity[2652]: Initializing monitor ... Apr 22 09:07:49 darkstar singularity[2652]: OneDrive monitor interval (seconds): 45 Apr 22 09:10:50 darkstar singularity[2652]: Moving xmodmap to xmodmap-copy-simonh-test Apr 22 09:11:35 darkstar singularity[2652]: Syncing changes from OneDrive ... Apr 22 09:12:17 darkstar singularity[2652]: Uploading new file ./a_test_file_with_a_long_name .\ .. Apr 22 09:12:19 darkstar singularity[2652]: [314B blob data] Apr 22 09:12:19 darkstar singularity[2652]: done. Apr 22 09:15:22 darkstar singularity[2652]: Syncing changes from OneDrive ... Apr 22 09:19:10 darkstar singularity[2652]: Syncing changes from OneDrive ...Yippee!
WFH Cat-on-Keyboard Problem (2020 March)
- Background
-
WFH because of the Covid-19 pandemic. Issue: working with laptop connected
to external mouse, keyboard and monitor. Cat likes to curl and and snooze
on laptop keyboard. Which is lovely. . .
. . . but uneven pressure results in random keys getting pushed downnnnnnnnnnnnnnnnnnnnnnnnnnn and havoc ensures! - Solution 1
- Place some padding in a folder on the keyboard. See photo. Limited success.
- Solution 2
-
Disable laptop keyboard:
prompt> xinput list Virtual core pointer id=2[master pointer (3)] Virtual core XTEST pointer id=4[slave pointer (2)] . . . . AT Translated Set 2 keyboard id=17[slave keyboard (3)] DELL Wireless hotkeys id=19[slave keyboard (3)] Logitech USB Keyboard id=12[slave keyboard (3)] prompt> prompt> xinput float 17 # ...soft-detach laptop kbd... # should I need to soft re-attach it: xinput reattach 17 3Complete success. Happy cat, happy me. - Phase Two
-
So, I swapped from my old Dell laptop (see pic) to a newish Lenovo Thinkpad.
Mungo (cat) found new ways to cause trouble:
- Turning the trackpad off via xinput was not enough; needed to explicitly turn off the buttons too.
- And there is also the function keys, which I don't seem to be able to turn off via xinput
xinput float 10 # ...disable kbd... xinput float 13 # ...extra buttons --- but what buttons?... xinput float 11 # ...disable trackpad... xinput --set-button-map 12 0 0 0 0 0 0 0 # ...disable trackpad buttons... # -- "xinput list --long" to get info on buttons xinput float 6 xinput float 8 # ...disable power and sleep buttons (sneaky cat powered off my laptop # at one point... # Q/ How to disable the function buttons?? # Q/ How to disable the function buttons?? # # !! Press Func+Esc !! # !! Works!
Secure Public Job Submission Engines (JSEs) via Shared Filesystems — Drag-n-Drop (2016 April 30)
- Background
-
Researchers' data is now commonly sensitive in some way (e.g.,
pseudo-anonymised patient data, or commercially-valuable data). So the
consequences of a research computing platform being compromised ("hacked")
and the data "leaked" are increasingly severe.
So our compute platforms live in private IP space (i.e., connections to/from the platforms are not routed on/off campus); firewalls are default-deny (often in both directions); that private IP space is being structured into unrestricted, restricted and highly-restricted zones, with sensitive data being located in the latter. - User Requirement — The Problem to be Addressed
-
A publically-accessible Web interface (e.g., something
Galaxy-based) which can submit
computational jobs to our research computing platforms.
This is not hard to set up — The Web site could use SGE's DRMAA API (e.g, Galaxy talks to SGE more or less out of the box). But that means we expose our compute platforms to the public internet via the Web site. Ain't Never Gonna Happen.
It would be a bonus if the solution adopted is flexible enough to work with other JSEs, for example, a drag-n-drop submission system, or submission via SFTP. - Solution Overview
-
Keep the Web site and the compute platform walled-off from eachother
by router ACLs and on-host firewalls, so network traffic cannot move from
one to the other. Instead, have the Web site and the compute platform's SGE
instance exchange messages and information via an dedicated, shared
filesystem.
- The Web site "drops" a standard SGE .qsub script into the filesystem, e.g., my_job.qsub.
- A cron job on the compute platform's submit node polls this spool area, parses the script for errors: if no errors are found the job is submitted — with a job name based on that which SGE would normally give it, but with a long, random string appended to ensure a unique name; if the submission is successful the name is written into a .qsubmit file, in this case my_job.qsubmit; if the submission is unsuccessful, a .qfail file is written containing the exit code and contents of STDOUT and STDERR.
- Each time the cron job runs, SGE is interrogated for the status of the job, by means of the unique name (qstat -j <jobname) and the status information written to a .qstat file, assuming the job is still running.
- If the job is found to have finished, a .qacct file is written containing information obtained from SGE's accounting system about the job.
- Example Files
- N.B. If the job name that SGE would by default generate is less than 10 characters long, then it is padded with spaces to make it look normal to command-line users of qstat on the SGE submit node.
NAT as a Service! (2016 April 09)
- Background
-
All Research IT computational platforms (e.g., the
CSF and
DPSF (nee Hydra))
are directly accessible from on-campus only, for very good security-related
reasons. Users who wish to access the platforms from off-campus must use
either the University VPN, or our SSH gateways.
In fact, no network traffic at all is possible, directly, from our computational platforms to/from off-campus. - User Requirement
- Increasingly, platform users which to access data-sets which are located off-campus as part of their computational work. This is commonly the case in bioinformatics work.
- Solution: Network Address Transation as a Service!
-
We provide a Linux server which offers Network-Address-Translation-as-a-Service.
Example:
- Connections come from campus-only hosts to a given port on the NATaas host;
- are DNATed to the required IP:port,e.g., 3306 at EnsembleDB for there SQL service;
- are SNATed to a public IP address and directed off-campus to the appropriate destination;
- return traffic also goes through the NATaaS host.
INteractive Computational LINux Environment — INCLINE aka iCSF (2013 May)
- Background
-
So, we have been running batch computational platforms for many years. Yet
many powerful workstations are procured each year for research groups and
individuals all over campus. Such workstations:
- are idle much of the time and therefore wasteful of financial resources;
- make user/researcher support by Research IT very innefficient as each is set up slightly differently from the others, they are geographically dispersed and each requires installation of software applications.
- User Requirement
-
Researchers:
- often need to run GUI-based applications — use of qrsh on batch/queue platforms offers a far from delightful experience;
- sometimes need to develop and debug code with frequent and short code runs;
- may need to pre-process or post-process data the main processing of which is performed on our batch compute systems.
- Solution Overview
-
Build a load-balanced SSH Linux
Virtual Server!
- Procure compute nodes with the same specification as those used in the CSF together with a LVS node for performing the routing and load-balancing.
- Using dedicated network infrastructure, make the same home and shared storage areas as are mounted on Redqueen and the CSF available on all compute nodes.
- Use mon to dynamically tweak the LVS config so as to perform load-balancing.
RI on 10.xyz.pqr.0/24, SSH Gateways and Virtual Desktops (201. . .)
- Background
- User Requirement
- Solution:
Redqueen and The CSF (201. . .)
- Background
- User Requirement
- Solution:
Coming R-S-N. . .
Future:
-- DPSF + RVMI + JSE + NAT -- vlan tagging
Millions of Small Files — SQL as a Service for HPC (2016: )
- Background
- User Requirement
- Solution:
HPC, Storage and VMs with Dedicated Networking — A Research Computing Ecosystem (2015. . .)
- Background
- User Requirement
- Solution:
INCLINE Part Two: Adding High-Memory Nodes and Virtual Desktop Servers (20. . .)
Add high-mem nodes:
PRE>
** added alias
em1 Link encap:Ethernet HWaddr B8:CA:3A:5F:D7:28
inet addr:10.99.203.43 Bcast:10.99.203.255 Mask:255.255.255.0
em1:0 Link encap:Ethernet HWaddr B8:CA:3A:5F:D7:28
inet addr:10.99.203.45 Bcast:10.255.255.255 Mask:255.0.0.0
--------------------------------------
ipvsadm -A -t 10.99.203.45:22 -p 86400
ipvsadm -a -t 10.99.203.45:22 -r 10.7.7.9:22 -m
ipvsadm -a -t 10.99.203.45:22 -r 10.7.7.10:22 -m
...so now have...
ipvsadm -L -n --persistent-conn
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Weight PersistConn ActiveConn InActConn
-> RemoteAddress:Port
TCP 10.99.203.43:22 wlc persistent 86400
-> 10.7.7.1:22 1 1 0 0
-> 10.7.7.2:22 1 0 0 0
. .
. .
-> 10.7.7.8:22 1 2 0 0
TCP 10.99.203.45:22 wlc persistent 86400
-> 10.7.7.9:22 1 1 0 0
-> 10.7.7.10:22 1 2 0 0
Add directly-connected virtual desktop server:
nyx3:/etc/hosts
10.5.5.250 incline.itservices.manchester.ac.uk incline 10.5.5.249 jabberwock
- So, on nyx3, enter "ssh incline" or "ssh jabberwock" and get routed as per the ipvsadm -L listing below.
nyx4:/etc/hosts:
10.5.5.248 incline.itservices.manchester.ac.uk incline 10.5.5.249 jabberwock
- So, on nyx4, enter "ssh incline" or "ssh jabberwock" and get routed as per the ipvsadm -L listing below.
ipvsadm -L IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 10.5.5.249:ssh wlc -> jabberwock:ssh Masq 1 0 0 TCP 10.5.5.250:ssh wlc -> incline01:ssh Masq 1 2 0 -> incline02:ssh Masq 1 3 0 -> incline03:ssh Masq 1 3 0 -> incline04:ssh Masq 1 2 0 TCP incline:ssh wlc -> incline01:ssh Masq 1 4 0 -> incline02:ssh Masq 1 4 0 -> incline03:ssh Masq 1 4 0 -> incline04:ssh Masq 1 4 0 -> incline05:ssh Masq 1 5 0 -> incline06:ssh Masq 1 5 0 -> incline07:ssh Masq 1 5 0 -> incline08:ssh Masq 1 5 0 TCP incline256.itservices.manche wlc -> incline09:ssh Masq 1 0 0 -> incline10:ssh Masq 1 0 0
- ...