Feed The Monster

Using Mbsync with DavMail and O365 (2020 August)

The Problem
Below we show how to set up DavMail with Mutt to access O365-based email over IMAP. It works, but its slow; the user experience is wait, wait wait. . . Can we use Mbsync to mirror remote 0365 to local Mail_Dir?
Config: ~/.mbsyncrc
# ------------------------------------------------------------------------------
# ------------------------------------------------------------------------------

#  mbsync -c ~/.mbsyncrc Personal-365

# ------------------------------------------------------------------------------
# -- IMAP for Personal 365 via DavMail :
# ------------------------------------------------------------------------------

IMAPAccount Personal-365
Port 1143
SSLType None
    # ...for DavMail...
User "simonhtest666@outlook.com"
Pass "mypassword"
####PassCmd "pass work/email"
AuthMechs LOGIN

# ------------------------------------------------------------------------------
# -- Remote and local "stores" :
# ------------------------------------------------------------------------------

IMAPStore Personal-365-remote
Account Personal-365

MaildirStore Personal-365-local
Subfolders Verbatim
    # ...Copy folder hierarchy...
Path ~/Mbsync-Personal-365/
    # ...The trailing "/" is important...
Inbox ~/Mbsync-Personal-365/Inbox

# ------------------------------------------------------------------------------
# -- Sync "channel" :
# ------------------------------------------------------------------------------

Channel Personal-365
Master :Personal-365-remote:
Slave :Personal-365-local:
Patterns *
    # ...Include everything...
Create Both
    # ....Automatically create missing mailboxes, both locally and on the server....
SyncState *
    # ...Save the synchronization state files in the relevant directory...

# ------------------------------------------------------------------------------
# END.
# ------------------------------------------------------------------------------
Run it:
    mbsync -c ~/.mbsyncrc Personal-365
Configure Mutt to talk to local Mail_Dir

Accessing OLE Embedded PDFs in a PPTX on Linux (2020 July)

The Problem
I received a .pptx file with embedded PDFs. Libreoffice can't see them; neither can Google Docs; neither can Microsoft 365!!!!!
Unzip the .pptx and cd to ppt/embedded to find the .bin files which are the PDFs — but with some header crap which upsets PDF readers. Remove the header crap using bbe:

    for i in *.bin; do bbe -b ':/%PDF/' -e 'D 1' $i > $i.nearly; done

    for i in *.nearly; do echo -n %PDF | cat - $i > $i.pdf; done
If you are more familiar with bbe that I am you might be able to tidy that up.

Accessing O365 2FA Email using DavMail and Mutt (2020 May)

UoM is migrating away from using on-site Exchange, with IMAP enabled to those what want it, to using Microsoft Office 365. This causes issues:
  • On-site, 2FA is done at the network level, i.e., to access IMAP you need to be connected to the Uni VPN, even if on campus. That's fine. Once connected to the VPN, IMAP works (with SFA) as normal.
  • Off-site/365 UoM 2FA is at the application level; not a good design but true. UoM's 365 is linked to UoM AD and hence Duo for 2FA. Alternatively one can use IMAP, with OAuth2.
2FA support from Linux email clients is patchy and in development. Evolution is ok; bleeding edge Thunderbird will do IMAP/OAuth2. But. . .
I love Mutt. I want to use Mutt. Google suggests trying an IMAP proxy: DavMail.
Quoting the Web site: Ever wanted to get rid of Outlook? DavMail is a POP/IMAP/SMTP/Caldav/Carddav/LDAP Exchange gateway allowing users to use any mail/calendar client with an Exchange server. More at http://davmail.sourceforge.net.
An Aside on Testing IMAP Connectivity and Authentication
  • Bad idea, as does not escape special characters, which can lead to a great deal of frustration until you realise, use Curl instead, below...
What about IMAPS?
  • use openssl _sclient as a test client :-)
  • https://tewarid.github.io/2011/05/10/access-imap-server-from-the-command-line-using-openssl.html
So, what do we do with Telnet/openssl? Basics of the protocol:
    openssl s_client -crlf -connect imap.gmail.com:993
    ...blah, blah...
    * OK The Microsoft Exchange IMAP4 service is ready. [QgBOADYAUA...AbwBtAA==]
    01 tag login simonhtest666@outlook.com mypasswordwithspecialchars
    01 OK LOGIN completed.
    02 LIST "" "*"
    * LIST (\HasNoChildren) "/" Archive
    * LIST (\HasChildren \Trash) "/" Deleted
    * LIST (\HasChildren) "/" Deleted/imaps:
    .    .
    .    .
    * LIST (\HasNoChildren) "/" Test_Folder
    02 OK LIST completed.
    * 3 EXISTS
    .    .
    .    .
    * OK [UIDNEXT 50] The next unique identifier value
    03 OK [READ-WRITE] SELECT completed.
    04 OK STATUS completed.
    05 FETCH 22:27 (BODY[HEADER])          # ...fetch headers of last few messages...
    * 22 FETCH (BODY[HEADER] {321}
    Date: Sat, 15 Feb 2020 10:09:06 +0000
    From: Simon Hood <simonhtest666@outlook.com>
    .    .
    FLAGS (\Seen))
    * 3 FETCH (BODY[HEADER] {698}
    .    .
    Sender: Simon Hood <simonhtest666@outlook.com>
    Date: Sat, 8 Feb 2020 09:08:18 +0000
    .    .   
    FLAGS (\Seen))
    05 OK FETCH completed.
    06 LOGOUT
    * BYE Microsoft Exchange Server IMAP4 server signing off.
    06 OK LOGOUT completed.
  • You must enter a lable (e.g., 01, 02...) at the start of each command, which prepends the last line of the output
  • More: https://tewarid.github.io/2011/05/10/access-imap-server-from-the-command-line-using-openssl.html
And Curl does the messy stuff for you:
  • Example
            curl --url "imap://localhost:1143" \
                  --user "simonhtest666@outlook.com:mypasswordwithspecialcharacters"
            # ...lists existing folders...
  • More on Curl and IMAP: https://debian-administration.org/article/726/Performing_IMAP_queries_via_curl
DavMail Version
The version in Mint 19.3 it tooooo old and does not have 0365Interactive which is essential.
  1. Get latest from DavMail site: davmail_5.5.1-3299-1_all.deb Version 5.1, for example, does not seem to store the oauth token (in .davmail.properties) so manual 2FA is required everytime, not every N days.
  2. Add dependencies, .deb packages from distro: libopenjfx-java, openjfx.
An Aside on Mutt Configuration Debugging
    mutt -D
        # ...defaults to compiled in defaults, with overrides from .muttrc...
SFA Testing with Personal Office365 Account
Start by solving half the problem:
  • I have both a UoM account and personal account at O365, the latter mostly so I can play (test things) without causing a disaster in a "production" setting.
  • This is a SFA, but good to get the basics tested. (Microsoft plan to disable SFA at some point.)
  • DavMail config — pertinent bits:
          Exhange Protocol:  EWS
          URL             :  https://outlook.office365.com/EWS/Exchange.asmx
  • Mutt config:
          source "~/.mutt/user-365-simonhtest-davmail-localhost-settings"
          source "~/.mutt/user-davmail-localhost-encryption-settings"
          source "~/.mutt/user-davmail-localhost-smtp-settings"
          set imap_user="$my_username@$my_domain"
          set imap_pass="$my_password"
          set header_cache     = "~/.mutt/user-simon.hood.uom-caches/headers"
          set message_cachedir = "~/.mutt/user-simon.hood.uom-caches/bodies"
          set folder="imap://localhost:1143/"
          set spoolfile ='imap://localhost:1143/'
          set trash = 'imap://localhost:1143/Trash'
    where user-365-simonhtest-davmail-localhost-settings
        set my_username = "simonhtestusername"
        set my_domain = "outlook.com"
        set my_password="mypassword"
        # -- Save sent messages:
        set copy=yes
        set record="imap://localhost:1143/Sent_Mutt"
            # ...davmail...
    and user-davmail-localhost-encryption-settings
        # ...turn it all off for use with DavMail on localhost...
        set ssl_force_tls=no
        set ssl_starttls=no
        # set ssl_use_tlsv1=yes
        # set ssl_use_tlsv1_1=no
        #### set ssl_use_sslv3=yes
        #### ...deprecated, no longer works...
    and user-davmail-localhost-smtp-settings
          set realname = "Simon Hood"
          set from = "$my_username@$my_domain"
          set smtp_pass = "$my_password"
          set smtp_url = "smtp://$my_username@$my_domain@localhost:1025/"
          set smtp_authenticators = "login"
UoM 365 Account with 2FA
DavMail settings:
    Exchange Protocol:  O365Interactive
Mutt settings, as above bar
    ####source "~/.mutt/user-365-simonhtest666-davmail-localhost-settings"
    source "~/.mutt/user-365-simon.hood.uom-davmail-localhost-settings"
where user-365-simon.hood.uom-davmail-localhost-settings
    set   my_username = "simon.hood"
    set   my_domain = "manchester.ac.uk"
    unset my_password

    set copy=yes
    set record="imap://localhost:1143/Sent_Mutt"
        # ...davmail...

    # set record="+imaps://outlook.office365.com/Sent_Mutt"
        # ...not via davmail...
        # ...creates nested:  imaps/outlook/office365/com/Sent_Mutt !!!
  • First time, DavMail pops up a GUI <--- the usual Duo "push": accept/save
  • Second time: just works. Duo only required every N days.
An Aside: Using a Perl Script as a Client
Perl: Mail::IMAPClient:
  1. Point the script at DavMail, i.e., localhost:1143.
  2. Crucial lines of code:
            my $yes = $imap->move("Logwatch", $uid);
            if ($yes) {  $imap->delete_message($uid);  }
    N.B. delete_message actually just marks a message for deletion and to actually delete messages we need to call expunge. At least via DavMail, this is a time-consuming call: DavMail works its way through all messages in the folder to get an update. Hence:
  3. Loop through say a few thousand messages marking some for deletion, then expunge; then loop through the next few thousand... And
  4. Turn OFF auto-expunge within DavMail otherwise IMAP move operations are veeeerrrryyyy sloooooowwwww. . ..
Remaining Issue
Performance is very poor. Potential solution, use Isync/MBsymc?

OneDrive on Linux — Singularity and Systemd (2020 April)

  • Microsoft do not offer or support a OneDrive client for Linux. Neither do they have plans to develop one (private communication, 2020, April).
  • Skilion is now abandoned, but there is an active fork, simply called OneDrive.
  • Some excellent intro notes to be found at Linux Uprising, including on integration with systemd, i.e., running the client as a service with continuous updates.
  • Other FOSS clients exist, such as rclone and CloudCross.
But Mint 19.3 (based on Ubuntu 18.x) is too old. . .
  • apt-get install onedrive yields v1, i.e., Skilion. Reject.
  • Newer Debian/Ubuntu-based distros include the v2, maintained version: Ubuntu 19.x, Debian Unstable/Sid and Debian 10/Bullseye, e.g., https://packages.debian.org/sid/net/onedrive. Mint 20.x, based on Ubuntu 20.x, is not due out until June.
Build from source on my Mint 19.3?
  • apt-get install ldc, the required D compiler is too old a version to compile v2.
  • How about a singularity container-based install?
    • apt-get install Singularity container on Mint 19.3 yields v2.4, which is too old for current Singularity images — need v3.x.
    • A manual install of Debian Sid packages does the trick:
    Bootstrap an appropriate Singularity image:
    cat bootstrap.debian-testing.docker:
      Bootstrap: docker
      From: debian:testing
      apt-get -y update
      apt-get -y install onedrive libcurl3-gnutls libcurl4
      export LC_ALL=C
      root@host> singularity build onedrive.sif bootstrap.debian-testing.docker
    Initial Testing:
      prompt> singularity shell onedrive.sif
        # -- enter the container and get a shell
      Singularity>  ondrive --logout
      Singularity>  Authorize this app [by] visiting:
        # -- Enter the above in a browser and ensure you point it to the right
        #    Office 365 account (e.g., personal vs work).  Once you click "yes,
        #    permission granted", you get a blank page --- that's fine, copy this
        #    blank page's URI back to your Singularity promp...
      Enter the response uri:
        # ...copy the blank page URI into the commandline...
      Application has been successfully authorised, however no additional command
      switches were provided.
      Singularity> onedrive --synchronize --verbose --dry-run
        # ...try it out --- files go to ~/OneDrive...
    More Testing
      prompt> singularity exec test.sif onedrive --synchronize --verbose
        # ...Run onedrive inside the container from outside the container...
    Integrate with Systemd
    We want a service — in particular a user service (not a system service). So we want a systemd unit — a user unit.
    • Great into to Singularity under Systemd at Sylabs but we don't need singularity instances in our case, nor forking and bear in mind that we are not running this systemd instance as a (root) service but a user service.
      cat onedrive.service
      Description=OneDrive Free Client for %i
      # Type=forking
      # Restart=always
      # ...what we might want for a singularity instance, e.g., a Web server...
      # User=%i
      # Group=users
      # ...we are starting our systemd instance as a user (not root) with "--user" so
      #    we don't want these:
      #     ...Failed to determine supplementary groups: Operation not permitted
      #        Failed at step GROUP...
      ExecStart=/usr/bin/singularity exec /home/simonh/Downloads/test.sif onedrive --monitor --confdi\
    Test it under systemd!
      host> systemctl --user enable onedrive
    	Created symlink /home/simonh/.config/systemd/user/multi-user.target.wants/onedrive.service --> /usr/lib/systemd/user/onedrive.service.
        host>  systemctl --user start onedrive
        host>  systemctl --user status onedrive.service
        onedrive.service - OneDrive Free Client for
        Loaded: loaded (/usr/lib/systemd/user/onedrive.service; enabled; vendor preset: enabled)
        Active: active (running) since Wed 2020-04-22 09:07:46 BST; 2s ago
        Docs: https://github.com/abraunegg/onedrive
        Main PID: 2652 (starter-suid)
        CGroup: /user.slice/user-1000.slice/user@1000.service/onedrive.service
        --2652 Singularity runtime parent
        --2695 onedrive --monitor --confdir=/home/simonh/.config/onedrive
        host> journalctl --user-unit onedrive -f
        -- Logs begin at Thu 2019-01-17 10:46:42 GMT. --
        Apr 22 09:07:46 darkstar systemd[1358]: Started OneDrive Free Client for .
        Apr 22 09:07:46 darkstar singularity[2652]: Notification (dbus) server not available, disabling
        Apr 22 09:07:49 darkstar singularity[2652]: Initializing the Synchronization Engine ...
        Apr 22 09:07:49 darkstar singularity[2652]: Initializing monitor ...
        Apr 22 09:07:49 darkstar singularity[2652]: OneDrive monitor interval (seconds): 45
        Apr 22 09:10:50 darkstar singularity[2652]: Moving xmodmap to xmodmap-copy-simonh-test
        Apr 22 09:11:35 darkstar singularity[2652]: Syncing changes from OneDrive ...
        Apr 22 09:12:17 darkstar singularity[2652]: Uploading new file ./a_test_file_with_a_long_name .\
        Apr 22 09:12:19 darkstar singularity[2652]: [314B blob data]
        Apr 22 09:12:19 darkstar singularity[2652]: done.
        Apr 22 09:15:22 darkstar singularity[2652]: Syncing changes from OneDrive ...
        Apr 22 09:19:10 darkstar singularity[2652]: Syncing changes from OneDrive ...

    WFH Cat-on-Keyboard Problem (2020 March)

    WFH because of the Covid-19 pandemic. Issue: working with laptop connected to external mouse, keyboard and monitor. Cat likes to curl and and snooze on laptop keyboard. Which is lovely. . .

    . . . but uneven pressure results in random keys getting pushed downnnnnnnnnnnnnnnnnnnnnnnnnnn and havoc ensures!
    Solution 1
    Place some padding in a folder on the keyboard. See photo. Limited success.
    Solution 2
    Disable laptop keyboard:
      prompt> xinput list
        Virtual core pointer                    id=2[master pointer  (3)]
          Virtual core XTEST pointer            id=4[slave  pointer  (2)]
          .   .
          .   .
        AT Translated Set 2 keyboard            id=17[slave  keyboard (3)]
        DELL Wireless hotkeys                   id=19[slave  keyboard (3)]
        Logitech USB Keyboard                   id=12[slave  keyboard (3)]
      prompt> xinput float 17
          # ...soft-detach laptop kbd...
          #    should I need to soft re-attach it: xinput reattach 17 3
    Complete success. Happy cat, happy me.
    Phase Two
    So, I swapped from my old Dell laptop (see pic) to a newish Lenovo Thinkpad. Mungo (cat) found new ways to cause trouble:
    1. Turning the trackpad off via xinput was not enough; needed to explicitly turn off the buttons too.
    2. And there is also the function keys, which I don't seem to be able to turn off via xinput
    Complete recipe:
      xinput float 10
          # ...disable kbd...
      xinput float 13
          # ...extra buttons --- but what buttons?...
      xinput float 11
          # ...disable trackpad...
      xinput --set-button-map 12 0 0 0 0 0 0 0
          # ...disable trackpad buttons...
          #     -- "xinput list --long" to get info on buttons
      xinput float 6
      xinput float 8
          # ...disable power and sleep buttons (sneaky cat powered off my laptop
          #    at one point...
      # Q/ How to disable the function buttons??
      # Q/ How to disable the function buttons??
      # !! Press Func+Esc !!
      # !! Works!

    Secure Public Job Submission Engines (JSEs) via Shared Filesystems — Drag-n-Drop (2016 April 30)

    Researchers' data is now commonly sensitive in some way (e.g., pseudo-anonymised patient data, or commercially-valuable data). So the consequences of a research computing platform being compromised ("hacked") and the data "leaked" are increasingly severe.

    So our compute platforms live in private IP space (i.e., connections to/from the platforms are not routed on/off campus); firewalls are default-deny (often in both directions); that private IP space is being structured into unrestricted, restricted and highly-restricted zones, with sensitive data being located in the latter.
    User Requirement — The Problem to be Addressed
    A publically-accessible Web interface (e.g., something Galaxy-based) which can submit computational jobs to our research computing platforms.

    This is not hard to set up — The Web site could use SGE's DRMAA API (e.g, Galaxy talks to SGE more or less out of the box). But that means we expose our compute platforms to the public internet via the Web site. Ain't Never Gonna Happen.

    It would be a bonus if the solution adopted is flexible enough to work with other JSEs, for example, a drag-n-drop submission system, or submission via SFTP.
    Solution Overview
    Keep the Web site and the compute platform walled-off from eachother by router ACLs and on-host firewalls, so network traffic cannot move from one to the other. Instead, have the Web site and the compute platform's SGE instance exchange messages and information via an dedicated, shared filesystem.
    1. The Web site "drops" a standard SGE .qsub script into the filesystem, e.g., my_job.qsub.
    2. A cron job on the compute platform's submit node polls this spool area, parses the script for errors: if no errors are found the job is submitted — with a job name based on that which SGE would normally give it, but with a long, random string appended to ensure a unique name; if the submission is successful the name is written into a .qsubmit file, in this case my_job.qsubmit; if the submission is unsuccessful, a .qfail file is written containing the exit code and contents of STDOUT and STDERR.
    3. Each time the cron job runs, SGE is interrogated for the status of the job, by means of the unique name (qstat -j <jobname) and the status information written to a .qstat file, assuming the job is still running.
    4. If the job is found to have finished, a .qacct file is written containing information obtained from SGE's accounting system about the job.
    The client (e.g. Galaxy) can watch for, and interrogate, the .qsubmit for the job name, the .qstat for job monitoring information and .qexit files for summary info about the completed (or possibly failed) job. And of course the client can access the usual job output files (both those from SGE) and any others generated by the job, by using the job name obtained from the .qsubmit file.
    Example Files
    N.B. If the job name that SGE would by default generate is less than 10 characters long, then it is padded with spaces to make it look normal to command-line users of qstat on the SGE submit node.


    NAT as a Service! (2016 April 09)

    All Research IT computational platforms (e.g., the CSF and DPSF (nee Hydra)) are directly accessible from on-campus only, for very good security-related reasons. Users who wish to access the platforms from off-campus must use either the University VPN, or our SSH gateways.

    In fact, no network traffic at all is possible, directly, from our computational platforms to/from off-campus.
    User Requirement
    Increasingly, platform users which to access data-sets which are located off-campus as part of their computational work. This is commonly the case in bioinformatics work.
    Solution: Network Address Transation as a Service!
    We provide a Linux server which offers Network-Address-Translation-as-a-Service. Example:
    • Connections come from campus-only hosts to a given port on the NATaas host;
    • are DNATed to the required IP:port,e.g., 3306 at EnsembleDB for there SQL service;
    • are SNATed to a public IP address and directed off-campus to the appropriate destination;
    • return traffic also goes through the NATaaS host.
    More. . .

    INteractive Computational LINux Environment — INCLINE aka iCSF (2013 May)

    So, we have been running batch computational platforms for many years. Yet many powerful workstations are procured each year for research groups and individuals all over campus. Such workstations:
    • are idle much of the time and therefore wasteful of financial resources;
    • make user/researcher support by Research IT very innefficient as each is set up slightly differently from the others, they are geographically dispersed and each requires installation of software applications.
    User Requirement
    • often need to run GUI-based applications — use of qrsh on batch/queue platforms offers a far from delightful experience;
    • sometimes need to develop and debug code with frequent and short code runs;
    • may need to pre-process or post-process data the main processing of which is performed on our batch compute systems.
    We therefore need to provide an interactive computational resource, which can support use by many researchers simultaneously, and which mounts the same home an shared filesystems as are used on our batch platforms such as Redqueen and The CSF.
    Solution Overview
    Build a load-balanced SSH Linux Virtual Server!
    • Procure compute nodes with the same specification as those used in the CSF together with a LVS node for performing the routing and load-balancing.
    • Using dedicated network infrastructure, make the same home and shared storage areas as are mounted on Redqueen and the CSF available on all compute nodes.
    • Use mon to dynamically tweak the LVS config so as to perform load-balancing.
    More. . .

    RI on 10.xyz.pqr.0/24, SSH Gateways and Virtual Desktops (201. . .)

    User Requirement


    Redqueen and The CSF (201. . .)

    User Requirement


    Coming R-S-N. . .


     -- DPSF + RVMI + JSE + NAT
     -- vlan tagging

    Millions of Small Files — SQL as a Service for HPC (2016: )

    User Requirement


    HPC, Storage and VMs with Dedicated Networking — A Research Computing Ecosystem (2015. . .)

    User Requirement


    INCLINE Part Two: Adding High-Memory Nodes and Virtual Desktop Servers (20. . .)

    Add high-mem nodes:

       ** added alias
        em1       Link encap:Ethernet  HWaddr B8:CA:3A:5F:D7:28
                  inet addr:  Bcast:  Mask:
        em1:0     Link encap:Ethernet  HWaddr B8:CA:3A:5F:D7:28
                  inet addr:  Bcast:  Mask:
        ipvsadm -A -t -p 86400
        ipvsadm -a -t -r -m
        ipvsadm -a -t -r -m
       ...so now have...
      ipvsadm -L -n --persistent-conn
      IP Virtual Server version 1.2.1 (size=4096)
      Prot LocalAddress:Port            Weight    PersistConn ActiveConn InActConn
        -> RemoteAddress:Port
      TCP wlc persistent 86400
        ->                  1         1           0          0
        ->                  1         0           0          0
           .         .
           .         .
        ->                  1         2           0          0
      TCP wlc persistent 86400
        ->                  1         1           0          0
        ->                 1         2           0          0

    Add directly-connected virtual desktop server:

    nyx3:/etc/hosts          incline.itservices.manchester.ac.uk incline          jabberwock

    nyx4:/etc/hosts:          incline.itservices.manchester.ac.uk incline          jabberwock

     ipvsadm -L
    IP Virtual Server version 1.2.1 (size=4096)
    Prot LocalAddress:Port Scheduler Flags
      -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
    TCP wlc
      -> jabberwock:ssh               Masq    1      0          0
    TCP wlc
      -> incline01:ssh                Masq    1      2          0
      -> incline02:ssh                Masq    1      3          0
      -> incline03:ssh                Masq    1      3          0
      -> incline04:ssh                Masq    1      2          0
    TCP  incline:ssh wlc
      -> incline01:ssh                Masq    1      4          0
      -> incline02:ssh                Masq    1      4          0
      -> incline03:ssh                Masq    1      4          0
      -> incline04:ssh                Masq    1      4          0
      -> incline05:ssh                Masq    1      5          0
      -> incline06:ssh                Masq    1      5          0
      -> incline07:ssh                Masq    1      5          0
      -> incline08:ssh                Masq    1      5          0
    TCP  incline256.itservices.manche wlc
      -> incline09:ssh                Masq    1      0          0
      -> incline10:ssh                Masq    1      0          0