Trust me when I say this is the perfect setup, I've tried pretty much all of them. This setup is designed to align with myos as described in my last post that I recommend reading. Here's what you'll get:
|Email Task||Software||Configuration Repository|
|Receiving||isync a.k.a mbsync||myOS-email|
|Organising + Filtering||notmuch||myOS-email|
However, that isn't enough to complete all your email tasks. You need to be able
to view and compose emails too i.e. you need an mail user agent (a.k.a email
client) that supports the above. To closer align with
myOS / policy 1 GPG must also be setup so
that you can sign, encrypt, and decrypt emails. Because GPG is used for more
than just email I have not included it in the
myOS / email system. Likewise,
as I am going to use notmuch-emacs as my mail user agent, and that is part of
myOS /emacs system, it is configured independently. Details of my emacs
configuration are left for later post.
|Email Task||Software||Configuration Repository|
|Viewing / Composing||notmuch-emacs||myOS-emacs|
|Encryption / Signing||gpg||myOS-distro|
By not including the mail user agent and encryption software in the
myOS / email does not force
you to use emacs as your email client – you may select one of your choice here.
However, it isn't completely independent as I rely on GPG to encrypt/decrypt
your email account credentials rather than doing the silly thing of storing them
in plain text in a file.
|Email Task||Software||Configuration Repository|
In general, you have complete freedom over the software to complete tasks that utilizes the system (Table 2) but not the software that the system utilizes (Table 3). However, if you have any alternative software to complete the task, it becomes a matter of find+replace or aliasing – interesting… That's started a train of thought that I'll publish next.
myOS / email
Here's how it all works, and how to get up and running. As I am using
Qubes and following myOS / policy 1 the
scope of my email configuration dotfiles are reduced to a single email
address. I no longer have to set up my email clients to deal with multiple
email addresses making my life easier. My dotfiles now live in
my main TemplateVM qube. However, if you're not using Qubes you can still
use this configuration but you would likely want to copy it directly to your
Before I go into all the dotfiles I will talk about what each piece of
software is doing. Emails are downloaded using IMAP and mbsync into a Maildir
~/Mail/<email@example.com>, you now have all your emails for
offline reading and searching. Tagging, filtering, and searching is done using
notmuch and it makes GMail look like a toy in comparison for these tasks.
Finally, emails are sent asynchronously using msmtp, you can stage emails to
be sent while offline.
The repository myOS-emacs has the most recent dotfiles and basic installation instructions for reference. I'll go over them in more detail here and give a quick example of how to get started with some basic tagging with notmuch. I will use setting up my personal GMail account for example, replace my email address with your own.
Clone the repo and from there it's two easy steps.
Step 1: Copy in the files and structure
Firstly, here is the file structure of the entire system configuration. This
is what you will be copying into your home folder. All of the real files are
~/Mail/config. Everything else is either a symbolic link or an
empty folder. Scripts in
~/Mail/config/bin are machinery around msmtp to
provide an mail queue (outbox).
. ├── .local │ └── bin │ ├── msmtp-enqueue.sh -> ../../Mail/config/bin/msmtp-enqueue.sh │ ├── msmtp-listqueue.sh -> ../../Mail/config/bin/msmtp-listqueue.sh │ └── msmtp-runqueue.sh -> ../../Mail/config/bin/msmtp-runqueue.sh ├── Mail │ ├── firstname.lastname@example.org │ ├── config │ │ ├── bin │ │ │ ├── msmtp-enqueue.sh │ │ │ ├── msmtp-listqueue.sh │ │ │ └── msmtp-runqueue.sh │ │ ├── credentials.gpg │ │ ├── mbsyncrc │ │ ├── msmtprc │ │ └── notmuch-config │ ├── .msmtpqueue │ └── .notmuch ├── .mbsyncrc -> Mail/config/mbsyncrc ├── .msmtprc -> Mail/config/msmtprc ├── .notmuch-config -> Mail/config/notmuch-config
Step 2: Edit the files in
There isn't much to edit, mostly standard email account things. Let's start with the trickier stuff.
~/Mail/config/credentials.gpg (Optional, Recommended)
You can just type your password into the other config files in plain text. I recommend that you generate an encrypted file, it's quite easy with GPG. Make a temporary plain text file with your password in it, encrypt it, shred the temporary file.
$EDITOR mypass.txt gpg -r "Your Name <email@example.com>" -e mypass.txt > credentials.gpg shred -n 200 -z -u mypass.txt
This file requires the most edits, it is about 50 lines long (including new lines). It controls how mbsync maps remote IMAP folders to the local Maildir. The following setup works wonderfully for GMail. I'm going to breakdown the file from the top. Starting with the basic account info:
IMAPaccount gmail host imap.gmail.com user firstname.lastname@example.org PassCmd "gpg -q -d --no-tty ~/Mail/config/credentials.gpg" SSLType IMAPS CertificateFile /etc/ssl/certs/ca-bundle.crt
The first three lines are self explanatory change these to suit your email
PassCmd line is the command to decrypt the file you made
above. It should simply return your password to stdout, test the command in
SSLType IMAPS is telling mbsync to connect to the server
using SSL/TLS on the secure IMAP port 993. If your server doesn't support
this you might want to try
SSLType STARTTLS instead. The last line is to
your systems default certificate file. The file above works for Fedora, on
other distrobutions the path is likely the same but the filename might be
Next we define where our emails are remotely, and where we're going to keep them locally. These are called stores:
IMAPStore gmail-remote account gmail MaildirStore gmail-local # trailing "/" is important Path ~/Mailemail@example.com/ Inbox ~/Mailfirstname.lastname@example.org/Inbox
The only edits you will want to make here are changing my email address to
your own, and perhaps any reference to
gmail to be consistent with
IMAPaccount defined above.
Next is the bulk of the file, it is the mapping between folder on the IMAP
store (Master) and a folder in the local store (Slave). For GMail based
accounts, it likely will require no more editing than making the names
IMAPaccount. For other IMAP servers, you may need to edit
Master line to match the remote folder correctly.
Channel gmail-default Master :gmail-remote: Slave :gmail-local: Patterns % "!Sent" "!Drafts" "!Trash" Create Both SyncState * Expunge Both Channel gmail-sent Master :gmail-remote:"[Gmail]/Sent Mail" Slave :gmail-local:Sent Create Slave SyncState * Expunge Both Channel gmail-drafts Master :gmail-remote:"[Gmail]/Drafts" Slave :gmail-local:Drafts Create Slave SyncState * Expunge Both Channel gmail-trash Master :gmail-remote:"[Gmail]/Trash" Slave :gmail-local:Trash Create Slave SyncState * Expunge Both
The last part of the file, again you will only need to edit to be consistent
Group gmail Channel gmail-default Channel gmail-sent Channel gmail-drafts Channel gmail-trash
This file configures msmtp. There should be nothing in this file you shouldn't be able to work out if you have properly understood the above. Edit appropriately.
defaults auth on tls on tls_trust_file /etc/ssl/certs/ca-bundle.crt logfile ~/Mail/config/.msmtp.log account gmail host smtp.gmail.com port 587 from email@example.com user firstname.lastname@example.org passwordeval "gpg -q -d --no-tty ~/Mail/config/credentials.gpg"
The last file to configure. You might need to edit
path to reference your
primary_email of course, and finally
if you need. Something to note while here are the
tags listed under
– these are applied to newly downloaded emails after running
[database] path=/home/user/Mail [user] name=Aqeel Akber email@example.com [new] tags=unread;inbox;new; ignore= [search] exclude_tags=deleted;spam; [maildir] synchronize_flags=true [crypto] gpg_path=/usr/bin/gpg
Now that you're all setup, download all your emails by typing the following:
After that is done, tag the newly downloaded emails:
When setting up your mail user agent use the following in place of
The above sends emails immediately if it can ping Google's DNS server:
Otherwise, you can check the outbox queue at the command line with:
If you want to force trying to send mail now, run:
You probably would like to set up
systemd timers or use
cron to run the
above commands periodically. Setting up your mail user agent is up to you, if
you use spacemacs and can't wait to get started check out my layer
A quick and dirty example highlighting why notmuch is awesome
Notmuch lets you search through your downloaded emails from the command line
with powerful queries and tag them, see
man notmuch tag for more info.
Notmuch also supports running executables/scripts after particular actions
using hooks, see
man notmuch hooks. By using
notmuch tag combined with a
post-new hook written in python you can do some pretty awesome things really
Try this, make a file at
~/Mail/.notmuch/hooks/post-new with the following
in it and make the file executable with
chmod +x. This is an example of a
quick and dirty script that has served me quite well, especially the "learn
from" filter which (dumbly) learns and mimics my behaviour. Before going
fully down the rabbit hole with this, you might want lean how to back up and
restore your tags database.
#!/usr/bin/python3 import subprocess def nmtag(tag : str, query : str): """Apply tag to all results of a notmuch query""" tag = tag.split(" ") cmd = subprocess.run(["notmuch", "tag"] + tag + ["--", query]) return cmd accname = "gmail" address = "firstname.lastname@example.org" """Start searching and tagging. When you're done with a particular subset of emails, remove the new tag so that it doesn't appear in subsequent queries. Working in this manner is simple but order matters. """ # tag account name tag = "+%s" % (accname) query = "(to:%s OR from:%s) AND tag:new" % (address, address) nmtag(tag, query) # from one of my accounts is probably important tag = "+inbox" query = "(from:%s) AND tag:new" % address nmtag(tag, query) # unless it's sent mail tag = "+sent -unread -inbox -new" query = "(folder:%s/Sent) AND tag:new" % accname nmtag(tag, query) # tag things stored as drafts tag = "+draft -inbox -unread -new" query = "(folder:%s/Drafts) AND tag:new" % accname nmtag(tag, query) # also, if it's IMAP trash tag = "+trash -inbox -unread -new" query = "(folder:%s/Trash) AND tag:new" nmtag(tag, query) ## ## FILTERING ## # I STRONGLY ENCOURAGE CALENDAR INVITES VIA EMAIL tag = "+calendar +filtering -inbox" query = "(attachment:.ics) AND tag:new" nmtag(tag, query) # LEARN MY BEHAVIOUR OVER TIME # Find email addresses in the database that have been tagged with LEARNFROM at # least 3 times. Tag new emails from these email addresses with the same tag. LEARNFROM = ["vip", "human", "client"] for t in LEARNFROM: tag = "+%s -inbox -new" % t prevfrom = subprocess.run(["notmuch","address","tag:"+t], stdout=subprocess.PIPE) prevfrom = prevfrom.stdout.decode("utf-8").split("\n") prevfrom = filter(lambda x: len(x)>3, prevfrom) for f in prevfrom: query = "(from:%s) AND tag:new" % f nmtag(tag, query) # FILTER TWITTER tag = "+social -inbox -new" query = "(from:twitter) AND tag:new" nmtag(tag, query) # FILTER SHOPPING UPDATES FROM BOTS tag = "+commerce -inbox -new" query = "(subject:receipt OR subject:invoice OR subject:order OR subject:'ship*' OR subject:'deliv*' OR from:auspost OR from:ebay) AND tag:new" nmtag(tag, query) # FILTER ALL OTHER GENERAL NOTIFICATIONS FROM BOTS tag = "+ping -inbox -new" query = "(from:'notif*' OR from:reply OR from:noreply OR from:donotreply OR subject:notification OR subject:reply) AND tag:new" nmtag(tag, query) # FINISHED ALL FILTERING tag = "-filtering -new" query = "tag:filtering" nmtag(tag, query) # IF IT ISN'T FILTERED, PUT INBOX FOR THE HUMAN TO DEAL WITH IT. tag = "+inbox -new" query = "tag:new" nmtag(tag, query)
Email is a fundamental part of myos, by introducing organisation by compartmentalisation with Qubes configuration of email is reduced to a single email account which simplifies the process. Further, by separating the task of viewing and composing emails from communication with the server and organising the emails has granted another level of flexibility. Software dependencies across myOS systems was briefly discussed which highlighted the possibility of aliasing software in configuration files to complete a task - this started a train of thought that will be the topic of the next blog post.
Installation and configuration of myOS-email was explained in detail. A notmuch tag + filtering script that learns and mimics your behaviour, and uses simple matching rules was given as an example highly effective method of organising mail with minimal effort.