How to encrypt your email, so the NSA can't read it.

Understanding Public Key Cryptography

When using encryption, it is helpful to understand some basic concepts. Don't worry, we won't get into the math, we are going to keep it really basic. But since encryption, like other digital technologies, is such an important part of modern life, this is really something that everyone should have some basic understanding of. I mean, this is the stuff that keeps your bank account and financial transactions safe online (I won't get into much of that because I'm focusing on encrypting email). You can skip this section if you want, but I highly recommend reading it.

When encrypting something you use some password, or key, to encode the information in such a way that it can't be deciphered or read without a key. It used to be that in order to use encryption, you had to arrange in advance for both parties to have the same key. It had to be exchanged in secret, because anyone who knew the key could also read the messages that where later encrypted with it. With this method, you need a separate key for each pair of people who want to communicate securely and they all have to be exchanged in secret! So this can quickly become unwieldy. This method is called symmetrical encryption, because the process works the same both ways. The same key that was used to encrypt the data can also be used to decrypt it.

Public key cryptography, however, is asymmetrical. The key that is used to encrypt a message cannot be used to decrypt it. Instead, there are two keys that are related: the two keys do the opposite of each other. So the keys come in pairs and something that has been encrypted with one key must be decrypted with the other key, and vice versa. Since the key used to encrypt the message can't be used to decrypt or read it, it can safely be shared in public. This is why it is called public key cryptography. Of course, this means that the other key of the pair must be kept secret, hence it is called the secret or private key. So, once you have created such a key pair, you put one away to be your secret key, and the other you can publish to all of your friends. Anyone can then use the public key to encrypt a message to you. Since the key you kept secret is the only way to reverse the process and decrypt the message, you will be the only one who can read these messages.

The only thing your friends need to do then is to verify that the key they are using actually belongs to you. Of course, if they use a key that really came from someone else claiming to be you, then that person can read the messages they intended to be secret. You can meet in person to verify that they have the right key, but it doesn't have to be in secret. The software will give you a fingerprint, a short sequence of characters, that identifies the key. You can compare the fingerprint of your key with the fingerprint of the key your friend has received, if the fingerprints are the same, so are the keys they came from.

Digital Signatures

Public key cryptography also gives us a way to verify who sent a message. Remember that anyone who has your public key (which really could be anyone), can encrypt a message to you. So how do you know who really wrote the message?

Let us suppose that you and a friend each have your own key pair, and that each of you has the other's public (but not private) keys. In order for your friend to send you a message, he encrypts it with your public key, so that only your private key can decrypt it. Now, what if he encrypts it instead with his secret key? Remember that the keys do opposite things, so if it was encrypted with his private key, it can be decrypted with his public key. At first this might seem useless. Since everyone has access to the public key, anyone can read the message. But once you do, you know something about it that you would not have known otherwise. That message could only have been written by someone who had access the secret key that goes with the public one you used to decrypt it. If you have verified that the public key belongs to your friend, and he has properly kept his secret key secret, you know the message is from him. This is called a digital signature.

Of course, you can use both processes on the same message, this way when your friend reads it he knows both that it came from you and that no one else could have read it. Or you can sign a message without encrypting it, so that anyone can read it, but they can verify who wrote it if they have ever verified your public key.

So, how do we go about creating a key pair and using it to encrypt (and sign) our email?

Installation and Configuration

While there are quite a few different applications that will allow you to encrypt your email, I'm going to explain how to use one particular set of software here. It is very likely that you are used to checking your email by logging into a website, but that doesn't work very well if you want to encrypt your email. Instead, we will be using Thunderbird, the email application from Mozilla, the makers of Firefox. It runs on Windows and Linux. (It also runs on Mac. I'm not giving specific directions for Mac here, but you can use a similar approach on a Mac.) Thunderbird doesn't come with the encryption features we want built in, but it has plugins just like Firefox does and we will use a plugin called Enigmail to add the encryption features we want. Enigmail doesn't do the encryption itself, it uses GnuPG, so you will need that as well. If you are running Linux, you probably already have GnuPG, on Windows you will need to install it as well.

Linux

If you are running Linux, you probably already have GnuPG installed and if you don't it should be available through your package management system. On Debian, you can install icedove (which is the Debian version of Thunderbird with a different name and logo). Ubuntu has a thunderbird pacakge. Or you can install the official Mozilla build. You may also be able to install Enigmail from your package management system to be available for all users, if you installed your distributions official package for Thunderbird or Icedove; however, I'm going to show later how to install the plugin from within Thunderbird for each user.

Windows

First we need to make sure that GnuPG is available. There is a handy Windows package called GPG4Win, there are several packages available but for what we are doing you only need to install the Vanilla version. If you install the full package it comes with a different mail client, Claws Mail, which takes the place of Thunderbird. Feel free to try it if you like, but my instructions will be covering Thunderbird. Run the installer and just click through the "Next", "Install", and "Finish" buttons.

Next, we need to download and install Thunderbird. This one is even a bit easier; it's just "Next", "Next", "Install", "Finish" and you're done.

Email Configuration

When you first run Thunderbird, it offers to set up a new email account for you. You can use this feature if you need a new address, but most people reading this will already have one. Thunderbird does a really good job of getting the correct settings for most major email providers, so usually all you need to provide is your name, email address, and password. If you have difficulty, you can contact your email provider to get the right settings to put in manually, or you can set up a new account to use with Thunderbird and Enigmail. In my examples, I'm using two of the more popular webmail services, Gmail and Hotmail, and Thunderbird determined the correct settings automatically.

Setting up a Hotmail account on Windows

 Setting up a Gmail account on Linux (KDE on Debian)

Installing Enigmail in Thunderbird

Now we need to install the Enigmail plugin in Thunderbird (if you didn't install it from your package management system on Linux as mentioned earlier). This is pretty simple and works the same whether you are running Thunderbird on Linux or Windows.

In the most recent versions of Thunderbird, there is a menu button on the right which looks like three thick horizontal lines. On this menu, click on "Add-ons" and it will open a new tab for the Add-ons Manager. You can use the search box in the upper right to search for enigmail. Click on "Install" and then you will need to restart Thunderbird once it is finished.

Creating and Sharing Keys

The next thing we need to do is create a key pair. Again, we will go to the menu button, this time under the "OpenPGP" menu we will click on "Key Management". This opens a new window for OpenPGP Key Management, from the menu across the top of that window click "Generate", and then "New Key Pair". You need to use a strong passphrase to protect your key pair. If you want to make your encryption stronger, under the "Advanced" tab change the key size from 2048 to 4096. (BTW, you DO want your key to expire.) Click "Generate key" and go do something else on the computer while you wait for it to finish (this can take a while). Once your key has been generated, it will offer to generate a revocation certificate. You should do this and then store it in a safe place. A revocation certificate is used to invalidate your key if it is ever compromised.

Once you have generated your own key pair, you will want to upload it to a key server to share with your friends. You will need to check "Display All Keys by Default" in order to see your own key, click to select the key you just generated, then go back to the menu, click "Keyserver", and then "Upload Public Keys".

In order to use a friend's key to encrypt messages to them, you must first download their key. In order to search the key server for their key, you need to go back to the menu on the OpenPGP Key Management dialog and click "Keyserver" (the "Search for:" box is only for searching through the keys you have already downloaded, not for new ones), and then "Search for Keys". Type in the email address (Key ID if your friend has given you the ID for his current key) you want to find a key for. Once you have found and selected the keys you want, click "OK" to download them.

In order to verify that you have the right keys for each other, you will want to compare the fingerprints. You can find the fingerprint for your key, or your friend's, by right-clicking the key and then clicking "Key Properties" from the pop-up menu. The key fingerprint is near the middle of the dialog. Copy it down and compare it with what your friend got for his key to make sure they match.

 

Once both users have verified that they have the right key for each other, we are ready to start sending encrypted email.

Sending Encrypted Email

In order to send an encrypted email, you begin composing a message just as you normally would. Enigmail provides several places to access the settings to sign and/or encrypt your mail. There is a menu button labeled "OpenPGP" with a padlock on it and a small arrow to the right. The arrow pulls up a dropdown menu which includes options to sign and encrypt a message, clicking on the button itself pulls up a dialog box with the same options. There is also a small pair of icons at the bottom right of the message composition window, a pencil and a key. This will be highlighted, or colored, once you have selected the corresponding option to sign or encrypt the message. This gives an easy way to see the status of the message before sending. You can also click on these icons directly to encrypt or sign your message. When you send the message, it may ask for your password to access your private key (depending on your preferences and how long it has been since you last entered your password).

Note that the subject line is not encrypted. So you want to make certain that you don't disclose any sensitive information in the subject line, either by keeping it vague (but potentially helping the reader classify their mail before decrypting it), or replacing it entirely with "Encrypted Message" or something.

Also, keep in mind that you can sign a message without encrypting it, or encrypt a message without signing it. Signing a message without encrypting it allows you to send a public message to a group while still allowing people to verify that the message is authentic and does come from you.

Reading Encrypted Email

When you open an encrypted email, it will appear only as a bunch of garbled nonsense until it is decrypted. Again, it may ask you for your password (depending on your settings, etc). Once we have entered the password, we can read the message.

If the message was signed, we can also see that information at the top in a colored banner. Here, it states that there is an "UNTRUSTED good signature". What does that mean, and why is it untrusted? It means the message is, indeed, signed with the public key we have for the sender; but although we verified the key, we haven't done anything to indicate that we have verified it. By default, Thunderbird allows you to use a key without have given any indication that we have verified the validity of the key. So what we are seeing here is actually just fine for now (in part 2, we will learn more about this and how to mark a key as verified or trusted).

Conclusion

At this point we have learned enough to set up and use encrypted email. I've kept it to the basics to make sure this is relatively easy to get into, but I'll be writing a part 2 to introduce some more features that will make things easier for you and your friends. With these simple instructions, you can install and set up the necessary software, create and exchange keys, and send and receive secure encrypted email.

So, how good is this encryption? Will it really prevent the NSA (and anyone else) from reading your email? According to Edward Snowden, "Encryption works. Properly implemented strong crypto systems are one of the few things that you can rely on. Unfortunately, endpoint security is so terrifically weak that NSA can frequently find ways around it." So what is that part about "endpoint security"? He is referring to the ends of the communication channel where the message is encrypted and decrypted, the two users' computers. If either of the systems is compromised, messages could be intercepted or forged. For instance, a virus on your computer could intercept all of your key strokes as you type them, sending the message to an attacker before it is ever encrypted. Or it could get your password when you type it, access your private key and use it to forge messages from you. It sounds rather scary, but it is worth considering that these sort of attacks are probably used primarily against priority targets, not generally as a tool of mass surveillance. So if you are simply wanting to thwart mass surveillance, and haven't done anything to actually draw the suspicions of powerful organizations like the NSA, these things may not be a major concern. Nevertheless, it would be wise to take steps to secure your system. Running an antivirus and preferably using a GNU/Linux operating system (the most popular version is Ubuntu, the one I'm using which is pictured here is Debian, from which Ubuntu is actually derived), would be a good start. But in general, yes, the techniques described here offer a high degree of protection for your digital communications.

Pop-up and email reminders

Previously, we used remind to calculate the next Thanksgivukkah (co-occurrence of Thanksgiving and Hanukkah). Today, we will learn to use remind for a much more typical use case, birthdays and anniversaries. But we are going to take it a little bit further and write some useful scripts using remind to send us reminders by email and as a pop-up on your desktop. I'm doing this a bit differently here, stringing several things together, so you can look through for the things you want, or you can follow through from beginning to end to get a tutorial that will introduce you not only to the specific tools used here, but also to programming via shell scripts. This tutorial should be simple enough to follow even if you don't have programming experience.

So, what can you find here?

1. Basic usage of remind for birthdays and anniversaries
2. Using kdialog, gdialog, and zenity for pop-ups
3. Sending email from the command line with mail
4. Checking local mail with Mutt or Thunderbird
5. Scheduling tasks with cron
6. Writing simple shell scripts with Bash

Below you will find sections dedicated to remind, the pop-up script, and then the mail script. We will be learning shell scripting through the pop-up and mail script sections. At the end of each, we will look at using cron to run the script automatically on a schedule. Then we'll see how to check local mail with Mutt or Thunderbird, and to forward system messages to your user. Finally, I'll present the complete scripts for review (or for those who just want to skip to the code). Let's dive in and have some fun!

Remind

Creating reminders for birthdays and anniversaries

The first thing we need to do is create a reminder file containing the birthdays and anniversaries. This is plain text file, so use any text editor you are comfortable with. Let us suppose that Justin Smith was born January 31, 1987. To create a reminder for his birthday, we write a line like this:

REM 31 JAN MSG Justin Smith's [ord(year(trigdate())-1987)] Birthday

Whoa! That looks a bit complicated; let's break it down. All reminders start with REM and this one occurs every 31st of January. The MSG means that when the reminder is triggered (on the 31st of January), it will produce a message. You can probably guess what the part in brackets should produce in the final message, but it may still not be quite clear what is happening. Since this section is nested, we will work from the innermost pieces outward. The trigdate() function gives the date on which the reminder was triggered. In 2014, that will be January 31, 2014. The year() function takes just the year portion out of a date, so year(trigdate()) is 2014. Subtracting Justin's birth year gives us 27, which is of course how old he is this year. The ord() function turns a number into an ordinal, that is, 1 becomes 1st, 2 becomes 2nd, 3 becomes 3rd, 27 becomes 27th, etc. Ordinarily, remind is expecting the message to consist simply of words to be printed, the square brackets tell remind that what is inside is code to be evaluated and the result is to be pasted into the message before it's printed.

And so we write a line the same way for each birthday. Anniversaries are, of course, quite similar. This file doesn't have to be in a particular place or have a particular name since we pass it to remind on the command line, so we will just save it somewhere convenient. For our scripts, that file name will be included in our code when we run the remind command.

Getting the reminders for the day, week, or month

If all you give the remind command is the file with your reminders, it simply prints the reminders for today:

remind /path/to/reminders.txt

We can also print a calendar with reminders on it. To print the calendar for the next month, all we need is:

remind -c /path/to/reminders.txt

To print more than one month, just put the desired number after the -c, so to print a calendar for the next 12 month we use:

remind -c12 /path/to/reminders.txt

Now, if we only want a week (or number of weeks) we use a + after -c (but before any number):

remind -c+ /path/to/reminders.txt

These calendars all have "lines" drawn with ascii characters (like hyphens, vertical bars, and plus symbols). We can make them look nicer with smooth, unbroken lines using unicode by adding a u after the -c. So to print a nicer version of the current week reminders:

remind -cu+ /path/to/reminders.txt

To learn more about remind, read the manual with:

man remind

Popups

There are a number of convenient programs that allow you to use graphical dialogs to interact with shell scripts. In this case, we simply want to generate an informational pop-up to remind us of the birthdays and anniversaries today.

I'm using kdialog which comes with KDE because that is the desktop environment I'm currently using. If you are using Gnome or another GTK desktop environment, and your system has gdialog available, the same syntax will work with it. If your system has the newer zenity instead of gdialog, the syntax is slightly different so I'll show that as well.

kdialog --title "Today's Reminders" --msgbox "Justin Smith's 27th birthday"

This example is pretty simple, the string after --title sets the title of the pop-up window, and the string after --msgbox is the message in the pop-up. Of course, when we put this into a script, we'll actually get the reminders from remind, and we will see how to pass that to kdialog (or gdialog or zenity) later.

As noted, gdialog would be the same, but zenity is slightly different:

zenity --title="Today's Reminders" --info --text="Justin Smith's 27th birthday"

Putting it together in a shell script

In this step we are actually writing our own computer program! It's not that scary, though. These will be very short and simple. Shell scripts are essentially a series of commands that you could have typed in at the command line, but they are written in a file that can be run as a program. The first thing we need is a line that tells the system how to run our program, that is, what other program is needed to interpret our commands:

#!/usr/bin/bash

Those first two characters are magic. No really, they represent what is called a magic number which indicates to Unix-like systems that the file contains a script and the path to the interpreter follows on the same line. Whenever the system sees a file starting with #! the next thing it looks for is a path to the program it needs to run the commands in the rest of the file. In this case, we are using Bash which is usually located at /usr/bin/bash.

Now we need to get the reminders for today from remind, and pass them to kdialog (or gdialog or zenity). For convenience, we will store the reminders we get from remind in a variable.

MSG=$(remind /path/to/reminders.txt)

There are two things going on here. The equal sign allows us to use the string on the left later to represent the thing to the right. In Bash, when we use that string later as a reference, we will put a $ in front of it to indicate we are using it as a variable to represent something else.

The next thing happening here is that the $() indicate that we want to run the command inside and substitute the output of the command here. This is called command substitution, because we are substituting the output of the command for the command itself. So later when we reference $MSG, it won't mean "remind /path/to/reminders.txt", it will mean "Justin Smith's 27th birthday". Now our kdialog command looks like:

kdialog --title "Today's Reminders" --msgbox "$MSG"

Now, if it's not anyone's birthday or anniversary, this will produce a pop-up that says, "No reminders." We probably don't want this, so let's add a condition so that it will only create a pop-up if we actually have a reminder. Here is how we do that in Bash:

if [[ "$MSG" != "No reminders." ]]; then
    kdialog --title "Today's Reminders" --msgbox "$MSG"
fi

If you are used to other programming languages like C, C++, JavaScript, Perl, etc, this may look a little strange; if you don't have any programming experience, it probably looks rather complicated. I won't go into all of the details, but let's break down the basics. The if-then part of the construct mirrors standard English usage. The double square brackets indicate that what is inside is some kind of test. As in many programming languages, == is used to test whether two things are equal. In this case, != tests whether two things are not equal. We will only do the things that come after then if the message in $MSG is not "No reminders." If it is, then we will not produce a pop-up. Note that the spacing is important, there are spaces between the brackets and what is inside, and between the != and the two things it is comparing. We end the conditional part of our code with fi (the opposite of if), so if there was any code after that, it would be run regardless of what $MSG turned out to be.

So far we have:

#!/usr/bin/bash
MSG=$(remind /path/to/reminders.txt)
if [[ "$MSG" != "No reminders." ]]; then
    kdialog --title "Today's Reminders" --msgbox "$MSG"
fi

Now, if you run this script from a terminal in your graphical desktop environment, it will work fine. But if you go to a virtual terminal (CTRL + ALT + F1, etc) and run it from there, you will get an error, "cannot connect to X server". The X server is what is responsible for drawing all of those windows, icons, etc that make up your graphical interface. And on a multi-user system, there could be several different displays at once, so it isn't sure what to connect to unless you tell it. Now if there are several users who could be logged into a graphical desktop and the same time using a switch user feature, which display is yours may depend on who logged in first. Since we don't know in advance what it will be, we need a way to find what it is.

There is a useful tool for obtaining the information we need, simply called w. It shows us who is logged into the system and what they are doing, including which X server display they are using. The problem is that it gives us more information than we need, so we will have to cut out just the relevant piece. There are a couple of tools we can use to cut out the parts we want, but before I get into those details, let me introduce you to the Unix pipe. What we need to do here is take the output of one program and pass it on to another (and then pass it's output on to yet another program). All Unix-type systems provide a simple way to do this. It is called a pipe because conceptually we are piping the output of one command directly into the input of the next. The syntax is very simple, we simply join the commands with a | symbol.

The first thing we will do is to find a line that has the information we are looking for. It needs to contain the user name and an X display number (if you log in from a virtual terminal or via ssh, there will be no display number associated with that login). First, let's find only the lines that have an X display number. We will use grep, a tool that has powerful pattern matching abilities and works line by line. Each X display number starts with a colon and a digit. But if we look just for a colon and a digit, we could be picking up lines that don't have an X display number but just have a time formatted with a colon. To avoid that, let's look for space in front of the colon instead of a digit. OK, now that we know what we are looking for in our first step, let's see how we do it:

w | grep "\s:[0-9]"

Without going into all of the details of how grep works, I'll just explain the patterns used here. The \s represents some kind of whitespace, such as a space or tab. The : simply represents a literal colon. The [0-9] looks for anything in the range of 0-9, a digit.

Now the next thing we need to find is lines that begin with the correct username. For this, we will use grep a second time. Just using the username for the second pattern would probably work, but we can protect against some odd cases by using a slightly more complicated pattern:

w | grep "\s:[0-9]" | grep -m1 "^username\s"

We already know that the \s represents whitespace, so this prevents us from picking up a longer username that actually begins with your user name (eg- if your username is jon, you don't want the script to try to send your reminders to a user named jonathan). The ^ means the beginning of the line, so if your user name is ash you won't get a line for some other user running bash, etc. The -m1 option returns only the first matching line. Since all of the lines for a user at this point have the same X display number, simply taking the first will do.

Finally, we need to cut just the X display number out of the line we have selected. We will use a very powerful program called awk (it's so powerful you can write whole scripts with it, but we will only look at what we need here):

w | grep "\s:[0-9]" | grep -m1 "^username\s" | awk '{print $3}'

This bit is simple enough, the $3 represents the third field, where fields are separated by whitespace, and this is the part that awk will print to it's output.

Now, the commands that generate pop-ups automatically look for the X display number in a special kind of variable, an environment variable, called DISPLAY. We set that using a command called export, and we will use command substitution the same way we did when storing the output of remind:

export DISPLAY=$(w | grep "\s:[0-9]" | grep -m1 "^username\s" | awk '{print $3}')

Finally, we have all of the pieces, but I'm going to suggest a couple more tweaks to the remind command we are using. It won't make a difference with what we are doing here, but if you have more advanced reminders in use, the -q option tells remind not to queue timed reminders for later execution and the -g option will sort them by date, time, and priority. So here is what we have now:

#!/usr/bin/bash
export DISPLAY=$(w | grep "\s:[0-9]" | grep -m1 "^username\s" | awk '{print $3}')
MSG=$(remind /path/to/reminders.txt)
if [[ "$MSG" != "No reminders." ]]; then
    kdialog --title "Today's Reminders" --msgbox "$MSG"
fi

Making it run automatically

While the script we have written is certainly handy, we don't want to have to run it every day, the point was to make it automatic. So now we are going to learn how to schedule a command to run on a recurring basis. The scheduler on a Unix-type system is called cron and the configuration file that specifies what commands to run and when is called a crontab. To edit your crontab, just run:

crontab -e

The format of the crontab file is a bit obtuse. Each line begins with a series of characters indicating the minute, hour, day of the month, month, and day of the week for the command to be run on. An asterisk indicates that any value is acceptable. So, to run our pop-up script every day we would use:

0 0 * * * /path/to/reminderpopup.sh

This will run the script at midnight. If we wanted to wait and have it run at 8AM, we would use:

0 8 * * * /path/to/reminderpopup.sh

Mail

Now, let's try a script that generates a calendar of upcoming events and mails them to us. I won't get into configuring your system to send email, but your system is probably configured by default to deliver mail locally, to users on the same system. If your system isn't going to be configured to deliver email to the outside world, or hasn't been yet, you can send mail to user@localhost.

We've already seen how to generate a calendar of the reminders for the coming month, or the current week. It turns out, there is a handy command for sending email which is simply called mail. The body of your message is sent to the command's standard input, so we will use a pipe, just like we did earlier. There is a convenient way to combine the output of several commands so that you can pipe them all into the same command by wrapping the commands in curly braces. So, we can send a reminder email with something like:

{
echo "Here are your weekly reminders:";
remind -c+ /path/to/reminders.txt;
} | mail -s "Weekly reminders" user@localhost

The -s option provides the subject for the email. Now, we could make that calendar look nicer with unicode, but standard email is ascii so if we just throw some unicode in there all we will get is garbage. In order to send unicode email, we need to specify the encoding in a header. We can add an arbitrary header to the email with the -a option. Here is what we need to use a unicode calendar in our email:

{
echo "Here are your weekly reminders:";
remind -cu+ /path/to/reminders.txt;
} | mail -a "Content-type: text/plain; charset=UTF-8" -s "Weekly reminders" user@localhost

This works pretty well already, but it would be nice to include which week the reminders are for in the subject. For this, we would like to find the dates of the first and last days of the week. Again, there is a handy command for working with dates which is simply called date. If run without arguments, it simply prints the current date and time. However, we can tell it to find a different date and we can also specify how to format it.

SUN=$(date --date='last Sunday' +'%B %d')
SAT=$(date --date='next Saturday' +'%B %d')

We have already seen how to store something in a variable and how to get the output of a command. The specifications for the dates are fairly obvious. The + indicates a string used to indicate the format we want the date in, %B is the name of the month and %d is the day of the month.

Now there is one catch with the dates here, if it is run on Saturday or Sunday, that date will be off (it will be last Sunday or next Saturday when what we really want is this Sunday or Saturday). We can avoid this by checking if it is Saturday or Sunday and using the current date if it is:

if [[ $(date +'%a') == 'Sun' ]]; then
    SUN=$(date +'%B %d')
else
    SUN=$(date --date='last Sunday' +'%B %d')
fi
if [[ $(date +'%a') == 'Sat' ]]; then
    SAT=$(date +'%B %d')
else
    SAT=$(date --date='next Saturday' +'%B %d')
fi

Here, the +'%a' requests the current date be formatted simply as the three letter abbreviation for the day of the week. So if today is Sunday, we use today's date for the Sunday on which our calendar begins, otherwise we use last Sunday. We do the same thing with Saturday.

Putting it all together we have:

#!/usr/bin/bash
if [[ $(date +'%a') == 'Sun' ]]; then
    SUN=$(date +'%B %d')
else
    SUN=$(date --date='last Sunday' +'%B %d')
fi
if [[ $(date +'%a') == 'Sat' ]]; then
    SAT=$(date +'%B %d')
else
    SAT=$(date --date='next Saturday' +'%B %d')
fi
{
echo "Here are your weekly reminders:";
remind -cu+ /path/to/reminders.txt;
} | mail -a "Content-type: text/plain; charset=UTF-8" -s "Reminders for the week of $SUN through $SAT" user@localhost

Since this script generates a weekly calendar of reminders (exercise for the interested reader: write a version that mails reminders monthly instead of weekly), we want to make it's crontab entry run the script weekly, each Sunday:

0 0 * * 7 /path/to/remindermail.sh

Checking local mail

Since I didn't get into configuring your system to send mail to the outside world, I will at least give you a quick look at how to check your local mail.

Mutt

You can check local mail from the command line with Mutt. Basic usage is pretty simple, it will take you to a list of messages, you can move up and down the list with the arrow keys and read the highlighted message by pressing "Enter", pressing the letter "q" will return to the list of messages or exit the application. For more information, you can read the manual.

Thunderbird

It's also fairly simple to configure Thunderbird to check your local mail, so you can see your local mail right next to your regular email. Under "Edit", select "Account Settings...", in the Account Settings dialog pull down the "Account Actions" drop down menu and select "Add Other Account..." (yes, it's Other Account, not Mail Account), select "Unix Mailspool (Movemail)" and click "Next", make sure the email address is user@localhost (or the domain name assigned to your machine), continue clicking "Next" (the account name is arbitrary, use whatever you like), and then click "Finish". Thunderbird is now configured to check your local mail, but in order for this to work, your user must be in the mail group. If you aren't in it already, you can add yourself by running this command as root:

usermod -aG mail user

Note that if you do this, it moves your mail from your system mailbox, to Thunderbird's. So once you have checked the mail with Thunderbird, it is no longer available to Mutt. Thunderbird seems to ignore the setting to automatically check mail from a Unix mailspool, you must explicitly click "Get Mail".

Getting system mail

Another handy use for checking local mail is to get important messages about the system. Many system utilities will send mail (which is why many systems like Debian automatically have an MTA installed and configured for local mail). This mail typically goes to root, so if your system isn't already configured to forward this mail to your account, you can do so by simply adding this line to the end of you /etc/aliases:

root: user

Summary

reminderpopup.sh

#!/usr/bin/bash
export DISPLAY=$(w | grep "\s:[0-9]" | grep -m1 "^username\s" | awk '{print $3}')
MSG=$(remind /path/to/reminders.txt)
if [[ "$MSG" != "No reminders." ]]; then
    kdialog --title "Today's Reminders" --msgbox "$MSG"
fi

remindermail.sh

#!/usr/bin/bash
if [[ $(date +'%a') == 'Sun' ]]; then
    SUN=$(date +'%B %d')
else
    SUN=$(date --date='last Sunday' +'%B %d')
fi
if [[ $(date +'%a') == 'Sat' ]]; then
    SAT=$(date +'%B %d')
else
    SAT=$(date --date='next Saturday' +'%B %d')
fi
{
echo "Here are your weekly reminders:";
remind -cu+ /path/to/reminders.txt;
} | mail -a "Content-type: text/plain; charset=UTF-8" -s "Reminders for the week of $SUN through $SAT" user@localhost

crontab

0 8 * * * /path/to/reminderpopup.sh
0 0 * * 7 /path/to/remindermail.sh

Notes

Bash versus generic sh

The scripts here are all written specifically for Bash, not portable shell scripting. All Unix-type systems have a shell, /bin/sh, that conforms to certain standards. Generic, portable shell scripts will work with any of them, but the code I have written here use extra features that Bash offers and so these scripts will only work with Bash. You can do the same things with portable shell scripting. So, if you are trying to write portable shell scripts, you will want to note the differences from what you have seen here.

Debian

I wrote and tested most of this on Debian. If you are using a different distribution, there may be some differences. I appreciate feedback if there is a significant difference affecting users of another distribution, etc.

Feedback

Questions, corrections, and feedback are welcome. You can comment on the post or email me directly. I'll try to answer any questions, but I can't guarantee a time frame. 

Update, 2015-01-12:

The previous version of the popup script broke when updating to Debian 8.0 “Jessie” as the output of w has changed. The following version works on the new version:

#!/bin/bash
export DISPLAY=$(w | grep -m1 "^blue\s\+:[0-9]" | awk '{print $2}')
MSG=$(remind -q -g /home/blue/Documents/FamilyBirthdays-remind.txt)
if [[ "$MSG" != "No reminders." ]]; then
    kdialog --title "Today's Reminders" --msgbox "$MSG"
fi

Calculating the next Thanksgivukkah with remind

Apparently, this year will see a rare coincidence of Thanksgiving with the first day of Hanukkah. It has been reported that this will not occur again for 79,043 years. I was suspicious of this and wanted to find for myself when the next Thanksgivukkah might occur. Now, this is a fairly complicated prospect as Thanksgiving (in the US) is defined as the last Thursday of November, while Hanukkah begins on the 25th of Kislev on the lunisolar Hebrew calendar. It turns out that there are some very powerful free software tools for GNU/Linux for calculating dates which can handle even this kind of complicated question.

I used a handy tool called remind. I'm sure you could do the same thing with Emacs calendar. (Actually by the time I was done, I was thinking I really should have tried Emacs instead...) With remind you store your reminders in a specially formatted text file. It is a command-line tool with many options and gui frontends available, but here I will just illustrate using the command-line tool to calculate the next date of Thanksgivukkah.

First, let's look at how to individually find the dates of Thanksgiving and Hanukkah. Here is what we need to set a reminder for Thanksgiving:

REM Nov Thurs 22 MSG Thanksgiving

This reminder will be triggered on the first Thursday on or after the 22nd of November, which is always the fourth Thursday.

Now, for Hanukkah remind thankfully supplies functions to support the Hebrew calendar, so here is what we will use for Hanukkah:

REM [trigger(hebdate(25, "Kislev"))] MSG Hanukkah

So remind calculates what day the 25th of Kislev falls on in our Gregorian calendar and all is right with the world. So how do we find when the two fall on the same day? This is where it gets complicated. First, we need to write a user-defined function that will check whether a date is Hanukkah:

FSET isHanukkah(date) \
     trigger(date) == trigger(hebdate(25, "Kislev", trigdate()))

This function compares the current date being evaluated with the date of Hanukkah for that year. Now, we will use the SATISFY clause to search for a Thanksgiving that is also Hanukkah and print a reminder:

REM Nov Thurs 22 SATISFY [isHanukkah(trigdate())] MSG Thanksgivukkah

So, to look just for the next date of Thanksgivukkah, create a remind file that contains only:

FSET isHanukkah(date) \
     trigger(date) == trigger(hebdate(25, "Kislev", trigdate()))
REM Nov Thurs 22 SATISFY [isHanukkah(trigdate())] MSG Thanksgivukkah

Now, in the process of writing this I discovered that in order to get this to work, you need a 64-bit system, and you need to recompile remind to take advantage of the larger dates a 64-bit signed integer can handle, otherwise you just get a date overflow error. (When I initially had used the pre-1942 definition of Thanksgiving, I did not run into this problem, see the note at the end. But since I had already published the mistake, I shall show the correct way even if we must edit and recompile from source. Onward!)

If you are brave enough to proceed with compiling your own modified remind (and have a 64-bit system), once you have downloaded the source and unpacked it, you will need to edit config.h.in and change

#define YR_RANGE 4000

to

#define YR_RANGE 17536635926176

To compile, just run:

wish ./build.tk

If you wish to install it system-wide, just do:

make install

Now, you can set you system clock to some time after Thanksgivukkah this year and print the next date with:

remind -n -x77030 /path/to/Thanksgivukkah-reminder-file

You can avoid the silliness with (falsely) resetting the system clock by printing a list of dates, but this performs very poorly and may take hours to run:

remind -s853000 -x77030 /path/to/Thanksgivukkah-reminder-file

 (Note that -xn sets how many times the SATISFY clause will be evaluated and the default is only 150)

Either way, we can confirm that the next date of Thanksgivukkah is November 23, 79043. So it's not quite 79,043 years in the future, but actually the year 79043.

EDIT

I initially used the pre-1942 definition of Thanksgiving, which was the last Thursday of November, rather than the fourth Thursday of November. For that we use:

REM Dec Thurs 1 -7 MSG Thanksgiving

Now I bet you are thinking that looks mighty odd, especially since Thanksgiving is in November, not December. It is not a typo, what we are doing here is to find the first Thursday in December and then counting back 7 days to get to the last Thursday in November.

So in this case we would use:

FSET isHanukkah(date) \
     trigger(date) == trigger(hebdate(25, "Kislev", trigdate()))
REM Dec Thurs 1 -7 SATISFY [isHanukkah(trigdate())] MSG Thanksgivukkah

If you didn't proceed with the recompilation described above, this example should still work fine for you, just run:

remind -s2965 -x247 /path/to/Thanksgivukkah-reminder-file

Now, the answer we get is:

2013/11/28 * * * * Thanksgivukkah
2108/11/29 * * * * Thanksgivukkah
2260/11/29 * * * * Thanksgivukkah

So the next date of Thanksgivukkah would only be 95 years away, on November 29, 2108, if we still celebrated Thanksgiving on the traditional date of the last Thursday of November. So, if you really want more Thanksgivukkah, you should petition Congress to change the date of Thanksgiving back! (Although this really only gives you two more Thanksgivukkahs before Hanukkah drifts away from Thanksgiving for millennia.)

It is also noteworthy that there will be a few more instances when Thanksgiving coincides with the first night of Hanukkah. That is, the first candle will be lit on the evening of Thanksgiving, but it won't technically be Hanukkah until sunset on Thanksgiving evening. We can find these dates simply by using the 24th of Kislev instead of the 25th, and this is another example you can run without modifying remind. This will give us:

2070/11/27 * * * * Thanksgivukkah?
2165/11/28 * * * * Thanksgivukkah?

Well, that's kind of like Thanksgivukkah, I suppose.

Creating animated GIFs and APNGs from videos

Linux Setup

For this project we will need two or three tools. We will need avconv to pull frames from a video file, and we will need gifsicle if we want to created animated GIFs and apngasm if we want to create animate PNGs. If you are on Debian, you can install libav-tools (which provides avconv) and gifsicle in the normal way, but see my previous post on how to install apngasm from source (you will also find an explanation of the differences between animated GIFs and PNGs, and when to use each). If you use another Linux distribution, check your package management system.

Windows Setup

To get this to work similarly on Windows, we will need some extra tools and the setup is a bit more complicated. To get glob expansion and redirection, we will install Cygwin. Cygwin provides a Linux-like environment on Windows, including the Bash shell and the GNU coreutils (basic commands like ls, cp, rm, mv, etc). The installer also serves as a package manager, allowing you to select the various software packages you want to install or upgrade. If you are not sure what you want, just install the default packages. This will provide a lot more than we need, but you will have a useful environment if you need it later. You can always run the installer again later if you want to install more packages.

You will need to install 7zip, since Libav binaries come in a .7z archive you will need 7zip to extract them. When using the most recent version of Libav (0.9.7) on Windows, all of the GIFs came out in grayscale, so I tried the same version that was included with Debian Wheezy (0.8.6) and found it to be working on Windows (YMMV).  Gifsicle and apngasm come in .zip archives. Gifsicle and apngasm both contain a single binary you can extract to wherever you would like. Libav is a bit more complicated, you will need to extract the bin, lib, and share directories. For simplicity, you could extract all of these directories to C:\, and then put the gifsicle.exe and apngasm.exe in C:\bin (or choose whatever directory structure seems suitable for you).

Now, it would be nice to put  the directory (or directories) containing our new tools in the system path variable, so Windows (and Cygwin) will know where to find them without us having to type the full path to the command. You can do that in cmd.exe like this:

setx path "%PATH%;C:\bin"

Now, for any of the following commands, we will use the Cygwin Terminal. If you are from a Windows background, you will notice that all of the paths are separated with forward slashes instead of back slashes (C:/Users/Name/Documents, rather than C:\Users\Name\Documents). If you have used Cygwin before, you will need to note that these are not Cygwin binaries we are using, so the path arguments they take will use Windows drive designations (C:/Users/Name/Documents, rather than /cygdrive/c/Users/Name/Documents). However, this only applied to arguments passed to the commands, if you are giving the full path to the command, you will be using the Cygwin form (/cygdrive/c/bin/avconv.exe).

Essentially, when you are running the following commands in Cygwin terminal on Windows, you will just add the drive letter (C:) in front of the file paths, and the commands will end with the .exe extension.

Extracting Images from the Video

The first step is to convert our video to a series of image files. Now when we do this, we need to know what frame rate we want our animation to run at, and how if we want to make it faster (or slower) than the original video. If we want it to run at the same speed, we will use the same frame rate when extracting the images as we will use when we create the animation. If we want to alter it's speed or playing time, we'll want to do a little math. For instance, suppose I want to make an animation of these roses blooming from youtube. The video runs for about a minute and a half, but let's say we want to see the whole process in about ten seconds in our animation. If we want our animation to run at 30 frames per second, then we need 300 frames for a 10 second animation. To find the frame rate we want to pull from the video, we take 300 frames divided by 90 seconds, which gives us a frame rate of 3.33 fps.

Now that we have chosen the frame rate, let's look at the command:

avconv -i /path/to/rosesblooming.mp4 -r 3 -vsync 1 /path/to/rosesblooming%03d.png

Of course, if you are creating an animated GIF, you will use the same command with a .gif extension. Now you should have a collection of PNG and/or GIF images extracted from your video.

Putting together an animated GIF

Now when we create the animated GIF, gifsicle takes a delay in centiseconds (or hundredths of a second) rather than a frame rate. So a frame rate of 30 fps is equivalent to 1 second divided by 30 and multiplied by 100, or about 3 centiseconds.

Here is the command:

gifsicle -d 3 -l /path/to/rosesblooming*.gif > /path/to/rosesblooming.gif

Putting together an animated PNG

When we create the animated PNG, apngasm takes a delay as two integers composing a fraction of a second, so if we have chosen a frame rate, the first integer is one and the second is the frame rate.

Here is the command:

apngasm /path/to/rosesblooming.png /path/to/rosesblooming*.png 1 30

Creating a thumbnail in an animated PNG

The APNG format allows you to specify that the first frame will be skipped during the animation so it is only shown in viewers that don't support animation. To those viewers, the APNG we just created with look like a bunch of leaves. If we want them to see the fully opened roses instead, copy the last frame to /path/to/rosesblooming000.png and add the /f option to the command, like this:

apngasm /path/to/rosesblooming.png /path/to/rosesblooming*.png 1 30 /f

Looping

The above commands create animations that repeat indefinitely. This is the default for apngasm, for gifsicle the -l option specifies this behavior. If you want the animation to play through only once, you can just omit the -l for gifsicle since this is the default, but for apngasm you will add /l1 to specify this. For apngasm, you can use any number in place of 1 to play through a given number of times. For gifsicle, the -l option will take a number as well, but it is separated by a space.

Conclusion

Now you should have all of the tools you need to create your own animated GIF and PNG images. And you can always learn more by reading the documentation for these tools.

Installing APNG Assembler (apngasm) on Debian Wheezy

Why you would use animated PNG instead of GIF

If you are reading this post, it is likely you may already know the virtues of APNG, but in case you've found this an aren't sure let me give a quick explanation. While animated PNGs are not as widely supported as animated GIFs, they are technically superior. GIF files support only 256 colors, while PNG supports 16,777,216. PNG files support alpha transparency, which means that parts of the image can be semi-transparent. With GIF files, a pixel is either completely transparent or completely opaque, which results in jagged outlines on images with transparency when the background is unknown. You can see the differences for yourself with this demonstration.

Unfortunately, as previously mentioned, support for APNG is somewhat limited. It was developed by Mozilla and has been supported by Firefox since 3.0. Opera supported it from 9.5-12.1, but unfortunately they gave up support for APNG when they moved from their own Presto engine to the Blink (formerly WebKit) engine used by Chrome. It is not natively supported by Chrome, currently the most popular browser in use, but there is an extension that will add support for APNG.

So, if you want the widest support, you may still want to use an animated GIF. But if you want quality, you need APNG and you can suggest your users try Firefox or the APNG extension for Chrome.

Installing APNG Assembler on Debian

APNG Assembler, or apngasm on the command-line, is a simple tool for creating APNGs from a series of PNG files. However, you might notice that it is not available in the official Debian repositories for Wheezy. The project has a Linux binary, but it didn't work for me on Debian Wheezy. However, building it from source is incredibly simple. Go to the project files on sourceforge, select the latest version, and download apngasm-x.x-src.zip. Then use the following commands (these work for 2.7, adjust accordingly if using a newer version):

unzip apngasm-2.7-src.zip
make
sudo cp apngasm /usr/local/bin

Now apngasm is installed and ready for use. (Note that you must be in the sudo group to use the last command.)

Usage

APNG Assembler doesn't come with a man page, but the command is documented in the readme.txt that comes with the source. Nevertheless, here is a quick overview.

 In order to make an animated PNG, you need a series of PNG images whose names end with sequential numbers, like this:

frame01.png
frame02.png
frame03.png

and so forth.

Basic usage is:

apngasm animation.png frame*.png

This produces an animated PNG that loops at 10 frames per second forever. If you want to change the delay between frames, you add two number to the command that represent a fraction of a second delay time. So, to produce an animated PNG that loops at 25fps (1/25 second delay) you would use:

apngasm animation.png frame*.png 1 25

If you what the animation to loop a limited number of times and then stop, you can use the /l options like this:

apngasm animation.png frame*.png /l2

This will cause the animation to play through twice, and then stop.

For backwards compatibility, animated PNGs can be displayed as a regular, non-animated PNG, but only the first frame will be shown. However, animated PNG also has a feature that allows you to skip the first frame. This way, you can use the first frame to display a thumbnail to non-supporting applications while omitting it from the animation. This thumbnail could be a single frame from the middle or end of the animation, possibly with a message informing the viewer that the image contains an animation their browser or viewer is not able to display. Once you have created this thumbnail as your first frame, simply add the /f option:

apngasm animation.png frame*.png /f

Now the first frame will be displayed for viewers that don't support APNG, but omitted from the animation.

Now you are ready go and fill the internet with much better animated images!

Protip #1

Some of you already do this and will think it is comically obvious. But someone out there will read this and think, "Oh, wow... that is such a great idea!"  and it will save them many a headache in the future. That person will achieve enlightenment.

Protip #1:

Whenever you are working on a computer and come across some oddity or have to configure things in an unusual or non-obvious way to make it work, make a note of it and document this information for future use. This will keep you from spinning your wheels trying to solve the same strange problem again later on (you don't think at the time that you will forget this stuff, but you will). One way to do this is to file this information away wherever you store important documents and information. This particularly works well if you maintain a lot of machines and want to keep your records in a centralized location. A simple alternative method (one I really like) is to simply write the information on a note card or something of that nature and put it inside of the case. That way, you can always find it when you are working on that machine.

Upon hearing this, the reader was enlightened.¹

Fixing ugly fonts on Tim Berners-Lee's (and other) webpages

Have you ever visited a webpage, such as Tim Berners-Lee's page (for those who don't know, he invented the World Wide Web), and found the font was wobbly, bumpy, jagged, or uneven? Various letters seem to be rendered at different sizes, making the text ugly and difficult to read. The problem is that on GNU/Linux and other Unix systems with FontConfig, URW Palladio L (which is the clone used for Palatino) suffers from really bad font hinting. Normally, hinting improves the appearance of fonts, particularly at small sizes and low resolutions. However, in this case it has gone horribly wrong. You can see the effect in these screenshots:

Fortunately, the fix is fairly simple once the problem has been identified. We can turn off hinting specifically for URW Palladio L with a little configuration. As the link above suggests, you can fix this on a per-user basis, but if you want to fix it system wide, simply create the file /etc/fonts/local.conf containing:

<?xml version="1.0"?>
<!DOCTYPE fontconfig SYSTEM "fonts.dtd">
<fontconfig>
  <match target="font">
    <test name="family">
      <string>URW Palladio L</string>
    </test>
    <edit name="hinting">
      <bool>false</bool>
    </edit>
  </match>
</fontconfig>

Then run:

fc-cache -f -v

to update the font cache information. You will need to close any applications that are currently open and reopen them for the changes to take effect. Now you should see much improved rendering for URW Palladio L, as seen in these screenshots:

Considering how bad the rendering is (I would consider it utterly broken), I'm not sure why hinting isn't turned off for URW Palladio L in the default configuration, but now at least you know how to fix it. And Tim Berners-Lee's home page should look much nicer.