Better speech compression with Opus

In a previous post, we looked at configuring Audex for ripping CDs to the superior Opus format. While that was good for general purpose audio encoding, it doesn’t let us take advantage of Opus’ immense versatility, it’s ability to get excellent results across a wide range of bit rates.

You will notice at the low end of that graph are codecs that typically specialize in speech, because speech is more compressible that complex audio sources like music. Speex, for instance, can get excellent compression of speech with acceptable quality, yielding much smaller file size than MP3 or Vorbis. One of the benefits of Opus is that it should be able to replace this usage as well.

As you might have guessed, the opusenc commands we gave Audex can also be used directly on the command line. If you want to target a lower bit rate for speech encoding, all you need to do is add the --bitrate option. For speech, try something in the range of 16-48. In Audex, you might add two different profiles for compressing speech with Opus, just as there are multiple profiles for different qualities of compression with Vorbis.

opusenc takes raw, Wave, AIFF, or FLAC input (FLAC is still the preferred format for archival storage). If you pass a FLAC file that is already tagged with metadata, opusenc will incorporate that metadata automatically, so there is no need to pass it on the command line.

opusenc --bitrate 16 input.flac output.opus

For comparison, a 40 minute audio recording ripped as an uncompressed Wave file was about 403 MB, compressed with FLAC it was about 103 MB, opusenc with default settings got it down to about 26 MB, while the above command reduced it to about 4.5 MB. So targeting the lower bit rate was a space savings of over 80% and made it small enough to share by email.

Ripping CD to .opus with Audex on Debian

The Opus codec is basically the one codec to rule them all. It is an open standard (RFC 6716) and it outperforms almost all other codecs, open and proprietary, over a huge range of bitrates. It also has a very low latency which is important for internet applications involving real-time communications and it is already supported by modern browsers like Firefox, Chrome, and Opera. But if you want to start storing your music in the best available format, you might have difficulty finding convenient tools that support it. Well, here is one that worked for me, so I’ll share it. It was pretty simple, although not entirely obvious.

Audex is a graphical CD ripper for KDE. It does not support ripping to Opus out-of-box, but it does have a feature to add a new encoder. For this, we will just use the command-line encoder found in opus-tools, so install the packages audex and opus-tools. Now, when you run Audex, go to “Settings” > “Configure Audex...”, select the “Profiles” section, and then “Add...”. Name the new profile “Opus”. Under the “Encoder” tab, select “Custom” from the first drop down. Under “Command pattern:”, put in “opusenc $i $o”, and under “Suffix:” put in “opus”. You can look at some of the other options if you wish (I set mine to replace spaces with underscores), but that is all that is necessary to get it working. Save your new profile, select the profile from the main window, and begin ripping your favorite discs.

If you also want to capture metadata, you will need a newer version of opus-tools than what is in Wheezy. You can download the source, and build it yourself. You will need to have the wheezy-backports repository enabled to get some of the dependencies:

sudo aptitude -t wheezy-backports install libogg-dev libflac-dev libopus-dev

Once the dependencies are met, you should be able to configure and install opus-tools:

cd opus-tools-src-dir
./configure
make
sudo make install

Now, you can change the “Command pattern:” in Audex to “opusenc --title "$ttitle" --artist "$tartist" --album "$title" --date "date" --genre "$genre" $i $o”. This will capture metadata and embed it in your new Opus files.

Update: If you are looking to get better compression for speech recordings, such as lectures, sermons, speeches, etc, see this post.

Setting up root SSH login on CyanogenMod

I did this with the latest CyanogenMod 11 snapshots on a Nexus 4 and a Nexus 5. I recommend buying a device with an unlockable bootloader, like the Google Nexus devices, because it makes rooting and installing custom ROMs, etc, much more straightforward. Besides, if you buy a device you ought to own it, so why give your money to a company that tries to lock you out of your own devices, as if they still own it even after you have bought and paid for it? If you already have a device that is locked down, you may have to search the web to find a hack to get access to it (Good luck!), but I won’t be covering that here. What I found to be difficult to find and poorly documented elsewhere was how to configure your device for root login via ssh, after installing CyanogenMod. This can be useful for a variety of reasons, for instance, you can easily make a full back up of the phone securely over your wireless network. But as always, exercise caution when using root!

Before you begin, make sure you have a few options set on the Android device. Under “Developer options” make sure that “Android debugging” is enabled, “Root access” is set to “Apps and ADB”, and while you are here, set “Device hostname” to something memorable. (You should have learned to access the hidden “Developer options” menu while install CyanogenMod.) Now, with phone connected by USB, login from your computer with:

adb shell

then start setting up ssh by copying over the template configuration file:

cp /system/etc/ssh/sshd_config /data/ssh
vim /data/ssh/sshd_config

and add the line:

PermitRootLogin without-password

This does not do what it sounds like. It will not allow you to login without authenticating, rather, it disables authentication with a password and requires you to use public key authentication which we will set up in a minute. Next:

mkdir /data/local/userinit.d
cd /data/local/userinit.d
cp /system/bin/start-ssh 90sshd
vim 90sshd

and change:

   # don't daemonize - otherwise we can't stop the sshd service
   /system/bin/sshd -f /system/etc/ssh/sshd_config -D

to:

   # don't daemonize - otherwise we can't stop the sshd service
   ## Actually, yes, do daemonize (remove -D option)
   /system/bin/sshd -f /system/etc/ssh/sshd_config

Now, if you don’t already have one, you will need to generate an RSA key for ssh. On your computer (not the adb shell that is already logged into your Android device) run:

ssh-keygen

and with the default options you will get a ~/.ssh containing id_rsa and id_rsa.pub. You will need to copy id_rsa.pub to your Android device in order to be able to login. Still working from your computer:

adb push ~/.ssh/id_rsa.pub /sdcard/

Now, on the Android device:

cd /data/.ssh
touch authorized_keys
cat /sdcard/id_rsa.pub >> authorized_keys
chmod 600 authorized_keys

Note that the authorized keys file must not be readable by anyone else or ssh will refuse to use it and authentication will fail. Now, you should be able to reboot and login to your Android device:

ssh root@AndroidHostname

If you set a password for the RSA key you generated for ssh, it will prompt you for that password, but it will not prompt for a password for root on the Android device (because it is using the key instead). If you want to login from other devices, make sure you have an authorized key on that device as well. To add more authorized keys, simply concatenate them onto the authorized_keys file, the same way we did the first one. Now you can remotely access your Android device via ssh.

Watch the NFL Sunday Ticket on Linux

If you subscribe to the DirecTV NFL Sunday Ticket, you can watch all of your Sunday football games on Linux. But it only works with certain browsers because it requires Flash Player 14 which is only available on Linux as a Pepper plugin. Only the Chromium family of browsers supports this API, and the NPAPI version that is used by other browsers is no longer developed on Linux by Adobe (version 11.2 is still receiving security updates). The easiest way to get this is to install Google Chrome. It is also possible to use it with the Chromium packages provided by various Linux distributions; packages are available to install it for both Debian and Ubuntu. For Debian it is in the contrib repository and for Ubuntu it is in Multiverse. It may also be possible to use it with the developer version of Opera, which is now Chromium-based as well, but I haven’t tested this.

Of course, this doesn’t mean that you must or even should swap browsers. You can use a Chromium-based browser with Flash for the NFL Sunday Ticket and still choose another browser, like Mozilla Firefox which seemed to come out ahead in our recent benchmarks, for other things. Ultimately, we would all really like to see web applications, including video streaming, developed with HTML5 and web standards. Mozilla is probably one of the biggest forces pushing for this. Some people even refuse to use the proprietary Flash plugin at all in order to help promote this lofty goal. But if you want to watch your NFL games on Linux this year, now you know how.

Protip #3: You do want your OpenPGP key to expire

You think you don’t, but you really do.You are probably thinking you don’t want to have to transition to a new key on some arbitrary date. But you can always extend the expiration later. And if you lose your key and don’t have a revocation certificate, the expiration date serves as a kind of “dead man’s switch”. If you don’t prevent it from being triggered by extending the date, your key will be automatically invalidated. This way, if you lose your key (and you don’t have a revocation certificate or have lost that, too), your key will not remain valid forever.

If you use Thunderbird with the Enigmail extension, per my previous tutorial, it is pretty easy to change your key’s expiration date from the Key Management interface. Right-click your key and select Change Expiration Date. Then you can select how many years, weeks, or days it should be until your key expires.

Of course, you will need to re-upload your public key so that your friends can get the extended expiration from the key server. Of course, you can also do the same with GnuPG from the command line, but I’ll leave that as an exercise to the reader.

And now you have no reason to have no expiration date on your keys!

Create your Own Word Searches on Debian, Ubuntu, Mac OS X, or Windows

Creating your own word search is really simple. They can be easily generated by software, given a simple word list, and I’m going to share just such an application with you today. Word Search Creator is a desktop word search generator licensed under the GPLv2 (so it is free as in freedom, as well as price). The downloads page has packages specifically for Ubuntu, Windows, or Mac OS X. These are pretty straight-forward to install. Ironically, there is no package for Debian (the parent distro from which Ubuntu is derived), and the Ubuntu package is not installable on Debian stable. But no worries, this is open source software! So for Debian we will just install from source.

Installing from source on Debian

Before we can install Word Search Creator, we need to install the Qt4 developer tools:

aptitude install qt4-dev-tools

Now, download the source, unpack it and cd to the unpacked source directory. To build the application, just:

qmake
make

In order to install it system wide, as root:

make install

If you are running KDE, run kbuildsycoca4 to rebuild your application menus.

Create a Word Search

When you first open Word Search Creator, there is a control box on the right side that you use to create the word search. Here you can type in a word list and set the size of the grid (or set it to automatically enlarge based on the words you give it). Type in a title and hit Create/Shuffle to generate your word search (hitting it again will, as the name suggests, shuffle your word search).

The menus give you a number of other options you can set. By default, words are only hidden left to right or top to bottom, but you can choose which directions are allowed. You can also choose how the word list is ordered, and edit the formatting. Perhaps one of the coolest features is the ability to change the shape of the puzzle by excluding squares from the grid.

Having fun with it

You can work the puzzle right there on screen, highlighting the words as you find them, and they will automatically get struck off the list.

You can also save it as a word search file, export it to pdf or svg, or copy it to the clipboard as an image (Ctrl + C). Working with it as an image, you can also use an image editor to decorate your word search. Here, I have simply filled in the empty squares and given it a yellow background color to highlight the smiley face.

And here is a full page puzzle.

Try it out yourself, and have fun!

Windows Browser Benchmarks

Ok, so this a bit off topic for this blog as it isn’t really a how-to, but I thought it might be interesting to my audience nonetheless. I ran the major browsers through four different benchmarks: Google’s Octane 2.0, Mozilla’s Kraken 1.1, and Apple’s new JetStream and Speedometer (the latter focuses on measuring the responsiveness of web apps by simulating user interactions). For graphing purposes, I normalized the scores on the various benchmarks so that the fastest browser was always 100 (this also means inverting scores for Kraken, which measures time rather than some unit of speed).

To make things interesting, I decided to include some less common variants, as well as the major browsers. I have been hearing about optimized Firefox-derivatives like Pale Moon and Waterfox and wanted to see how they stack up. Here is what I found:

The official Mozilla build was generally faster, although Waterfox was a bit ahead on the Kraken. Pale Moon was consistently last. Both Waterfox and Pale Moon are based on older versions of Firefox. Apparently, the optimizations don’t make up for the improvements in newer versions of Mozilla Firefox, at least not at this time.

And here is how Mozilla Firefox stacks up against other major browsers:

Firefox was the leader except on Octane, where it was pretty close behind Chrome and Opera (which is now based on Chrome, hence the very similar scores). Internet Explorer was consistently near the bottom, losing on both Octane and the Kraken by a wide margin. Although it wasn’t last on Speedometer and JetStream, even coming in second on Speedometer, it was still nearer the bottom than it was to catching Firefox in the lead.

And here I put them all on one chart for a more complete overview:

Finally, I decided to do a combined comparison by looking at how far behind the lead each browser was on each benchmark. This graph is like a target with the bullseye in the center. A browser that was fastest on all benchmarks would be dead center, and the one losing by the widest combined margin would be furthest from the center:

Here we can see that Firefox was the clear winner. Waterfox still outperformed Chrome and Opera, while Pale Moon fell behind them. Internet Explorer was clearly last. Note, however, that the benchmarks are not really meant to be combined in this way and you could probably easily shift these rankings by choosing different benchmarks. Nevertheless, I think the results are interesting.

And for anyone who is particularly interested, here are the original numbers:

	Octane	Kraken	Speedometer	JetStream
Firefox 30	9169	3709.8	25.57	65.589
Pale Moon 24	7005	4413.8	19.15	57.805
WaterFox 28	8483	3405.5	21.5	62.103
Chrome 35	9698	3824.3	16.1	55.168
Opera 22	9551	3853.7	15.7	46.022
Internet Explorer 11	6416	6286.4	18.14	50.282

Generating and learning strong passwords, Python version

Just for fun, I rewrote my password shell scripts in Python. These scripts work essentially the same way and are compatible with the scripts from the previous post. The real purpose of this exercise is to give you the chance to compare some of the syntax and features of the languages.

This is my first post using Python, but I’m not going to explain all of the basics of Python syntax here. If you aren’t familiar with Python, you may want to read a tutorial or introduction to the language first.

Here is what genpw.py looks like:

#!/usr/bin/python3

import random
import sys

chars = ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z', '!', '"', '#', '$', '%', '&', '\'', '(', ')', '*', '+', ',', '-', '.', '/', ':', ';', '<', '=', '>', '?', '@', '[', '\\', ']', '^', '_', '`', '{', '|', '}', '~' ]
length = 13
pw = ""

for i in range(1, len(sys.argv)):
    if sys.argv[i] == "--alnum":
        chars = chars[0:62]
    elif sys.argv[i] == "--length":
        length = int(sys.argv[i+1])
    elif sys.argv[i] == "--forbidden":
        forbiddenchars = sys.argv[i+1]
        for ch in forbiddenchars:
            chars.remove(ch)

end = len(chars) - 1

for i in range(0, length):
    index =  random.randint(0, end)
    ch = chars[index]
    pw += ch

print(pw)

First, we start with a list of printable ascii characters. Since it has been sorted into numbers, letters, and symbols; we can get the alphanumeric subset with a simple slice operation. And we can remove any forbidden characters by simply calling the remove() method of the list of characters. Then, we find how many characters are in the resulting list and generate a random number within that range for each character in the final password. We simply use that number as the index to choose one of the characters from our list and then print the completed password string.

The approach here is slightly different from the one that I used in the Bash script. There, we generated a stream of random numbers and filtered out only the desired characters until we had the desired length. Here, we create a list of desired characters and select the desired number of characters randomly.

And here is what setpw.py looks like:

#!/usr/bin/python3

import sys
import os
import hashlib

if len(sys.argv) > 1:
    name = sys.argv[1]
else:
    name = "default"

pw = input().encode('ascii')
pwhash = hashlib.sha512(pw).hexdigest()
hashfile = os.path.expanduser("~/." + name + "pwhash")

print("\033[1A\033[0K", end="")
with open(hashfile, 'w') as f:
    f.write(pwhash)

This one is even more similar to the Bash version, it’s approach does not substantially differ.

Finally, here is what ppw.py looks like:

#!/usr/bin/python3

import sys
import os
import hashlib
import getpass

if len(sys.argv) > 1:
    name = sys.argv[1]
else:
    name = "default"

pw = getpass.getpass("").encode('ascii')
pwhash = hashlib.sha512(pw).hexdigest()
hashfile = os.path.expanduser("~/." + name + "pwhash")

with open(hashfile, 'r') as f:
    storedhash = f.read()

if pwhash == storedhash:
    print("Correct")
else:
    print("Wrong\a")

Again, the approach here is essentially the same as the Bash version. You may notice that instead of input(), I used getpass.getpass(""). This prevents it from echoing the characters to the terminal. By default, getpass() uses the prompt Password:, so to make it more consistent with the Bash version I passed it an empty string instead.

Hopefully you found this little exercise entertaining and educational.

Generating and learning strong passwords

In the wake of the recent catastrophic security vulnerability known as “Heartbleed”, many people have been tasked with thinking of new strong passwords for their online accounts and learning them. I’m not writing about Heartbleed, per se, but suffice it to say that you need to change passwords for any affected sites (after the vulnerability has been patched) and any sites you may have reused those passwords on. What I’m sharing here is an approach to generating and learning strong passwords. There are many approaches to password security, including password managers and using long passphrases instead of simple passwords, but I’m just sharing one approach here.

Humans are not very good at generating random passwords, so it can be helpful to use a proven computer algorithm and then simply work to memorize the password that was generated. We are going to use a bit of Bash scripting with the OS pseudo-random number generator and some basic Unix utilities for this. One of the best ways to memorize a strong, random password is to practice typing it. To help with this, we are also going to use a bit of Bash scripting that will let us type the password repeatedly and check if it is correct.

We could create three different shell scripts for this and keep them somewhere like /usr/local/bin (for system-wide use) or somewhere in your home directory (for personal use). Or, we could define them as functions in /etc/bash.bashrc (for system-wide use) or ~/.bashrc (for personal use). I’ll present them both ways, first as individual scripts and at the end as a series of functions. Putting them in separate script files would make them available from other shells, etc. (If you login to csh and invoke one of the scripts it will simply call Bash to run it. If you defined them as functions they would be unavailable in csh.)

First, here is a one-liner to generate a random password:

cat /dev/urandom | tr -cd "[:graph:]" | head -c 13 && echo

The first part reads from the pseudorandom number generator and passes it to the next part, the tr command removes all characters that are not printable ascii (you could also use "[:alnum:]" to generate an alphanumeric password), head -c takes only the specified number of characters and then terminates the pipeline, and the echo command simply outputs a newline, so that we don't end up with the command prompt being printed on the same line at the end of the password.

Now, we’ll look at a more complete example that takes command line options instead of manually editing our command:

#!/bin/bash
chars="[:graph:]"
length="13"
forbidden=""
for i in $(seq 1 $#); do
    if [[ "${!i}" == "--alnum" ]]; then
        chars="[:alnum:]"
    elif [[ "${!i}" == "--length" ]]; then
        ((n=$i+1))
        length="${!n}"
    elif [[ "${!i}" == "--forbidden" ]]; then
        ((n=$i+1))
        forbidden="${!n}"
    fi
done
cat /dev/urandom | tr -cd "$chars" | tr -d "$forbidden" | head -c "$length"
echo

This script takes several arguments. The --alnum argument limits the password to alphanumeric characters rather than printable ascii. The --length option is followed by the number of characters to generate and --forbidden is an additional list of forbidden characters (useful for sites that accept special characters with a few stated exceptions). The default is 13 characters consisting of printable ascii characters. 13 random ascii characters meets the NIST recommendation for 80 bits of entropy for a strong password (learn more about password strength on Wikipedia).

The for loop here counts the number of arguments passed to the script (stored in $#) and loops over them. The ${!var} notation treats $var as the name of another variable. In other words, if $i is 1, then ${!i} is the same as $1 which is the first argument that was passed to the script. The double parentheses are used to evaluate a mathematical expression. After evaluating the command line arguments, we have essentially the same pipeline we used before. The -d option for tr deletes characters from the input, while -c means to delete everything but the specified characters (the “complement” of the specified character set). So the first tr command removes all of the characters except for printable ascii (or alphanumeric, if specified), the second removes additional characters specified with --forbidden.

Now, on to our password practicing tools. First, we need a way to set the password:

#!/bin/bash
name=${1:-default}
read pw
echo -ne "\033[1A\033[0K"
echo -n $pw | sha512sum | tr -d ' -' > ~/.${name}pwhash

The read command takes input from the user and stores it in a variable named pw. By default, read prints what you are typing to the terminal. We allow it to do so here, so that you can make certain you are typing the password correctly the first time. However, as soon as we have finished typing and hit “enter”, we clear that line so the password is no longer visible. The -n option tells echo not to automatically output a newline at the end, and the -e tells it to interpret escape sequences. The sequence \033[1A moves the cursor up one line, and \033[0K deletes the current line. Rather than storing the password itself, we store a hash of the password for a bit of extra security (hopefully, of course, the machine we are doing this on is already secure, but this is a simple precaution to take). The sha512sum prints a couple of spaces and a hyphen at the end; the tr -d ' -' removes these. This script optionally takes one argument, a name so that you set and practice multiple passwords. The notation ${1:-default} is equivalent to $1 if it is set, otherwise it defaults to default.

Now, we need a way to practice typing the password we set:

#!/bin/bash
name=${1:-default}
read -s pw
userhash=$(echo -n $pw | sha512sum | tr -d ' -')
storedhash=$(cat ~/.${name}pwhash)
if [[ $userhash == $storedhash ]]; then
    echo "Correct"
else
    echo -e "Wrong\a"
fi

This time we used the -s option so that read does not print what you are typing to the terminal. Similar to the first script, this one optionally allows you to specify a name and then compares the hash of the password you type to the one that was previously stored. If they match, it informs you that you have typed the password correctly; if not, it let’s you know it was wrong. The \a is the bell character; it may give an audible alert, or in some cases a visual alert or nothing at all, but it is a nice touch to get your attention when the password is typed incorrectly.

Putting them all into functions is quite simple:

function genpw() {
    chars="[:graph:]"
    length="13"
    forbidden=""
    for i in $(seq 1 $#); do
        if [[ "${!i}" == "--alnum" ]]; then
            chars="[:alnum:]"
        elif [[ "${!i}" == "--length" ]]; then
            ((n=$i+1))
            length="${!n}"
        elif [[ "${!i}" == "--forbidden" ]]; then
            ((n=$i+1))
            forbidden="${!n}"
        fi
    done
    cat /dev/urandom | tr -cd "$chars" | tr -d "$forbidden" | head -c "$length"
    echo
}
function setpw() {
    name=${1:-default}
    read pw
    echo -ne "\033[1A\033[0K"
    echo -n $pw | sha512sum > ~/.${name}pwhash
}
function ppw() {
    name=${1:-default}
    read -s pw
    userhash=$(echo -n $pw | sha512sum)
    storedhash=$(cat ~/.${name}pwhash)
    if [[ $userhash == $storedhash ]]; then
        echo "Correct"
    else
        echo -e "Wrong\a"
    fi
}
function unsetpw() {
    name={1:-default}
    shred -uxn1 ~/.${name}pwhash
}

I added an extra one here to unset the password by removing the hash from your system, although this one is fairly trivial. In addition to learning one handy way to generate and learn strong random passwords, hopefully this little exercise has also given us a look at some handy Unix tools and Bash scripting features. For comparison, I’ve also written a Python version of these scripts.

Installing your CUPS shared printer on Windows

This is one of those things that really isn’t that complicated, but it doesn’t work as intuitively as it should and I find myself looking it up every time I have to do it again, so I’m going to go ahead and document the process here in a clear and easy to follow fashion. I’m not showing how to set up the printer with CUPS on Linux, I’m assuming that is already done and just showing how to add that printer on Windows (Windows 7 is shown here).

First, you need to know the name of your printer. If you have forgotten, you can easily find this by using your web browser to pull up localhost:631, this is an administrative interface for your CUPS server. Click the Printers tab and you will see your printer’s name under Queue Name in the first column.

Now that you know the name (and the IP address or domain name for the computer), we are ready to add the printer in Windows. Pull up Devices and Printers from the Start menu. Click the Add Printer button to pull up the Add Printer dialog. Of course, we will be selecting Add network, wireless, or Bluetooth printer. It won’t find the printer, but don’t worry, just select The printer I want isn't listed. Now use Select a shared printer by name. Ignore the examples, the correct format to enter is:

http://ipaddress:631/printers/Printer_Name

or:

http://domainname:631/printers/Printer_Name

Now, you will have to select the manufacturer and model (or series) of your printer. Now you’ve installed the printer and you will presented with an option to print a test page to make sure it works.

As I said, it was pretty easy, just not exactly intuitive. And now you should be able to print from Windows to the printer attached to your Linux machine (or other Unix system, including Mac).

Fun Unicode Characters for Facebook (and Generally Anywhere).

 Here is just a sample of fun unicode characters for Facebook. You can use these pretty much anywhere (including Google+), but I’ve specifically selected a few that don’t have emoticon equivalents supported by Facebook (at least that I’m aware of), don’t get converted to graphic emoticon representations by Facebook, and do display properly on most systems.

There are several different ways you can type the characters, depending on your system. You can use the hex codes on Windows and Linux (at least GTK apps). Using hexadecimal input for unicode characters on Windows can be a little tricky, though. Older alt codes can be used for some, but not all of the characters. Although the characters you can type with alt codes is limited, they have two advantages: they don’t require any special configuration and since they are decimal numbers they don’t contain any letters that are likely to cause conflicts with program shortcuts, so I’ve included them in the chart where applicable. To use the alt codes, you simply hold the Alt key while typing the number (you don’t use the + key like you do for the hex codes). Please note the leading 0 is important, alt code 145 produces a different character (æ) from 0145.

The easiest and most convenient input method is the compose key on Linux. It doesn’t cover all unicode characters, but it covers more useful characters than alt codes and they tend to be much easier to type and remember because they tend to be mnemonic (while not on the list because Facebook provides emoticon equivalents, the compose sequence for a smiley is :), a frowney is :(, and a heart is <3). There is a caveat, however, for the compose key on GTK apps. GTK overrides the configurable behavior of the compose key, unless you specifically configure to it use the underlying configuration from X. In order to do this, you will want to add a line to your ~/.Xsession like this:

export GTK_IM_MODULE="xim"

Then log out and back in. You can also type the above line directly into a terminal and then launch the GTK app from that terminal if you want to just quickly try it. The highlighted compose key sequences below may not work in GTK apps—including Firefox and OpenOffice—without this extra configuration, however.

Of course, if you have trouble typing any of the characters, you can also copy and paste them from here. That isn’t exactly the point here, but it will work.

Description	Compose	Alt	Hex		Character
left single quote	<'	0145	2018	‘
right single quote (apostrophe)	>'	0146	2019	’
left double quote	<"	0147	201C	“
right double quote	>"	0148	201D	”
en dash	--.	0150	2013	–
em dash	---	0151	2014	—
hedera			2766	❦
side-ways hedera			2767	❧
degree symbol	oo	248	00B0	°
copyright symbol	oc	0169	00A9	©
registered trademark symbol	or	0174	00AE	®
trademark symbol	tm	0153	2122	™
bullet point	.=	7	2022	•
cross			271D	✝
outlined cross			271E	✞
check mark			2714	✔
x mark			2718	✘
snowman			2603	☃
infinity symbol	88	236	221E	∞
radiation symbol			2622	☢
skull and crossbones			2620	☠
eighth note	#e	13	266A	♪
beamed sixteenth notes	#S		266C	♬
musical sharp symbol	##		266F	♯
musical flat symbol	#b		266D	♭

Bonus: Tux

It is often asked if there is a unicode character for Tux, the penguin who serves as the Linux mascot. There is no character for Tux in the unicode standard, however there is a private area for fonts to include non-standard characters. Linux Libertine, an excellent free font, includes Tux at code point e000. You can use this character if you can specify the font, such as on a web page (better use @font-face or the browser will fallback to something else if it isn’t installed) or in a document where you select the font. If you use this on Facebook, it will only display properly for those who have the font installed and have a browser that automatically falls back to a font that has the character available if it isn't in the currently selected font.

Here is what it looks like, if you have the font installed: 

And here is what it looks like a bit larger: 

Note: I changed the font color for Tux to black and the background to white, because it does look rather odd in reverse. I also made sure to remove the text-shadow effect.

Typing Arbitrary Unicode Characters in Linux

Ok, this one is a little strange. The X server that provides a graphical environment on Linux does not offer a feature to input arbitrary unicode characters by code point. It does provide a powerful feature to enter some unicode characters by mnemonic sequences with the compose key. GTK, a toolkit used by Firefox, OpenOffice, and Gnome applications, does provide a way to input arbitrary unicode characters. If you are using these and have not reconfigured it, you can input arbitrary unicode character by holding down the Ctrl and Shift keys while you type the letter u followed by the hex code for the unicode character you want. To be clear, the only keys you hold are Ctrl and Shift, the rest are typed in a sequence and then you release Ctrl and Shift.

Now, here is where it gets interesting. GTK overrides the default input for the X server and provides it’s own set of compose key sequences. This is nice for consistency, but the default configuration for the X server may provide compose sequences for characters that are not supported by GTK. Additionally, the X compose feature is configurable: you can add new sequences for characters you want to use. This is very convenient. It is possible to use the default X input method in GTK apps, but you lose the ability to input arbitrary unicode characters. If you prefer to use the default X input method, you can accomplish this by adding a line to your ~/.Xsession like this:

export GTK_IM_MODULE="xim"

You can also change this for a single app by entering the same thing on the command line in a terminal app and then launching the desired application from the same terminal. Now you will be able to use compose sequences such as Compose + #e to get a musical eighth note like this: ♪. You can also add your own compose key sequences by editing /usr/share/X11/locale/en_US.UTF-8/Compose. Personally, I think having a powerful, configurable compose key is more valuable than arbitrary unicode input, but I’m still looking for a way to have both.

Typing Arbitrary Unicode Characters in Windows

There is actually a way to directly type in unicode characters by code point in Windows, but it may take some configuration to get it to work properly. To use this method, you have to memorize the hex code for the characters you want to use, but it will allow you to type in obscure characters like the ornamental typography symbol called a hedera ❦ or a musical note ♪.

This method is supported on Windows 2000 and up; I'm demonstrating it on Windows 7. You will need to create a registry key to enable this feature and log out and back in (or reboot on older versions). Caution! Editing the registry can be dangerous, follow the directions and don't mess with anything else.

First, open regedit. Now, in the tree in the left panel, expand the path HKEY_Current_User/Control Panel/Input Method. In the open space on the right, right-click and select New > String Value. Type in the name EnableHexNumpad. Now, right-click the value and edit it, type 1 in the Value data field and click Ok. Now log out and in (or reboot).

Now, all you have to do is hold the Alt button, press + on the numpad, and then enter the hexadecimal code for the unicode character you want. Unfortunately, shortcuts in some applications may interfere with this. For instance, in Firefox (English language versions) there is a bug, preventing you from entering codes containing the letters B, E, and F since these trigger the Bookmarks, Edit, and File menus. Linux doesn't seem to have this problem since it typically uses Ctrl + Shift instead of Alt (and starts the sequence with a u, rather than +). Nevertheless, this can be a handy technique, so give it a try. Unicode contains a huge number of characters, you can find handy codes with a simple web search. (A few cools ones are in my newer post.)

Easily Type Accented Characters, Curly Quotes, and Other Symbols in Linux with the Compose Key.

Setup

While there are a variety of ways to input special characters not found on your keyboard, the compose key provides one of the easiest. The key sequences are fairly mnemonic, making them easier to remember, and they are often shorter than equivalent hex codes and easier to use than copy and paste. But in order to use these, you may first have to make sure your system is set up to use a compose key.

KDE

In KDE, pull up your System Settings and select Input Devices. Under the Keyboard section, select the Advanced tab and check the Configure keyboard options box. Click the arrow to expand the Compose key position section, and then check the box for the key you wish to use. I’m using the right win key, which on my keyboard is actually the Tux key.

Gnome

In Gnome, pull up your System Settings and select Keyboard. Under the Typing tab, select Layout Settings. Then, under the Layouts tab select the Options... button. Under Compose key position, select the key you want to use as the compose key. Again, I’m using the right win key, which is actually the Tux key on my keyboard.

Usage

Now that your desktop environment is configured to use the compose key, all you have to do is hold down the compose key while typing the key sequence for the character you want to type. Note that some of these characters—such as the caret, tilde, and double quotes— require the shift key. You can press and release the shift key as needed while holding the compose key.

Accents and Diacritics

You don't really have to memorize the combinations for each accented character, there is generally a character for each accent, so you simply type [compose key] + [accent character] + [letter]. For instance, you use a single quote for an acute accent, so on my system Tux + ' + e gives me the accented characters for typing “résumé”.

Accent	Character	Example
acute	'	á
grave	`	à
circumflex	^	â
breve	b	ă
diaeresis or umlaut	"	ä
tilde	~	ñ
cedilla	,	ç

Punctuation

Name	Key Sequence	Characters
curly double quotes	<" and >"	“ and ”
curly single quotes	<' and >'	‘ and ’
en dash	--.	–
em dash	---	—

The right single quote character is also the preferred character for an apostrophe. The en dash is used to indicate a range of values, and for attributive compounds; the em dash is used to indicate a break in thought or interruption of speech, and to set off the attribution of a quote. Wikipedia has more information on dashes, if you are uncertain which to use.

Currency

Name	Key Sequence	Characters
cent	|c	¢
pound	-L	£
euro	=c	€

Other Symbols

Name	Key Sequence	Characters
degree	oo	°
copyright	oc	©
registered trademark	or	®
trademark	tm	™

Batch file processing from the command line (photo-editing, renaming, etc)

One of the convenient things about the command line is being able to easily perform batch operations on a large number of files, things that are systemically applied to each file in the same way, such as replacing all of the spaces in file names with underscores or hyphens, resizing images or converting from .png to .jpg, etc. Here I'm going to present some of the basic tools for performing these kind of operations and you can then put them together in various ways to perform any number of customized tasks. Note that I'm using Bash for the interactive shell (or interpreter) here, and the parameter expansions presented here, although useful, are not portable.

Looping

The most common idiom you will see for iterating over a set of files is using the glob to match a set of files:

for f in /path/*

The * matches any file in /path, so it becomes the list of files in that directory. The variable f becomes each file (or rather, the full path to each file) in turn, and we can use that in the code that follows. The command following the for should begin with do and the loop will continue until it reaches the done command and then repeat for each file. The glob can be used to match part of a file, so we could operate on only the .png files in /path like this:

for img in /path/*.png

In addition to the glob, you can also use other things like brace expansion. For instance, /path/*.{png,jpg,gif} will give you all of the files ending in .png, .jpg, or .gif. If you have a set of pictures with names like DSC_nnnn.JPG, but you only want to work on images 5 through 22, /path/DSC_{0005..0022}.JPG would let you do this. You can also use command substitution to loop over the output of a command by wrapping the command with $() or ``.^[1]

Parameter Expansion

Bash parameter expansion provides a really handy way of manipulating file names. Here we will see how to separately get the filename, path, and extension, and substitute one character for another.

Get the base name of a file:

base=${file##*/}

Get the path to a file:

path=${file%/*}

Strip the last extension from a filename:

new=${file%.*}

Strip all extensions from a filename:

new=${file%%.*}

These work by stripping a prefix (# and ##) or suffix (% and %%) from $file. In the first example, the * comes before the / because we are stripping a forward slash and everything that comes before it, while in the latter two examples the * comes after the . because we are stripping a dot and everything that comes after it. The double forms strip the longest match, while the single forms strip the shortest match. But with the latter two examples, beware files and directories with unexpected dots! For instance:

file="/home/user/my.pics/selfie.jpg"
echo ${file%%.*}

will print "/home/user/my",

file="/home/user/web2.0essay"
echo ${file%.*}

will print "/home/user/web2". Errors like these are easier to avoid when you are typing a quick command and know what kind of files you are dealing with. If you are writing a script that might later be used in different context, you must be extra careful that it doesn't break when file and directory names don't conform to your initial expectations.

Replacing spaces in a filename with hyphens:

new=${file// /-}

This form actually does pattern matching. You can also require it to match at the beginning or end by replacing the second forward slash with % or #, respectively. If you leave out the last part, whatever matches the pattern will be replaced by nothing, that is, it will be deleted. In this case, you can also omit the final forward slash. So you could remove only a three character file extension with:

new=${file/%.???}

Or you could remove a two digit prefix with:

new=${file/#[0-9][0-9]}

Resizing Images

For command line image processing, we will be using convert from the imagemagick package. The basic command for resizing an image is:

convert picture.jpg -resize 1232x816 smallerpicture.jpg

Note that this will not necessarily make the image exactly 1232x816, it will make it fit inside a box that is 1232x816; it will not squeeze, stretch, or crop the image to fit that exact size. If you want to distort the image to fit:

convert picture.jpg -resize 1232x816\! smallerpicture.jpg

Distortion, however, is often undesirable and it may be better to crop the image. Suppose you wanted to create a series of 64x64 thumbnails without distorting the images. You could crop them with:

convert picture.jpg -resize 64x64^ -gravity center -extent 64x64 thumbnail.jpg

The ^ means to make the image fill, rather than fit into, the 64x64 box. The extent crops the image to 64x64, and the -gravity center means to center the image when cropping so the thumbnail comes from the center of the original image.

Now suppose that you wanted to shrink a series of larger images, but you don't want to enlarge any that are smaller. You can do that like so:

convert picture.jpg -resize 64x64\> resized.jpg

You will notice that all of the above examples require a file name to be supplied for the output, so if we actually want to use them on a batch of files, we will need to combine them with a loop, like this:

mkdir /path/images/thumbnails
for img in /path/images/*.{jpg,gif,png}
do basename=${img##*/}
name=${basename%.*}
convert "$img" -resize 64x64^ -gravity center -extent 64x64 "/path/images/thumbnails/${name}-thumb.jpg"
done

For each image.jpg this will produce an image-thumb.jpg in /path/images/thumbnails. Note that this will create jpg thumbnails for gif and png images, too. You may also note that I didn't bother to write out a script for this, these commands can easily enough be typed in on the command line for a simple task, as explained in Protip #2. Of course, you could write a script if you will be doing the exact same operation frequently.

Convert Image Types

You will notice that our thumbnail example actually converted any gif or png images to jpg when generating thumbnails. Similarly, it is very simple to convert one image type to another, such as if you wanted to use a series of jpg images to make an animated gif or png (similar to this), you just leave out the resize part:

for img in /path/*.jpg
do convert "$img" "${img%.*}.png"
done

Renaming Files

We saw the basics for this when we introduced parameter expansion, but I'm going to go ahead and show a full example. Suppose that we have some files with spaces in their names and find this annoying when manipulating them from the command line, so we decide to replace the spaces with underscores. But some of the files have hyphens and the resulting "_-_" just doesn't look right, so you want to collapse the spaces surrounding a hyphen. Easy enough, we'll just use two steps:

for f in /path/*
do f1=${f// - /-}
new=${f1// /_}
mv "$f" "$new"
done

But what if the path contains spaces? That would be a problem here, but it's not too hard to get around:

for f in /path/*
do basename=${f##*/}
f1=${basename// - /-}
new=${f1// /_}
mv "$f" "${f%/*}$new"

We simply removed the path first, and added it back at the end.

And More!

You can do many more interesting things on the command line with Bash, and ImageMagick offers a huge selection of image editing features not mentioned here. If you want to learn more, you may want to check out the Bash Reference Manual and Examples of ImageMagick Usage. I also recommend GreyCat's Wiki, which has several resources on Bash, including an introductory guide, pitfalls, FAQ, and quick reference sheet.

Protip #2: Using loops from the commandline

Remember that the same things you can do in a shell script, you can also do directly from the command line. This includes looping constructs like for, while, and until. In a shell script you might write:

#!/bin/bash
for f in /home/user/Documents/*; do
    new=${f// /-}
    mv "$f" "$new"
done

to rename all of the files in your Documents folder, replacing spaces with hyphens (note that the parameter expansion, ${f// /-}, is not portable). But for something this simple, you don't need to write a script, you can just type this in on the command line, like this:

for f in /home/user/Documents/*
do new=${f// /-}
mv "$f" "$new"
done

or on one line:

for f in /home/user/Documents/*; do new=${f// /-}; mv "$f" "$new"; done

The indentation is irrelevant to the interpreter, we just use indentation in shell scripts for human readability. Those of you who have worked with a programming language with an interactive toplevel, such as Lisp or Python, may recognize how handy this is. Of course, many of you already know this and probably just went, "Duh!", but someone out there either didn't realize they could use loops from the command line, or just didn't really think of it because we usually see them in scripts. That person, upon reading this, was enlightened.¹

How to Encrypt your Email so the NSA can't read it (Part 2)

Trusting Keys

In part 1, we downloaded our friend's public key from a key server and verified it by comparing the fingerprint in person. That was enough to get us started and let us send encrypted emails, but it left some things to be desired. It is still somewhat cumbersome; it would be nice not to have to verify every key in person. For instance, it would be very helpful if we could trust all of the keys that our friends have verified, so that we could help each other out with the task of verifying keys. As long as it has been verified by someone I trust, I should be able to trust the key. It turns out there is a way to do exactly this.

Obviously, this requires a way to indicate that you have verified a key, and who you trust to verify keys for you. We not only need to indicate that a key was verified for our own use, we need to communicate this to our friends in such a way that they can be assured it was you, and not an imposter, who verified the key. We have already seen a way of doing this. Remember that signing an email allows your friends to verify that it came from you? Similarly, we can sign keys to indicate that we have verified them. Rather than sending an email, we upload the signed key to a key server. This way, when your friends retrieve a key from the key server, they will also get your signature and automatically know that you have verified that the key belongs to the correct person. Now, they won't have to meet with that person to verify the key, the work is already done!

So let's look at how to sign a key with Thunderbird and Enigmail. Again, we will open the OpenPGP Key Management dialog. Now, find the key you've just verified and right-click it, the click "Sign Key". This will pull up a new Sign Key dialog, which will ask how well you have verified this key. Hopefully you have verified it thoroughly. You will also see an option for a local signature. If you choose this option, it will be signed only for your purposes and the signature will never be uploaded to a key server for your friends to see. Again, signing a key may require your password, depending on your settings and when you last entered it, to use your secret key. Once you have signed the key, go to the menu on the OpenPGP Key Management dialog, click "Keyserver" and then "Upload Public Keys". This will upload the signed copy so that next time your friends check, they will see that you have verified and signed the key. You will also notice that once you have signed the key, emails that were signed by that key simply show "Good signature from..." rather than, "UNTRUSTED Good signature from..."

It should probably go without saying that since all of your friends are counting on you, you should only sign a key that you have really properly verified, but I'll say it anyway, just in case. If you are signing a key for someone you do not know well, you should check some form of ID, but remember you are also verifying the email address.

Now to make use of the keys our friends have signed, we need to also indicate which of our friends we trust to verify and sign a key. The setting for this is called Owner Trust. Again, from the OpenPGP Key Management dialog, right-click the key of the person you want to trust to verify and sign other keys, then click "Set Owner Trust". If you set someone as fully trusted you will trust any key they have signed, if you set them as marginally trusted you will only trust keys that have been signed by several other marginally trusted people. By default, GnuPG will only follow a chain of trusted keys for a maximum of five steps, and the number of marginally trusted signatures required for a key to be considered valid is three. These numbers are configurable, but this is not an option in the Enigmail interface and for our purposes I'll just assume the defaults are suitable for most people (the relevant options are max-cert-depth and marginals-needed if you are interested in changing the gpg configuration directly, though).

By default, Enigmail trusts all keys, so if you want to start managing trust manually, you will also probably want to change this setting. This time go to "OpenPGP" on the Thunderbird menu and click "Preferences". In the preferences dialog there is a button that says, "Display Expert Settings", click on this and you will get a series of new tabs to choose from. In this case, click on the "Sending" tab, and then uncheck the box that says,"Always trust people's keys". Once you have done this, Enigmail will no longer use a key for encryption unless it is trusted, because you or someone you trust has signed it.

Synchronizing and Backing Up Your Keys

With our original setup, we can only send or read encrypted email from the one computer we originally set up. If you repeat the process on a second computer, generating a new key, you will have two different keys and mail sent to one can't be read from the other device so this is clearly not a proper solution. Of course, many people will want to use more than one device to read their mail so we are going to see how to do that here, as well as how to backup your keys. Now to be clear, you do not want to put your keys on any machine you don't trust, you should limit this to devices you control and you should take steps to protect them.

The easiest way to synchronize your devices so you can use the same keys on both is to copy over GnuPG's configuration folder. On Linux, this is in your home folder /home/username/.gnupg, on Windows it is %APPDATA%\gnupg which is usually C:\Documents and settings\username\Application data\gnupg\. This folder contains your public and secret keys, as well as they keys of others you have collected and any custom configuration you have made, so once you've installed the necessary software you won't need to worry about generating keys or re-downloading your friends', it will all be there.

Similarly, if you make regular backups of your full system, you are already covered. If you only make backups of select files and folders, make sure you include your GnuPG configuration folder. However, if you use a cloud-based backup system, you may want to consider whether you want to trust it with your keys and consider alternatives. Keep in mind that if you loose your secret key, you will lose access to all email encrypted to you with that key, and if your secret key. On the other side, if you allow your secret key to fall into the wrong hands, that person can read any mail encrypted with that key and can also forge new messages appearing to be from you. Everyone should have a strategy for backing up valuable data (especially those things you can't replace, like family photos, etc) but your GnuPG key is particularly important to not only have backed up, but to keep the backups secure as well.

Summary

Hopefully, what we have covered here will make it easier to use encryption to keep your email secure and private. Of course, you won't have the convenience of accessing your email from any where you can get on the web, it wouldn't really be practical to keep it secure that way. But with these steps, you can access your email from multiple secure devices. There are also apps for using the same kind of OpenPGP encryption on your Android mobile devices, although I haven't covered that here (I've not used them personally). Please share your questions, comments, and suggestions! I'm glad to answer questions and appreciate any feedback.

How to encrypt your email, so the NSA can't read it.

Understanding Public Key Cryptography

When using encryption, it is helpful to understand some basic concepts. Don't worry, we won't get into the math, we are going to keep it really basic. But since encryption, like other digital technologies, is such an important part of modern life, this is really something that everyone should have some basic understanding of. I mean, this is the stuff that keeps your bank account and financial transactions safe online (I won't get into much of that because I'm focusing on encrypting email). You can skip this section if you want, but I highly recommend reading it.

When encrypting something you use some password, or key, to encode the information in such a way that it can't be deciphered or read without a key. It used to be that in order to use encryption, you had to arrange in advance for both parties to have the same key. It had to be exchanged in secret, because anyone who knew the key could also read the messages that where later encrypted with it. With this method, you need a separate key for each pair of people who want to communicate securely and they all have to be exchanged in secret! So this can quickly become unwieldy. This method is called symmetrical encryption, because the process works the same both ways. The same key that was used to encrypt the data can also be used to decrypt it.

Public key cryptography, however, is asymmetrical. The key that is used to encrypt a message cannot be used to decrypt it. Instead, there are two keys that are related: the two keys do the opposite of each other. So the keys come in pairs and something that has been encrypted with one key must be decrypted with the other key, and vice versa. Since the key used to encrypt the message can't be used to decrypt or read it, it can safely be shared in public. This is why it is called public key cryptography. Of course, this means that the other key of the pair must be kept secret, hence it is called the secret or private key. So, once you have created such a key pair, you put one away to be your secret key, and the other you can publish to all of your friends. Anyone can then use the public key to encrypt a message to you. Since the key you kept secret is the only way to reverse the process and decrypt the message, you will be the only one who can read these messages.

The only thing your friends need to do then is to verify that the key they are using actually belongs to you. Of course, if they use a key that really came from someone else claiming to be you, then that person can read the messages they intended to be secret. You can meet in person to verify that they have the right key, but it doesn't have to be in secret. The software will give you a fingerprint, a short sequence of characters, that identifies the key. You can compare the fingerprint of your key with the fingerprint of the key your friend has received, if the fingerprints are the same, so are the keys they came from.

Digital Signatures

Public key cryptography also gives us a way to verify who sent a message. Remember that anyone who has your public key (which really could be anyone), can encrypt a message to you. So how do you know who really wrote the message?

Let us suppose that you and a friend each have your own key pair, and that each of you has the other's public (but not private) keys. In order for your friend to send you a message, he encrypts it with your public key, so that only your private key can decrypt it. Now, what if he encrypts it instead with his secret key? Remember that the keys do opposite things, so if it was encrypted with his private key, it can be decrypted with his public key. At first this might seem useless. Since everyone has access to the public key, anyone can read the message. But once you do, you know something about it that you would not have known otherwise. That message could only have been written by someone who had access the secret key that goes with the public one you used to decrypt it. If you have verified that the public key belongs to your friend, and he has properly kept his secret key secret, you know the message is from him. This is called a digital signature.

Of course, you can use both processes on the same message, this way when your friend reads it he knows both that it came from you and that no one else could have read it. Or you can sign a message without encrypting it, so that anyone can read it, but they can verify who wrote it if they have ever verified your public key.

So, how do we go about creating a key pair and using it to encrypt (and sign) our email?

Installation and Configuration

While there are quite a few different applications that will allow you to encrypt your email, I'm going to explain how to use one particular set of software here. It is very likely that you are used to checking your email by logging into a website, but that doesn't work very well if you want to encrypt your email. Instead, we will be using Thunderbird, the email application from Mozilla, the makers of Firefox. It runs on Windows and Linux. (It also runs on Mac. I'm not giving specific directions for Mac here, but you can use a similar approach on a Mac.) Thunderbird doesn't come with the encryption features we want built in, but it has plugins just like Firefox does and we will use a plugin called Enigmail to add the encryption features we want. Enigmail doesn't do the encryption itself, it uses GnuPG, so you will need that as well. If you are running Linux, you probably already have GnuPG, on Windows you will need to install it as well.

Linux

If you are running Linux, you probably already have GnuPG installed and if you don't it should be available through your package management system. On Debian, you can install icedove (which is the Debian version of Thunderbird with a different name and logo). Ubuntu has a thunderbird pacakge. Or you can install the official Mozilla build. You may also be able to install Enigmail from your package management system to be available for all users, if you installed your distributions official package for Thunderbird or Icedove; however, I'm going to show later how to install the plugin from within Thunderbird for each user.

Windows

First we need to make sure that GnuPG is available. There is a handy Windows package called GPG4Win, there are several packages available but for what we are doing you only need to install the Vanilla version. If you install the full package it comes with a different mail client, Claws Mail, which takes the place of Thunderbird. Feel free to try it if you like, but my instructions will be covering Thunderbird. Run the installer and just click through the "Next", "Install", and "Finish" buttons.

Next, we need to download and install Thunderbird. This one is even a bit easier; it's just "Next", "Next", "Install", "Finish" and you're done.

Email Configuration

When you first run Thunderbird, it offers to set up a new email account for you. You can use this feature if you need a new address, but most people reading this will already have one. Thunderbird does a really good job of getting the correct settings for most major email providers, so usually all you need to provide is your name, email address, and password. If you have difficulty, you can contact your email provider to get the right settings to put in manually, or you can set up a new account to use with Thunderbird and Enigmail. In my examples, I'm using two of the more popular webmail services, Gmail and Hotmail, and Thunderbird determined the correct settings automatically.

Setting up a Hotmail account on Windows

 Setting up a Gmail account on Linux (KDE on Debian)

Installing Enigmail in Thunderbird

Now we need to install the Enigmail plugin in Thunderbird (if you didn't install it from your package management system on Linux as mentioned earlier). This is pretty simple and works the same whether you are running Thunderbird on Linux or Windows.

In the most recent versions of Thunderbird, there is a menu button on the right which looks like three thick horizontal lines. On this menu, click on "Add-ons" and it will open a new tab for the Add-ons Manager. You can use the search box in the upper right to search for enigmail. Click on "Install" and then you will need to restart Thunderbird once it is finished.

Creating and Sharing Keys

The next thing we need to do is create a key pair. Again, we will go to the menu button, this time under the "OpenPGP" menu we will click on "Key Management". This opens a new window for OpenPGP Key Management, from the menu across the top of that window click "Generate", and then "New Key Pair". You need to use a strong passphrase to protect your key pair. If you want to make your encryption stronger, under the "Advanced" tab change the key size from 2048 to 4096. (BTW, you DO want your key to expire.) Click "Generate key" and go do something else on the computer while you wait for it to finish (this can take a while). Once your key has been generated, it will offer to generate a revocation certificate. You should do this and then store it in a safe place. A revocation certificate is used to invalidate your key if it is ever compromised.

Once you have generated your own key pair, you will want to upload it to a key server to share with your friends. You will need to check "Display All Keys by Default" in order to see your own key, click to select the key you just generated, then go back to the menu, click "Keyserver", and then "Upload Public Keys".

In order to use a friend's key to encrypt messages to them, you must first download their key. In order to search the key server for their key, you need to go back to the menu on the OpenPGP Key Management dialog and click "Keyserver" (the "Search for:" box is only for searching through the keys you have already downloaded, not for new ones), and then "Search for Keys". Type in the email address (Key ID if your friend has given you the ID for his current key) you want to find a key for. Once you have found and selected the keys you want, click "OK" to download them.

In order to verify that you have the right keys for each other, you will want to compare the fingerprints. You can find the fingerprint for your key, or your friend's, by right-clicking the key and then clicking "Key Properties" from the pop-up menu. The key fingerprint is near the middle of the dialog. Copy it down and compare it with what your friend got for his key to make sure they match.

 

Once both users have verified that they have the right key for each other, we are ready to start sending encrypted email.

Sending Encrypted Email

In order to send an encrypted email, you begin composing a message just as you normally would. Enigmail provides several places to access the settings to sign and/or encrypt your mail. There is a menu button labeled "OpenPGP" with a padlock on it and a small arrow to the right. The arrow pulls up a dropdown menu which includes options to sign and encrypt a message, clicking on the button itself pulls up a dialog box with the same options. There is also a small pair of icons at the bottom right of the message composition window, a pencil and a key. This will be highlighted, or colored, once you have selected the corresponding option to sign or encrypt the message. This gives an easy way to see the status of the message before sending. You can also click on these icons directly to encrypt or sign your message. When you send the message, it may ask for your password to access your private key (depending on your preferences and how long it has been since you last entered your password).

Note that the subject line is not encrypted. So you want to make certain that you don't disclose any sensitive information in the subject line, either by keeping it vague (but potentially helping the reader classify their mail before decrypting it), or replacing it entirely with "Encrypted Message" or something.

Also, keep in mind that you can sign a message without encrypting it, or encrypt a message without signing it. Signing a message without encrypting it allows you to send a public message to a group while still allowing people to verify that the message is authentic and does come from you.

Reading Encrypted Email

When you open an encrypted email, it will appear only as a bunch of garbled nonsense until it is decrypted. Again, it may ask you for your password (depending on your settings, etc). Once we have entered the password, we can read the message.

If the message was signed, we can also see that information at the top in a colored banner. Here, it states that there is an "UNTRUSTED good signature". What does that mean, and why is it untrusted? It means the message is, indeed, signed with the public key we have for the sender; but although we verified the key, we haven't done anything to indicate that we have verified it. By default, Thunderbird allows you to use a key without have given any indication that we have verified the validity of the key. So what we are seeing here is actually just fine for now (in part 2, we will learn more about this and how to mark a key as verified or trusted).

Conclusion

At this point we have learned enough to set up and use encrypted email. I've kept it to the basics to make sure this is relatively easy to get into, but I'll be writing a part 2 to introduce some more features that will make things easier for you and your friends. With these simple instructions, you can install and set up the necessary software, create and exchange keys, and send and receive secure encrypted email.

So, how good is this encryption? Will it really prevent the NSA (and anyone else) from reading your email? According to Edward Snowden, "Encryption works. Properly implemented strong crypto systems are one of the few things that you can rely on. Unfortunately, endpoint security is so terrifically weak that NSA can frequently find ways around it." So what is that part about "endpoint security"? He is referring to the ends of the communication channel where the message is encrypted and decrypted, the two users' computers. If either of the systems is compromised, messages could be intercepted or forged. For instance, a virus on your computer could intercept all of your key strokes as you type them, sending the message to an attacker before it is ever encrypted. Or it could get your password when you type it, access your private key and use it to forge messages from you. It sounds rather scary, but it is worth considering that these sort of attacks are probably used primarily against priority targets, not generally as a tool of mass surveillance. So if you are simply wanting to thwart mass surveillance, and haven't done anything to actually draw the suspicions of powerful organizations like the NSA, these things may not be a major concern. Nevertheless, it would be wise to take steps to secure your system. Running an antivirus and preferably using a GNU/Linux operating system (the most popular version is Ubuntu, the one I'm using which is pictured here is Debian, from which Ubuntu is actually derived), would be a good start. But in general, yes, the techniques described here offer a high degree of protection for your digital communications.