• User: root, Password: aardvark. (Repeat with various well-known usernames and every word in the English dictionary)
• User: abe, Password: abe. (Repeat with all common first/last names)

It’s not hard to think of simple methods to cut off these attacks without altering or otherwise restricting one’s ssh service (by, say, running sshd on a non-standard port):

• Throttle login attempts on a per-host basis. i.e. if a host tries and fails to log in N times in M seconds then start making it wait longer and longer between successive logins (or even blacklist the host entirely).
• Throttle login attempts on a per-account basis. i.e. if user bob gets N failed login attempts within M seconds then throttle or blacklist future login attempts on that account.
• Auto-blacklist hosts that attempt to log in to accounts that should never be logged in to. e.g. if somebody tries to ssh in as user “games” they’re toast.

Indeed, there are any number of programs out there that aim to provide exactly these capabilities.¹ What these techniques have in common is the need to measure the number of failed login attempts associated with a host and/or account over a period of time. So how do the existing programs do it? Sadly, a large number do it by scanning log files. In other words, a scanner will look for a certain pattern (typically using regular expressions) in the logging output for your system and assume that it indicates a failed login attempt. This is not a great idea.

What’s wrong with log file scanning, anyway? The first (and least important, IMHO) mark against it is performance. A typical system has dozens or hundreds of processes running at any time, each generating log output to some degree. Well-written programs generate a trickle, but it’s not unusual for a program to accidentally ship (or be misconfigured) with debug-level logging enabled and thus generate a torrent of debug output. By running a log file scanner you’ve now got a program that has to sift through every byte of that output looking for just one type of event! This is certainly not the most efficient way to find out about login attempts.

Another bad aspect of log file scanning is that it relies on the specific formatting of log messages from specific programs. If you replace OpenSSH with BogoSSH and BogoSSH uses a slightly different log output format then your scanner breaks. Similarly, if the OpenSSH people ever decide to change their logging format (not that this is likely) then your scanner breaks. If you install some new service that the scanner doesn’t know about it’s probably going to be totally unprotected unless it happens to format its log messages just like some other service the scanner understands.

But the worst thing about log file scanning is that sloppy implementations can actually open your system up to denial of service attacks! I recently came across this article by Daniel B. Cid that describes vulnerabilities in several popular log scanners that allow remote attackers to cause your machine to falsely blacklist any other machine of their choosing. Even worse, one scanner allowed attackers to cut a machine off from the network entirely! I refer you to the article for details, but the root causes of this problem are easy to describe:

• Regular expression matching is error-prone. People aren’t terribly good at understanding the full class of strings that a given R.E. will match. I’ve been doing complicated R.E. stuff for years now and I still have a terribly hard time writing bug-free regular expressions.
• Log files are chock full of strings that come from insecure sources, like, say, random users on your machine and script kiddies all across the internet who want to crack your machine. You do not want to be taking action automatically based on the contents of such an untrusted data source. The article doesn’t even begin to mention what an attacker with local access could do by writing and executing a program that mimics sshd’s log output.

So how should this problem be solved? By going straight to the horse’s mouth. If you want to know about authentication attempts, talk to the authentication system. Every modern *NIX system supports PAM, the Pluggable Authentication Module system. With a PAM module you can perform an action on any failed login attempt from any PAM-enabled service. This allows you to create a concise, trustworthy authentication log file with a consistent format, which can then be monitored by a very simple (and thus secure) scanner.

One project that takes the PAM approach is pam_abl (auto-blacklist). While it doesn’t use the specific strategy I described above (it builds a database rather than producing an authentication log file) it does get its info from the right source and is thus immune to these bogus log message attacks. Unfortunately pam_abl is a bit inflexible and limited in its capabilities. It doesn’t provide any way to blacklist a host if it tries to log in to a bogus account like ‘games’. It doesn’t provide a way to throttle rather than blacklist. And although it effectively ensures that an attacker will never have a chance to guess your account passwords it doesn’t provide a way of firewalling off a malicious host entirely. Even worse, the maintainer of pam_abl doesn’t seem to have time to work on it any longer. Hopefully somebody else will step up and turn this into the tool for stopping dictionary attacks dead in their tracks.

¹ Here are a handful:

• denyhosts
• fail2ban
• blockhosts
• blocsshd
• crackblock
• shellter
• sshdfilter
• sshit
• sshutout

Updated: Removed ssh-faker from the footnote since it effectively alters the sshd service.

So I’m supposed to upgrade my kernel without knowing anything about what’s changed?? Brilliant!

Let’s do a little Linux magic. We’re going to create a RAID 1 (mirrored) pair, transform it to a 2-disk RAID 5 array (you know, the kind that “requires” 3 or more disks), then grow it to a 3-disk RAID 5 array.

Just so nobody gets too confused, I’m using a fairly fresh install of Ubuntu 6.06 Dapper Drake, which includes EVMS 2.5.4.

First let’s make some 10MB “disks” to play with.

$ mkdir disks
$ cd disks
$ dd if=/dev/zero of=img0 bs=1M count=10
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.734714 seconds, 14.3 MB/s
$ # These produce similar output:
$ dd if=/dev/zero of=img1 bs=1M count=10    
$ dd if=/dev/zero of=img2 bs=1M count=10

And let’s mount them as loopback devices:

$ sudo su  # From here on out we'll need root privileges
# losetup /dev/loop0 img0
# losetup /dev/loop1 img1
# losetup /dev/loop2 img2

Now we’re ready to work in evms. Launch the NCurses interface to evms with the command evmsn. Our first step is to delete the “compatibility volumes” that evms automatically creates for us. Select Actions > Delete > Volume (by typing a d v) and select all of the /dev/evms/loopX volumes listed (using the space bar and arrow keys). Don’t select anything else that might be listed! Hit enter to go ahead with the deletion. The dialogs that you’ll see next are not important for our purposes — just chose the defaults until you get back to the evms interface and see the message “The Delete command(s) completed successfully.” at the bottom of the screen.

Next, we will build our first RAID, a level 1 mirrored pair. Select Actions > Create > Region, select the MD Raid 1 Region Manager, and hit enter. Now select loop0 and loop1 and hit enter. Leave the configuration options alone, hitting enter once more to create the Raid region. You’ll get a congratulatory message that informs you that the storage object md/md0 has been produced.

Now if this were a real storage array we were building we would probably put an LVM2 container on top of md0 and create some logical volumes, but in this tutorial we’re just going to create a volume from md0 directly.

Select Actions > Create > EVMS Volume, select md/md0 and give the volume the name RaidVol. Hit enter to go through with the creation. Now we should see /dev/evms/RaidVol in the list of logical volumes.

Let’s put a filesystem on it, so we can save some test data to the volume. Select Actions > File System > Make, select Ext2/3 File System Interface Module, and hit enter. Now choose RaidVol and hit enter twice to finish up with the default configuration options.

At this point your logical volumes list (hit zero to show it if it’s not there already) should look something like this:

 Name                          Size Modified Active R/O  Plug-in   Mountpoint
 -----------------------------------------------------------------------------------------
 /dev/evms/RaidVol           9.9 MB    X                 Ext2/3

EVMS is a cautious system and doesn’t actually change anything until you save your changes. At this point we need to save our progress, so select Actions > Save. Once you’ve saved your changes you can mount your shiny new volume! Select Actions > File Systems > Mount and mount RaidVol at /mnt/raidvol. We’re done with evmsn for the moment, so select Actions > Quit.

It’s easy for me to tell you this works, but if you want to have confidence in this procedure you’ll want to verify it for yourself. Copy some files to /mnt/raidvol — images, text files, whatever. Try to fill up the drive as completely as possible. (Hint: One way to fill it up is dd if=/dev/zero of=/mnt/raidvol/zeros) Once you’ve filled the disk, run md5sum on your files:

# ls -l /mnt/raidvol/zeros
-rw-r--r-- 1 root root 8971264 2006-06-30 00:03 /mnt/raidvol/zeros
# md5sum /mnt/raidvol/zeros
37e6cfefc792d550d54f0422c8521fea  /mnt/raidvol/zeros

You might also want to cat /proc/mdstat to see that the Raid is doing its thing:

$ cat /proc/mdstat
Personalities : [raid1] [raid5]
md0 : active raid1 loop1[1] loop0[0]
      10176 blocks [2/2] [UU]

unused devices: <none>

Now we’re ready to start making magic. (Hint: You might want to back up img0 and img1 now so you can skip many of the previous steps if you want to experiment in the future.) Launch evmsn once more. Unmount the volume with Actions > File System > Unmount.

We’re going to be doing some things behind the back of evms, so let’s convert the volume to a “compatibility volume” with Actions > Convert > EVMS Volume to Compatibility Volume. Save your changes and quit.

The next bit is somewhat voodoo. It’s based on the semi-obscure fact that the RAID 5 algorithm can indeed be applied to two disks, and when you do so the disks end up mirrored! This works because the parity of a single block is equal to the block itself, though the normal way of setting up RAID 5 arrays involves computing the parity of at least two blocks. As a consequence, the only difference between a 2-disk RAID 1 pair and 2-disk RAID 5 array is the metadata! By rewriting the RAID metadata, we can instantly convert our array to RAID 5.

We’re going to use mdadm to rewrite the RAID metadata. First we’ll need to stop the array:

# mdadm --stop /dev/evms/md/md0

Now we’re going to “create” a RAID 5 array using our existing loopback devices. In effect, this should just change the metadata and give us a functional array. mdadm is going to get suspicious, but it’ll let us proceed:

# mdadm --create /dev/md0 --level=5 -n 2 /dev/loop0 /dev/loop1
mdadm: /dev/loop0 appears to contain an ext2fs file system
    size=10172K  mtime=Fri Jun 30 00:28:04 2006
mdadm: /dev/loop0 appears to be part of a raid array:
    level=1 devices=2 ctime=Thu Jun 29 23:20:15 2006
mdadm: /dev/loop1 appears to contain an ext2fs file system
    size=10172K  mtime=Fri Jun 30 00:28:04 2006
mdadm: /dev/loop1 appears to be part of a raid array:
    level=1 devices=2 ctime=Thu Jun 29 23:20:15 2006
Continue creating array? y
mdadm: array /dev/md0 started.

Now the moment of truth! Let’s try mounting /dev/md0:

# mount /dev/md0 /mnt/raidvol
# ls /mnt/raidvol
lost+found  zeros

Hooray! Let’s make sure nothing got corrupted:

# ls -l /mnt/raidvol/zeros
-rw-r--r-- 1 root root 8971264 2006-06-30 00:03 /mnt/raidvol/zeros
# md5sum /mnt/raidvol/zeros
37e6cfefc792d550d54f0422c8521fea  /mnt/raidvol/zeros

And let’s convince ourselves that we really have a RAID 5 array:

# cat /proc/mdstat
Personalities : [raid1] [raid5]
md0 : active raid5 loop0[0] loop1[1]
      10176 blocks level 5, 64k chunk, algorithm 2 [2/2] [UU]

unused devices: <none>

Part II: Growing the Array

Hot-damn, it worked! Now let’s add another disk to the RAID 5 array. First we unmount the volume, then go back to evms:

# umount /mnt/raidvol/
# evmsn

Growing a RAID 5 array is actually quite easy in evms, but the documentation gives one little-to-no help understanding how it’s done. Trial-and-error led me to the following procedure:

Actions > Convert > Compatibility Volume to EVMS Volume
    use the name RaidVol and select /dev/evms/md/md0
Actions > Expand > Volume
    choose RaidVol
    choose md/md0 as the "expand point"
    choose loop2 as the object
Actions > Save

You should get a list of messages that the procedure has produced, and, with luck, none of them should be error messages! Choose cancel to dismiss the dialog, and notice that RaidVol is now 20MB! Mount the volume (Actions > File System > Mount), quit evmsn and let’s make sure everything is ok:

# ls /mnt/raidvol/
lost+found  zeros
# df -h /mnt/raidvol/
Filesystem            Size  Used Avail Use% Mounted on
/dev/evms/RaidVol      20M  9.7M  9.0M  52% /mnt/raidvol
# ls -l /mnt/raidvol/zeros
-rw-r--r-- 1 root root 8971264 2006-06-30 00:03 /mnt/raidvol/zeros
# md5sum /mnt/raidvol/zeros
37e6cfefc792d550d54f0422c8521fea  /mnt/raidvol/zeros

Nifty, eh? Now before you run down to your data center and try this procedure on your client’s drives, you should be aware that I’m pretty clueless about all this stuff and there may very well be extremely good reasons not to do this. If you do attempt it, make sure you have recent backups, and don’t say I didn’t warn you!

When I do that with the Backspace key I get:

KeyPress event, serial 24, synthetic NO, window 0x1000001,
    root 0x57, subw 0x0, time 41167852, (177,28), root:(1136,123),
    state 0x0, keycode 59 (keysym 0xff08, BackSpace), same_screen YES,
    XLookupString gives 1 bytes:  "

KeyRelease event, serial 24, synthetic NO, window 0x1000001,
    root 0x57, subw 0x0, time 41167935, (177,28), root:(1136,123),
    state 0x0, keycode 59 (keysym 0xff08, BackSpace), same_screen YES,
    XLookupString gives 1 bytes:  "

By reading this output we see that the first event is the key press and the second is the release. Both send the keycode 59 (not keycode 22 as on James'
system), which is correctly mapped to the keysym BackSpace.

Thankfully, this BS vs. Delete issue is one that rarely comes up these days.
All of the Unix systems I've used in the last 5 years or so have been configured
correctly on delivery.