Like most ssh servers on the internet, the Mojave group server gets lots of ssh dictionary attack attempts. These attacks are carried out by scripts that try to find dumb passwords on a machine by brute force. There are a few different tactics these scripts use for picking username/password pairs, including:

• User: root, Password: aardvark. (Repeat with various well-known usernames and every word in the English dictionary)
• User: abe, Password: abe. (Repeat with all common first/last names)

It’s not hard to think of simple methods to cut off these attacks without altering or otherwise restricting one’s ssh service (by, say, running sshd on a non-standard port):

• Throttle login attempts on a per-host basis. i.e. if a host tries and fails to log in N times in M seconds then start making it wait longer and longer between successive logins (or even blacklist the host entirely).
• Throttle login attempts on a per-account basis. i.e. if user bob gets N failed login attempts within M seconds then throttle or blacklist future login attempts on that account.
• Auto-blacklist hosts that attempt to log in to accounts that should never be logged in to. e.g. if somebody tries to ssh in as user “games” they’re toast.

Indeed, there are any number of programs out there that aim to provide exactly these capabilities.¹ What these techniques have in common is the need to measure the number of failed login attempts associated with a host and/or account over a period of time. So how do the existing programs do it? Sadly, a large number do it by scanning log files. In other words, a scanner will look for a certain pattern (typically using regular expressions) in the logging output for your system and assume that it indicates a failed login attempt. This is not a great idea.

What’s wrong with log file scanning, anyway? The first (and least important, IMHO) mark against it is performance. A typical system has dozens or hundreds of processes running at any time, each generating log output to some degree. Well-written programs generate a trickle, but it’s not unusual for a program to accidentally ship (or be misconfigured) with debug-level logging enabled and thus generate a torrent of debug output. By running a log file scanner you’ve now got a program that has to sift through every byte of that output looking for just one type of event! This is certainly not the most efficient way to find out about login attempts.

Another bad aspect of log file scanning is that it relies on the specific formatting of log messages from specific programs. If you replace OpenSSH with BogoSSH and BogoSSH uses a slightly different log output format then your scanner breaks. Similarly, if the OpenSSH people ever decide to change their logging format (not that this is likely) then your scanner breaks. If you install some new service that the scanner doesn’t know about it’s probably going to be totally unprotected unless it happens to format its log messages just like some other service the scanner understands.

But the worst thing about log file scanning is that sloppy implementations can actually open your system up to denial of service attacks! I recently came across this article by Daniel B. Cid that describes vulnerabilities in several popular log scanners that allow remote attackers to cause your machine to falsely blacklist any other machine of their choosing. Even worse, one scanner allowed attackers to cut a machine off from the network entirely! I refer you to the article for details, but the root causes of this problem are easy to describe:

• Regular expression matching is error-prone. People aren’t terribly good at understanding the full class of strings that a given R.E. will match. I’ve been doing complicated R.E. stuff for years now and I still have a terribly hard time writing bug-free regular expressions.
• Log files are chock full of strings that come from insecure sources, like, say, random users on your machine and script kiddies all across the internet who want to crack your machine. You do not want to be taking action automatically based on the contents of such an untrusted data source. The article doesn’t even begin to mention what an attacker with local access could do by writing and executing a program that mimics sshd’s log output.

So how should this problem be solved? By going straight to the horse’s mouth. If you want to know about authentication attempts, talk to the authentication system. Every modern *NIX system supports PAM, the Pluggable Authentication Module system. With a PAM module you can perform an action on any failed login attempt from any PAM-enabled service. This allows you to create a concise, trustworthy authentication log file with a consistent format, which can then be monitored by a very simple (and thus secure) scanner.

One project that takes the PAM approach is pam_abl (auto-blacklist). While it doesn’t use the specific strategy I described above (it builds a database rather than producing an authentication log file) it does get its info from the right source and is thus immune to these bogus log message attacks. Unfortunately pam_abl is a bit inflexible and limited in its capabilities. It doesn’t provide any way to blacklist a host if it tries to log in to a bogus account like ‘games’. It doesn’t provide a way to throttle rather than blacklist. And although it effectively ensures that an attacker will never have a chance to guess your account passwords it doesn’t provide a way of firewalling off a malicious host entirely. Even worse, the maintainer of pam_abl doesn’t seem to have time to work on it any longer. Hopefully somebody else will step up and turn this into the tool for stopping dictionary attacks dead in their tracks.

¹ Here are a handful:

• denyhosts
• fail2ban
• blockhosts
• blocsshd
• crackblock
• shellter
• sshdfilter
• sshit
• sshutout

Updated: Removed ssh-faker from the footnote since it effectively alters the sshd service.