Six Apart Developer Wiki

The Movable Type and Professional Network Wiki has been moved to wiki.movabletype.org.

Recent Changes

Spam Fighting in Movable Type

[Login] Change #29 by OpenID Identitybyrne.vox.com at 2007-06-25 19:52:07.

Many nefarious types on the Internet try to take advantage of the feedback features on blogs, such as comments and TrackBacks, to try to publish their own links on other people's sites. To help combat the practice, we include a few tools with Movable Type to help fight junk and spam on blogs, and the tips below are contributed by the community to help you get the most out of the available tools.

Recommendations fall under three categories:

  • Plugins - Extensions to Movable Type that help fight spam
  • Fighting Comment Spam Attacks - Server Recommendations for Administrators
  • Services - Web Services that can reduce spam
  • Troubleshooting and Resources

Plugins

These add-ons for Movable Type add significant spam-blocking capabilities.

SpamLookup

SpamLookup is a plugin that comes with every version of Movable Type. The design of SpamLookup was inspired by SpamAssassin, a popular email spam detection and filtering program. The concept behind the design is that each comment coming into the system passes through a serious a filters, each filter assigning a score to the comment. A filter can give a positive score if the comment looks legitimate, or a negative score if it looks spamish.

SpamLookup Extension

SpamLookup Extension is a variant of the SpamLookup plugin with bug fixes, the ability to filter on the content of specific fields, and white listing (on any field). It is backwards compatible with SpamLookup. An extensive set of filters is available from the documentation.

OpenID Comments

OpenID Comments lets anyone with one of the more than 100 million OpenID identity URLs sign in to your blog. Requiring (or preferring) authentication can help decrease the amount of junk comments submitted, and can also help you by providing consistent identities for people who make comments on your site.

Comment Email Filter

Comment Email Filter allows you to trust and ban commenters based on their email address. This is especially useful for regular commenters who don't use Typekey. Comment Email Filter goes one step further and allows you to block any comments that don't contain a trusted email address.

URLess

The URLess plugin provides a new comment policy option: to disallow anonymous (unauthenticated) commenters from leaving links within their comments. Since this is a common signature of blog spam, this is an effective way to eliminate most all comment spam you receive.

Anything that does get by this plugin will be filtered by MT's regular antispam filters.

Comment Challenge

Comment Challenge adds an extra field to comment forms which requires commenters to answer a simple question before their comment can be submitted. Comments where the question hasn't been answered can be junked or rejected outright. Helps to stop most automated comment spam.

BanPing2OldEntry

Rejects or junks trackback pings sent to entries older than a certain date.

Real Comment Throttle

Defines a maximum number of comments that a weblog should accept each hour or each day, to reduce the damage caused by spam floods.

SCode

Forces commenters to type in a series of numbers shown in an image (known as a CAPTCHA) before being able to submit a comment, to stop automated comment spam.

AutoBan

Technically AutoBan is not a filter but works as a complement to other filters. It uses the junk objects generated by those filters to maintain a list of banned IP addresses via the Apache webserver access file.

Form Action Hiding

Not a plugin, but this technique uses a default action on the comment form that is NOT the MT comment script, but rather a static HTML page. A little bit of Javascript executed on load sets the correct action, transparently to automated comment spam-bots. The template changes to do this are detailed here: Dealing with Comment Spam. Have been using it with great success on a highly trafficked site Ask Leo! for some time, and continue to get zero automated comment spam.

Fighting Comment Spam Attacks

Much of the following tips address the problem of detecting spam once it has entered into your system. All of them require system resources and the more you employ the more taxed your system may become when you are being hammered by a spam attack.

It is not uncommon for spam to come in waves as bots attempt to infiltrate your blog spamming all of your entries. When this happens, these onslaughts can often turn into Denial of Service attacks. So how do you combat spam at this level?

Step 1: Install mod_security

First and foremost, you need to keep spam from ever hitting your app server. You need to stop it at the networking or web server level if possible. The mod_security module "is an open source intrusion detection and prevention engine for web applications. It can also be called an web application firewall. It operates embedded into the web server, acting as a powerful umbrella, shielding applications from attacks."

This module will help detect and block a flurry of requests coming in from the same source - or in other words, it can help block spam bots.

Step 2: Use Fast CGI

If spam hits your app server, which it will eventually even under the best of circumstances, you need a way to keep the load down. With Fast CGI installed on your web server (and MT configured to route requests through it), MT is capable of loading itself into memory so that you no longer incur the penalty associated with initializing the platform each every time you receive a request. Now, the platform is persisted in memory.

That means that even when spam hits the application server, the app is able to respond quickly to free up the port for another request.

Services

OpenID

OpenID is a free, decentralized identity system which allows commenters to use their own URL as an identity to authenticate their comments. The OpenID Comments plugin provides this functionality for Movable Type.

TypeKey

The TypeKey service was initially released to help combat comment spam by forcing users through an "authentication firewall" prior to leaving a comment. It is not meant to be an identity authority, in fact TypeKey is pseudo-anonymous as best.

Those who have turned TypeKey on have seen a reduction of spam by virtually 100%.

Byrne's Experience

I felt for a long time that TypeKey was a barrier for people commenting - and that if I turned TypeKey off I would get more legitimate comments in addition to more spam. So I conducted a dead simple experiment: turn off TypeKey. The result was surprising.

  • SpamLookup caught 95%-97% of my spam. Turning off TypeKey produced a little more overhead than normal in managing spam.
  • That being said, NO SPAM was published to my blog. All spam was caught and held for moderation.
  • I did not see an increase in the number of comments I was getting. Meaning to me that TypeKey was NOT a barrier to commenting. My blog was simply not as popular as I thought it was. :'-(

Akismet

Akismet is a service created by the folks at Automattic that provides a centralized spam database and spam identification service. The service works by comments being transmitted to Akismet which compares the comment to known spam in the database.

Troubleshooting and Resources

I have words in my SpamLookup keyword filter, but spam keeps getting through - what's wrong?

  1. Look at one of your comment's junk log -- just go to the individual comment's editing page in MT and look at the bottom. It should display the spam's final feedback rating as well as a "junk log" which details exactly which tests were performed and their results. Most problems can be diagnosed in this manner. If there is no junk log, then perhaps one or more filters have been disabled? You should also check your SpamLookup settings and make sure the tests you're expecting to run are in fact enabled.
  2. Check to see if you have any blog-level SpamLookup settings in effect. If you do, there will be a "Reset" button along with the "Save Settings" and "Cancel" button. If the "Reset" button is there, then the blog settings are overriding the global settings (they aren't additive settings). You'll probably want to use the "Reset" to clear any blog-level keywords to use the system-wide settings.
  3. On your blog's "Feedback" settings, check the value for your "Junk Score Threshold". We strongly recommend that this setting is left at zero. It is intended to compensate for anti-spam plugins which are effective but "noisy" (score hits too high, either negatively or positively). To date, no plugins are known to do this so "0" is the correct threshold setting.

  4. Are you running under mod_perl? If so, add each plugin's "lib" directory to the Perl include path. So edit your apache configuration file to say:

    use lib 'PATHTOMT/plugins/spamlookup/lib';

Six Apart Guide to Comment Spam

Published a few years ago, this guide to Comment Spam is slightly out of date but still contains some very useful general information about the problem of comment spam. Also included is an overview of some suggested solutions.

  • http://www.sixapart.com/pronet/comment_spam
[Login] Change #29 by OpenID Identitybyrne.vox.com at 2007-06-25 19:52:07.
Six Apart
Makers of weblog software and services for individuals, organizations and businesses.