Can technology can spam?

It seems to be impossible to implement a law against spam – unsolicited bulk email – without making a hash of it. At best, anti-spam laws are ineffective; at worst, they cause more problems than spam itself (see Can the law can spam?, by Sandy Starr). Can technology fare any better?

There is a technology arms race between spammers and anti-spammers, with each side constantly looking to outwit the other. This contest has recently been stepped up, with Microsoft in particular pursuing a restless set of strategies – it has set up an Anti-Spam Technology and Strategy Group; launched a Coordinated Spam Reduction Initiative and an Anti-Spam Technology Roadmap; developed the new anti-spam technology SmartScreen; and has created a set of Microsoft Network Spambuster websites (1).

But despite this flurry of initiatives, we are yet to see a definitive answer to the spam problem. An Anti-Spam Technical Alliance has been formed by Microsoft, America Online, Yahoo! and EarthLink, but these companies continue to proffer competing solutions. Meanwhile, the technology being deployed in the spam wars is causing collateral damage, in the form of ‘false positives’ – email that is incorrectly categorised as spam, and so never reaches its intended recipient. Only recently, for example, a spam filter implemented by the broadband provider Comcast inadvertently prevented its customers from communicating with anyone who happened to have a Russian email address (2).

According to one widely reported study, people are starting to give up on email as a result of spam, either using it less or abandoning it completely. Meanwhile, frustrated techies try to stamp out the problem themselves, resorting to acts of ‘cybervigilantism’ (3). But there is, in fact, reason to be optimistic.

A major breakthrough was the development of Bayesian filters, which aggregate statistics to determine the likelihood of email being spam, rather than concluding bluntly that it either is or isn’t. The success of these filters is evident in the desperate and often self-defeating lengths to which spammers now go to try to get around them. As the Guardian reports: ‘sending unsolicited mail is much harder now. To get around filters, you have to play dirty. You need to use those virus-infected machines; misspell your product; surround your message with enough textual chaff to get past the filters.’ (4)

To get around filters, some spammers have tried inserting spaces or invisible HTML tags between words in their emails, or intentionally mispelling words, so that suspect words will not trigger filters. Other spammers incorporate ‘hash busters’ into their emails – erudite and rare words, or large portions of out-of-copyright classic literature. Either way, the resulting email, even if it does manage to convince a machine that it is legit, will quite clearly be gibberish in the eyes of a human. The Register points out that ‘by talking gobbledygook, spammers have found the perfect way to eliminate themselves’ (5).

It is also encouraging that Google, the innovative company behind the world’s most popular search engine, proposes to tackle spam with its new email service, Gmail. Gmail, Google tells us, not only ‘turns annoying spam email messages into the equivalent of canned meat’, but is ‘built on the idea that you should never have to delete mail and you should always be able to find the message you want’ (6). In other words, Gmail promises to do for email what nuclear energy promised to do for electricity – make it too cheap to meter.

But it is symptomatic of the confused battle lines in the spam wars that Google has found itself tarred with the same brush as the spammers, with its critics seeking to use legislation to thwart plans for Gmail (7). Because Google proposes storing and scanning the emails of Gmail users, and using an automated system to target adverts at them, it is being accused of countenancing the same kind of indiscriminate marketing that is practiced by spammers, not to mention infringing upon people’s privacy. This, despite the fact that Google’s search technology is far more likely to land you with an advert for something you’re interested in than spammers are.

Yet Gmail’s emphasis upon unlimited storage space is appealing, because it tackles head-on the underlying cause of spam – the fact that most of the cost of processing an email is borne by the recipient, rather than the sender. As long as this remains true, there will always be an economic incentive to spam, because only a tiny proportion of those being spammed need to buy the generic viagra or lend money to the fake Nigerian dignitary in order for the spammer to turn a profit.

Gmail isn’t the only proposal to impose an economic disincentive upon spammers. Computer scientist Paul Graham, who helped to pioneer Bayesian filtering, has proposed a regime where ‘auto-retrieving’ spam filters follow all of the links in spam emails, driving the spammers’ bandwidth costs up (8). Others have proposed ‘hash cash’ schemes, where sending emails involves some sort of micropayment.

Unfortunately, a system that required a payment – no matter how small – for sending email would be difficult to implement, and would arguably be a technological step backward. The fact that email is basically free to send represents an advance on more primitive forms of communication, and to turn back the clock by contriving a system of payments would be a tacit admission of defeat.

Microsoft is proposing a distinctive form of hash cash payment, with its Penny Black project, which proposes that senders of email be required to make a payment in the form of computer processing power rather than money. We would give up a small amount of computer processing power for every email that we send, and unless we send bulk email we shouldn’t notice any difference. While this would be less unwieldy than a system of financial payments, it is still not ideal. The fact that Microsoft’s project is named after Britain’s first postage stamp, originally introduced in 1840, does not exactly suggest a forward-looking project worthy of the leader of the global software market (9).

There are other points of attack, as well as the economic. Increasingly, there is a focus on the fact that there is no comprehensive way of verifying the identity of the sender of an email. This is seen as a problem not just in the sense that email can be sent anonymously, but also in the sense that a sender’s email address and the server from which their email originates can be ‘spoofed’, or forged.

We are now seeing a raft of proposals to create new standards of authentication for sent messages, such as Microsoft’s Caller ID for Email, Yahoo!’s DomainKeys, and the Sender Policy Framework and the Trusted Email Open Standard – both of which are endorsed by a variety of organisations. Even more radically, we are seeing proposals to revise or replace the basic technical standards that underpin email. The standard that is seen as especially problematic is the simple mail transfer protocol (SMTP), which is key to the sending and receiving of all email. SMTP is thought to be so inadequate, that even one of the authors of the protocol that preceded it now recommends that ‘they just write a new protocol from the beginning’ (10).

In principle, there is nothing wrong with altering technical standards – even ones that are ubiquitous – if they are failing to meet our requirements. Indeed, to insist upon sticking with an inadequate standard, on the grounds that it would be too much bother to change it, smacks of luddism. The problem, however, when it comes to the internet, is that there is no initial consensus as to what our requirements are. Furthermore, there is no institution with the authority to make as fundamental a change as altering SMTP.

Because the foundations of the internet were largely built within US military and academic circles before the internet assumed global importance, the question of who governs the basic technology is now highly contentious. The organisations that oversee the technical administration of the internet, such as the Internet Corporation for Assigned Names and Numbers (ICANN), the Internet Engineering Task Force, the Internet Society, and the World Wide Web Consortium, are avowedly apolitical and open to participation by all. But while this utopian vision of global democratic input is admirable, it simply doesn’t square with the divisions of power that exist in the real world (11).

As these supposedly apolitical organisations have become more significant to policy, regulation and business, so they have become more open to influence from those who wield political and economic power – it’s just that this influence is exercised in an informal fashion. As ICANN’s European elected director Andy Müller-Maguhn noted ruefully about his organisation’s work developing internet technology, at a conference in 2003, ‘to attend and follow this process, you really need money’ (12).

So while we should not be afraid to make changes to the technology that underpins the internet, we should nonetheless be highly sensitive to who is driving these changes and who benefits from them. There is legitimate scope for changing standards such as SMTP in order to prevent spammers from spoofing email addresses. The danger though, in today’s climate, is that we go to the other extreme – and end up with technology that facilitates regulation, stifling all flexibility in the way that we send and receive email. Additionally, the possibilities for state surveillance that are opened up by tinkering with basic internet standards, make Google’s proposals for targeted email promotion look positively benign by comparison.

So while technological solutions to spam can be effective – unlike the legal solutions – each specific technology must nonetheless be assessed on its own merits, and its consequences carefully thought through. Unfortunately, rather than considering specific solutions, opponents of spam are currently adopting a blunderbuss approach, latching on to every anti-spam technology going in the vain hope that one of them might do the job. John Levine, co-chair of the Anti-Spam Research Group, argues that ‘dealing with spam is like curing cancer…cancer isn’t one disease; it’s 100 diseases, and you will need to come up with a 100 cures for it’ (13).

Even leading technology companies are running at spam with all guns blazing. While the companies comprising the Anti-Spam Technical Alliance pursue a relentless series of high-profile court cases against alleged spammers – recently leading Microsoft to issue an embarrassing apology to a telecoms engineer from Merseyside whom it had falsely accused of spamming – they also push just about every conceivable anti-spam technology going (14).

This is disappointing, when these same companies have the nous to develop sensible anti-spam solutions and to recognise problematic solutions. Microsoft chairman Bill Gates has argued correctly that ‘although a lot of spam is pure junk, not all of it is clearly distinguishable based solely on broad, global criteria. Deciding precisely where to draw the line must ultimately be up to the individual’ (15). But rather than take responsibility for defending this view, and building a specific technology around it, Gates is hedging his bets. His company is attacking spam with every method and from every direction, in order to be seen to be doing something.

Spam has been made into a moral issue, where those accused of sending it or apologising for it are automatically vilified, and those who oppose it are thought to occupy the moral high ground. This has left us without the perspective necessary to deal with spam effectively. The sooner we recognise that spam is nothing more than a thorny practical problem, the sooner we will develop technology that can solve that problem.

Sandy Starr has consulted and written on internet regulation for the Organisation for Security and Cooperation in Europe, and for the European Commission research project RightsWatch. He is a contributor to Spreading the Word on the Internet: Sixteen Answers to Four Questions, Organisation for Security and Cooperation in Europe, 2003 (download this book (.pdf 576 KB)); From Quill to Cursor: Freedom of the Media in the Digital Era, Organisation for Security and Cooperation in Europe, 2003 (download this book (.pdf 399 KB)); and The Internet: Brave New World?, Hodder Murray, 2002 (buy this book from Amazon (UK) or Amazon (USA)).

(1) Microsoft Anti-Spam Virtual Pressroom section of the Microsoft Corporation website; The Coordinated Spam Reduction Initiative: A technology and policy proposal (.pdf 943 KB), Microsoft Corporation, 13 February 2004; Microsoft’s anti-spam technology roadmap, Microsoft Corporation, 24 February 2004; Remarks by Bill Gates, COMDEX Las Vegas, 16 November 2003; Dealing with junk email section of the Microsoft Network website

(2) Technology solution to slicing spam lags, Stefanie Olsen, CNET News.com, 22 March 2004; Comcast goofs in Russian spam blockade, Paul Festa, CNET News.com, 2 March 2004

(3) See Spam: how it is hurting email and degrading life on the internet (.pdf 359 KB), Deborah Fallows, Pew Internet and American Life Project, 22 October 2003; On the Web, vengeance is mine (and mine), John Schwartz, New York Times, 28 March 2004

(4) Incredible bulk, Danny O’Brien, Guardian, 8 April 2004

(5) Spammers struggle with words, Jan Libbenga, Register, 15 October 2003

(6) Google gets the message, launches Gmail, Google, 1 April 2004; Gmail, on the Google website

(7) See UK lobby says Google mail may violate privacy laws, Lucas van Grinsven, Reuters, 5 April 2004; Google’s Gmail could be blocked, BBC News, 13 April 2004

(8) See Filters that fight back, Paul Graham, August 2003

(9) See the Penny Black Project, on the Microsoft website

(10) See Caller ID for E-Mail technical specification, Microsoft, 24 February 2004; Sendmail and Yahoo! mail collaborate to develop and deploy DomainKeys, Yahoo!, 24 February 2004; the Sender Policy Framework website; Trusted Email Open Standard, on the ePrivacy Group website; Simple Mail Transfer Protocol, Jonathan B Postel, Request for Comment 821, August 1982, on the Internet Engineering Task Force website; End of the road for SMTP?, Paul Festa, CNET News.com, 1 August 2003

(11) See the Internet Corporation for Assigned Names and Numbers, Internet Engineering Task Force, Internet Society, and World Wide Web Consortium websites; Developing IT, by James Woudhuysen

(12) ‘Responsible’ regulation, by Sandy Starr

(13) Finding a way to fry spam, Marguerite Reardon, CNET News.com, 24 February 2004

(14) Microsoft says sorry to ‘spammer’, BBC News, 8 August 2003

(15) Toward a spam-free future, Bill Gates, Microsoft, 24 June 2003