Microsoft

Email Spam Coordination Across Different Services?

Send to Kindle

Surprise! It’s been 1,743 days since my last post. This isn’t a music post, so even if you are momentarily pleased to see a new one pop up in your feed, you’ll likely be disappointed by this “techie” one…

Disclaimer: this is 100% speculation on my part, I have done zero research to see if my theory is correct. I’m sharing it mostly to remember these thoughts as they occur, and perhaps in the distant hope that someone will either confirm or disprove the hypothesis with real proof.

Hypothesis

There is high level coordination among disparate email providers to determine bulk senders, in a completely unintuitive but ostensibly extremely clever manner.

I put the above hypothesis first, so that you can stop reading right now if you have no interest in this topic, because as is my usual style, this is likely to get long, quickly…

Background

I have operated my own email server for over 20 years. My CTO friends think I’m insane (perhaps for other reasons as well, but definitely for running my own email server). For clarity, it’s Postfix, not an email server that I wrote, just one that I operate on a dedicated server (where this WordPress blog is hosted as well).

I’ve been through the wars with email, sometimes caused by my misconfiguration or misunderstanding, but sometimes entirely out of my control (e.g., when my dedicated server was transitioned to a new data center and the static IP changed, and it had previously been on many RBL blacklists!).

Over time, I’ve tamed the configuration into a very stable setup. That has included complying with SPF, DKIM, DOMAIN-KEYS, DMARC, etc. Basically, anything that Google or Microsoft claim will help them validate email from me as a sender, and not mark it as spam or worse, just bounce it back to me.

The Problem

While my setup works flawlessly most of the time, on occasion, Lois or I will get a bounce back from someone (typically Google/Gmail, but sometimes Microsoft/Outlook/Hotmail). Once that bounce occurs, we’re often shut out from sending email to that service for a full day (rarely, longer!).

As you can imagine, it’s wildly frustrating to not be able to send an individual mail (this is not spam or bulk mail, but rather one to one emails to friends).

This is the error message we get in the bounce:

               The mail system

XXX@gmail.com: host gmail-smtp-in.l.google.com[74.125.20.26] said:
550 Action not taken (in reply to end of DATA command)

Wow, very helpful, “Action not taken”. Nothing indicating what we did wrong and why Google rejected the email. It feels like it should be a transient error, but it not only persists, it typically stops us from sending any further emails to anyone on that service for the rest of the day.

This has been going on for at least a couple of years. Just not often enough for me to pull out my (one remaining) hair trying to track it down.

What is going on?

Until this week, I literally had no idea (perhaps I should have diagnosed it earlier…). While I can’t say with certainty that my new understanding covers all aspects of this error, I can say with certainty one use case that definitely causes the error, and it might explain every single occurrence that we’ve had in this regard.

We (Lois more than I, but I do it too) share identical emails individually with a variety of friends. Specifically, if a group that we love puts out a new music video, Lois will send a link to that video to a group of people, but each will get their own separate email with the same email body and subject, so that they can reply just to us and not be BCC’ed in a large group.

I had never made the connection before that this somehow triggered the bounces, even though they were short emails, sent to people we’ve sent emails to 100’s of times, that almost always respond to those emails, and that we’re likely in the contacts list of the receivers. I couldn’t imagine that we were tripping any spam filter.

What do I think is going on?

I now believe (very firmly, with zero proof) that the body of the email (not including the subject, or the receivers) is being hashed. When another email with the identical hash (of the body) comes through (not sure if there is a time-limit or not), the service bounces it immediately with the above error message of “Action not taken”.

How did I come to this conclusion?

A week ago, I sent out invitations to a number of people to a house concert in March. Most of them went out fine, a very few bounced, so I didn’t have any suspicions over those bounces just yet.

This week, we discovered that one of the band members couldn’t make it, and we agreed with the head of the band that we would simply cancel the show. So, I sent an email to everyone that I had previously written to (except those that already said they couldn’t make it) telling them that the show was cancelled.

The cancellation emails were all bouncing, except for the first one!

I complained to Lois that the cancellations were bouncing, and that we would likely have to wait at least a day for the bounces to clear before I could send them again (still having no idea why they were bouncing).

Lois asked me why I thought the vast majority of the invitations didn’t bounce (to the same people, and those had links to the band in them, so if anything, they would have appeared to be more spammy).

It was a good question, to which I had no answer.

But, my brain often needs to sleep on problems before enlightening me, and indeed, the next morning I woke up with a theory to test.

While the bodies of the invitations were nearly identical, they each started with “Dear XXX,”. So, they couldn’t have hashed into the same exact body. On the other hand, each of the cancellations were identical in every way (copy/paste) without the lead Dear XXX. So, they indeed would hash into the same body.

To test the theory, I added back the “Dear XXX” to the cancellations, and sure enough, every single one went out without any bounces!

How is that Cross Provider Coordination?

Aha! It turns out that once Gmail (for example) bounced an email, so did Microsoft, Verizon/AOL/Yahoo, Apple (via @mac.com) and likely others (like Comcast).

To be clear, once Gmail bounced an email, if the next email went to hotmail.com, but was the very first such email to any Microsoft address, it was bounced immediately.

That implies (to me, proves) that it’s not just that they too hash incoming emails and bounce duplicates, but that they share the hash (somehow) among the competitive service providers, in order to more efficiently identify bulk senders quickly.

Please don’t ask me to conjecture on how they do that, I don’t have a clue.

Summary

We are both unbelievably relieved to understand what has been going on with these bounces (or at least to delude ourselves into thinking we understand it now).

Postscript

I doubt anyone who normally reads my blog will have an interest in this, but I needed to get it off my chest anyway.

I can only hope that an expert will weigh in and either confirm or provably deny my hypothesis.

Also, I have at least one more (completely picayune) email issue (rant?) that I will share if I get any feedback that this kind of stuff is interesting to anyone…

Microsoft Madness

Send to Kindle

Yesterday, I read the following article on PC World’s website. It mirrored my thoughts about Windows XP vs Windows Vista perfectly, including direct experience not just theory.

What I learned in that post (which I probably should have known earlier but didn’t) is that Microsoft intends to stop most sales of Windows XP as of June 30th, 2008. I’m not really sure what most means in this context, but either way, it’s boneheaded.

I just did a quick search, and apparently it means that they likely won’t be offering it to OEMs, so if you expect to get Windows pre-loaded on a new laptop after June 30th, you’ll have a choice of Vista or Vista (or Vista or Vista, given that there are four version of Vista available!).

John Heckman questions whether Microsoft won’t bow to pressure and push back the June 30th date.

The minute I read the article I knew I was going to post this. My first instinct was to title it Wake Up Microsoft. Then this morning, it came to me, this is the perfect season to aptly and correctly use the term Madness.

It’s clear that Vista is a bomb. You’d be hard pressed to find anyone without an ax to grind that would seriously defend the merits of Vista over XP. It’s not the first time Microsoft has bombed with an entire operating system. How many of you are still running Windows ME?

At least with Windows ME, it died a relatively quick and painless death. With Vista, for any number of reasons, Microsoft isn’t willing to give up. Given enough time (and money), they will likely make it decent, though it’s unlikely to ever be great (given it’s core), and it’s not even likely to get decent given that they are already working on it’s successor.

The madness isn’t in not killing Vista (I understand that the investment and marketing bets that they’ve made are too big to simply throw away). The madness is taking away the only viable choice that still puts money in Microsoft’s pocket!

Folks, there’s no doubt that XP is eating into Vista sales. That’s the only reason that Microsoft wants to stop selling XP, they want to remove the competitive choice and force new computers to be pre-loaded with Vista! Will it work? Of course, there are many people who wouldn’t consider Linux or Mac under any circumstance, and they will grudgingly (or ignorantly) accept a machine with Vista on it, if they have no other choice.

This doesn’t make it a smart strategy. The sane move would be to keep offering XP as a choice (while heavily promoting Vista). Then, whenever Vista truly rivals XP (don’t hold your breath), or Windows 7 (or whatever it will be called when it finally arrives) is available, stop selling XP.

In the best case scenario, Microsoft will sell exactly the same number of licenses in total (Vista only, instead of a mix of Vista and XP). They will get to declare a huge PR win for Vista (look how sales ramped so nicely!). They will not get any additional profit (since they will be maintaining XP for years to come anyway). They will create a slew of miserable users who will equate Microsoft with pain (or worse).

In the worst case scenario, they will push people toward alternative operating systems like Mac and Linux.

I haven’t done a scientific survey, but I honestly believe that nearly every technology professional (business people too, not just developers) that I know has switched to using a Mac as their primary computing platform (most on laptops, but I know a number of people who use iMacs as well!). When I say “nearly every” one, I believe the number is pretty close to 90%.

Examples include Zope Corporation. While 100% of our services to customers are delivered on Linux-based servers, there is only one developer in the company that hasn’t switched to a Mac. Even the SAs (System Administrators) all got Macs recently (though one of them decided after the fact that he’s more productive on his Linux laptop).

My friends (you know who you are) have been needling me for years to switch to the Mac. I have very long experience with the origins of Mac OS X (NeXT), so no one needs to convince me of the power and the beauty of the underlying software.

I haven’t switched for two reasons:

  1. There are programs (some cool, some necessary) that only run on Windows, or at the very least, run on Windows way earlier than they become available on Mac.
  2. The value proposition of generic hardware (laptops and desktops) is overwhelming vs the Mac stuff. The Mac stuff is gorgeous, and brilliantly designed. Ultimately, it’s not worth the money and locks you in. They also have enough quality problems to make me pause.

My non-technology professional friends (neighbors for example) still prefer Windows. There are a number of reasons but they are all valid (games for their kids, Windows is used at the office, I know Windows, I don’t want to have to buy new copies of software I already paid for, etc.).

In April 2004 I bought my current laptop. In fact, I just wrote about that in this post. I bought it without an operating system pre-loaded because I was committed to switching to Linux full time. The experiment lasted six weeks (not too bad), but once I started running Windows in Win4Lin, I realized that I wasn’t quite ready to cut the Windows cord full time, and I installed Windows XP Pro.

There were two reasons that I switched back:

  1. 95% of the day I was happier on Linux than on Windows. 5% of the day I required a program that was only available on Windows. That 5% started to bug me more each day until I switched back.
  2. Linux was great in 2004, but it wasn’t quite as good on cutting edge hardware as it is today, and I had some real problems on my (at the time) brand new beast. It’s possible that I would have toughed it out if Linux had worked perfectly on my laptop back then. I have no doubt it would work flawlessly today.

My one direct experience with Vista came when my next door neighbor bought a new Dell Laptop for her mother. There was no choice, Vista only. I am their tech support team and she asked me to customize the machine for her mother when it showed up. I was amazed at the hoops I had to jump through to install programs onto the machine. I couldn’t begin to imagine what someone who was less technical would have done (other than throw the machine out!).

In addition, the machine crashed on me at least 10 times in one day during the setup. Sheesh.

Since then, I have been asked for laptop recommendations at least five times. In all cases, the buyer wanted Windows. In all cases I have vehemently recommended XP, and (amazingly enough) it was now available again as an option. None of those users has had a single problem with their new laptops.

Where does that leave me? As I mentioned in my spring cleaning post, I will likely be buying two new laptops at some point (possibly this year, but definitely next year if not in 2008). I have thought about this (before knowing about the demise of XP) for much longer than I care to admit, and I decided that I was going to stick with Windows. Sorry Mac fanboys. 😉

If Vista is my only choice, I can guarantee you that I won’t be buying it. Best case scenario (for Microsoft) is that I will buy a retail CD of XP and load it myself. Much more likely scenario is that I will install Linux on the machine, and try really hard to avoid the few Windows-only programs that I’ve come to rely on. The least likely choice is that I will break down and buy Mac laptops, but it’s not impossible (the possibility is at least on my radar for the first time ever).

So, coming full circle to my original post title: Wake Up Microsoft!