email

Email Spam Coordination Across Different Services?

Send to Kindle

Surprise! It’s been 1,743 days since my last post. This isn’t a music post, so even if you are momentarily pleased to see a new one pop up in your feed, you’ll likely be disappointed by this “techie” one…

Disclaimer: this is 100% speculation on my part, I have done zero research to see if my theory is correct. I’m sharing it mostly to remember these thoughts as they occur, and perhaps in the distant hope that someone will either confirm or disprove the hypothesis with real proof.

Hypothesis

There is high level coordination among disparate email providers to determine bulk senders, in a completely unintuitive but ostensibly extremely clever manner.

I put the above hypothesis first, so that you can stop reading right now if you have no interest in this topic, because as is my usual style, this is likely to get long, quickly…

Background

I have operated my own email server for over 20 years. My CTO friends think I’m insane (perhaps for other reasons as well, but definitely for running my own email server). For clarity, it’s Postfix, not an email server that I wrote, just one that I operate on a dedicated server (where this WordPress blog is hosted as well).

I’ve been through the wars with email, sometimes caused by my misconfiguration or misunderstanding, but sometimes entirely out of my control (e.g., when my dedicated server was transitioned to a new data center and the static IP changed, and it had previously been on many RBL blacklists!).

Over time, I’ve tamed the configuration into a very stable setup. That has included complying with SPF, DKIM, DOMAIN-KEYS, DMARC, etc. Basically, anything that Google or Microsoft claim will help them validate email from me as a sender, and not mark it as spam or worse, just bounce it back to me.

The Problem

While my setup works flawlessly most of the time, on occasion, Lois or I will get a bounce back from someone (typically Google/Gmail, but sometimes Microsoft/Outlook/Hotmail). Once that bounce occurs, we’re often shut out from sending email to that service for a full day (rarely, longer!).

As you can imagine, it’s wildly frustrating to not be able to send an individual mail (this is not spam or bulk mail, but rather one to one emails to friends).

This is the error message we get in the bounce:

               The mail system

XXX@gmail.com: host gmail-smtp-in.l.google.com[74.125.20.26] said:
550 Action not taken (in reply to end of DATA command)

Wow, very helpful, “Action not taken”. Nothing indicating what we did wrong and why Google rejected the email. It feels like it should be a transient error, but it not only persists, it typically stops us from sending any further emails to anyone on that service for the rest of the day.

This has been going on for at least a couple of years. Just not often enough for me to pull out my (one remaining) hair trying to track it down.

What is going on?

Until this week, I literally had no idea (perhaps I should have diagnosed it earlier…). While I can’t say with certainty that my new understanding covers all aspects of this error, I can say with certainty one use case that definitely causes the error, and it might explain every single occurrence that we’ve had in this regard.

We (Lois more than I, but I do it too) share identical emails individually with a variety of friends. Specifically, if a group that we love puts out a new music video, Lois will send a link to that video to a group of people, but each will get their own separate email with the same email body and subject, so that they can reply just to us and not be BCC’ed in a large group.

I had never made the connection before that this somehow triggered the bounces, even though they were short emails, sent to people we’ve sent emails to 100’s of times, that almost always respond to those emails, and that we’re likely in the contacts list of the receivers. I couldn’t imagine that we were tripping any spam filter.

What do I think is going on?

I now believe (very firmly, with zero proof) that the body of the email (not including the subject, or the receivers) is being hashed. When another email with the identical hash (of the body) comes through (not sure if there is a time-limit or not), the service bounces it immediately with the above error message of “Action not taken”.

How did I come to this conclusion?

A week ago, I sent out invitations to a number of people to a house concert in March. Most of them went out fine, a very few bounced, so I didn’t have any suspicions over those bounces just yet.

This week, we discovered that one of the band members couldn’t make it, and we agreed with the head of the band that we would simply cancel the show. So, I sent an email to everyone that I had previously written to (except those that already said they couldn’t make it) telling them that the show was cancelled.

The cancellation emails were all bouncing, except for the first one!

I complained to Lois that the cancellations were bouncing, and that we would likely have to wait at least a day for the bounces to clear before I could send them again (still having no idea why they were bouncing).

Lois asked me why I thought the vast majority of the invitations didn’t bounce (to the same people, and those had links to the band in them, so if anything, they would have appeared to be more spammy).

It was a good question, to which I had no answer.

But, my brain often needs to sleep on problems before enlightening me, and indeed, the next morning I woke up with a theory to test.

While the bodies of the invitations were nearly identical, they each started with “Dear XXX,”. So, they couldn’t have hashed into the same exact body. On the other hand, each of the cancellations were identical in every way (copy/paste) without the lead Dear XXX. So, they indeed would hash into the same body.

To test the theory, I added back the “Dear XXX” to the cancellations, and sure enough, every single one went out without any bounces!

How is that Cross Provider Coordination?

Aha! It turns out that once Gmail (for example) bounced an email, so did Microsoft, Verizon/AOL/Yahoo, Apple (via @mac.com) and likely others (like Comcast).

To be clear, once Gmail bounced an email, if the next email went to hotmail.com, but was the very first such email to any Microsoft address, it was bounced immediately.

That implies (to me, proves) that it’s not just that they too hash incoming emails and bounce duplicates, but that they share the hash (somehow) among the competitive service providers, in order to more efficiently identify bulk senders quickly.

Please don’t ask me to conjecture on how they do that, I don’t have a clue.

Summary

We are both unbelievably relieved to understand what has been going on with these bounces (or at least to delude ourselves into thinking we understand it now).

Postscript

I doubt anyone who normally reads my blog will have an interest in this, but I needed to get it off my chest anyway.

I can only hope that an expert will weigh in and either confirm or provably deny my hypothesis.

Also, I have at least one more (completely picayune) email issue (rant?) that I will share if I get any feedback that this kind of stuff is interesting to anyone…

Droid Rocks

Send to Kindle

For those who don’t like to read, I’ll start with my bottom line, then give lots of details:

I love the Droid, it’s one of the best devices I’ve ever owned, and I’m a gadget freak!

A little background before I begin what will inevitably be a very long post. I’m a geek, but I’m often late in jumping on new trends. In fact, many times I get on board because a friend decides to buy me a gift and I become an instant fan.

Back in 2000, I received a Blackberry (BB) email device as a gift. That was before there were any BB cell phones. It was purely for email, and I thought I died and went to heaven (not that I would have bought it for myself).

A couple of years later another friend bought me a Treo 600p (a Palm-based smart phone). While it wasn’t as good an email device as the BB, it was an all-in-one mobile powerhouse, and I quickly got used to the convenience. Since then I’ve upgraded to a Treo 650p, 700p and then a 755p, which I was still using (reasonably happily) until I got a Droid as a gift on November 17th.

I was also reasonably happy with Sprint, so I seriously considered a Palm Pre before deciding it just wasn’t the right device for me (I’m very impressed with it, just not enough to have locked myself in at this time).

Finally, this is not an iPhone bash, or an iPhone vs Droid post. I’ve never been interested in an iPhone, and I’ve never regretted avoiding one. That said, the overwhelming majority of my friends carry iPhones and they love them. The only consistent complaints are about AT&T, not the iPhone.

The person who gave me the Droid is a major iPhone lover. In fact, he’s all things Apple, all the time. He knows how I feel, and I believe was tired of me yapping his ear off about whether the Droid was right for me, or the Pre, etc. I think he decided to shut me up by getting me the Droid.

He told me that I had 30 days to return it, no questions asked, but at least I’d know. Within two days I knew I wasn’t returning the phone. It still took me 11 days to decide to port my old number, but I’ve done that too, which means my old Treo 755p is now a paper weight.

This won’t be anything like a generic review. There are a ton of them out there, and all of them have done a better job than I could ever do. So, this will be a randomly ordered collection of my personal observations and usage patterns, collectively accounting for the reasons that I love this device.

Let’s start with one of the very few complaints I had about the Treo and build from there. While the Treo was very early in having a very usable browser, the web has moved on since it came out, and the Treo browser, while functional, is hardly a pleasure to use. There are many pages it can’t render (correctly or at all), and large pages render so slowly as to be practically useless even if they render correctly.

That meant that I only used the browser for necessary information, never for pleasure or to get things done while out and about. One of the most incredible features of the Droid is the browser. I’m sure that all Android phones share that great browser under the covers, but at the moment, the covers on the Droid are tops, with it’s large screen and super-hi resolution (480×854).

The browser can pretty much render any page I throw at it accurately, and pretty quickly, especially when using Wi-Fi (though Verizon’s 3G network has impressed me so far). One of my test pages is PeopleBrowsr, which is a browser-based Twitter client (and much more). It’s the most sophisticated JavaScript based website I’ve come across.

A friend tried real hard one night to convince me to get a Pre. He brought his over to spend the evening wowing me. Another friend came over with his iPhone the same evening. The Pre choked trying to bring up the PeopleBrowsr page (WebOS has been updated since, so it might work now). The iPhone did better, rendering most of the page, but it too choked eventually.

The page comes up on the Droid. When I log in to PeopleBrowsr, things get a little less usable, but the mere fact that I can log in was dramatically better than the other two phones (that’s the last phone-to-phone comparison!).

Why is having a first-class browsing experience so critical? The web has something for everyone, and more sites are becoming sophisticated end-points (full blown applications) every day, rather than just content repositories. It’s true that smart phones are often defined by their App ecosystem (and I’ll cover the ones I use on the Droid shortly), but, when your browser can utilize pretty much any web-based app correctly, having a local app available becomes less important.

Droid comes pre-loaded with a Facebook App. It’s looks nice, and works as expected. If you want to update your status, and see recent status updates from your friends, great. On the other hand, if you’re a Facebook power user, and like to see the real-time activity stream, you’re out of luck.

In fact, even if you visit the Facebook website with your mobile browser, you will be redirected (by default) to their mobile site. It seems that the Facebook App simply implements the limited functionality of the mobile site.

On the Droid, that’s not a problem. I fire up the browser and visit the Facebook Home Page URL directly. Currently, that bypasses their sniffing of the User Agent for the browser, and delivers the full Facebook experience to the Droid browser. I can tap on News Feed, or Live Feed, and see everything. It’s quite snappy. I never visit the app.

I am not saying that local apps aren’t useful or necessary, just that the web will grow and innovate more rapidly than apps will, and no approval process is necessary, and if the browser can deliver a first-rate experience, it becomes the most important app on your phone (IMHO).

Multi-tasking is much talked about on smart phones as well. It’s definitely cool to be able to listen to music or a podcast while doing something else. Some might shrug that off as just entertainment. There are other, more important benefits to multi-tasking.

One example is IM. If you want to be reachable to others initiating an IM with you (over various IM networks), you don’t want to stare at the IM screen for hours waiting for that message to come in. You want to fire up your IM client and move on to other things. People can and will send you messages whenever they want, just like they do when you are using your desktop IM client.

Add to that location-based (or location-aware) applications, that notice that you’re on the move in the background, and alert you to things you might be interested in right now. They need to be running in the background as well, while you continue to talk on the phone, check your email, keep the IM conversations alive, listen to music, etc.

It all just works, and it’s pretty slick.

Just in case you don’t believe the Verizon commercials, let me assure you that call quality and coverage is exceptional. In addition to the network, I’m really impressed with the Droid hardware as well. The speakerphone can be as loud as you’d like.

Then there’s Google Voice. Many people have written about the amazing things that Google Voice (GV) can do, so I won’t bother. I have been signed up for service since it was a startup company called Grand Central Communications. Google bought them, and enhanced the service dramatically.

Even though I’ve had a GV number for years, I never gave it out to anyone. I rarely used it for anything. The biggest reason is that even if you only gave out that number, if you returned a call on your cell phone (without jumping through hoops), whoever you call back now also has your cell phone number.

With the GV App, that all changes. I now make 100% of my calls through GV, so anyone I call will only ever see my GV number (even if they called my cell directly and I was returning that call). If you already know my cell number (remember, I ported my old one), then fine, call me. Otherwise, I’ll be giving out my GV number in the future.

The point is that I get more control. If I don’t want my cell to ring after 10pm? No problem (except for those pesky people I trusted with my real cell number…). Battery run out on the cell phone? Tell GV to ring your wife’s cell phone instead. 🙂

Also, GV rings my house and my cell at the same time. So, if I call you from my cell (you only see the GV number, remember?), and you return the call later, when I am home, I can pick up the call on my home phone. You don’t need to know or think about which device I prefer to be called on. That’s my problem, now solved by GV.

Speaking of porting. This has nothing to do with the Droid, but rather with Verizon, but I’ll get it out of the way now and move on to the apps. Since the phone was gifted to me, my friend was smart enough not to have them port it when he was in the store (what if I didn’t like the Droid, or Verizon?).

After waiting 11 days, I called customer service to port my Sprint number. I talked to a porting specialist who took down all my information. He told me that it rarely takes 24 hours, but that since it was Saturday, it’s remotely possible that it wouldn’t be done until Monday. Still, he thought it would happen that day.

On Tuesday morning, when the number still hadn’t ported, I called again and asked what the status was. I was told that everything was set up, but that I hadn’t confirmed to the automated system that I wanted this done. Huh? Speaking to a porting specialist isn’t confirmation? Oh well.

The representative transferred me to the automated system and stayed on the line as well while I pressed a number to confirm. Within an hour, my number was ported.

At the end of that week, Verizon was offering a BOGO (Buy One Get One free) offer for any Droid or Droid Eris, even on one that was purchased less than 30 days earlier. Another friend of mine who loves her iPhone, but hates AT&T (can’t get coverage at her house) went into the same store that my Droid was purchased from, and got her Droid free (using mine as the first of the BOGO). She ported her number at the store, and it was working three minutes later!

On to the apps. First, I haven’t bought any yet, so any app I mention here is free, or has a lite version that’s free. My friend who bought me this phone says that this is bad sign for the app ecosystem that I haven’t parted with any money yet. He’s sort-of right.

In the beginning, I wasn’t willing to buy any apps because I wasn’t sure I’d keep the phone. Then, it became more of a commitment thing than a money thing. I can afford the $2 average price for an app (trust me on that), but I’m more likely going to lock myself in psychologically (stupid, I know), and I’m still in the discovery phase. Luckily, there are some pretty awesome free apps, so I have much to explore before I decide to lock myself in.

I’ll cover apps in the order of most used to least used, not most important or best. Let’s start with email. As much as I’m crazy about the browser on the Droid, I wouldn’t keep any phone that didn’t at least have a reasonable email app. First and foremost, I require the device to handle emails well. Even more so than being a good phone!

Because Android is a Google product, there are two separate email programs in Android 2.0. One is for GMail only and the other is for any other mail server (IMAP and POP). I’ll finish this post later on with thoughts on Google and their apps, specifically.

The GMail app is excellent (as many others have noted), and if you’re primary email is GMail, this phone will make you feel right at home. I rarely use GMail (though I have an account, and I do get 10 emails a week there), so it hasn’t been critical to me, though I appreciate how well integrated it is and how well it works. In fact, seconds after an email hits my GMail account, my phone is alerting me to that fact.

The generic email client is OK, but barely. I find it reasonably attractive (font and layout) and reasonably reliable. That said, it’s so bare bones that if there were no alternatives, I might get tired of it quickly. Interestingly, I could consider using the browser for webmail access, since the browser is that good!

Two of the deficiencies are:

  • no identities (I send email from a work persona, home, etc.)
  • no auto-signature (at least not that I could easily find)

The good news is that there is an open source alternative called K-9 (available in the Android Market) that is much more configurable, including the two deficiencies noted above. The bad news is that it’s significantly less attractive (font/layout). I can live with less attractive (for now) but more functional.

Google Voice Search (has nothing to do with the GV service). This isn’t unique to Android, as you can do this on an iPhone as well. The slight difference is that there is a dedicated button available on the Droid (the magnifying glass on the bottom right hand corner), that when held for 2 seconds, launches Voice Search.

I use this all the time. It’s fun, but it’s incredibly accurate as well. In a quiet room, I’d say it approaches 99% accuracy for non-funky words. Even with the TV on, or other people having conversations in the room, the accuracy is over 80%.

Hold the button, say something, see a browser loaded with search results for your query. Click a link, off you go. It saves a lot of typing, and like I said, it’s plain old fun! You can use it to dial the phone number of someone in your contact list as well, get driving directions, or find some other stuff on your local phone, but for the most part, it’s about searching the web without having to type.

I’ve chosen IM+ Lite for IM. It’s multi-protocol. Actually, I don’t care. I do all my IM via Jabber to my own server, and I run transports on that server for AIM and ICQ. So, if I can connect to an arbitrary Jabber server with even a single-protocol client, I can chat with all my friends/colleagues, easily. IM+ Lite shows scrolling ads at the top of the screen, which I can live with. It works really well.

Android also comes with built-in GTalk for IM (I don’t think it supports the GTalk Voice part). I’ve tested it, and it works well (since it’s part of the Google suite, it doesn’t need to be launched automatically). Unfortunately, not one single friend of mine uses GTalk as their default chat service, so I’m not likely to get much use out of it either.

At home and the office I run an Asterisk PBX. That makes VoIP services a nice to have (definitely not necessary). I greatly prefer IAX2 for the VoIP protocol, but there are no IAX2 clients available for Android. Oh well. There isn’t a good selection of SIP clients either (a little more surprising), but I’ve found one that’s somewhat workable, SIPDroid.

It’s quirky and picky about which services it will connect to, partially due to the fact that it seems meant to promote a public PBX service called PBXes. Still, I can (somewhat reliably) connect to just one of my Asterisk servers, and from there, make/receive calls as if I was sitting at my desk. Cool, but it could be a bit less rough around the edges.

Fring is a more general VoIP and IM client, and I believe that it will be a winner. It seems to have a lot of buzz and momentum. I don’t use it for IM, and I haven’t had success with the SIP part either (I get one-way audio, so connection isn’t the problem).

What do I use Fring for then? I have had perfect success with them as a Skype gateway. I can make/receive calls to/from any of my Skype contacts, and the call quality is excellent (over Wi-Fi or 3G). Amazingly, at least for now, the Skype Lite app (provided by Skype!), can’t do Skype-to-Skype calls!

I don’t typically launch Fring, just as I don’t typically launch Skype on my desktop. If someone emails me, or IMs me that they would like to speak, I’m happy to launch Skype/Fring.

I was sitting in Birmingham, AL over the Thanksgiving weekend, and I received a notification on the Droid that there was an update available for Fring. I downloaded and installed it, and launched the app to test out the update. Within 30 seconds of launching it, I was getting a Skype call.

The caller is a life-long friend of mine who lives in Thailand! The first thing I said to him is “You probably won’t believe this, but you’re reaching me on my cell phone!”. His response? “I know that. My contact list shows you as Hadar on mobile Skype via Fring!” Wow, very cool. Like I said earlier, the quality was superb.

Google Maps. Like me, you probably use Google Maps on your desktop browser. I am surprised by how often I use it on the Droid. Not really to get directions (in the car, I’m still a fan of my Garmin Nuvi, more on that later). One of the reasons I tap on the Maps icon a lot is because of Latitude.

Google Latitude is an opt-in service which will show your position on a map, and broadcast that position to anyone who you’ve friended and permitted to see it. Of course, you see where your friends are on a map as well, if they’ve reciprocated. This is not Android only, it works on an iPhone too. But, since the iPhone doesn’t multi-task, unless your friend brings up Maps, their position doesn’t update (sorry, I guess that counts as another comparison…).

It only takes a second to see where people are, and I pull it up way more than I thought I would. It’s ideal for when you’re meeting one of those friends somewhere, and there’s no wondering how close they are, and whether they are making progress toward the meeting place or not. 🙂

ConnectBot is an excellent SSH client. I use it to connect to and control my server whenever necessary. It supports public/private keys. Because the Droid has a hardware keyboard, there is no screen space obscuring the TTY output from the server. A lifesaver on those occasions when you’re out, but the server requires some attention.

Update: Evernote just released their Android app today (or I became aware of it today, 12/16/2009). I tested it briefly, and I like what I see so far. I’m guessing that it will become a frequently used app on my Droid.

Barcode Scanner is an amazing app. It’s used as the base for other cool apps which rely on it to provide the barcode recognition. For one, if I weren’t lazy (and I am tiring myself out with all this typing, so I am being lazy in other respects), I could provide a QR code (a 2-dimensional barcode) for each app that I’m mentioning here. You point Barcode Scanner at the QR code, click the Open in Browser button, and that app would be available instantly for download from the Market.

On top of Barcode Scanner, I have ShopSavvy, Amazon, Key Ring and Handy Cards. Using ShopSavvy, you scan a barcode, and it searches online stores to find the best price for that product. Amazon works the same way, searching only their site. More on the Amazon app in a second.

Key Ring and Handy Cards are competing apps (I have both installed). You scan in your loyalty cards (CVS, Borders, super market, etc.), and instead of carrying them all around, you fire up the Droid and let the clerk scan from the high-res picture on your phone instead. Cool!

Amazon is my favorite online store. I shop from and trust a number of different online sites, but unless there is a big price difference, I choose Amazon first. their customer service is unparalleled, and the entire shopping experience is pleasant and optimized (especially for repeat customers who ship to many places).

The Amazon app on the Droid replicates that full experience, without needing to fire up the browser (not that I mind firing up the browser!). 🙂

Twitter. The most written about Android Twitter client hasn’t made it onto my phone yet, TwitDroid. There is a free version and paid one. When I read about the differences, on day one of having the Droid in my hands (when I was committed not to pay for any app yet), I wasn’t impressed by what I read about the free version.

At the moment, I have three Twitter apps installed on my phone. All three are reasonably good, and capable. The first one I installed was renamed later to Twigee. I no longer use it (not that it was bad or gave me any troubles), but I should revisit it since they’ve updated it to be multi-account.

I have two Twitter accounts, one that I lock for purely private tweets to my friends. I use that mostly as an offline status message of where I am, and the other for public tweets (the public one is @hadarvc). Neither of the Twitter clients that I use is multi-account at the moment, so I use one for my public account, and the other for my private one.

For the public app, I am using Seesmic at the moment. I occasionally use Seesmic Desktop on my laptop, and in general I’m very impressed with the company. The Android app is brand new and works reasonably well. I am sure it will improve dramatically, quickly, if their pace of innovation on the web and the desktop is any indication.

For my private account, I use Swift. It’s very attractive, works well, and if I had to lock in a single app for a single Twitter account now, I might actually select that over Seesmic. Thankfully, I don’t have to lock in that choice now, or likely ever. 🙂

I really thought that Twitter would be one of the more heavily used apps on my phone (something I never did on the Treo), but I actually rarely bother. On occasion I’ll update my status (which on the Treo I did via SMS), but reading the timeline is something I more typically wait to catch up on when I’m back on the laptop. YMMV, and the apps are more than capable of doing it exclusively on the phone.

KeePassDroid. This is an app that keeps my passwords for web sites handy. It copies username/password for easy pasting into the browser. I was using a different password database on the PC, but when I saw that this app existed for Android, I switched to KeePass on the PC as well. Keeping the two in sync is quite easy.

I thought I would use DropBox for syncing files (including the KeePass DB). They haven’t yet come out with a native Android app, though they claim that their mobile site works perfectly on Android. I didn’t check that claim out.

Instead, I signed up for a free account with SugarSync, which has a native Android app. It works perfectly and instantly. If I add a password to my KeePass DB on the PC, I tap SugarSync on the Droid, copy the file to a different folder, and voila, my updated DB is available on the Droid as well. So far, while there are a few manual steps involved, I’m happy with having full control and no surprises in the syncing process.

I have also used SugarSync in the other direction, to get photos taken on my phone over to the PC without hooking up the USB cable.

Google Finance is extremely cool if you follow any stocks. I’m shocked that it displays most stocks in real-time. Having worked on Wall Street long ago, for many years, I know what hurdles data vendors have to go through to provide such a service. You can maintain a portfolio on Google’s finance site and it will automatically sync with the phone.

Foursquare. I have read about Foursquare for a long time now, because I subscribe to Fred Wilson’s blog, and he is one of the VC’s that funded Foursquare. Even though I saw the Twitter buzz about it, I wasn’t ever tempted to check it out. When I saw that they released an Android app, I decided to check it out just for yucks.

What do you know, I like it. It’s fun. If I had more friends on it, I can even see it having tons of utility, just like Google Latitude, but different enough. Currently, all of my Foursquare friends live very far away from me, so I am aware of what they’re up to, but can’t readily join in the fun.

I use AndroZip as my file manager and I’m quite pleased with it. I use Advanced Task Killer as my Task Manager, and I’m quite pleased with it as well.

I’ve installed a number of Note taking apps but I haven’t had the occasion to use them frequently because I haven’t found a good way to sync my Outlook Notes with any.

I never intended to use my cell phone (any, not just the Droid) as a primary music/podcast device. I am quite happy with my iPod Nano when I’m running around, and the iPod Classic is the main device that Lois DJs from when we’re in the car.

Even after owning the Droid for a few weeks, I hadn’t budged on that decision (though music sounds exceptionally good on it, can be run in the background, and the MicoSD card has plenty of storage space). There’s the slightest chance that I might be rethinking that in the coming months.

For one, streaming music on the device is a fantastic experience. While I haven’t chosen to do so much yet, I have Pandora, Imeem (don’t know if it still works now that MySpace took them over!) and Last.fm all installed (there are more available!). Since you can stream in the background, you can get fresh music all day long. Of course, you can play your own MP3’s as well. I use DoubleTwist (on the PC side) to sync over music (at least I did for the music I was testing).

I installed Google Listen for finding and subscribing to podcasts. This morning, while waiting for my car to be inspected, I used it for the first time. I streamed a podcast live over 3G. It worked flawlessly. Another reason why I am at least thawing to the idea of listening a bit more on this phone.

The built-in YouTube app works flawlessly. It can also deliver HD quality videos when those are available on YouTube. Given the screen res, the quality of the speakerphone (if you’re sharing the experience with friends), the Droid is a great device in this regard. When Flash comes to this phone (likely in the next few months), that will open up even more great video services that I’m likely to take advantage of.

I have a few location-based apps installed. The one I use most frequently is Google Places Directory. Occasionally I launch Where and Yelp. I have tested Aloqua as well, but rarely launch it.

I like Movies by Flixter, Inc. I have used it a number of times. The other day, even though I was on the laptop, when Lois asked me what time a movie was starting, I reached for the Droid instead of doing a web search on the laptop.

Apps I launch occasionally, and like: SportsTap (for sports scores, duh), USA Today (to kill a few minutes and be up-to-date), Radar Now (shows local radar weather map), Open Table, Shazam (to wow people that might not have seen it on an iPhone before), NYC Subway and Bus Maps (because it’s cool to have the entire system in the palm of your hand), Aldiko Book eReader (just installed and tested, no books read just yet).

Another wonderful feature of Android (largely enabled because of multi-tasking) are Widgets. These are apps that display something useful directly on a home screen. I use the built-in Power Widget to turn on/off Wi-Fi, Bluetooth, GPS, Sync and Screen Brightness.

One of the more useful third-party widgets that I just discovered a week ago is Silent Mode Jammer. One ill-thought-out feature on Android was silent mode. If you hold the power button down for a second, you get a menu where you can select Silent Mode, Airplane Mode and Power Off. Airplane mode shuts off all radios making the device a local one only. I use it when I go to sleep to avoid alerts coming in.

Silent Mode simply silences alerts. Unfortunately, it also turns off all vibrations. Not so smart. If I’m in a restaurant, I don’t want the phone to ring, but I want to feel it vibrate. If I’m not expecting anything urgent, I can pretend it never vibrated. Otherwise, I can check at my convenience.

Silent Mode Jammer overrides this behavior with a toggle that keeps vibrate on when in silent mode, or turns it off (back to the default Android behavior). Wonderful!

ScreenModeWidget performs an analogous function with the screen timeout. By default, the screen dims after 30 seconds of inactivity, and shuts off after 60 seconds. That’s adjustable in preferences. ScreenModeWidget let’s you perform adjustments from the home screen. You can tell the screen to turn off but not lock, or never turn off, or go back to the default. Wonderful!

I have a few other apps installed that I haven’t really played with yet, including Voice Recorder and the brand new Google Goggles.

The phone is quite thin, especially for one that has a full qwerty keyboard that slides in/out. I bought a hard plastic case from Verizon that has a separate belt clip that the case slides into. It’s a terrific package ($29.99). There is only one thing I don’t understand about the case. The speaker grill on the back of the phone spans nearly the entire width, which is likely one of the reasons it’s so good.

The case has an opening that exposes only half of the speaker. Perhaps the rest of the speaker grill is only for show, and the entire working part is exposed, but I don’t think so. So, with the case on, I think you don’t get the full effect of how loud this phone can be. It hasn’t been a problem for me yet, but I’m surprised they didn’t make a bigger cutout on the case.

I also purchased the in-car mount for the eventual convenience of the hot Google Navigation app. Unfortunately, the phone only fits into the car mount when it is removed from the nice case I keep it in at all other times. Bummer, because I don’t want to remove it from the case, and I suspect that if the case were popped on and off many times, it eventually wouldn’t fit so well.

Thankfully, I’m still thrilled with my Garmin Nuvi 265WT, so in the short run, I don’t intend to take my phone out of the case to use in the car. That might change in the future, in which case I’ll have to consider another type of case for non-car use.

If you’ve made it this far, I’m surprised and impressed! 😉

Switching gears to a more philosophical discussion of Google and Android.

I totally understand the concerns that privacy advocates have about Google, and respect their position. For a number of reasons, I don’t personally guard myself as well as they would recommend. While I have had a Google account for a long time, and use a number of their services (notably, Analytics, Feedburner and Webmaster Tools), I don’t tie myself entirely to Google (not because of privacy concerns).

I had a GMail account, but rarely used it. I had a GTalk account, but rarely used it. I had a Calendar account but rarely used it. I didn’t populate my GMail Contacts. I had a GV account that I never used, etc.

Google might prefer to be associated with Android from the perspective of making mobile search a more pleasant experience, but their vision is much deeper than that (at least that’s what I suspect, strongly). If you buy into an Android phone, and you don’t give over your life to Google, you will be one of the rarest Android users on the planet.

Why? Because when you marry an Android phone with an online Google service, everything just works, seamlessly, typically nearly instantaneously as well. That’s a very sexy siren song, difficult not to succumb to. I have succumbed to all but GMail at the moment, and I admit to at least giving that some thought as well.

I have installed Google Calendar Sync into Outlook. I still use Outlook as my primary calendar, but that’s not necessarily going to remain true for long. When I put something in Outlook, it shows up in Google Calendar on the schedule I set, and then instantly appears on the phone. If I update the phone calendar, the next time I check Outlook, there it is!

I duplicated all of my contacts in both Outlook and Thunderbird (keeping them in sync by hand after the initial dump/load). That’s because my Treo sync’ed with Outlook, not Thunderbird, which is my main email client. There is no good free solution for syncing Outlook Contacts with GMail contacts. But, there is a very good (and free) Thunderbird extension called gContactSync that does a nice job of syncing GMail contacts with Thunderbird, bi-directionally. I now use that, and again, there it is on my phone.

Contacts are still a little rougher around the edges. Some edits don’t show up on the phone. When I notice that, I can always manage to edit some other field in GMail on the web directly, causing it to update on the phone. I don’t know if that’s a Thunderbird problem, a gContactSync problem, or a GMail one. So far, it’s not a big deal, but I’ll have to keep an eye on it.

Google Tasks isn’t even anemic, it’s essentially non-existent. I use Outlook Tasks quite a bit, and haven’t found a good solution to sync them all the way through. I like ToodleDo online enough, and there is a good syncing extension for Outlook. Unfortunately, the one ToodleDo Android client I have is horrible (no need to name it). So I never launch it, and still haven’t settled on a good todo manager locally.

Astrid looks to be a good free one, and it can sync with Remember The Milk (RTM) online, but RTM can’t sync with Outlook (absolutely incredible!), so for the moment, I’m avoiding it as well.

I don’t yet sync Outlook Notes (as mentioned above), and I’m seriously considering just putting them in my KeePass DB and maintaining them there only, stopping to use Outlook for that at all. If that happens, I’ll be down to only using Outlook for tasks (as my primary client, since I could easily give up the Calendar if I wanted to). We’ll see…

The point is that Google isn’t forcing me to use more of their services (yeah, right), they are enticing me to do it (hence my reference to sirens!). They’re smart. They are making my life easier, and in the process capturing everything there is to know about me, so they can sell their knowledge of me at higher prices to the world’s advertisers… I understand the bargain, and for the moment, and happily and willingly making it.

Converting from Procmail to Maildrop

Send to Kindle

I’ve been using procmail to filter mail on the server forever. I like it, so it’s important to note that even though I switched, I have nothing bad to say about procmail.

So, why did I switch? Procmail can be a little terse to read (obviously, I’m used to it by now). Over the years, I have built a large set of rules. There is a ton of cruft in there. If I wanted to clean it up, I had to rewrite it. Rewriting it in procmail was definitely a possibility.

But, over the years, I was also aware of maildrop as a filtering solution. It has a cleaner (more accurately, a more straightforward) syntax. The documentation is a little sparse (missing a few key examples IMHO). There are also thousands (if not millions!) of example lines of procmail available on the net, and it can be hard to find complex real-world examples of maildrop filters.

But, I knew that if I rebuilt my filters in maildrop, I’d be forced to rethink everything, since I couldn’t get lazy and just grab hunks of procmail from my current system and plop them into the new one. So, maildrop it was going to be!

One last time, just to make sure I don’t offend lovers of procmail (of which I am one!), everything that I did in maildrop could easily have been done in procmail. I just happened to choose maildrop for this rewrite, and for now, will stick with it. Perhaps if I ever revisit this project, I’ll iterate the next time back in procmail.

The goal of my filters is to toss obvious spam (send it to /dev/null). Likely spam gets sorted into one of three IMAP folders. The only reason I split it into multiple folders is so that I can test rules before turning them into full-blow deletes. Finally, mail that falls through those is delivered to my inbox.

Over the years, I added numerous rules to filter classes of spam (stocks, watches, viagra, insurance, etc.). Without a doubt, I introduced tons of redundancy. I didn’t scan all of the previous rules to see where I might be able to add one more line because it would be too tedious vs just adding a new rule.

I was reasonably satisfied with the result, but over time, became less aggressive about deleting mail automatically, preferring to stuff it in a spam folder for visual scanning during the day.

Since I create my own rules (I don’t run a system like SpamAssassin, which I did for a while), I can start to see patterns and simplifications over time, which was the impetus for the rewrite. In other words, there are more commonalities across classes of spam, and I don’t have to spend as much time categorizing things as I was bothering to do.

I’ve now made my first cut of the maildrop-based system. It’s been in production now for seven days, and I’m very happy with it so far. The one major change I made is to default to deleting things (in other words, much more aggressive than the previous system), but, I keep a copy of all mail in an archive IMAP folder that I will prune through a cron job, and never scan visually.

I review my delete logs once a day, so if I spot an email that looks like I shouldn’t have deleted it, or someone contacts me asking why I didn’t respond, I will be able to check the archive and have the full mail there (for some reasonable period of time).

Here’s the result of the rewrite:

The original procmail system had roughly 3800 lines it in (including comments and blank lines). The new maildrop system has under 550 lines, including comments and blanks. I delete more mail automatically, and in a week, haven’t deleted a single mail that I didn’t mean to. I am getting a few more emails sneaking into my inbox, but each day, I add a few more lines and the list gets shorter the next day.

Now that I am getting a bit more spam each day into my inbox, Thunderbird junk filters are getting more to train on, and they are getting better too, so even the junk that is getting in, is mostly getting filed in the Junk folder locally, automatically.

Here are two things that took me longer than they should have to figure out with maildrop (they are related, meaning the solution is identical in both cases, but it wasn’t obvious to me):

  • How to negate a test using !
  • How to use weighted scoring correctly (very simple in procmail)

Here’s a line in maildrop format:

if ($TESTVAR =~ /123/)

do something useful if true…

The above will “do something useful” if the variable TESTVAR contains the pattern 123. What if I want to “do something” if TESTVAR does not contain 123? Well, until I figured it out, I was making an empty block for “do something”, and adding an else for the thing I really wanted. Ugly.

My first attempt was to change the “=~” to “!~” (seemed obvious). Nope, syntax error. I then tried “if !($TESTVAR =~ /123/)”. Nope, syntax error. I then tried “if (!$TESTVAR =~ /123/)”. No syntax error, but it doesn’t do what I wanted.

I stumbled on the solution via trial and error:

if (!($TESTVAR =~ /123/))

Ugh. The ! can only be applied to an expression, which is normally (but not always?!?) enclosed in parens. But, the if itself requires an expression, so you need to put parens around the negated expression as well. At least I know now…

The second problem was weighted matches. I was having the same problem. Once I put parens around my expressions, it started working. That’s one of the few places where the procmail syntax feels a drop cleaner:

COUNT=(/123/:b,1)

COUNT=$COUNT+(/456/:b,1)

COUNT=$COUNT+(/789/:b,1)

echo $COUNT

So, the above sets the variable COUNT to the number of times that the string 123 exists in the body of the message. That is then added to the number of times that the string 456 exists in the body, finally adding the number of times that the string 789 exists in the body. The total is then echoed to the console. Without the parens, no workie.

I don’t like the fact that I have to maintain the running count myself. In procmail, you basically set a limit and the tests stop once the limit is reached (which feels way more efficient). There might be a way to accomplish that with maildrop too, but I haven’t found it as yet…

While I fully expect to add more rules, or lines to existing rules, I can’t imagine a scenario where my file will even double from here, so it will end up at less than 1000 lines. That will be easier to maintain for a number of reasons, most notably syntax readability.