February, 2007:

SPAM Problem Solved!

Send to Kindle

OK, not really. That was only a slight exaggeration 😉

Seriously, the specific spam problem that I complained about in my “technology is random” posting is what I’ve now solved.

As I mentioned in that post, I had a combination of procmail rules and SpamBayes filtering, etc. I completely turned off the old SB filtering, because at first I thought that somehow it was causing the emails with attachments to be deleted. Only when I did that, did I notice that it was throwing away other emails simply because it was incorrectly tagging them as certain spam (score of 1.0). I couldn’t believe that, but like I said, since I wasn’t updating the db, it was degrading.

So, I turned off the SB filtering, and still, emails were being sent to /dev/null on the server if they had large-ish attachments. That meant that one of my other procmail rules was kicking in. I looked at each (I have many) very closely, and couldn’t imagine which might be causing this.

Also as mentioned in the previous post, I temporarily fixed this by creating a procmail-based white list, which (unfortunately) was both after the fact, and growing steadily.

I also went back and with a few carefully crafted grep and tail pipelines, was able to identify other emails that had quietly been thrown away, and then contacted those (very surprised) authors, and asked for a resend.

OK, on to the solution (almost). Yesterday, an old boss of mine (no, he’s not that old, but I haven’t worked for him directly since 1989!) asked me to review a 384 page document that he had written (no, I’m not kidding about the size). People who know me, know that I (and Lois) are like an echo when it comes to email (think “ping pong”). When he didn’t get an acknowledgement from me within an hour, he assumed that something was wrong.

He sent me another email, asking if I’d gotten the file. Of course, /dev/null had eaten it…

I white listed him, and got the file (which is how I know the size, as at first he scared me by telling me that it was 400 pages) 😉

That got me to thinking that I now had a specific attachment that I knew would fail. I ended up sending it to myself from an account that wasn’t white listed. It got thrown out immediately. Bingo! Now I was at least in control of my own destiny, since I could provoke the problem any time I wanted to.

The next step was easy (and obvious). I turned on verbose logging in procmail and resent the email. You might ask “Why the hell didn’t you turn on verbose logging earlier?” Good question. Aside from not really thinking about it, I must have known (intuitively) that my disk would have filled up waiting for a “bad” email to come in and provoke the problem. Even asking someone to resend would have an unacceptable lag in waiting for them to see my email and act on it, etc.

Logging showed that I was being completely stupid in one specific rule. As the rest of you must know, one of the most popular email annoyances are the pump-and-dump stock schemes. They promote a specific stock as the next moon shot. Many are traded on an exchange with a code of PK (for the few of you who don’t know, that’s the Karache Stocke Exchange in Pakistan, a place where I am dying to find a good stock deal!) 😉

So, I started a little procmail rule that added any symbol in those emails that I was sure (and here comes my ultra-stupidity) that couldn’t occur in a normal email. So far so good, right? As an example, let’s say that one of the symbols was “JMNX.PK”. Come on, would I worry about accidentally deleting an email that had that string of characters in it?

Well, mistake number 1 (the tiny one) is that without escaping the “.” in the above symbol, it would have substituted for any character, so if a buddy sent me an email saying “Howdy, check out JMNXOPK”, I would never have seen it. Hopefully, I’d survive such a faux pas. But, over time, I added shorter symbols. Notably, one was PHYA. Again, I wasn’t “worried” that someone would send me a legitimate email with that in it. This was mistake #2, and clearly the biggie…

When someone sends you an attachment, it gets encoded, typically in base64, which is an ascii encoding. That means that it is converted into a series of apparently random characters. The bigger the attachment, the more of these random characters, and the more likely that any 4-letter combination will appear.

So, it turned out that the 384 page document had the string “pHYa” in it. Note that procmail was kind enough to be case insensitive so that “pHYa” matched my input of “PHYA”, reducing the number of random combinations I had to sweat out.

Of course, in retrospect, I was an idiot, and the inevitability of the match is obvious. The solution is trivial too: delete the rule 🙂 Now that it’s gone, it’s just as simple to add at least another step to check for any number of other typical pump-and-dump keywords along with the ticker symbol, and that should work just fine. In the end, it was both laziness on my part, coupled with the fantasy of catching every occurence of that particular type of email that did me in.

All I can say is amen, a modicum of sanity has returned to the world…

Debugging Firefox and WordPress 2.1 UI issues with Firebug – SUCCESS

Send to Kindle

I’m typing this to provoke the following error:

uncaught exception: Permission denied to call method XMLHttpRequest.open

So far, not provoked, which means that I think I figured out what was killing me ever since I upgraded to WordPress 2.1.

Before I declare victory, let me try to insert a link, which is what I was having the most trouble with before. Yippee!

The part of the post from the above link that made the lightbulb go off in my head was where megatron5151 points out that links with “evolvefuel.com/blog” were different than “www.evolvefuel.com/blog” for purposes of the browser thinking that there was a cross-scripting domain issue.

I realized that I was redirecting all URLs with either wp-admin or wp-login in them to require https. That made the base URL of the site different than the administrative part of the site (only due to my redirection, not because of anything that WordPress was aware of!), and so the new AJAX niceties that were introduced in WordPress 2.1 were being turned off by Firefox (correctly!), like autosave, etc.

The simple solution (and I’m not sure whether I have compromised security here or not, so if anyone is indeed reading this, which I doubt, and you know the answer, please let me know!), is that I made only the wp-login redirect to https, and once logged in (presumably avoiding my password being transferred in the clear over the wire), I revert back to plain old http. If the rest of the authentication is done via cookies, or sessions ids, I guess/hope that I’m fine from a security point of view. If not, then I guess that Firefox will be continually sending my password in the clear in the background (unbeknownst to me), in which case I need a better long-term solution than this.

In the meantime, I am immensely relieved to have put this headache behind me. Further, it turned out to be an interesting first use of Firebug, which is clearly awesome 🙂

P.S. I am very happy to have been able to link to Firebug as a result of it helping me to track down this problem 😉

Why does most technology feel “random” so often?

Send to Kindle

I’ve been involved professionally with technology since 1980. So, you’d think that I understand it (and how it works) reasonably well by now. On some levels, sure, but on others, I feel as helpless as the proverbial mother-in-law or grandparent in the “clueless users” examples people always give…

Conceptually, I understand how “small tweaks” can lead to large unexpected results. It’s a variation on chaos theory. Practically, it’s still annoying. What is harder (for non-techies) to understand is when things break down after no changes (that they are aware of!). Of course, it’s the parenthentical comment that is the clue.

With modern operating systems, the vast majority of users have some form of automatic updates turned on. That being the case, things are chaging frequently, and possibly in very significant ways. It just so happens that the user doesn’t associate different behavior in their favorite applications with an invisble update.

The above was just generic whining to get to one or two rants that have been bugging the hell out of me lately…

The first topic is spam filtering. For many reasons (most of them rational ;-)), I am Windows user (specifically, WinXP Pro, but that’s not important). I don’t think it’s superior, etc., but many applications that I find convenient (and in some rare cases even necessary) are always available first on Windows, and often only on Windows… C’est la vie…

So, being a comitted Windows person (no, the irony of that statement doesn’t escape me ;-)), for many years, I was a tried and true Outlook user. In fact, I started with Outlook 97, moved to 98, then 2000, and then 2003 (no, I didn’t have the pleasure of Outlook XP).

In the early years, there was no need for spam filtering. Not only was the volume of spam low, my Internet activities were reasonably limited, so I wasn’t on many spam lists anyway. Of course, being a VC now, and having my name on many public sites, along with being subscribed to many mailing lists (public as well as publically available internal company lists), has changed that fact melodramatically.

On some days, I get well over 1000 spam messages (through the variety of means that email can wind up in my real account). Clearly, that isn’t a sustainable number of mails to have to delete by hand (even though I am ultra fast at spotting spam and hitting the Junk key).

So, a few years ago, I installed the free SpamBayes plug-in for Outlook. (This now requires a minor side-rant) 🙁

<Side Rant>

Ever since I upgraded to WordPress 2.1, I can’t create any links with their “visual” tab. I wanted to link to the SpamBayes project page above, and got a blank pop-up box where the form is supposed to be. Firebug shows errors with TinyMCE, and before that, an error with an XHTPPRequest, so it’s likely Firefox config that’s causing the problem, but I have no idea whatsoever what else to try (obviously, I’ve tried a lot of things…)

</Side Rant>

So, I ran SpamBayes for a long while, and also ran a commercial derivative of it, InBoxer (should have had a link to that as well…)

It did a pretty good job. Still, it wasn’t all that satisfying, because every message needed to be downloaded to my laptop, before SpamBayes (SB) could analyze it. That meant that on a heavy spam day, if I was on a slow link (let’s say dial-up, gulp), I had to wait for all of the spam to come down to find the few gems that I was breathlessly waiting to read.

So, after doing that for quite a while, and building up a large SB db, I decided to get creative. I installed SB on the server as well (I control my own server), and regularly uploaded my local (meaning laptop) SB db to the server. Then I added a procmail rule that filtered each message using the locally trained db (but now up on the server), and then did one of three things with the result:

  1. If it was marked as “ham” (definitely not spam), it was just passed through normally.
  2. If it was marked as “unsure” (the range is user-definable), then it was moved to another account on the server, so that it didn’t auto-download on each email check (this solved the problem of slow links with lots of possible spam)
  3. If it was marked as “spam”, it was deleted right then and there on the server.

This worked very nicely for quite a while as well.

Then, I woke up, and decided to break myself of the Outlook Addiction. I’m still firmly in the Windows world, and have been ever since I decided to stop using Outlook for email (over 2 years ago now!). Even though I own a legal copy of Office 2003, I now only use Outlook for Calendar, Tasks and Notes, and that only because it syncs reliably with my Treo 700p.

I switched to Thunderbird, and have never regretted doing that. I’ll save any niggling complaints about Thunderbird for some future post when I am really bored, since for the most part, I am extremely happy with TB.

Now, the first part of the problem. TB has built-in Junk filters, which work OK (but not that great), but that puts me back to having to download everything to have it analyzed. The second part of the problem is that I can continue to use the old (static) SB db on the server to help cut down on spam, but the real beauty of SB is the B (Bayes), which continually learns. Since spammers constantly change their strategies to stay ahead of the anti-spam companies, having an outdated SB db degrades its usefulness over time.

Wow, I can’t believe how much background I just gave in order to get to the actual point…

Recently, emails that were previously being marked as “ham”, or “unsure”, were getting tagged as guaranteed “spam”, meaning SB was assigning them a spam score of 1.0! Of course, my server-side filter was dutifully tossing them to /dev/null as instructed, and I was blissfully unaware of that.

I discovered that when another phenomenon began. Any emails with large attachments were going directly to /dev/null. Since most of my procmail rules are also duplicated for Lois, she was complaining before I noticed, that people were writing tons of “follow up” emails to her, wondering why she hadn’t responded to their last email. Those follow up emails were getting through, because they didn’t have attachments. I am still not sure that this was because of the old SB db, but at least that caused me to find the other emails that were definitely being miscategorized…

In any event, I turned off the SB db, and the flood of spam started up again. About a month ago, I turned off SpamAssasin on the server side, because while it was somewhat effective, it was also one of the biggest resource hogs I had ever seen on the server, and the “reward” wasn’t worth it…

So, now, I’m spending a little too much time hand-tuning procmail rules to get the spam back down to a mangeable range. So far, so good, but with lots more effort than I would have hoped to expend, given the nice steady state I had for a reasonably long time.

Anyway, this post has turned out way longer than I expected, so I will save the other “random” events for some future post, when they bubble to the top of my frustration queue.

P.S. I am still not sure I’ve “solved” the large attachment problem. My temporary solution was to specifically whitelist those senders in procmail, which works, but begs the issue of whether others are being thrown away that I’ll never find out about, or find out about too late 🙁

I’m truly a glutton for punishment…

Send to Kindle

OK, so here I am again, updating my progress on the old Dell Latitude L400 saga…

Before I begin, let me assure the millions of readers of this blog that I have no illusion that I will ever be able to use this laptop in anything other than as a “guest surfing” node in my apartment. It’s not the case that I think if I just “bull through this”, I will figure out how to make it reliable for the originally intended purposes…

So, why do I continue banging my head against this rock-solid wall? Because I like to understand things, even things I can’t make work. Since some of the failure modes of the machine are reproducible, I have an inclination (or perhaps fantasy is a better term) that I can at least figure out what is failing. Even if I do, there’s no way that it will be economically viable to “fix it”, but knowing will make all the difference (to me at least) 😉

Deciding that it was worth the minor effort to turn this into the equivalent of a “thin client” browsing machine, I wanted to pick a Linux distro that would require little tweaking for my purposes. While Ubuntu 6.06 was working reasonably well on the machine previously (fewer halts than Windows), I wasn’t crazy about putting it back on. This actually introduces a little side rant…

(Pardon the interruption of the real point of this thread…)

I get that Linux distros want to offer some sort of “stability” promise, and as such, are supposedly careful about upgrading apps too aggressively. There’s a general “goodness” associated with that concept. That said, it’s also annoying that there isn’t an easy way to over-ride that.

An obvious example is Firefox. Ubuntu 6.06 ships with 1.5.0.3. Assuming that it is correct not to upgrade to 2.0.x, is it therefore correct to not automically update (through their automated updating service) to 1.5.0.9? After all, the concept here is those are bug releases only (no new features).

It’s bad enough that I can’t just get that update automatically, but I’m not even given a choice to upgrade to it optionally, even with a warning that it hasn’t been fully tested. However, what really bugs me (again, remembering that I understand the concept of stability) is that within Firefox itself, logged in as root, I can’t hit the “Check for Updates” button, as it is greyed out. Obviously, I’m being saved from myself, and I don’t like it.

(OK, back to our regularly scheduled saga…)

So, I wanted a distro that had the latest versions of Firefox, Thunderbird and OpenOffice.org. Even Ubuntu 6.10 doesn’t qualify. The current “snapshot” does, but then I don’t get the easy-to-install-from CD, etc.

A little searching, and I found PCLinuxOS version 2007.1 test release. With a little trouble (all caused by me, not PCLinuxOS!), I got it installed. I’m done then right? Wrong! Why? Because, PCLinuxOS doesn’t start up the wireless networking correctly (it works fine with the wired port). Somewhere, deep in the bowels of the kernel, it recognizes my Lucent Wavelan (Orinoco) card, as “dmesg” shows that there is a card inserted in slot 0. All attempts to load the right things (including: “modprobe hermes”, “modprobe wavelan_cs”, “modprobe orinoco_cs”) fail to make the card work, but all succeed in loading the appropriate modules.

Anyway, I boot with Damn Small Linux 3.2 (DSL), and from the Live CD, it recognizes the card correctly, and can access the Net just fine. Damn (so to speak) 🙁

This is likely a general problem with this specific distro (PCLinuxOS), as they have deferred their final release due to the number of problems found in the test release…

So, I’ll let this one go for now (though my general complaint about distros upgrading to current app releases within a bug-fix range stands!) 🙂

So, back to the original intent of this post, which was to explain why I even bother to continue working on this machine, when I know it can’t fulfill my true need for it…

It turns out, that when I put PCLinuxOS on the machine, and found out that it didn’t work with my wireless card, I decided to “upgrade” from their repository, hoping to install additional wireless tools. That worked fine, with one notable exception.

No matter how little or how much I asked for the updater to download (the largest was 80 files with 38MB, the smallest was 12 files with 2+ MB), it downloaded them all just fine, but the instant it started to process the files, the machine halted! Sound familiar?

It then occured to me that every time the machine halted, it was doing something related to the network. This was across operating systems (Windows XP, Windows 2000, Ubuntu 6.06, PCLinuxOS, etc.). So, perhaps a bad NIC? I doubt it highly. Why? Because in the case of PCLinuxOS, it was the wired port, in the case of Windows 2000, it was operating 100% wirelessly, etc. Also, it was able to do hundreds of MBs and hundreds of individual updates for Windows XP and 2000. It was specific use cases of networking that caused it to crap out!

In the PCLinuxOS case, after rebooting and starting Synaptic again, and asking it to update again, it found the packages locally (intact!), and installed them all without problem. On the next set of updates, it failed in exactly the same way. Download them all cleanly, and upon attempting to install them, the machine halted.

So, I’m no closer to understanding the what is happening, but it seems to be related to some kind of networking issue, perhaps related to task-switching from networking to disk, etc. Who knows…

I’m hopeful that this is the last post on this specific laptop (as I’m sure the throngs of my readers will appreciate!) 😉