upgrade

Announcement went out 35 minutes ago, and I’ve already successfully upgraded. Haven’t touched anything that’s new, but here it is, waiting to be abused.

I’ve been involved professionally with technology since 1980. So, you’d think that I understand it (and how it works) reasonably well by now. On some levels, sure, but on others, I feel as helpless as the proverbial mother-in-law or grandparent in the “clueless users” examples people always give…

Conceptually, I understand how “small tweaks” can lead to large unexpected results. It’s a variation on chaos theory. Practically, it’s still annoying. What is harder (for non-techies) to understand is when things break down after no changes (that they are aware of!). Of course, it’s the parenthentical comment that is the clue.

With modern operating systems, the vast majority of users have some form of automatic updates turned on. That being the case, things are chaging frequently, and possibly in very significant ways. It just so happens that the user doesn’t associate different behavior in their favorite applications with an invisble update.

The above was just generic whining to get to one or two rants that have been bugging the hell out of me lately…

The first topic is spam filtering. For many reasons (most of them rational ), I am Windows user (specifically, WinXP Pro, but that’s not important). I don’t think it’s superior, etc., but many applications that I find convenient (and in some rare cases even necessary) are always available first on Windows, and often only on Windows… C’est la vie…

So, being a comitted Windows person (no, the irony of that statement doesn’t escape me ), for many years, I was a tried and true Outlook user. In fact, I started with Outlook 97, moved to 98, then 2000, and then 2003 (no, I didn’t have the pleasure of Outlook XP).

In the early years, there was no need for spam filtering. Not only was the volume of spam low, my Internet activities were reasonably limited, so I wasn’t on many spam lists anyway. Of course, being a VC now, and having my name on many public sites, along with being subscribed to many mailing lists (public as well as publically available internal company lists), has changed that fact melodramatically.

On some days, I get well over 1000 spam messages (through the variety of means that email can wind up in my real account). Clearly, that isn’t a sustainable number of mails to have to delete by hand (even though I am ultra fast at spotting spam and hitting the Junk key).

So, a few years ago, I installed the free SpamBayes plug-in for Outlook. (This now requires a minor side-rant)

<Side Rant>

Ever since I upgraded to WordPress 2.1, I can’t create any links with their “visual” tab. I wanted to link to the SpamBayes project page above, and got a blank pop-up box where the form is supposed to be. Firebug shows errors with TinyMCE, and before that, an error with an XHTPPRequest, so it’s likely Firefox config that’s causing the problem, but I have no idea whatsoever what else to try (obviously, I’ve tried a lot of things…)

</Side Rant>

So, I ran SpamBayes for a long while, and also ran a commercial derivative of it, InBoxer (should have had a link to that as well…)

It did a pretty good job. Still, it wasn’t all that satisfying, because every message needed to be downloaded to my laptop, before SpamBayes (SB) could analyze it. That meant that on a heavy spam day, if I was on a slow link (let’s say dial-up, gulp), I had to wait for all of the spam to come down to find the few gems that I was breathlessly waiting to read.

So, after doing that for quite a while, and building up a large SB db, I decided to get creative. I installed SB on the server as well (I control my own server), and regularly uploaded my local (meaning laptop) SB db to the server. Then I added a procmail rule that filtered each message using the locally trained db (but now up on the server), and then did one of three things with the result:

1. If it was marked as “ham” (definitely not spam), it was just passed through normally.
2. If it was marked as “unsure” (the range is user-definable), then it was moved to another account on the server, so that it didn’t auto-download on each email check (this solved the problem of slow links with lots of possible spam)
3. If it was marked as “spam”, it was deleted right then and there on the server.

This worked very nicely for quite a while as well.

Then, I woke up, and decided to break myself of the Outlook Addiction. I’m still firmly in the Windows world, and have been ever since I decided to stop using Outlook for email (over 2 years ago now!). Even though I own a legal copy of Office 2003, I now only use Outlook for Calendar, Tasks and Notes, and that only because it syncs reliably with my Treo 700p.

I switched to Thunderbird, and have never regretted doing that. I’ll save any niggling complaints about Thunderbird for some future post when I am really bored, since for the most part, I am extremely happy with TB.

Now, the first part of the problem. TB has built-in Junk filters, which work OK (but not that great), but that puts me back to having to download everything to have it analyzed. The second part of the problem is that I can continue to use the old (static) SB db on the server to help cut down on spam, but the real beauty of SB is the B (Bayes), which continually learns. Since spammers constantly change their strategies to stay ahead of the anti-spam companies, having an outdated SB db degrades its usefulness over time.

Wow, I can’t believe how much background I just gave in order to get to the actual point…

Recently, emails that were previously being marked as “ham”, or “unsure”, were getting tagged as guaranteed “spam”, meaning SB was assigning them a spam score of 1.0! Of course, my server-side filter was dutifully tossing them to /dev/null as instructed, and I was blissfully unaware of that.

I discovered that when another phenomenon began. Any emails with large attachments were going directly to /dev/null. Since most of my procmail rules are also duplicated for Lois, she was complaining before I noticed, that people were writing tons of “follow up” emails to her, wondering why she hadn’t responded to their last email. Those follow up emails were getting through, because they didn’t have attachments. I am still not sure that this was because of the old SB db, but at least that caused me to find the other emails that were definitely being miscategorized…

In any event, I turned off the SB db, and the flood of spam started up again. About a month ago, I turned off SpamAssasin on the server side, because while it was somewhat effective, it was also one of the biggest resource hogs I had ever seen on the server, and the “reward” wasn’t worth it…

So, now, I’m spending a little too much time hand-tuning procmail rules to get the spam back down to a mangeable range. So far, so good, but with lots more effort than I would have hoped to expend, given the nice steady state I had for a reasonably long time.

Anyway, this post has turned out way longer than I expected, so I will save the other “random” events for some future post, when they bubble to the top of my frustration queue.

P.S. I am still not sure I’ve “solved” the large attachment problem. My temporary solution was to specifically whitelist those senders in procmail, which works, but begs the issue of whether others are being thrown away that I’ll never find out about, or find out about too late

I googled to see whether other people were having trouble with the upgrade to WordPress 2.1. Most were successful on their first try, but a few people had problems that were similar to me.

I made two changes and tried again:

1. I ensured that my user had full permissions on the DB
2. I performed the upgrade from a fresh directory untar of 2.1 (previously, I untarr’ed on top of 2.0.7)

I doubt that #2 had any effect, but I may as well be complete in my report.

This time, everything went fine. This post is being brought to you courtesy of WordPress 2.1. Whew

I’m not a happy camper. I started out with WordPress 2.05. After the first post or two (or three), I upgraded to 2.07. That upgrade went flawlessly.

This past weekend, I upgraded to WordPress 2.1. The upgrade “failed”. The previous blog posts showed up fine, but I saw a SQL error in the part that shows which categories each post is in. A little digging, and it became obvious that one of the schema changes failed. Turns out that my “user” (at the MySQL level) had insert/update/delete rights, but not alter, etc. Ugh.

Nothing I did after granting those rights made it possible to re-run the “upgrade.php” script correctly. At least not that I could tell…

After reading some forum posts, I installed phpMyAdmin, and while looking at the updated schema file, I hand-edited the DB to add the appropriate columns and one new table. It seemed to work, as the SQL error went away.

Then yesterday, I wrote another long post complaining about my saga with the old Dell laptop. When I hit save, it “appeared” to save, but the post was gone, forever… The title was there, with a blank post. Another test, and sure enough, no posts were making it to the DB.

Man, the people that “encouraged” me to start blogging owe me quite a number of hours to be added back to my life (not that I’m anything less than a world-class time-waster all on my own, but I really didn’t need any help here)

So, I exported all of my previous posts (yep, all 5 of them). Today, I created a new DB, updated the wp-config.php to point to the new DB, made sure to grant my user ALL privileges on that DB, and re-ran the “wp-install.php” script. After that worked, I imported my posts, and voila, they are all back.

This post will be the “test” as to whether that solves my problem of posting in general…

Finally, while WordPress 2.1 seems to have cooler features in it, so far, in Firefox 2.0.0.1, the UI is hosed. The textarea that I am writing in spills into the sidebar on the right hand side (that worked perfectly in 2.07), and the only “fix” is to maximize the window to the entire screen (I’m running 1600×1200), which is annoying, to say the least…

P.S. No such luck. This post failed too. I have reverted back to 2.0.7. Even with a fresh install of 2.1, and no imports, I can’t post anything by a blank with a title . This is a fresh install of 2.0.7, with the original posts imported, and this one as the “new” one…