July, 2008:

Recovering From a Server Disk Crash

Send to Kindle

The machine that is delivering this blog to you is a standalone Dell server, running CentOS 5.2. It resides in a data center managed by Zope Corporation (ZC) Systems Administrators (SAs). I perform the majority of the software administration and maintenance myself, and they maintain all of the hardware along with some of the software.

The single most important software maintenance that they are responsible for is backing up the system. We’ll get to that in a minute.

This past Sunday, we were in our usual hotel room. I was logged on for roughly eight straight hours, and all was well on this server. I shut my laptop off at around 10:15pm. When we woke up in the morning, Lois checked her email on her Treo. It kept hanging. We thought it was a Treo/Sprint problem, but even rebooting the Treo didn’t correct the hang.

When we got into the office (at around 7am), our laptops couldn’t retrieve mail from the server either. Pinging the server worked, but I couldn’t SSH on to it. In fact, most services delivered by this server were gone (including this blog, which was unreachable). The only service (aside from ping/ICMP) that was obviously up was Zope! If you just wanted to visit our corporate site, and learn about our VC business (all of which is handled by Zope), that was running fine!

The head of the SA group at ZC was also in that early (thankfully!). After poking around a bit, I encouraged him to power off the machine remotely, and them power it back up remotely. He did. Unfortunately, the machine didn’t come back up (or at least it wasn’t responding to pings any longer, so something was wrong).

Another SA was called and directed to go to the data center rather than the office. When he got there, and hooked up a console to the server, he saw that the disk was failing. Attempts to get it started (running fsck, etc.) proved fruitless. It decided to die completely that morning.

Yet another SA was dispatched (from the office) to the data center with a fresh disk (he had other work to perform at the data center, or I would have delivered the disk myself, happily!). We knew in advance that this disk might be slightly problematic, as it had CentOS 4.6 installed on it.

When the disk was inserted into the machine, it booted up immediately. I was able to successfully SSH to the machine, but of course, nothing of mine was on there. That’s when the original SA (who also happens to be our expert in backups) started restoring the machine from our backups.

For a long time, we used to run the Amanda backup program at ZC. I don’t know why, but our experience with Amanda was never good. I am not suggesting that others don’t find it to be a perfectly acceptable solution, but for us, for whatever reasons, it wasn’t good enough.

After searching and evaluating a number of alternatives, the ZC SAs selected Bacula. We’ve been using that for a reasonable period of time now. The Bacula restore of my machine didn’t take all that long, and every file that I wanted/needed was successfully and correctly restored. In fact, the nightly incremental backup had run successfully before the disk decided to die (how polite), so even some non-critical files that I touched on Sunday were successfully restored! Whew!

That said, it was hardly a trip to the candy store. All of the applications that I compiled myself (more than you’d think, I’m a geek at heart!), didn’t work because of the OS (operating system) mismatch. My programs required newer versions of various libraries (possibly even the kernel itself in some cases), so starting from a 4.6 machine and restoring files that required 5.2 wasn’t as clever as we’d hoped.

Still, with some pain, theoretically, one can ugrade a 4.6 machine to 5.2 over the network. That was my plan… Well, the best laid plans, as they say…

I released the SA from the task of baby-sitting my machine, because I knew that a network upgrade would take a while, at best. After doing some network magic, the machine was in a little bit of a funky state, but there was a chance that a reboot would bring the machine up in a 5.x state, making the rest of the upgrade fairly straightforward.

Unfortunately, when I rebooted, the machine hung (wouldn’t actually go away, still pingable, but otherwise non-responsive). Again I asked the head of the SA group to remotely power down/up. Again, it powered down properly, but didn’t come back up.

In fact, it likely did come up, but because of the funky state that I left the machine in, it couldn’t be seen publicly due to network configuration issues. This time, we decided to take a more conservative approach, because opticality.com was down for at least 8 hours already (not a happy situation for either Lois or me).

The original SA went back down to the data center. This time, he burned a CD with a network install ISO of CentOS 5.2. After installing the correct OS onto the machine, he again restored the disk with Bacula. This time, everything matched. Still, there were problems…

The biggest issue (by far!) was foolishness on my part in mapping out what I wanted backed up on the machine to begin with. Sparing you the gory details, I ended up restoring the Yum database from my backup over the actual Yum database from the installation, so the system didn’t really know what was installed and what wasn’t. Not a good thing.

I only really cared about email to begin with. I built a few things (pretty quickly) by hand, and got email running. Then I got the web stuff up pretty quickly too. Finally, IM. Those are my big three hot buttons, everything else could be dealt with later on.

I didn’t want the SA to leave until we could prove that the machine could be booted correctly remotely. That took some time as well, as a number of the services that are started automatically weren’t installed on the machine (though RPM/YUM thought they were!). We (or rather he, at the console) disabled them one by one, until the machine came up.

After I restored the more critical ones, we tested a reboot again, and it came up fine. Whew. I released him again, this time for the last time.

I cleaned up a few more things and went to bed reasonably happy (it was now close to 10pm on Monday night). Over the next two days, I spent a few hours cleaning up more things. Yesterday, I completed the cleanup.

A series of shell scripts and filters, doing things like the following:

yum list | grep installed | cut -d ‘ ‘ -f 1 > /tmp/installed

Then running the resulting packages (the ones the system thought were installed!) through:

rpm -V package-name

Filtering that output for lines starting with the word “missing”. Then removing those packages (they weren’t there anyway, so it was a database cleanup activity) and then installing them via Yum again. It wasn’t actually as painful as it sounds, but it wasn’t pain free either.

The biggest headache occurred when removing a non-existent package also moved a config file (that was being used by a package I built separately, so Yum was unaware of it). At one point yesterday, without realizing it, I killed our SMTP (email) server. Oops. We were down for about 10 minutes, before I realized it, and got it back up and running.

At this point, all is back to normal, and Yum correctly knows what’s on the system.

Here are the lessons:

  1. Back Up Regularly
  2. Have a great group of dedicated SAs behind you
  3. Have some spare parts close by
  4. Think hard before taking some stupid actions (all my fault!)

I’m truly amazed, pleased, impressed, etc. that we lost nothing. Of course, we were down for 12 hours, but Internet email is a truly resilient thing. All mail that wanted to be sent to us was deferred when our SMTP server didn’t answer the call. When we came back up, mail started to be delivered nearly instantaneously, and by morning, all mail had been correctly delivered.

Here’s hoping I don’t go through this experience again, and here’s hoping that if you do, this post might help a bit… 🙂

July 2008 Poker

Send to Kindle

I won’t be playing tonight, so I can safely report this month’s results now.

Given the past few months, this one was financially successful. That said, it was extremely unsatisfying financially, while simultaneously being extremely satisfying from a having a good time perspective. I guess you can’t have it all. 😉

So, the bottom line first, then a few details that I want to report just to get them off my chest (and make the above paragraph a little less opaque).

I finished the month with a profit of $359.02. Not too shabby.

Still, it was a financially frustrating month. If you read this space regularly, you know that I play a lot less than I used to (as in a ton less). That means I have much less time to play in cheaper qualifiers, so I pay a lot more to enter the bigger tournaments. That means one or two bad breaks a month and it becomes very hard to show a profit.

This month, I lost in all four attempts at the big Sunday tourney. That was an $860 hole that I dug for myself! In addition, they now have a nice Omaha Hi/Lo on Saturday afternoon (perfect time for me!) that costs $162 to enter. I entered that one twice as well, so that’s nearly $1,200 just to enter these six tournaments.

In the big Sunday one, I just missed this week. I came 109th out of 944. They paid the top 100. I likely could have drifted into the money, and gotten back $300 for my $215. Psychologically, I needed to do that (or rather, could have benefited from a cash), but I knew that playing my JJ was the right thing to do. Amazingly, after a very big stack called my all-in, another guy went all-in (bigger stack than me, but less than the stack that called me!) and he only had AK. He risked his entire tournament on a draw with one person all-in, and a bigger stack calling it.

The guy who called me had 66, so I was ahead of him, and the AK got lucky and hit, so I was out. A week ago, I came 164th out of 901, so I’m getting close, but just not getting there all the way…

That said, in the new Saturday Omaha Hi/Lo, I had very good success this past Saturday. I came in 4th out of 72 players. That was good enough for a $1,500 payday (bring the month back into the black). That was my single largest cash in online poker. That’s why I say it was a frustrating month, as I needed that large of a cash just to put me somewhat ahead for the month.

My other minor frustrations were in some qualifiers. I missed winning seats to the Omaha Hi/Lo and the Sunday tourney by one spot at least three times this month. In one, I was the overwhelming chip leader and when it came down to two people, the other person hit every flop five hands in a row, and there was nothing I could do but come second. The free seat only came to first place… On the flip side, after paying the full freight, that’s the one I made my big cash in, so I can’t complain too loudly…

Anyway, on to another month. 🙂

Dave Mason at BB King

Send to Kindle

Last night couldn’t come soon enough for us. We loved Dave Mason’s show at Blend Bar on March 6, 2008 (covered in this post). The experience at Blend Bar was less than special, but the show was awesome. We immediately grabbed two tickets to see Dave again on April 4th at BB King, a place we frequent often and love.

As covered in this post, Dave called in sick that day, but we didn’t check in advance, and ended up showing up nonetheless. That show was rescheduled for sometime in June (a date that worked for us). That date was quickly canceled and rescheduled for last night (July 22nd, 2008). We held on to our original tickets, which were honored last night. Third time was indeed the charm at BB King.

The show was close to the same set list from Blend, but not identical (one example, they didn’t play Every Woman last night). In terms of their sound, and individual talents, every word in my previous post applies, so I won’t repeat that. Let me just say what was (slightly) different about last night’s show.

John (Johnne) Sambataro was as good as he was at Blend, and killed the crowd on at least two spectacular solos, but he was actually highlighted a little less last night than he was at Blend.

Johnne Sambataro

Johnne Sambataro

Bill Mason was actually highlighted a drop more last night. He’s a fantastic keyboards guy. His fingers were flying on his Roland, all night long, in particular on the numerous solos that he took.

Bill Mason

Bill Mason

There was a new guy added to the lineup last night, but even though Dave introduced him at least twice, I didn’t catch the name (Chris something I think). He played percussion (mostly bongos, a little tambourine, etc.). I couldn’t make out a single sound that I could attribute to him, but his hands were clearly moving fast, and in perfect rhythm with the music, so I assume he’s really good. No idea really. 🙁

Even this photo is the fuzziest of the bunch. Perhaps he wasn’t even really there! 😉

Unknown Band Member

Unknown Band Member

Alvino Bennett was awesome. In my last review I understated his playing a bit. Partially, it was because he was obscured in the corner, and partially, it was because he never took a solo. Last night, we sat at the right-most edge of the stage, a few feet back from it. Alvino was in the right-hand corner of the stage, so my view was of his forearms forward. In other words, all I could see were the drum sticks, his wrists, and some forearms, flying all night long.

Ironically, that gave me a very deep appreciation for how good this guy is. He had to lose 10 pounds while playing last night. Even though he doesn’t take any solos, his beats are fast, furious, constant, and perfect. To be clear, he wasn’t worse at Blend, I just didn’t get to pay as much attention to him, as I was so mesmerized to see Dave in person for the first time then. Anyway, Alvino is truly a great drummer!

Here’s a good shot of Alvino waving goodnight to the crowd after the show:

Alvino Bennett

Alvino Bennett

Lois had a slightly better angle on Alvino than I did, so she was able to catch him at the drums when he leaned forward slightly. 🙂

Alvino Bennett Drums

Alvino Bennett Drums

Alex Drizos was incredible on the bass. I could almost mirror the words I said about Alvino with regard to the drums, and apply them to Alex on the bass. He impressed me tremendously at Blend as well, but last night, we sat closer to Alex than to anyone else on the stage, so I got to watch his fingers in action a bit more.

While he too doesn’t get highlighted for solos, his bass lines are extremely interesting, often complex, and never overwhelm the rest of the band. He was great too!

Alex Drizos

Alex Drizos

That leaves the great man himself, Dave Mason. Wow. It was worth the wait. As I noted in the Blend review, his voice is still superb and his fingers are still silky smooth and super fast on the guitar (both six and 12 string). He’s not the most talkative guy on stage, but when he does speak, he’s warm and funny (and interesting). The band clearly loves each other, and they play incredibly tightly together!

Dave Mason

Dave Mason

For those who don’t feel like reading the Blend review (why not?!?), I’ll repeat here that Dave also has new music (in fact, he’s releasing a new CD on September 30th). At Blend, he played at least three new songs. Last night, he played two. They are awesome, so he’s still got it, in every way. Of course, his old stuff is as good as it gets as well, so he is incapable of disappointing no matter what they play!

They were announced at 8:03pm and all but Dave came out on the stage. Within 10 seconds, Alex moved to the center microphone and apologized and asked the crowd to give them a minute. They all left the stage.

After a 10 minute break, at 8:14pm, they were announced again, and all of them, including Dave, came on this time. Whew. 🙂

They played for 75 minutes and ended with a rousing standing ovation. When they came back out for the encore, they had a special guest star, Jimmy Vivino. I hadn’t heard of him before, but the bio on his site is quite interesting and impressive.

Jimmy Vivino

Jimmy Vivino

Dave let Jimmy play his guitar (something that is reasonably rare in my experience). They played Stormy Monday (one of my all-time favorite blues tunes, I listen to the Allman Brothers Band version all the time). Dave sang (and didn’t play guitar at all). He was awesome, of course. Jimmy played lead throughout. He’s fantastic. I’d be happy to catch him in any other show and get to know his music and enjoy his talent more thoroughly!

After that, Jimmy left the stage, Dave took back his guitar, and they closed with the same number they closed Blend with, Feelin’ Alright. The crowd stood and danced (or swayed in my case) 😉 throughout the finale. We left on a complete high.

They were on stage for 90 minutes in total, and rocked the house down for every second of it.

For whatever reason, we decided that we were willing to risk slightly less-than-perfect seats last night, in exchange for not standing on line in the heat for too long. So, we left a bit later than we normally would. Since we both showered right before leaving, we also decided to take the bus (even though it’s a short walk to BB King) to enjoy the air-conditioning bliss that often is delivered on modern NYC buses.

Lois was quick to get her camera out of her bag when she spotted this colorful Statue of Liberty on the sidewalk (photo taken from inside the bus):

Statue of Liberty

Statue of Liberty

We got to BB King at 6:15pm (doors officially open at 6pm). There was still a short line outside, but it worked out exactly as we hoped (dare I say, planned?) and we moved inside within five minutes. We were seated by 6:25pm, in pretty good seats.

We both had the Chopped Salad (yes, I am soliciting polite golf claps for being good). It’s a really excellent salad. I had Pulled Pork on top (OK, not quite as good as Lois) 😉 and we split a side of Red Cabbage Slaw (they make that really well there too!).

In all of the shows that we’ve seen at BB King (which is many) even when there is an opening act, that opener comes on stage at 8pm. Last night, at 7:15pm, the lights dimmed a bit, and they announced David Jacobs-Strain. He came on the stage with an acoustic guitar.

David Jacobs-Strain

David Jacobs-Strain

He opened with an instrumental, and it was obvious throughout that this guy is an extremely talented guitar player. He played a number of blues songs, the rest all included him singing as well as playing slide guitar. He has a good voice as well.

While we both enjoyed his performance, especially that it was a true bonus, not cutting into any Dave Mason time, nor lengthening the evening at all, it’s not particularly my style of music. That should take nothing away from David, who is wildly talented, and those that like raw blues, will love this guy!

That said, here’s the really cool part of the story. The night before, David opened for Dave Mason at Rams Head in Annapolis, MD. After the show, Dave’s road manager called him in for a chat. David was obviously nervous that somehow, he blew it. Instead, he received a pleasant surprise, which was hearing that Dave was inviting him to open at BB King the next night!

David was supposed to head to California yesterday, but made the correct choice (IMHO) of joining Dave for another night, and playing at BB King. Kudos to Dave for wanting to highlight a real talent (when Dave came on, he retold the story that David did, and said that David was the real deal, and we could all use some more real deal in our lives). Kudos to BB King for being flexible, and starting an opening act at 7:15pm, accommodating everyone involved!

David played for exactly 30 minutes.

After the show, David was selling CDs in the lobby. Even though this isn’t exactly the style of music we normally buy, we both like to support live music in general, and talented musicians in particular, so we bought David’s CD: Liar’s Day. Lois got him to sign it for us as well. 🙂

WordPress 2.6 Goodness

Send to Kindle

So, I tweeted the other day that I updated to WordPress 2.6 successfully, but didn’t blog about it. I am very pleased with the changes, though I had two minor frustrations, one was WP’s fault, the other not.

First, the goodness. When 2.5 came out, they completely changed the media/image management, changing the upload and selection process to be AJAX based. I really liked the look, but hated the feel. It took so many clicks and so much scrolling to edit the various image fields, and to select and insert an image in a post.

2.6 fixes this completely (for me). This is done by making two (theoretically) trivial changes:

  • Make the pop-up window larger, so that all of the fields for an image fit without scrolling
  • auto-scroll to the selected image when the user clicks on show

That’s it! Now I don’t have to scroll after I click show.

Another goodness is that the number of plugins that have newer versions available is now highlighted on every admin screen, with a cute little bubble above the plugins link. Very nice. I was very good about checking regularly anyway, but this will save me a few clicks every day.

There are other new features, most of which I haven’t played with yet, but they certainly sound useful. The video that shows off the new features is crisp and gets to the point quickly.

I’m happy with 2.6.

Now for my two frustrations. The first is a WP problem. I noted in a recent post that WordPress Ate My Posting Date. I patched it successfully (a one-liner). The patch documentation led me to believe that it was already in the trunk, and therefore would be part of WP 2.6. It wasn’t. 🙁

When I posted about Chuck Mangione the other day, the RSS feed delivered that post with a date of 1999 again. I reapplied the patch, and I believe all is well again. It was disappointing that this simple one-liner missed the 2.6 release…

The second frustration had nothing to do with 2.6. After I saw the new plugin update notification, I updated the two plugins that had newer versions available. When I went back to the admin screen and reloaded, it still claimed that both plugins needed updating, and the new bubble still showed a two in it.

After some playing around, I decided to clear the XCache cache completely. That solved the problem. So, somehow, there was PHP code cached that didn’t update when I installed the new WP 2.6. I will try to remember not to make that mistake again, and just clear the cache whenever I update WP. Now the question is whether I need to clear the cache when a plugin gets updated, etc. Frustrating that somehow date stamps on the files don’t over-ride the cacheing…

In any event, all is well again (at least for the moment). 😉

CMA Writers Series at Joe’s Pub

Send to Kindle

We’ve been to three previous shows in this series and loved all of them. The first one, was the only show we’ve had to stand the entire time at Joe’s Pub. We were away a few weeks ago and missed one show. Last night was another in the series. We didn’t find out about it until last week, but were lucky enough to get tickets. These shows always sell out.

Here is a shot promoting the sponsors of the series. They deserve the attention!

CMA Sponsors

CMA Sponsors

As I’ve mentioned in the past when reviewing these shows, typically, these aren’t the most polished performances by seasoned performers. It doesn’t matter. There is a raw power to hearing a song (that you know well!) sung by the person that actually wrote it. You often get a completely different image than the one you associate with the star who made it famous. They also tell some great back stories, which are interesting and often hysterical as well.

In the past, there has been an occasional ringer in the bunch. By that I mean someone who does indeed perform a lot, and therefore raises the level of the performance beyond the others. One notable example of that was Ronnie Bowman. He’s been a star in his own right for ages, and deservedly so. As you’ll see below, all four guests (Bob DiPiero is always there) were great performers!

Last night there were five people on stage. The one staple is the host for each of these shows, who is also in the CMA Writers Hall of Fame, Bob DiPiero. He always does a great job as both host, and performer, and last night was no exception. The crowd loves every story he tells and every song he sings.

Here’s a broad shot of the stage:

CMA Lineup

CMA Lineup

He sat in the middle of the stage. The format is the same each time, all of the artists sit on the stage together, but they go one at a time in line singing one of their famous songs. The others join in on occasion but often, the songwriter is doing their song solo (Bob is the most common accompanist).

Bob DiPiero

Bob DiPiero

With that, I’ll name last night’s performers as they sat on the stage, left-to-right:

Chris Tompkins had an electric keyboard in front of him. He’s excellent on the keyboards and sings quite nicely. His songs are fantastic. The crowd loved every second of Chris. He’s a Grammy winning songwriter, for the Carrie Underwood hit Before He Cheats.

Last night was Chris’ wife’s birthday, and she’s pregnant as well. I had the privilege of sitting shoulder to shoulder with her, so I got to congratulate her personally, and tell her how awesome her husband is. They were definitely having a blast being in the Big Apple. 🙂

Chris Tompkins

Chris Tompkins

Karyn Rochelle was next, with her acoustic guitar. Karyn only played the guitar on her own songs, but she also sang backup for some of the other artists. She has a stellar voice, and of course, by virtue of being invited to this series, writes great songs as well! She penned Kellie Picklers hit Red High Heels (for example). Bob joked (though it was clear that it wasn’t really a joke!) that Kellie sings it like Karyn, rather than the other way around!

Karyn Rochelle

Karyn Rochelle

On the other side of Bob:

Dave Berg played acoustic guitar. He has an excellent voice, plays the guitar well, and writes fabulous songs. He has a great stage presence as well. He did the first verse of one of his big hits in Bob Dylan’s voice, and it was a hoot!

Dave Berg

Dave Berg

Jedd Hughes (an Aussie!) rounded out the group, playing the acoustic guitar. Of all the performers we’ve seen in the four times we’ve been to this series, Jedd is the best musician. His guitar playing is superb and the others often highlighted him playing lead during their numbers, which was always good for a big ovation from the crowd. He sings well too, and writes beautiful songs.

Jedd Hughes

Jedd Hughes

So, we have loved all four of these. Last night we already purchased tickets for the next one in the series, September 9th. We also bought tickets to one of my favorite groups, The Duhks, for the next night, September 10th. The Duhks are the reason we discovered Joe’s Pub to begin with, so it will be great to see them there again!

Last night was different (and in that sense very special) from the other CMA Writers Series shows. All of the performers were individually polished and collectively much more polished than previous shows. We’re happy even when that isn’t the case, but when the performances also match the quality of the writing, it’s just double goodness! 🙂

Since we purchased our tickets late, we got a little bit of a shock when we were told that no dinner reservations were available. That meant that we’d be standing again, like we did the first time we attended one of these shows. That’s simply not fun for us (even though we did love that show!). Luckily, and thankfully, Lois called two days in a row to check, and indeed one table opened up. It turned out to be our favorite table, so that ended up being a wonderful surprise as well!

My Seared Tuna was perfect, as was my chocolate martini (which I hadn’t had at Joe’s in many months!). Welcome back chocolate martini, I missed you! 😉

Chuck Mangione at Blue Note

Send to Kindle

On June 7th, Lois and I saw Chuck Mangione at Tarrytown Music Hall. Last night, we saw him (with the same group) at the Blue Note Jazz Club in NYC. I wrote a long post about the Tarrytown show, all (except for the tribute to Jim McKay) applied to last night’s show, so I encourage you to read that one for more detail on the musicians and the music.

At the bottom of that post, I mentioned that they would be at the Blue Note for six straight nights (two shows a night!) this week. There are still four nights left to this run, and I encourage you to get to the Blue Note if you can, you won’t be disappointed in this show!

Lois and I had planned to see them again last night. It was a loose plan made with a very good friend of mine who has a girlfriend (do you still use that term at our age?) that Lois and I haven’t yet met (though we want to meet her badly, and she wants to meet us as well). In the end, with life’s complications, she couldn’t make it this week. I didn’t end up making reservations for the show.

On Tuesday, Lois asked me when we were going. I told her that because our friends couldn’t make it, I didn’t make the reservations. She was disappointed. So, she called on the spot and made them for last night (the only night that would have worked for us). Our godddaughter and husband had just officially moved to NYC that day (Tuesday), so she made the reservation for four, hoping they would be available. They were. 🙂

So, we went last night. We got there at 5:55pm (doors open at 6pm). We were seated at the stage, dead center (the band had to walk by our table to get up on the stage). We don’t have any good photos of them playing, because we were really close up (which might have been really distracting to them even without a flash) and because the band members all wear black on a black background, so the contrast was really bad. So, here is one photo of each band member (taken with a flash) while they were setting up. Chuck wasn’t on the stage, so no photo of him.

Gerry Niewood

Gerry Niewood

Apologies to Corey Allen, who had his back turned at the time Lois snapped the photo:

Corey Allen

Corey Allen

Charles Frichtel

Charles Frichtel Kevin Axt

Dave Tull

Dave Tull

Coleman Mellet

Coleman Mellet

So, they were as good last night (all of them) as they were in Tarrytown. That said, because we were right up front, in a much more intimate environment over all (which we prefer in general), there was an even better feel to the show. When Chuck played the Flugelhorn, he was all of four-five feet from us. When he played the keyboards, he was two feet from us. Dave Tull, one of the most incredible drummers you will ever hear, was less than five feet from us. Wow!

We got to shake every one of their hands after the show, and tell them how incredible they all are. Like I said earlier, they had to walk right by us, whether they liked it or not. 😉

As noted above, and in more detail in the previous post, Chuck paid a great tribute to Jim McKay’s passing at the Tarrytown show. Lois was very moved by it, and last night handed Chuck a note to that effect as he left the stage. It was another benefit of being right there. We certainly hope Chuck appreciated the sentiment.

The four of us all enjoyed our meals very much. I had the marinated skirt steak (I’ve always been a big fan of skirt steak, even though it’s a cheaper cut of meat). I had one extremely minor disappointment. My entire odyssey of chocolate martinis began two years back at the Blue Note with their Nutty Angel.

Unfortunately, it’s no longer on the menu. All of the bartenders are new (according to our waitress) so they didn’t know what was in it. I setlled for an Italian Chocolate Martini. It was good, but was thicker than any other one I’ve had (more like chocolate milk with vodka).

To make up for it, I had a Nutty Irishman coffee as my dessert. It had Bailey’s Irish Cream in it, but more importantly, it had the word Nutty in the title, so all ended well with the evening. 🙂

Catching Up

Send to Kindle

It’s been exactly a week since I last posted. Usually, if I take that long a break, it’s a combination of not much to say and not much time to say it in. This time, I had a number of things to say (one in particular) and plenty of time to say it. I purposely didn’t post, because I wanted The Wedding post to stay on top on the main page (without explicitly pinning it), to savor the wonderful memory, just a little longer. Alas, life goes on, and so will this blog. 🙂

So, I’ll cover a number of things in this post, trying to keep each much shorter than they might have been had they been given their own space. Hopefully, the entire thing will be reasonable in length as well. Seperately, perhaps tomorrow, I’ll write a general music catchup post, so I’ll leave music out of this one.

The one post that was hard to avoid writing last week would have appeared on Thursday morning (congratulate me on my restraint). We have really good friends that over the past few years we’ve probably seen more often in NYC than any other couple. We used to grab a meal together roughly every other trip back to NYC.

For a variety of reasons, the last time we saw them was December 2007, mostly because their lives got really complicated. He got pneumonia that lasted a month, and immediately started a new job right after that (we had no idea about either event) and she was busier than ever in a wonderful job she landed six months earlier.

Cutting to the chase, we reconnected via email a few weeks back, and the best night for them to get together was last Wednesday. We were delighted to oblige. What they had no idea about (yes, we’re sure) is that it was our Anniversary. Even without that knowledge, they insisted on picking the place and treating.

They took us to Butai, a very nice Japanese restaurant. We had a fantastic meal with wonderful company. I ordered a fancy drink (I haven’t had a fancy drink in a while) that included Prosecco (a champagne-like sparkling wine) and Pear puree (among a few other ingredients). It seemed fitting on our Anniversary. Lois ordered straight Prosecco (she didn’t realize my drink had any, and she hasn’t ordered a drink in a restaurant in nearly a year!).

Anyway, Butai is highly recommended, and we’re glad to have reconnected with great friends. Thanks guys! 🙂

I know how late I am to the party, but I simply can’t let the Jesse Jackson – Barack Obama comments go by without mention. Here’s the only thing I want to say on that subject (would have been much more if it were its own post, in a timely manner): Jesse Jackson’s apology was beyond laughable.

I’m not surprised he apologized. I’m not surprised he’s still backing Obama (could you imagine him supporting McCain?). So, I’m not calling him a hypocrite for still wanting Obama elected, badly. But, could he not have injected an iota of reality into the apology? After all, he was quoted as threatening to castrate Obama (literally!). Here’s the apology I would have liked to have heard:

I sincerely apologize to Barack Obama for my comments yesterday. While I have some fundamental differences with him on a number of issues, which caused me to privately lash out, they pale in comparison to the numerous issues where I agree with him completely. Further, even in those issues where I disagree with him, I am closer to his position on those than I am to John McCain’s, so my support for Obama continues to be as strong today as it was previously.

Simple, but believable. Don’t pretend that it was all just taken out of context, and that it’s a non-stop love-fest between the two of you. It’s obvious to any thinking person that Jesse Jackson can’t stand Obama whatsoever. That’s fine, they don’t have to love each other in order to be supportive of each other. Bottom line, with friends like Jackson, Wright and Phleger, Obama certainly doesn’t need any enemies…

I’ve been good about keeping up with my exercise routine. I walked my 8+ mile jaunt in NYC three times this week, Thursday, Saturday and Sunday. I don’t typically walk two days in a row (especially after taking off only one day in-between the first two walks), but the weather was perfect here this weekend, we were atypically in the city over the weekend, and they were predicting rain all day today.

In any event, I had great walks all three times. It’s helped with my weight as well, as I hit at a new low this morning since I reported on my dramatic weight gain back in this April post. I’m sure it will fluctuate up and down a bit more, but the fact that I’m at a new (interim) low, a week after a wedding where I didn’t hold back on desserts, is a good thing. 🙂

To be clear, I’m still way above my low since beginning to lose weight in 2001, but headed back in the right direction, finally!

Lastly, there aren’t any particularly insightful words I can add to the numerous praises that have been heaped on Tony Snow after his passing this weekend. Lois and I watched Tony for years and were always impressed with him. He was as geniunely a good person as one could aspire to be. He was also only one year older than me, so I know (personally) how unbelievably short his life was. Rest In Peace Tony, you well deserve it!

The Wedding

Send to Kindle

The day finally came and is now firmly entrenched in the memories-of-a-lifetime category. As godparents, we enjoyed it without having to endure all of the amazing hard work and planning (not to mention costs!) that the actual parents undertook. Since the result was perfect (in every way), the heartiest of thanks and congratulations go to the parents of the bride, our dear friends, Bob and Sally! Bravo! 🙂

On Saturday, I wrote a shortish (for me) post on how we spent Friday and early Saturday. What was special about the Jamestown trip (beyond the normal wonders of learning about history) was that it was the perfect way to introduce a group of strangers (many of whom were about to become related to each other) in an interesting and relaxed manner. Kudos to the father of the bride for having the idea, and executing it so well!

As noted in that post, we had a fantastic meal and fellowship together at the rehearsal dinner on Friday night. For that, we thank the father of the groom, for putting together such a splendid evening!

The festivities and preparations were ongoing throughout the big day. There were lots of details to be taken care of as well as the typical female primping. Smartly, I avoided all of it. Since Lois was involved to an extent, and the hubbub was happening in the rooms all around ours, I was peripherally aware of some of the activities. Here’s a single example of some primping:


The wedding was called for 5pm at the Wren Chapel at William and Mary in Williamsburg, VA. You can read the storied history of the Wren Chapel, but here’s the opening paragraph from that link to give you a flavor of the magic of the place:

The Sir Christopher Wren Building is the oldest college building in the United States and the oldest of the restored public buildings in Williamsburg. It was constructed between 1695 and 1699, before the city was founded, when the capital of the colony of Virginia was still located at Jamestown, and the tract of land which was to become Williamsburg was populated by simple timber buildings and known as “Middle Plantation.”

They hired a limousine to shuttle the wedding party and out-of-town guests from the hotel to the chapel. It’s a short trip, so the limo was to make continuous trips back-and-forth starting at 4pm. I got to the lobby at 4:28pm and didn’t see anyone I knew. When I stepped outside, something akin to an Airport Shuttle Bus pulled up. I asked if he was heading to the Wren Chapel, and he said he was. When I stepped inside, I saw that it was as plush as you could imagine (no hint from the outside), including two flat-screen TVs (they were off). A limo indeed! 🙂

After waiting five minutes, I was shuttled alone to the chapel. I felt a bit cramped in the back of a vehicle that could comfortably seat 16-18 people, but I made do. 😉

Lois saved me a seat near the front of the chapel, so getting there a little later didn’t cost me. You can see from the stock photo that the place is gorgeous, but in addition to how beautiful and well-preserved the place is, it has a very wonderful feel to it as well. To my left was a great friend of the bride, and for a couple of years now a great friend of ours as well, playing the harp. She was awesome, as was the organist (playing in the balcony) who was her teacher when she was at William and Mary herself. They serenaded us as people arrived.

Here’s the harpist, with her harp. This photo is from the rehearsal since Lois didn’t get any good shots of the harp on the wedding day:


At 5pm sharp, the electricity was in the air, as the music shifted and the guests hushed. The procession began. The Wren Chapel is set up perfectly to enjoy exactly this kind of event, because the pews face the center aisle. No neck-craning required to see all of the attendants and the main attraction. I purposely avoided the wedding rehearsal (but cleverly, not the rehearsal dinner!) 😉 so I got to soak it all in without knowing how it would play out.

When the beaming bride walked down the aisle on her father’s arm, it was truly a joyous site to behold. They were fortunate enough to have the minister from their church in Williamsburg (who therefore knew the couple well) preside over the ceremony. He was amazing. His wit, wisdom and spirituality were deeply inspiring.

The bride’s uncle is a Pastor as well, and he read a moving passage and added some inspirational words of his own. Mirroring that, a close friend of the groom’s family read a moving passage as well. On every level, the ceremony was just wonderful. On the most important level, the joining of the happy couple, it was heart-warming (and mesmerizing) to see their eyes locked on each other, with deep love and anticipation for their future together. There was no nervousness at the altar.

One grainy photo of the married couple:


When Laura (the bride) was younger, she had the honor of being a Junior Bridesmaid at her cousin’s wedding. Laura is very close to her cousin and to her cousin’s children. She had the pleasure of honoring her cousin’s oldest child by including her as a Junior Bridesmaid this time around (full circle!). Here is a photo of the Minister calling on her and the final groomsman to close the procession:

Junior Bridesmaid

When the ceremony was over, we were all asked to gather as quickly as possible outside for a group photo. The cause for the rush was the impending storm. Luckily, aside from a very few drops, the clouds held it together for what will hopefully be a really great shot from a balcony above us.

We hitched a ride with a couple that we’re deeply fond of and rarely get to spend quality time with, so that turned into a surprise pleasure. One minute after we got into the reception area (a wonderful air-conditioned tent on the veranda of a beautiful country club in the outskirts of Williamsburg) it started to pour. We were very lucky. Others needed umbrellas to make their own luck. 😉

Appetizers and drinks were served while people selected which table to sit at. We gathered with a group of Richmond-based friends and the merriment began immediately.

I was honored to be asked to introduce the wedding party upon their arrival. At roughly 6:50pm I was alerted that my duties would be discharged shortly. I waited patiently on the dance floor by the DJ, microphone in hand. At one point there was a possibility that I would also be introducing the parents of the couple, but that didn’t happen. Here’s what I would have said had the opportunity presented itself:

It is clear that the love shared between these two couples, for each other and for their wonderful children, is a model that our new couple will follow. Guided by their parents, they were destined to find each other, sharing a faith that is truly inspirational. Please join me in welcoming the proud parents of the bride and groom!

Those words didn’t get spoken that night, but they are memorialized here. 🙂

Here’s what I did say:

My name is Hadar Pedhazur, and I have two distinct privileges tonight. The first is that of being godfather to Laura, which is what allows me to enjoy the second privilege, of announcing the wedding party.

And, of course, then I announced them, followed by being the first to call the bride and groom “Mr. and Mrs.” (at least the first with a microphone in his hand!). 😉

Hadar Introduction

Here is a grainy photo of them arriving, waiting for me to call out their names:

Wedding Party

Here is the happy couple, introduced by me as “Mr. and Mrs.”:

Happy Couple

When I was done, the father of the bride took over. His speech was very moving, ending with everyone joining hands and being led in a wonderful blessing by him. He could have been a Minister, had he so desired!

On to the festivities. The food was served buffet style. It was the first buffet of this size that I attended that was run rationally. Each table was told when to get up so that we didn’t have wrap-around-the-block lines. It was quick and painless to fill your plate. The food was outstanding. Kudos to the kitchen staff for preparing a delectable feast.

Many people traveled great distances to attend. The groom’s family came from California. The bride had family from Nebraska and Texas. None of that matched the trip undertaken by one of the bride’s cousins (uncle to the Junior Bridesmaid above). He came from Capetown, South Africa, with his 14-month-old daughter (her first trip to the US). She had the distinction of both being the youngest guest, and the one that traveled the furthest. She was an angel in every possible respect, and was likely the best behaved person (adults included!) at the wedding and reception. 🙂

Lois Olivia

Immediately after eating, I got to catch up with a number of incredible people who we see all-too-infrequently. That was another blessing associated with this wedding, that it brought together all of the people that hold this couple and their extended families so dear. A breath of fresh air to collect so many nice and extraordinary people under one roof (OK, tent) for such a happy occasion.

Finally, the dancing. Of course, the youngsters were on the dance floor the rest of the night, non-stop. Many of the older crowd were cutting up a rug as well. Normally, you can’t pay me to get on a dance floor, but when they called up every married couple, it was hard to pretend that I wasn’t. 😉

So, even I danced with my lovely bride:

Lois and Hadar Dancing

Here is a photo of the Groom’s landlady, dancing with one of the groomsmen. He had trouble keeping up with her, I kid you not!

Landlady Dancing

Of course, one of the more important dances of the night, the famous Father/Daughter dance. Unfortunately, another grainy photo (sorry folks):

Father Daughter Dance

One Father/Daughter picture deserves another. The father of the groom has two lovely daughters, both of whom were bridesmaids:

Father and Daughters

Since we’re showing off our pride and joy(s), we may as well complete the scene with two photos. The first is of the proud godparents (us) with the bride (our goddaughter), groom and our godson. The second is with our godson only, mostly because it’s a much clearer picture of the three of us:

Proud Godparents

Proud Godparents Clearer

There were three toasts given to the happy couple. The first by the Maid of Honor (a good friend of ours too). The second by the Matron of Honor (another good friend of ours, whose wedding we attended just last summer!). Finally, the Best Man (and father of the groom!) spoke.

All three speeches were moving and captured the spirit of the bride and groom beautifully. That said, because they were so moving, they weren’t easy to get through. Both the Maid and Matron of Honor broke down, multiple times. I was impressed that the bride kept it together as well as she did! The father of the groom kept it together a drop better, but it was a struggle for him as well, as he was bursting with pride and love for his son.

A very grainy photo of the Best Man (father of the groom) giving his toast:


After significant additional merriment, we finally said goodbye to the bride and groom, by forming two lines and giving them a Sparkler Sendoff (exactly as we did with the Matron of Honor’s wedding the year before). Last year, someone put a still-lit sparkler into the bucket of fresh (unused) sparklers, creating a gigantic blast and flame. No one made that mistake this time around. 🙂

Sparklers Canopy

The mother of the bride leading the sparklers farewell:

Mother of the Bride

Since we hitched a ride over, we needed to hitch a ride back. We were less fortunate this time, and ended up splitting up, each taking one empty seat with people who were kind enough to put up with us.

Back at the hotel, we topped off this most extraordinary evening even more so. First, our room ended up being a temporary gathering place for most of the wedding party, as they prepared to do stuff to the couple’s car (I didn’t want to know the details). 😉 Having energetic young folk around keeps us young (at heart at least), so we both love that.

Shortly after they left, the parents of the bride stopped by our room to finally breathe a well-deserved sigh! Their son (our godson) joined as well, and the five of us just quietly basked in the glow of a perfect ending to a perfect day together.

The next morning was filled with lots of present stuffing in our car. We drove most of their presents to NY yesterday, since they’ll be living in the same building as us. We then spent the next 7.5 hours in the car, and were very happy to finally see our own bed, after being on the road for two weeks!

Congratulations Laura and Chris, we couldn’t be happier for you, and we can’t wait to see you in NYC next week as you kick off the next phase of your life together! 🙂


Send to Kindle

The wedding weekend is upon us. In fact, the wedding itself is just over two hours away.

Yesterday was a delightful day. We’re staying in Williamsburg, VA (where the happy couple both went to school at William & Mary). At 10am, 25 of us (both sides of the family) drove the 10-15 minutes from the hotel to the Historic Jamestown site. Our host, the father of the bride, arranged for a private tour guide who took us through the settlement and did a wonderful job of bringing life in the 1600’s alive in the present. We have nothing to complain about nowadays…

After the 2.5 hour tour, we dispersed for lunch. I chose to spend the rest of the afternoon in bed (I’m exhausted). I didn’t really nap, but I definitely rested. After a shower, I joined a much larger group for the rehearsal dinner at The Blue Talon restuarant. A great meal and a great time (including a wonderful compilation of photos of the couple). I had the Lamb Shank, it was yummy.

It was raining pretty hard during dinner so Lois and I decided to skip the fireworks and head back to the hotel. We walked. The rain had stopped, but after one block, it started coming down again, accelerating into a full downpour for the last few blocks. Lois wasn’t happy with me, but I didn’t really mind bonding with nature for a few minutes.

Unfortunately, even though the room and the bed are very comfortable, I didn’t get any meaningful sleep last night. Nothing I can do about that now. After a wonderful buffet breakfast in the hotel with most of the men-folk in our group, I went back to our room to catch up with email. I didn’t turn the laptop on at all yesterday!

The hotel charges for WiFi (through Wayport). Rather than giving 24 continuous hours of access, they give access until Check-in time. So, if you start at 10am, you get only five hours of air time before having to pay again. Not interested. I overpay for the privilege of rarely using my cell phone as a high-speed modem for my laptop. I’ve been logged on for five hours in a row now, through my cell phone. I am getting nearly 800Kbps downstream, and 130Kbps up, so I have no complaints (many hotels are much slower over their shared WiFi).

I received two phone calls on my cell, and the connection to the Internet wasn’t dropped (I was flabberghasted!). My connection dropped once (the normal sound I hear when my cellphone has a weak signal that it reacquires), and one click on the Sprint Connection Manager software reconnected me. I am certainly delighted with the ease and reliability, if not with the price of my insurance policy.

I am (mostly) caught up now, and once I log off to attend the wedding, won’t be back on until tomorrow evening, since we’re heading out tomorrow morning for NY.

Update: Now that we’re home, Lois uploaded her many photos. I’ve selected a few to support my tale. 🙂

I spent most of the social time at Jamestown chatting with the groom’s grandfather. He is a fascinating man and I thoroughly enjoyed every second that I spent with him. Thanks Bill!

Hadar and Bill

Here are a few Indian Carvings of the Powhatan Tribe:

Indian Carvings Jamestown

The youngest member of our 25-person-strong brigade was a 3-year-old. I’m thinking that she didn’t think that living in a hut full-time would be all that bad. 😉

Indian Hut

Part of our group about to board the lead ship in Jamestown:

Jamestown Ship

Our littlest one learning to construct things the old-fashioned way, inside the Jamestown Fort:

Jamestown Fort

Topping off the day, here is a photo of the bride-to-be herself (now, officially married an on her honeymoon!), cutting me a slice of her world-famous apple pie! It went amazingly well with the chocolate Groom’s cake with ice cream. 🙂

Apple Pie