Technology

Asus Wifi Router AiMesh Odyssey

Send to Kindle

This post was rattling around in my head before the current stay-at-home situation arose. Somehow, it never found its way to my fingers, until now.

This will be a typical techie post by me, in that it will be long, rambling/meandering, and likely bore the people who used to mostly read my music blogs.

tl;dr

Most newer (even years old) Asus routers now support AiMesh, a method of turning normal routers into a mesh system. When configured correctly, it works remarkably well and has some serious advantages over pre-built mesh systems (like Google WiFi, Eero, etc.).

My start with Asus Routers

For years, I purchased cheap (often refurbished) routers. I typically flashed them with dd-wrt firmware to make them better than new.

On December 13th, 2013 I broke the mold. I purchased my first high-end router, the Asus RT-AC66U. It was my first “AC” router (now called WiFi 5) and the first time I ever spent that much money on a router of any kind ($179.99 before tax!).

It has a (theoretical and nonsensical) top speed rating of 1750 megabits per second (mbps). That’s still pretty good even by today’s standards.

Back then, this router was glorious. I was very happy with my purchase and it was my main router for many years. It only got replaced when I switched to Verizon FiOS which at the time came with their own branded router.

This router still gets firmware updates (what a credit to Asus!). More on that later in this (happy ending) saga.

The big move

In 2015 we moved from NY to VA. Our house in VA is large, but not that large. The layout puts our master bedroom far away from every other room (no centrally placed router would reach that room).

Mesh routers hadn’t exploded in popularity yet, so I never considered one at the time. Given that I was installing FiOS there (the previous owner had FiOS as well), I knew that my main router would be a Verizon branded one.

I had an electrician put in a hard-wired Ethernet port in three rooms, in addition to the closet where the FiOS router lives. There is an Ethernet port in the wall in our office, the master bedroom, and the basement (where the TV is). Each of the three rooms is a home-run back to the closet, so each of the rooms connects into one of the four LAN ports on the back of the FiOS router.

The office is very close to the closet with the main router in it, so to begin with, I didn’t put another router in there.

The master bedroom and basement each got routers. Those routers were put in AP mode (Access Point, or Bridged). That meant that the main FiOS router still handed out all IP addresses (no matter the room or which WiFi router our devices were connected to).

For simplicity, each of the AP routers got a separate SSID (the WiFi Network Name). Devices that were fixed in a particular room always connected to their local SSID/Router (e.g., the TV, or a Fire Stick/Roku, etc.). Our cellphones had to switch to a new SSID when we moved from one room to another.

Since the main router signal really doesn’t reach the master bedroom at all, the phones would switch pretty quickly (without any manual intervention), simply because they would lose the signal completely and immediately begin searching for a new signal.

It wasn’t instantaneous, so you would have to think about starting something before you walked out of a room, because the connection would definitely drop for a few seconds.

After a few months, the FiOS router started getting flaky on the AC channel (this eventually straightened out years later after some arbitrary firmware update). This led me to pull out my old Asus RT-AC66U router (which was in a drawer for a while by then) and install it in the office.

Even though the office was close to the main router, I could no longer trust the WiFi on the main router, so I used the wired connection in the office to hook up the Asus as an AP and all was well again.

Growing the Asus family

The above setup continued for a while and worked fine. In the master bedroom I had an old TP-Link router (another brand that I really like). It worked fine, but wasn’t the fastest thing and would rarely cause me tiny frustrations.

Six weeks before buying my first Asus router, I bought a Nexus 5 phone from Google. That marked my switch from Verizon (where I was using the Galaxy Nexus) to T-Mobile. I’ve been a happy T-Mobile customer ever since.

My house had effectively zero coverage on T-Mobile when we moved in (it’s marginally better now, though I rarely ever have to use it in the house). After a few months in the house, I saw an ad for a T-Mobile Cellspot. It’s a rebranded Asus RT-AC68U router (one step up from the 66U that I owned).

T-Mobile was selling them very cheaply (much cheaper than the Asus branded 68U). I bought it not only because the price was excellent (if memory serves me well, I think I paid $75 for a new one, when the 68U was going for $199 new), but also because it was (supposedly) optimized for T-Mobile WiFi calling (which given my cell signal at home sounded like a great addition).

I replaced my (rarely) flaky TP-Link router with a T-Mobile Cellspot one in the master bedroom.

The dreaded, complicated upgrade

The new router ran great for months, but I never used the WiFi calling feature (that’s a long story for another post that I will likely never write).

At some point I stumbled on a post about “Turning a T-Mobile Cellspot into a Full Blown Asus RT-AC68U“. If you skim that article, and more importantly the tons of comments with a lot detail in them, you’ll see that this is not for the feint of heart.

Fortunately or otherwise, this is exactly the type of thing I enjoy (hey, you get your kicks your way, I’ll get mine my way…).

With some twists and turns, and needing to read a bunch of comments when I got stuck, I eventually turned the Cellspot into a full blown Asus router.

Why bother?

That’s a fair question. The Cellspot worked perfectly (for some definition of perfect). There are a few good reasons to consider the upgrade. By far the most important one is that the Cellspot firmware was way behind, and while occasionally upgraded, pretty rarely.

That meant that when the various kracks that have been discovered in recent years against WiFi routers are revealed, a real Asus router will be patched months (or years!) before the Cellspot will. That’s reason enough to bother if you don’t require any special handling of the WiFi calling feature for T-Mobile specifically.

Another reason to upgrade is to flash different firmware, e.g., dd-wrt (mentioned above). That’s not possible from the base Cellspot firmware, but is from the Asus firmware.

Finally, if you want to run the AiMesh software, you need to be on the real Asus firmware.

A quick look ahead

I’ll deal with this later on, but there are newer (shorter, better) ways of upgrading a Cellspot, and very important warnings and caveats which didn’t exist at the time I upgraded my first one.

The point of this interlude is to tell you to read on rather than follow the instructions in the article linked above.

Another Asus added to the family…

A year later, my better TP-Link router running in the basement started to have some issues (again, I think it was a firmware issue that I later resolved). I decided to replace it with another upgraded Cellspot.

I bought one refurbished on Amazon for $48. I followed the same upgrade instructions linked above, and had another working Asus RT-AC68U router installed in the basement the same day it arrived.

I now had four routers in the house. The main FiOS one which was mostly acting as a wired router to the Internet (a few legacy devices, security cameras, etc., were connected directly to the 2.4Ghz channel on that router), plus the original RT-AC66U in the office, and the two upgraded Cellspots, in the master bedroom and basement.

What? Asus doesn’t make infinitely perfect hardware?

About six months ago, I walked into the office and the old 66U router was dead. No lights, no Internet (obviously).

I disconnected the cables and pulled out the TP-Link Archer C9 that had previously been running in the basement. That’s the one that I asserted was flaky only because of a specific firmware.

I reconfigured it to take the place of the old 66U, made sure it was current on firmware, and turned it on. Problem solved, we were back in business.

I decided to try and diagnose what went wrong with the old 66U (just out of curiosity, as it was 6 years old at the time and didn’t need to provide any additional service to make it one of the more outstanding tech purchases).

I connected it directly to my laptop using the wired port and fired it up. The lights blinked for a second and then went dead. It only took me one more try to realize what was wrong. The power button was broken. It simply wouldn’t click and stay on.

In typical MacGyver mode, I found a round hard piece of plastic, scotch taped it on to the power button, then put a rubber band around that to ease the pressure on the scotch tape.

Voila! A working 66U router, once again…

I swapped it back for the TP-Link, which was now perfectly configured to be an instant backup router should my MacGyver skills prove unworthy.

Wasn’t this supposed to be about AiMesh???

Oh yeah, though I did specifically mention that this would meander and ramble and I didn’t want to disappoint on that front either…

Unfortunately, a bit more meandering is necessary, just for historical accuracy, not to discuss the merits of AiMesh.

Getting into trouble

Before getting the second Cellspot router, I upgraded the first one using the built-in Asus firmware upgrade tool once, and it worked great.

When I got the second one, I (of course) upgraded it to the same version of firmware as the first one was on, with no issues.

Toward the end of last year, another krack was discovered, and I checked whether Asus had an updated firmware to mitigate it. They did.

I updated the old 66U first, and it upgraded perfectly.

I updated the first 68U and it reverted back to the original Cellspot firmware (which had even more issues than I was currently trying to fix!).

Whoah, what just happened?

A bit of Googling and I found that Asus decided that if they noticed that a Cellspot router was being flashed with Asus firmware (rather than a T-Mobile branded firmware), they would roll it back to the original.

Darn!

Silver Lining?

This caused me to find newer methods of turning it back into an Asus router, including ways to thwart Asus from rolling it back. The old method (linked above) still works, and has the appropriate warnings and methods to avoid the rollback, but it’s still more complicated than the new ones.

This is a link to the instructions that I used the second time around. It looks long and complicated, but that’s because there are three different (analogous) methods for accomplishing the upgrade. The key point is that this avoids the full downgrade which the original method requires.

When I did this and got my first router back to the new firmware, and made sure that it wouldn’t downgrade again, I flashed the second one as well, apparently successfully.

Apparently???

Wait, what? Either it worked or it didn’t. Well, yes (and no).

I was so smart (being a seasoned techie) that I was incredibly stupid (being a seasoned know-it-all).

The first time that I upgraded each router, I meticulously followed the dozens of steps in the original article linked above. Amazingly, following detailed instructions worked (can you believe it?).

This second time around, using the simplified instructions (which are 100% accurate and would work if you followed them exactly!), I skipped one crucial section (a few commands) because I assumed that they were unnecessary the second time around (meaning, I thought I had the correct files from the first time around just sitting in my folder waiting to be reused).

Again, why apparently then? Because I don’t end up using the router in the basement all that often, and it took trying to get AiMesh to work (still coming, I promise) to finally see what I had done so wrong…

AiMesh, finally!

Well, actually, getting closer, not there quite yet…

Now that everything was running fine (or so I thought), I decided to finally experiment with turning on AiMesh in all three Asus routers.

I really didn’t need it, my setup was working well enough, but I was curious and now that I was running the latest version of Asus firmware on all three routers, I was in a position to find out. I could always roll back to non-AiMesh mode if it wasn’t to my liking.

Unfortunately, I hit a snag immediately. It turns out that the old 66U is not capable of running AiMesh software. There is a newer revision (RT-AC66U rev B) that can run full AiMesh, but mine is too old and can’t do it.

So, I popped on to Amazon and ordered another Cellspot for $48 (this time, it was labeled renewed rather then refurbished).

Unfortunately, I compounded my error by skipping the exact same block of steps on this newest router as I had on my others, because I hadn’t yet noticed the problem that I had introduced into my network…

Turning on AiMesh, finally, really, this time

I flashed the real Asus firmware onto the newest Cellspot and retired the old 66U once again. I was now ready to flip the switch and turn the three routers into an AiMesh mesh network.

All attempts to get them talking to each other failed! After some searching, it seemed that some people had more success using the Asus Router App on their phones, than using the web browser interface.

I broke down and installed the app on my phone. I did seem to get a bit further, but if I got something going on one router, another would disappear, and then that would be flipped. It was maddening.

Discovering the problem

Well, the problem was entirely created by me, so I was the problem. The crucial steps involved the following:

Upload original_cfe.bin to https://cfeditor.pipeline.sh/
Select 1.0.2.0 US 1.0.2.5 US for AC68P or 1.0.2.0 US AiMesh for AC68U with AiMesh as Source CFE
Download the new .bin
rename it to new_cfe.bin

I assumed that having done that once on the original router, I had the correctly modified CFE (now called new_cfe.bin). Meaning, I thought that all Asus routers (at least of the same model number, which mine were), shared the same identical new cfe.bin file.

You’ve all heard what the definition of the word assume is, right? I’ll spare the gentler ears/eyes from seeing it here again…

It turns out that the file is unique to each and every router. Why? Because among other things, it contains the MAC Address of the routers ports (both Ethernet and WiFi) embedded in it. So, by reusing the same cfe.bin file on all three routers, they were all running with the same exact MAC Address.

To be clear, they each had different IP addresses assigned, but that doesn’t make the problem better. The way local networking works, there is an ARP table maintained that tells the network how to reach the physical machine associated with an IP address, by translating it into the MAC Address.

So, when I tried to reach any of the three routers via their IP address, all of them returned (at a very low level) the same MAC Address, and therefore it was entirely random (perhaps based on distance in the house) as to which router would see my request!

Ugh.

The solution

Once I understood the problem, the solution was obvious and straightforward, but by no means simple. I needed to fix the individual cfe.bin files, but I could no longer follow the original instructions (uploading them to a website which would edit them) because I didn’t have the original files to upload!

Worse, I needed to figure out which MAC Address was correct for which router, which meant going to each of them and finding the stickers with the serial numbers and MAC Address printed on them.

Once I did that, I had to use a HEX Editor to load up each file, find the wrong MAC Addresses (yes, plural, since there are multiple interfaces in each router) and type over them (very carefully).

Then I needed to copy them over to the correct router, flash them, reboot the router, and pray.

Yes, that worked!

Are we there yet?

So, was I really done? Unfortunately, not quite.

I was able to get AiMesh going, but the speed in the bedroom was pathetic (reliable, but pathetic). The speed in the basement was great!

This one didn’t take long to diagnose, but it did take a while to fix…

By default, AiMesh sets the backhaul (how the mesh routers communicate with each other, rather than with the client devices or the Internet) to auto.

In the case of the basement router, that ended up using the wired connection over the Ethernet cable (which is exactly what I expected the default to be).

In the master bedroom, even though the router is fully wired like the basement one is, auto defaulted to wireless backhaul.

If you recall from a few days ago, when you started reading this post, the master bedroom is too far away to get a reliable signal, so the backhaul was awful (amazing that it worked at all!).

The solution is simple, force the backhaul to be wired. Yes, simple, in theory, but I couldn’t find any way to do that!

More searching on the Internet and I finally found a single forum post where someone linked to the official guide with highlighted screenshots.

Bless that individual, and Google, for surfacing the correct post (after much tribulation).

Here is a link to the guide, with step 6 being the secret sauce to finally see where the default backhaul could be changed/

Conclusion

So, was it all worth it? Yes, of course.

First, I love technology puzzles, even ones created by me. Once I screwed up the settings really badly, I just had to figure out how to get myself out of it. It wasn’t fun (on any level), but it was instructive, informative, and satisfying (in the end).

Much more importantly, I am now running a full mesh network and I like it. Our phones don’t drop when walking from our office to our bedroom. All three routers are effectively managed from the one main AiMesh one.

Why AiMesh is really cool

Most importantly, it’s a mix and match network. You don’t have to buy kits. You don’t have to have identical routers at each node. As long as a router supports the AiMesh firmware (which many Asus routers do!), it can be a node (or the master) of your AiMesh network.

This is crucial. Today, I don’t own a single WiFi 6 (AX) device. So, it would be overkill for me to buy a WiFi 6 router, let alone a WiFi 6 Mesh Kit.

However, if/when I get a new laptop (I’m typing this on a 6-year-old one) that has WiFi 6 in it, or a new phone (mine is 2.5 years old), I’ll be able to get an Asus WiFi 6 router (any of them!) and use it as my main AiMesh node (and place it wherever I use the laptop most frequently, which now, is in the office).

I won’t have to change the other routers, or change any settings on the other routers either. They will all just work. My laptop (and phone) will work with WiFi 6 when they’re connected to the new router, and automatically and gracefully downgrade to WiFi 5 when they roam to another AiMesh node that’s still on WiFi 5.

Further, I can even do the WiFi 6 upgrade piecemeal. For example, I could get a lower-end WiFi 6 AiMesh router first, and make that the master. Then, as I have more devices that can take advantage, I can get a higher-end WiFi 6 router, make that the main one, and move the older WiFi 6 router into the bedroom.

The ultimate beauty is that each of the routers can always be instantly returned to be non-AiMesh routers. So, I can pass them on to friends when I replace them with a WiFi 6 one and those people can use them as standalone routers, AP bridges, or create or augment an AiMesh of their own.

That’s what makes these more flexible than full-time mesh systems.

Also, it doesn’t hurt that the models I’m running can be picked up for $48, if you’re willing to be super careful and avoid the stupid mistake that I made.

Starting a new life, Post Zope Corporation

Send to Kindle

Starting a new chapter is a common expression. Lois and I have had a different expression to describe the transformations we’ve experienced as a couple and as individuals, over the 30 years that we’re together. We say we’ve lived many lives. We’re about to start a new life (to us, not just a new chapter in an existing one).

In 1997 I contacted Digital Creations, a small Python shop in Fredericksburg, VA, to ask if they were looking for an investment. I was looking to put some personal money into an innovative software company. We hit it off, but I was new and naïve about angel investing and they were overly-cautious and the deal never happened.

I fully credit my involvement with that initial failure for giving me the taste for technology venture capital. A couple of months after our deal disintegrated, I made my first investment. That turned into me founding a venture capital firm. I never lost my interest or desire to be involved with Digital Creations. I approached them again in 1998. Digital Creations ended up being the fourth investment in my new VC firm.

In 2001, we renamed the company Zope Corporation, to reflect the success that our product, Zope, had in the market. Most of the people reading this will probably have never heard of Digital Creations.

Also in 2001, the company repurchased the shares of two venture capital firms that invested in late 2000. That left the company with limited financial resources in a very difficult time. Lois and I dedicated ourselves to operational roles to help rebuild the company. We believed that we were making a one-year commitment to working at/with/for the company. Here we are, nearly 11 years later.

Back in 2001, it was my idea to make the offer to buy out the other VCs. This time, my old playbook was employed by rest of the management team, who felt that they were in a position to make me a similar offer. After working out myriad details, a deal was struck. The transaction closed yesterday afternoon.

Lois and I are no longer involved with Zope Corporation, after it was the central part of our working lives in our most recent past life.

We wish them continued success in everything they do, as a company and as individuals who we have known and loved for a very long time.

A natural question is “What will you do in your next life?”. The honest answer is I have no idea. After all, I’m a newborn again. I have some time before I’m expected (or able) to walk. Come talk to me when I’m crawling and we’ll see if a plan has formed.

Installing CrashPlan on a PogoPlug Pro

Send to Kindle

There are critical updates embedded in this post, added on 9/3/2011, all preceded with Update:. You can apply those instructions separately if you’ve already completed the rest of this installation. Updates will be marked with End Update to allow for multi-paragraph updates. I will strike-through the parts that were replaced, so that you can safely ignore them if you’re going through this guide for the first time.

PogoPlug Pro is an amazing device (coupled with an amazing service). CrashPlan is an amazing piece of software (and also provides a fee-based amazing service). I’ve had both for a while and think very highly of them.

To solve a number of my own problems (not caused by either service!), I decided to investigate marrying them (the PogoPlug device, with the CrashPlan software). To be slightly more accurate, I wanted the device to perform an additional function (beyond backups). I wanted it to be my primary DLNA server.

Caution: none of what follows is supported by either company. You will be voiding your PogoPlug warranty and CrashPlan does not support this configuration of their Linux software. Proceed at your own risk!

My primary reason for installing CrashPlan on this device is to compensate for the pathetic upload speed provided by Time Warner, all of 485Kbps, shared with my wife, for normal Internet use, VoIP calls and backups. In other words, not really a useful real-time backup solution. Since we are often at other, very high-speed locations, I still believe that paying for the CrashPlan online backup service is the way to go (and gives me great comfort), but when I’m home, I wanted a local solution (that didn’t involve plugging in a hard drive to my laptop).

Since I was able to do this, successfully, the instructions can be found on the Net. Since it took me way longer to find the various pieces, let alone get them to work, than I felt it should have, I’m writing this (for myself, as well as for others who might give up more easily than I did). None of the credit goes to me, I’m just collecting the wisdom of others in one place, hopefully an easy one to find.

Update: All sections marked Update: that apply to Java and udev were courtesy of Ankit Mohan. Ankit used this guide to get CrashPlan running on his PogoPlug, then dug in a lot more than I did to solve the problems I described. I am indebted to him. End Update.

There are a number of Linux distributions available for the PogoPlug Pro (an ARM-based device). I chose Arch Linux because it was the most prominent one I found and because it has a very good reputation independent of the PogoPlug. The specific ARM implementation has it’s own site, which is where I started my odyssey.

After reading the overview on that page, I clicked on the Installation tab. The instructions there are extremely clear. The first time I followed them, the formatting of my external drive failed. I ended up formatting the drive while it was connected to a laptop running Linux, but all of the other instructions on that page worked. I will repeat them here, so that anyone who doesn’t want to link off of this page can simply follow all the way through.

You have to register your PogoPlug at my.pogoplug.com. This is required, even though we will shortly be disabling the service, since this is the only way to get SSH access to the device. You will be able to reverse the process, returning the device to a full PogoPlug in the future, should you desire that, but it’s not a dual-boot situation where you decide which version you want it to be.

Once you’ve enabled SSH on the device, you can set your own password. The username will be root. The default password for a PogoPlug Pro is ceadmin (as noted, you can change it via the website, or once you log in, with the passwd command. Change it!

One of the steps that they don’t cover is discovering the IP address of your PogoPlug, so that you can SSH to it. In a home environment, this is relatively easy (for the geeks among us). You login to your router, look at the list of attached devices and easily spot it.

I’m installing the software on a second device as I type along. I’m in an office environment and don’t have access to the router. There are hundreds of devices in the office. I had to write down the MAC address of the PogoPlug, go over to a system administrator and ask him to search the DHCP log files for that MAC address. He did and I found out the address that was assigned to it. Whew.

I successfully logged into the device, just to make sure it worked, while it was still an official PogoPlug. That step was optional, but comforting. Since the next step is to power down the device (which you can safely do by just pulling the power cord, especially if you have no hard drives attached yet), since I’m logged in as root, I typed /sbin/halt instead, to be a little safer. Wait 60 seconds (for added safety), then pull the power cord.

We’re going to install Arch Linux on an external drive. The only thing that will be changed on the PogoPlug itself is the boot sector, which will now point to the external drive (that’s what would need to be reversed to restore the PogoPlug functionality).

With the PogoPlug powered down, attach only the drive that you intend to install Arch Linux on. This way there will be no confusion or errors. Later on, if you want multiple drives attached (for backups and/or media files) it will be easy to add them. I am using a 2TB Fantom Drives to do my install.

Once the drive is attached (and turned on), plug the power cord back into the PogoPlug and wait for it to boot. Then SSH back on to it (the root password will be what you set it to, or ceadmin if you didn’t change it). The box is still running the PogoPlug software, with your drive attached to it.

Type: killall hbwd

That will stop the PogoPlug software from running on the box. We don’t want it to interfere with the installation of Arch Linux. You might have to wait a bit for the service to stop. If you want to check, type the following:

ps | grep hb

The only result should your grep process. Then, you can type:

/sbin/fdisk –l

The last line of output should start with /dev/sda1. That means your disk drive was found and has a partition on it (it’s likely formatted already). We are about to erase everything on the disk, so be absolutely sure that you want to continue with this adventure before doing that! If you’re ready, type:

/sbin/fdisk /dev/sda

That will bring up the fdisk program on the entire drive (sda as opposed to sda1 which is the first partition). You will now have a prompt that is directly from the fdisk program. We will type a number of one character commands. Right after you type the character and press enter, fdisk will go off and do what you asked it to.

Type: o

That will clear the partition table so that the disk will become unusable (for the moment). As you can see from the messages, nothing has been committed in stone as yet (very soon). This has modified an in-memory copy of the partition table.

Type: p (to verify the above, that there are now no partitions)

Type: n (press Enter, this will create a new partition)

Type: p (to make it a Primary partition. At this point, I’ll stop saying “Press Enter”, but you still have to!)

Type: 1 (to make this partition #1)

At the next two prompts (First and Last cylinders), just Press Enter to accept the defaults (you are making the entire disk available as the first and only partition).

Now comes the destructive part. This will actually wipe out any data you had on the disk (but still doesn’t modify the PogoPlug in any way!).

Type: w (this writes the partition table back out to the disk)

You are now back at the command line. If you’re a paranoid type (or just careful), you can verify that things worked by repeating the fdisk command and listing out the partition table, all in one shot:

/sbin/fdisk –l

This is the output on my system:

Device Boot      Start         End      Blocks  Id System
/dev/sda1               1      243201  1953512001  83 Linux

2TB, in one partition, marked for use by a Linux system. Now we need to actually create the filesystem, which in our case will be an ext3 one. This will require downloading some commands that will need to be executed. Here are the steps:

Type: sync (to flush any filesystem changes)

Type: cd /tmp (to change to a temporary, writable working directory)

Type: wget http://archlinuxarm.org/os/pogoplug/mke2fs (to retrieve the program mke2fs)

Type: chmod 755 mke2fs (to make it executable)

Type: ./mke2fs -j /dev/sda1 (the leading dot is critical. This will format the partition to an ext3 filesystem)

The above command can take quite a while, depending on the size of your external drive. This is the command that failed for me on my first PogoPlug, so I ended up having to detach the drive, connect it to a Linux laptop, perform the same exact command as above (which was already available, I didn’t need the wget part) and then reattach the drive to the PogoPlug.

Note: it failed for me again. I was able to format it using the built-in /sbin/mkfs.ext2 command (passing in the “-j” flag), but I didn’t trust that it was building a true ext3 filesystem (ext2 + journal). So, I disconnected the drive from the PogoPlug, attached it to a VirtualBox VM on my Win7 laptop, and formatted it there as a real ext3. Took forever, but it worked.

Whether the mke2fs command worked for you or whether you had to format the drive separately, like I had to on two separate installations, you’re now ready to install Arch Linux on the external drive. You should already (still) be in /tmp, but to make sure, feel free to type: cd /tmp

Type: wget http://archlinuxarm.org/os/oxnas/oxnas-install.sh (that retrieves the install script)

Type: chmod 755 oxnas-install.sh (this makes the script executable)

Type: ./oxnas-install.sh (this starts the script, which will send lots of messages to your terminal window. It also downloads the root filesystem image, which is roughly 129MB, so it can take a while if you don’t have a fast network connection.)

When it’s finally done (took between 5-10 minutes on a very high speed connection in the office), the output should look like this if it succeeded (at the very end):

#############################
## Looks good!
# Sync …
# Unmount
# Reboot to enter into Arch Linux ARM

Note the looks good! and then the instructions to reboot. That’s what we’re going to do next.

Type: /sbin/reboot (cross fingers!) Winking smile

This will immediately disconnect you from the terminal window you were in. You need to wait a few minutes for the orderly shutdown of the PogoPlug, followed by the booting up of Arch Linux ARM. You can watch the lights on your external drive to see when there is activity on it, indicating that the booting has begun. When the light settles down, the boot is complete.

We’re ready to log back in (this time to the new operating system), and the password has been changed to a new default one. The user is still root, but now so is the password (root), which you should change right away with the passwd command. It’s quite possible that the IP address of the box has changed during the reboot, so please verify the new (or existing) one, before SSH’ing back on.

If the IP address did not change, then you might have to remove the old key associated with it, or ssh might refuse to connect, thinking it’s a security violation. If you get the same IP address again, you may need to run the following command first (on your local machine, the one you are SSH’ing from):

Type: ssh-keygen -R 192.168.1.123 # (using your device’s IP, which won’t likely be 192.168.1.123)

The following First Steps page on the Arch Linux ARM Wiki explains the above, and gives you a number of other useful tips. I followed them religiously the first time through, but I changed the order a bit this time around and it saved a bit of typing (or I think it did).

Instead of going through the above, this time I updated all of the packages right away. I believe that it installed openntpd and updated the /etc/rc.conf file (one of the first steps that I performed manually the last time). You can do what I did, then check if openntpd is installed and running.

Type: pacman –Scc (clear out old packages. I said YES to the first, and NO to remove unused repositories)

Type: pacman –Syu (this will do a large update, first syncing the repositories, then updating all packages)

Now comes a crazy part. I say crazy because by the time you read this, perhaps the maintainers of Arch Linux ARM will have updated the repositories and this will no longer be necessary. Then people will think I’m an idiot, so be it, I’m putting it in here because it can’t hurt!

Type: pacman –Sy udev-compat (to fix a problem with udev + syslog-ng taking up 100% of your CPU)

The 100% cpu problem might be happening as you read this (if you’ve done the previous steps already). It might be filling up your disk in /var/log as well. We can check that in a minute (here’s the thread that helped me: there’s a typo in that thread, “sleep3” should be “sleep 3”), but first, let’s do a few more things and then reboot.

Update: Type pacman -Sfy udev-oxnas udev-automount (this fixes the udev problem noted above, now struck-through. I added the f option to pacman to force the removal of the bad udev, or udev-compat that you installed if you’ve already completed these instructions. You will have to say Y to the prompt to remove udev, which defaults to N.) End Update.

Let’s create a swapfile:

Type: dd if=/dev/zero of=/swapfile.img bs=1M count=1024 #for a 512MB swapfile, use count=512

Type: mkswap /swapfile.img (to turn the file we just created into a valid swapfile)

Type: swapon /swapfile.img (to see whether you get any errors, you shouldn’t)

Now we’ll edit /etc/rc.local (use your favorite editor, I use “vi”, you might prefer “nano”) to add exactly four one lines after the comments:

swapon /swapfile.img
kill $(pidof udevd)
sleep 3
udevd &

The first line turns the swap on after each reboot. The next three lines kill the bad udev (even after updating the udev-compat the process sometimes misbehaves at boot), then sleeps for three seconds and restarts udev, which makes it seem to behave correctly so far.

OK, time to reboot and see if we have a stable system:

Type: sync (flushes the memory to disk)

Type: reboot (you should lose your ssh connection right away)

Wait until the disk activity settles down a bit, then ssh back in.

Type: top (to see what processes are running. If udev and/or syslog-ng are at the top, something isn’t good)

Type: q (to exit top, whenever you’re done looking around)

You can follow any of the additional instructions for setting up a user, adding sudo, changing TimeZone settings, etc. All are linked from the First Steps above.

Since the reason I did this was to install DLNA, I’ll cover that first (it’s really short), then we’ll move on to the heavier CrashPlan setup. Skip the next few lines if you have no interest in DLNA.

Type: pacman -Sy minidlna jack (this names two packages, but will install something closer to 44, with dependencies)

You should edit the file /etc/minidlna.conf and change any variables (where the files are stored, where to store the DB, what you want to call your DLNA server, etc.). You can read the Wiki page (linked above) to see the more important entries.

Then add the word minidlna at the end of the DAEMONS= line, which should be at the bottom of the /etc/rc.conf file. This will auto-start the DLNA server every time the PogoPlug is booted. To start it right now, type: /etc/rc.d/minidlna start

Update: I discovered two things. 1) Many devices, e.g. Google TV and Sony Bravia TV, don’t show any of the files with a filetype of FLV. It turns out that if you simply rename the filetype to AVI (probably others), the files play fine (assuming they were encoded with H.264). 2) After you populate your media directory, run the following command and wait patiently: minidlna -R to force a build of the database. You probably want to kill all minidlna processes when this is done and start it again (minidlna alone on the command line is good enough). You can tail the minidlna.log file (in your database directory) to know when the database rebuild is done. End Update.

Whew. Finally ready for the very tricky and long installation of CrashPlan. This is not for the faint of heart, nor is it in any way supported by CrashPlan. It works for me (and obviously others), but you’ll have to be the judge as to whether the hassle is worth it for you.

Let’s start with crediting the place that got me unstuck, the CrashPlan support forums! Kudos to CrashPlan for allowing this type of discussion on their forums, even though they don’t support this configuration. Here’s the thread, but all of the interesting bits are in the long comment by Torbjorn. It was really hard for me to find, because I was searching for the word PogoPlug. This solved the problem for Sheeva (the baseline of the Pogo), but it’s not quite identical.

Type: pacman –Sy openjdk6 cpio (we need to get a working Java installed and CrashPlan will require the cpio package separately)

The next step (according to Torbjorn) is to download an ancient (2005) libjtux source package, apply a patch and compile it. He supplies a pointer to the source (amazingly, still available) and the patch file is available as an attachment to his comment. You can grab both from the the thread linked above. If you do, you will likely have to download a bunch of development packages (using pacman), starting with gcc.

Instead, I will attach the completed libjtux.so that I built (following those instructions), to save you time, effort and potential errors. I also just grabbed it from my first build and applied it to my second, for the same reasons.

Now we need to install CrashPlan itself by heading to the download page for Linux. I right-clicked on the download button (currently version 3.0.3, but the software auto-updates after the first install). I copied the link location. Back on the PogoPlug:

Type: wget http://download.crashplan.com/installs/linux/install/CrashPlan/CrashPlan_3.0.3_Linux.tgz (that should work, but starting at the http part, just paste in the link you copied if it’s newer than 3.0.3)

Type: cd /tmp

Type: tar –xzf WHEREVER_YOU_DOWNLOADED_THE_CRASHPLAN_FILE (which could be /tmp to simplify matters)

Type: cd CrashPlan-install

Type: ./install.sh (all of the defaults seem reasonable to me, though I did put my archives in another directory. You will need to page through the license file with the space bar and accept that as well. The init scripts on Arch are in /etc/rc.d, which is the other thing I changed from the default.)

When this is done, it will report that it has successfully started the CrashPlan service. It did not. That’s because we haven’t yet replaced the libjtux.so that comes with CrashPlan. The problem is that it was compiled with an Intel i386 architecture.

Type: cd /usr/local/crashplan

Type: mv libjtux.so libjtux.so-ORIGINAL (no real need to save it, other than to memorialize the changes we’re making)

Type: cp WHERE_YOU_DOWNLOADED/libjtux.so . (this copies my version from wherever you downloaded it, or you can right click my link above, and wget directly from my website to this directory)

Torjborn mentions editing a file to add jna.jar to it. I didn’t need to do that, and the directory he references doesn’t exist. I think it’s from a different installation of Java (for the original Sheeva) and not necessary when using the openjdk6 package.

Update: This next part solves the timing delays, apparently among a number of other issues that I wasn’t even aware of! Once again, thanks to Ankit for figuring this all out.

You will need the nss package installed. Mine was there after the major update above. If you don’t have it installed, type: pacman -Sy nss

The file that’s missing from openjdk, which is causing all of the problems with CrashPlan, is libjnidispatch.so. It’s part of the jna package (Java Native Access). You’ll have to download a Debian package and extract the file.

Select a location near you from this link: http://packages.debian.org/sid/armel/libjna-java/download. In the US, I chose this link directly: http://ftp.us.debian.org/debian/pool/main/libj/libjna-java/libjna-java_3.2.7-4_armel.deb

Assuming you downloaded that file to /tmp, here are the commands to extract the file we’re interested in:

Type: cd /tmp

Type: ar vx libjna-java_3.2.7-4_armel.deb (this will extract three files, one of which contains the file we’re interested in)

Type: tar -xzf data.tar.gz (this will unpack the tar file, creating a series of directories in /tmp/usr)

First we need to create the target directory for this. Since mkdirs (make nested directories) doesn’t exist by default, we’ll execute a number of consecutive mkdir commands. Hint: you can hit up-arrow after each command and just append the next directory name in the series of mkdir commands.

Type: cd /usr/local/crashplan/lib

Type: mkdir com

Type: mkdir com/sun

Type: mkdir com/sun/jna

Type: mkdir com/sun/jna/linux-arm

Type: cp /tmp/usr/lib/jni/libjnidispatch.so com/sun/jna/linux-arm/ (now we have the file in the correct directory)

Type: cp -p jna-3.2.5.jar jna-3.2.5.jar-ORIGINAL (this isn’t strictly necessary. Aside from documenting our change, it allows us to recover from errors more easily)

Type: jar uf jna-3.2.5.jar com/sun/jna/linux-arm/libjnidispatch.so (this is the critical step, inserting the library into the jar file that CrashPlan uses. I’m not sure the com/sun/jna/linux-arm directory creation is necessary, but better safe than sorry)

It seems that in addition to solving the timing issues (long delays in starting up), this actually dispatches (duh) maintenance tasks like pruning, synchronizing, etc. It’s all good, but there is a side-effect (well worth it!) that the Java process now takes up a significant amount of CPU, even when backups aren’t active, since it appears to actively maintain the system. Previously, Java would consume 0% of the cpu when it wasn’t backing up.

End Update.

We’re ready to start CrashPlan (at least to see if it comes up and stays up). We can do it by hand, but I am going to add it to the DAEMONS list at the end of /etc/rc.conf (like we did with minidlna) and reboot the machine to ensure that it comes up on it’s own. The daemon name to add right after minidlna is crashplan (lower case).

Type: /sbin/reboot

If you did everything above correctly, when you log back in, there should be a java process running, with CrashPlan as the application. It can a few minutes to completely initialize. In that case, we’re done, right? Wrong! CrashPlan is indeed sitting there, waiting to go to work, but it needs to be configured to allow your machine(s) to start backing up to it. Under normal circumstances, this would be trivial to do, but since the PogoPlug is a headless server, we have to jump through a few final hoops.

Once again, CrashPlan support to the rescue (again, for a completely unsupported feature). If you want to understand the details, or are enough of a techy to prefer the theory, I urge you to just read the CrashPlan document and skip to the final section of this post. If you want fewer steps (and warnings) to follow, I’ll give you the bare necessary steps here.

They key is that on every machine that has CrashPlan installed, there are two programs: 1) the server that does all the work and 2) the GUI (graphical user interface) that connects to the server when launched, and allows you to configure and monitor the server process. On the PogoPlug, we only have #1. The good news is that the GUI speaks to the server over the network. By default, that network is local to the machine that the server is running on, but with some ssh magic (and a little editing of a configuration file), we can make that a remote connection.

All of the work is going to be done on your desktop or laptop, where you already (presumably) have CrashPlan running. This is likely the machine that you want to backup to the PogoPlug. Let’s just call it laptop so that it’s obvious it isn’t the PogoPlug.

On laptop, make sure that the CrashPlan GUI isn’t running. If it is, exit the application. Find the conf directory associated with CrashPlan on your system. On my Windows 7 x64 machine, it’s this directory: “c:\Program Files\CrashPlan\conf”. In that directory is a file called ui.properties. Edit that file. The following line is in that file: “#serviecPort=4243”. This line is commented (#), because 4243 is the default value. You can leave that line commented and add a line below it:

servicePort=4200

(You could also remove the comment and replace 4243 with 4200, but I recommend adding a new line.)

Save the file to disk. While still on laptop, open a terminal window and execute the following SSH command (if you’re using Putty to do this on Windows, rather than cygwin, I recommend reading the post back on the CrashPlan site).

ssh -L 4200:localhost:4243 user@192.168.5.71

In the above command, substitute your username (or root) where I put in “user” and the IP address of your PogoPlug where I put in the “192.168.5.71”. This command makes port 4200 on laptop magically redirect to port 4243 on the PogoPlug, which is where the CrashPlan server is listening by default already.

Now launch the CrashPlan GUI on laptop by double-clicking the icon (on Windows, it’s probably in your system tray). You should see a request to create a new CrashPlan account (free) or log in to an existing one. Since I already have one (and presumably you do too), just put in your email address and password. It took quite a while for it to log in and download the configuration, but it worked. I think when I first tried it on my first PogoPlug it actually timed out, but worked the second time.

Once that’s done, you can exit the GUI, as all of the defaults are exactly what we want/need. The only exception to that is if you want to let others (that aren’t part of your account) backup to your PogoPlug. Then you will need to write down the CODE to enable that (it’s toward the bottom of the front page on the GUI and also on a Settings tab).

You can now exit from the ssh session that was started above (Type: exit or hit Ctrl-d in that terminal window).

Once the GUI is shut down, edit the ui.properties file again and delete the extra line we added with “servicePort=4200” (or place the comment “#” back in front of it). Save the file.

Launch the GUI again. This time it will connect to the local CrashPlan server on laptop. Now click on Destinations (bottom left entry on the left-hand navigation). Then click on the icon labeled Computers in the center (the PogoPlug is a real computer!). Whatever you called your device should be in your list, no code necessary, since you should have used the same account on both machines. If you didn’t name your device, then Arch Linux ARM defaults your host name to alarm (get it? ArchLinuxARM?).

Now you’re truly done. If you have a large amount to backup, it could take a couple of days to complete the first backup, even though it’s on a LAN. It will also alternate between the various backup locations (including CrashPlan Central), which is one of the reasons it will be somewhat slowed down.

For reasons I can’t explain, it can take a very long time to start the initial backup, even if you pause the other locations. The GUI correctly communicates with the server instantly, since I can see the correct directory created on the PogoPlug, but the destination still shows up as unavailable for some period of time. Eventually, it gets going, and appears to be quite reliable from that point on.

Update: I struck out the previous paragraph since I’m hopeful that with the addition of libjnidispatch.so to the jna jar file, you won’t experience the long startup delay.

cp -p jna-3.2.5.jar jna-3.2.5.jar-ORIGINAL