CrashPlan is dead

It looks like Code42 isn’t interested in home users anymore, and they’ve announced that they are shifting focus to enterprise users. Even if you were backing up to your own computers, your account is still going away, and with it your ability to back up your stuff.

Now is the time to start looking for alternatives. I don’t have a recommendation yet, but I’m looking into it. I’m interested in hearing what everyone else is using, and how it’s been working out. I’d like to be able to recommend a drop-in replacement for the CrashPlan workflow, but nothing has quite fit the bill just yet.

For the short-term, Windows 10 users like myself can use Windows’ built-in backup system with a network share living on a Raspberry Pi. You can also use Resilio Sync or SyncThing to mirror your important files to the Pi.

Posted in Computers and Internet | 12 Comments

Upgrading from Jessie to Stretch

On Wednesday, Aug 16th 2017, a new major Raspbian OS version was released. The “Stretch” release replaced the previous “Jessie” version, and makes a number of changes that may or may not affect you. I’m editing this post as I go through the process of upgrading my servers to the Stretch release, to let you know what I saw.

This will likely be another “live” post for a little while until I’m sure everything is stable, so remember to check back now and then, and please mention any problems you’ve seen in the comments.

Known issues (so far)

This upgrade is not quite ready for everyone to just jump in and do. I’m finding that a few things aren’t working correctly already.

Samba shares

My shares were gone after the upgrade, and even purging and reinstalling Samba couldn’t bring them back. I tried for hours to figure out what the problem was before deciding to go whole hog and do a dist-upgrade, which I normally don’t recommend. A lot of things can go wrong with a dist-upgrade since you’ll sometimes get newer versions of packages that aren’t quite ready for the real world. In this case, it worked. My shares are back online. DO NOT try this without a full backup. You’ve been warned.

Network UPS Tools (NUT)

The first time I tried the upgrade process, I got a rather troubling error message at the end saying that there were errors while processing “nut-client”, “nut-server”, and “nut”, so I gave apt-get upgrade a second pass just to make sure everything else was updated properly. Only these three packages had failed to upgrade, and it appears it’s because the nut-client service failed to restart. This is because the configuration files were overwritten by the upgrade, and didn’t have any useful information in them. After I restored my configuration settings, I was able to complete the installation. See below for more details.

Take a Backup

I should even have to tell regular readers to do this. We’re about to make major changes to the OS itself. You should shut down the Pi, and take a backup of the SD card before going any further. If you’re booting from a hard drive, then you’ll want to attach that to another computer and backup the root partition as well. After all, that’s where most of your stuff actually lives.

Consider upgrading from the desktop

If you can boot to the desktop, then I’d consider doing that. There are a couple times during the upgrade process where I found it convenient to open a second terminal window to examine something the upgrade process was going to change. If you connect through SSH, then you can use multiple sessions to achieve the same effect. If you’re running Raspbian Lite, and only use a directly-attached monitor and keyboard, then you may want to keep a note pad nearby in case you need to take note of proposed config file changes so you can restore your customizations afterward.

Update APT sources

To get the new software packages, APT will need to know where they live first. All you need to do is edit the two “source” files to point to the new “stretch” repositories.

sudo nano /etc/apt/sources.list

Change all instances of “jessie” to “stretch” and save the file. You can do this by hand, or you can let nano do some of the work for you. To do a search and replace, press ctrl-\. That’s the control key and the backslash. You’ll be prompted for the text to search for (jessie), and what to replace it with (stretch). If you haven’t done much to these files, then you should only find two matches, one on the first line, and one on the last line, commented out. Press “Y” for each match, or “A” to replace them all at once.

When you’re done, close the file, saving your changes (ctrl-x, y, enter). Next, do the same thing again, for a second file.

sudo nano /etc/apt/sources.list.d/raspi.list

Change all the “jessie”s to “stretch”es, and save the file. Finally, do a standard update/upgrade.

sudo apt-get update
sudo apt-get upgrade

What to expect during the upgrade

This is a pretty massive update, so you can expect this to take a while. Don’t just leave and come back later though because you’ll be prompted for answers several times during the upgrade process. The first prompt will be to approve the proposed changes, just like every time you do an upgrade, but the list will be huge. I’m talking hundreds and hundreds of packages have updates. You’ll also be prompted when configuration files have updates. I got prompted for changes to the following files

  • /etc/skel/.bashrc
    I’ve never made any changes to this file by hand, so I just took the new version by pressing “y” and then “enter” when prompted. Note, these kinds of changes will default to “n”, so fight the urge to just hit enter like you’re used to. If you’ve made any customizations to this file, then you might consider opening a copy of the file in another window, and then reapplying your changes by hand when the update is complete. See the dhcpcd.conf step below for details.
  • /etc/login.defs
    I took the new version of this file as well since I’ve never touched it myself.
  • Graphical prompt for the keyboard language
    I let this one “guess”, which is the default option.
  • /etc/dhcpcd.conf
    Now this IS a file that I’ve messed with. It’s how you set up static IP addresses on the Pi these days, so I first used the “D” option to examine the differences between the proposed new version and what I currently have. There were changes to things other than defaults; things I had modified by hand. I opted to open a copy of the file in nano from a second command prompt, and then hand apply my customizations when the upgrade completed. Press “y” to take the new version of the file. Don’t forget to come back and apply your customizations later on, though.
  • /etc/lightdm/lightdm-gtk-greeter.conf
    Another file I haven’t touched by hand. I took the new version.
  • /etc/nut/nut.conf and /etc/nut/upsmon.conf’
    I’ve definitely customized these as part of installing the battery backup (see Network UPS Tools), but the customizations aren’t that extensive. I opened a couple new terminal windows, opened the files in nano, and then took the new version of the file (“Y” option). The installation will fail to complete, but once we restore the customizations to these files, you’ll be able to pick back up and complete the process.

    • /etc/nut/ups.conf
      The part you’re interested in is at the bottom, and it’s where you set up the driver for your particular UPS. Mine looks like this:

      [RPHS]
       driver = usbhid-ups
       port = auto
       desc = "CyberPower SX550G"
    • /etc/nut/upsmon.conf files.
      This sets up the UPS monitor that’s in charge of actually shutting down the Pi when the power goes out. There are a few sections you’ll need here.
      The first is the MONITOR section. Mine looks like this:

      MONITOR rphs@localhost 1 upsmon NOTMYREALPASSWORD master

      The second section is the NOTIFYCMD. You will only have touched this part if you set up email notifications for power events. Mine looks like this:

      NOTIFYCMD /etc/nut/upssched-cmd.sh

      Finally, there’s the NOTIFYFLAG section. This tells NUT which power events you’re interested in getting notifications for. Not just email notifications though, this includes “wall” messages. Mine looks like this:

      NOTIFYFLAG ONLINE SYSLOG+WALL+EXEC
      NOTIFYFLAG ONBATT SYSLOG+WALL+EXEC
      NOTIFYFLAG LOWBATT SYSLOG+WALL+EXEC
      # NOTIFYFLAG FSD SYSLOG+WALL
      # NOTIFYFLAG COMMOK SYSLOG+WALL
      # NOTIFYFLAG COMMBAD SYSLOG+WALL
      # NOTIFYFLAG SHUTDOWN SYSLOG+WALL
      # NOTIFYFLAG REPLBATT SYSLOG+WALL
      # NOTIFYFLAG NOCOMM SYSLOG+WALL
      # NOTIFYFLAG NOPARENT SYSLOG+WALL

      It’s not important that your configuration looks like mine, and it probably won’t. The important thing is that you’re writing down, saving off, or opening a second window with your customizations so that we can restore them later on.

  • Graphical prompt for the “/etc/apt/apt.coasdfnf.d/50unattended-upgrades” file.
    I haven’t touched this one either, so I took the new version.

Removing PulseAudio

The Jessie release of Raspbian used the PulseAudio library for Bluetooth audio. If you’re not using it, you can safely remove it.

sudo apt-get -y purge pulseaudio*

Restoring NUT

I decided to take the new version of the configuration files simply because I don’t know what else has changed in the newer versions, and my own customizations aren’t that extensive. Taking the new files will cause the upgrade to fail because part of the upgrade involves restarting the services, but the new configuration files are missing all of the vital information about your UPS. We’ll restore these files one at a time.

/etc/nut/ups.conf

Scroll to the bottom and restore your UPS information from above.

sudo nano /etc/nut/upsmon.conf

Restore the MONITOR, NOTIFYCMD, and NOTIFYFLAG sections from above. Then, we’re ready to take another shot at completing the NUT upgrade. Pick up where we left off with the following command.

sudo dpkg --configure -a

Apt-get is just a polite shell around the dpkg command which is really doing all the work behind the scenes. This command tells dpkg to finish configuring any outstanding packages. You’ll get prompted again to keep or overwrite your files. We’ve already overwritten them once, and then reapplied our customizations, so this time, choose the default option of “N” to keep your configuration files the way they are now, and the installation should complete successfully this time.

Restoring Samba

As I mentioned above, my Samba shares stopped working after the upgrade, and the only thing that seems to have helped bring them back to life is this:

sudo apt-get dist-upgrade

Normally, I’d say don’t do this. I used to do it all the time until I got burned. dist-upgrades are the bleeding edge of upgrades. Not everything has been tested to make sure it gets along well. Most of the time you’re probably okay, but it’s that one time in ten that takes your machine down that you can avoid by only doing normal upgrades.

Checking one last time

Just to be sure nothing got left behind, I did another apt-get upgrade. I noticed a note about packages that were no longer needed. I decided to leave them alone for now. There is also an extensive list of packages that have been “kept back”. You can force these packages to update with a “sudo apt-get dist-upgrade”, but I advise against that. You can read more about it here, but the practical explanation is that dist-upgrade can leave your system pretty broken.

You’re welcome to try it if you’re feeling daring, but I’ve had bad luck with it in the past and generally avoid it. There’s definitely no way you should even consider this without a fresh backup. You’ve been warned.

Changes in Stretch

One that I’ve read about is a change to the way network interfaces are named. From what I’ve read, this only affects new installations, and upgrades will retain their previous naming scheme, so an upgrade should be safe. Previously, you could count on the Pi’s Ethernet port being named “eth0”. Much like hard drives though, if you happened to have more than one Ethernet port, there is the possibility that their names could end up in a different order on any given day. That’s a pretty rare case though. Most Pi’s are only ever going to have the one port that they came with.

Posted in Computers and Internet | Leave a comment

Raspbian Stretch Released

The next major version of the Raspbian OS was released today, and I wanted to take a moment to let you know that I’ll be walking my way through the entire series again on a fresh install, as well as upgrading my existing installations in-place. I’ll write a follow-up post detailing what I’ve found, but it might take me a while to complete.

In the meantime. If you try it on your own, please let me know if you’ve found it to be better or worse, and what issues you’ve run into along the way. Instructions for doing an in-place upgrade are available on the official Raspberry Pi blog (https://www.raspberrypi.org/blog/raspbian-stretch/), but it boils down to:

  1. Take a backup
  2. Edit your apt-get source lists and change all the instances of “jessie” to “stretch”.
  3. apt-get update
  4. apt-get upgrade

The instructions then tell you to purge the pulseaudio library unless you’re using it for Bluetooth audio. I’m not sure what that’s all about, since they didn’t go into detail, but maybe more details will come later on.

Posted in Computers and Internet | 1 Comment

CrashPlan Experiment

Important: This is an experiment, and should still be considered preliminary. I’m putting this out there “live” because I know a lot of people are interested in this topic. I’ll be updating this post as the experiment progresses, but just know up front; I’m not promising that this is a viable long-term solution. I have something that seemed to be working, but I’m still testing whether it’s stable or not (and it looks like “not”). Make sure you read all the way to the bottom, and pay attention to the updates there. These will be a running log of the experiment until such time as I’ve determined whether or not this is a viable solution. At that point, you can expect this whole post to be updated with the final results.

Things I’ve learned so far:

  • This didn’t work very well at all on a Pi 2, but seemed happier on a Pi 3
  • It ran very well on the Pi 3, and then stopped. The basic CrashPlan logs have nothing interesting to say, but the service.log.0 file is positively massive, and not entirely happy
  • Just when you’re about to give up on it, it starts working again.
  • Just when you think it’s working, it stops again.

Basically, yes, it runs, but not for very long. Maybe it’s a memory restriction, maybe it’s some other instability, I don’t know just yet, but one thing’s for sure. CrashPlan won’t stay running long enough to count.


Ever since Code42 started distributing pre-compiled binaries as part of CrashPlan, I’ve been hoping that they would realize just how many people want to run CrashPlan on Raspberry Pis and other similar devices, and start compiling binaries for ARM processors. I realize that they have no real incentive to do so since their business model relies on subscriptions to their cloud services, but still, it would be nice. That does not appear to be happening, and so I’ve been looking for other alternatives.

My first solution was to configure the Raspberry Pi like any normal file share, and simply use it as a backup location rather than counting on it to actually run CrashPlan. This has been working, but has two major problems.

  1. It requires a read/write share for storing the backups. If your main computer gets hit with a virus, it could destroy the files in this location, including the backups of other computers. You could mitigate this by creating different shares for different computers, and assigning them different permissions.
  2. The Pi itself has no way to back up its own files. We can fix this through regular backups of the SD card, or the boot partition on the hard drive, but it’s not automatic.

I’ve been looking for other alternatives, and found something that I thought would work. A company called Eltechs makes a product called Exagear Desktop that is an x86 emulator. I’m not talking about emulating an entire computer the way you would use RetroPie to run an old arcade game. Exagear Desktop is more of an emulated execution environment. I don’t want to get into splitting semantic hairs here, but it’s a way for us to run the regular x86 desktop Linux versions of programs like CrashPlan on the Raspberry Pi. We’re going to take a performance hit, of course, but the hope is that it’ll get us our automated backups again… once the kinks have been worked out.

It’s worth noting that Exagear Desktop is not free. While I’ve been able to pull off everything in this series at little or no cost so far, this is one area where you’ll need to spend some more money. A perpetual license for a single Raspberry Pi costs between $17 and $33, depending on which model of Pi you’re buying it for. In some cases, that’s more than the Pi itself costs, but consider this; A $35 Pi 3, plus a $33 license is still only $68, and uses a penny a day in electricity (not counting the hard drive), so it’s still a cheaper option than dedicating a regular desktop machine to the job. If you have an old leftover desktop machine, then by all means repurpose that, but if you’re stuck on the idea of a CrashPi like I am, then Exagear Desktop may (eventually) be a viable option. You can even try it out for free for three days to see how it performs on your Pi before deciding whether to spend the money or not.

Note: I’m not getting paid for this or anything. When I told them about the blog series, and specifically about me wanting to test CrashPlan on Exagear, Eltechs did give me a longer license for a single machine to do my testing with, but I didn’t get a free perpetual license or anything. If a viable free alternative x86 emulator comes out, I’ll be covering that too. This just seems to be the only emulator that actually works right now. Also, while I was just beginning the process of writing this, they posted their own blog on exactly this topic, so they beat me to it, but I don’t care much for their installation instructions, which are terribly manual, and won’t result in a free trial license, so I’m continuing on with my own post.

I’m examining the possibility that following their strict instructions may result in a more stable system… just not this weekend. I burned all of Saturday trying to coax this thing to life, without success.

Ready? Let’s get started.

Install Exagear Desktop

As I mentioned, there is a blog post on Eltech’s site already (drat, they beat me to it), about getting CrashPlan up and running, but I don’t advise following their instructions for installing their own product because their instructions only work for a purchased, licensed copy, and I’m presuming you want to try it before you buy it.

If you have a license for the product, then following their instructions will possibly get you a more recent version, but installing through apt-get will mean more convenient updates later on. You’ll only get one shot at a trial per machine though. Even if you wipe yourself back to a previous image, the trial will fail on a second attempt. I found this out while trying to verify my installation instructions. Still, if you just want to see how it will behave, this is the only way I know of to kick off the three-day trial period. The regular installation instructions on Eltech’s blog post will not get you a trial, you’ll need a full license for them to work.

First, make sure your Pi is either hooked up to a monitor, or you are connected through a VNC connection. Part of the installation process is going to require you to interact with a pop-up dialog in order to start your trial, so you need to be doing this from a desktop environment. If you already boot to the desktop, then great. Otherwise, you should be able to start the desktop manually with “startx” from the command line. If you built your Pi from a Raspbian Lite image, then you have no desktop, so you won’t be able to get a trial. Sorry. You could probably still install the full version using Eltech’s instructions though, but you’ll need a license.

We’re going to install Exagear Desktop using apt-get, and we won’t even have to go messing around in the source files this time because it’s already in the public repositories. Open up a terminal window, and install Exagear Desktop.

sudo apt-get install exagear-desktop

When the installation is complete, you’re ready to start Exagear, but first, check your current processor architecture.

arch

The output should indicate that you are on an “ARM” architecture. The exact string will vary depending on the model of Raspberry Pi you’re using. I’m installing CrashPlan on a Model 2B that used to be my main server, but got demoted to being the CrashPi when the Model 3s came out, so my output says “armv71”.

Now, start the virtualized x86 environment.

exagear

Since this is the first time we’ve run Exagear Desktop, we’ll get a pop-up asking for contact information, and for us to accept the EULA. Fill in your information, and click “Activate Trial”. After a couple seconds you’ll see a confirmation dialog. Click “close” to dismiss it, and you’ll find yourself back at the terminal window again. Nothing looks any different, but you’re now operating in an x86 “guest” environment. Check the processor architecture to convince yourself this is really happening

arch

It should say i686, which is a total lie, but exactly what we’re looking for. That’s it, we can run x86 Linux programs now. If you really want to get weird, you could even install Wine, and run an emulated Windows environment from your Raspberry Pi. Don’t expect any kind of performance though.

Install CrashPlan Prerequisites

There are a few prerequisites, but we need to make sure we are downloading the proper versions first. Since we’re in a virtualized x86 environment, the repositories that apt-get knows right now are the wrong architecture. Update your sources and install the prerequisites like this:

sudo apt-get update
sudo apt-get install lxrandr libgtk2.0-0 libXtst6 cpio

You can expect this to take a while. The prerequisites have their own prerequisites and so on. It’s prerequisites all the way down. If we were installing things directly on the Pi’s native Raspbian environment, a lot of these things would already be in place. So yeah, we’re going to end up with two copies of some stuff, but it’ll be worth it.

Install CrashPlan

We’re now ready to install CrashPlan, and the instructions are very much like what I originally wrote years ago, only without the hacking of Java files and the swapping of binaries. We’re down to a very clean install now.

Since you’re already in a desktop environment on the Pi, just go ahead and use the browser to download CrashPlan from https://www.crashplan.com/en-us/thankyou?os=linux. The downloaded file should end up in your Downloads folder on the Pi. While we’re here, we may as well make use of the convenient built-in archiver as well to extract the downloaded CrashPlan_?.?.?_Linux.tgz file. Right-click on the file and select “Extract Here”.

ExtractingCrashPlanReturn to your terminal window, which should still be emulating an x86 execution environment, and navigate to the new “crashplan-install” folder. From there, run the CrashPlan installation script.

Note: You need to do this from the emulated environment. If you have multiple Terminal windows open, make sure you’re in the right one with the “arch” command again before continuing.

cd ~/Downloads/crashplan-install/
sudo ./install.sh

Just like in the good old days, you should take the default answers for everything except the backup location. Make sure you point that somewhere on your external hard drive where you want the backups to go. In my case, that’s /mnt/data/backups/.

CrashPlanOptions.PNG

Let the CrashPlan installer complete, and it will ask if you want to run CrashPlan Desktop now. This part won’t work quite right, so answer “no”. There will also be a CrashPlan shortcut on your desktop. It won’t work either, so you may as well delete that. The only shortcut that’s going to work is the one that got created in the start menu, but don’t launch it just yet.

If you’re running this on a fresh Raspberry Pi, or at least one that isn’t trying to run the whole “Home Server” show, you may see a warning about how many files it is configured to watch in real time. We addressed this issue in the MiniDLNA post, but if your Pi isn’t running MiniDLNA, then you most likely didn’t configure the “watches” on it. If you’re only planning to use this machine as a backup destination, and it won’t be backing up its own contents somewhere, then you can ignore this warning, since CrashPlan won’t need to pay attention to local files anyway.

If you want to configure the Pi to watch more files at once, we’ll need to edit the sysctl.conf file. This virtualized environment doesn’t know what nano is though. I’ve gotten used to nano, and I’m sure it will come up again in the future, so I’m going to install it and use it to make the required changes.

sudo apt-get install nano
sudo nano /etc/sysctl.conf

At the bottom of the file, add a line that says

fs.inotify.max_user_watches = 65536

That should do it. Save the file and exit nano (ctrl-x, y, enter). Next, double check that the CrashPlan service is running.

sudo service crashplan status

You should see a sad, grey message that the service is inactive (dead), so I guess the installer didn’t get that part quite right either. It should have set the service up to run automatically though, and we’ll want to check that this is working, so all you should need to do is reboot. You don’t want to issue this command from the emulated command line, though. Either use the desktop menu to reboot the system, or exit the emulated x86 environment first.

exit
sudo reboot

When the system has restarted, get to a plain old terminal window (no need to run exagear again) and check the CrashPlan service.

sudo service crashplan status

It should be a happier green color now. Not only that, but if you read carefully, you’ll even notice the word “exagear” in the path to the service. Even though the service runs in the virtualized x86 environment, it can auto-start and be seen from the native Raspbian environment. Cool, right?

Now that the background CrashPlan service is running, let’s get it configured and attached to your CrashPlan account. Remember, you don’t have to pay for a subscription, but you do need to establish an account. That’s what allows all of your different machines to discover and talk to each other in order to coordinate things.

Start CrashPlan desktop using the main desktop menu, not the shortcut on the desktop, which you should just go ahead and delete if you didn’t do that already. You may want to go grab a snack or a drink or something. CrashPlan desktop is going to take a while to start up. Even in the original post on this topic, before we introduced any kind of emulation into the mix, this step took ages. Now, with the emulated x86 execution environment thrown into the mix, it takes even longer. Just keep looking at the CPU meter in the upper right corner of the desktop. As long as it looks busy, there’s still hope. Mine stayed at 30%-50% for a few minutes before I finally saw the CrashPlan splash screen.

After you’ve waited long enough, or left and come back, you should see the CrashPlan sign-up page.

CrashPlanSignUp.PNG

You can refer back to the original article if you want, but I’ll summarize here. Either start a new account, or sign in to your existing account if you already have one. Given how slow the Pi can be sometimes, I’d establish an account using your desktop computer, and then choose “Existing Account” here, but that’s just me. Either way, there’s going to be another very long delay, so be prepared to wait a little while. Keep your eye on the CPU meter again if you want assurance that the system hasn’t gone to sleep.

In fact, just about everything having to do with the CrashPlan UI is going to be incredibly slow on the Pi. You’ll just have to get used to that. Fortunately, once CrashPlan is up and running, you won’t need the desktop application very often. When the sign-in page finally goes away, you should be able to set up backups, listen for incoming backups from other computers, and all the other things you’re used to.

So far, I’ve noticed that after rebooting the Pi, it isn’t always noticed right away on the network. I’ve had to reboot my main computer before it has picked up on the CrashPi being available, but maybe I just wasn’t being patient enough.

Please make sure you give this a test drive using a trial installation before plunking down money on a license. I have not quite figured out the magic to getting the system to be totally stable yet. Sometimes kicking off the desktop application seems to wake things up, but not always. I imagine the CrashPlan engine just has a lot of housekeeping to do on startup, and it’s just not ready to listen for connections for a while after a reboot. The testing is continuing on this end.

I’ll be updating this post as time goes by to review how well the system is performing over time. Keep watching.

Update 1: It’s only been a few hours, and the CrashPi disappeared on me. The processes seems to have stopped, so I rebooted the Pi, but not before making some tweaks. I overclocked the Pi (Model 2B) to 1000mhz, and reduced the GPU memory to 16. That should free up some resources, I hope. Things may run better on a Pi 3. I’ll be experimenting with that in the near future. Also, this particular Pi does not boot from the hard drive, which means that its swap file is still on the SD card. I’ll be moving things to the hard drive next to see if that helps.

Update 2: A few more hours, and I’m already wishing I hadn’t posted this so early. Initial tests looked good. It was slow to start, but it worked. After leaving it running for a few hours, I’m very disappointed with the stability, at least on the Pi 2. I’ve moved the SD card to a Pi 3 to see if it makes any difference, and so far it’s looking pretty good. Things seem to start up a bit faster. One thing to note is that the engine still takes some time to get up and running, so there’s still a good five to ten minutes after rebooting when the desktop application will keep complaining that it can’t reach the backup engine. Give it some time, tell it to retry, and eventually it will connect. On the Pi 3, my backup is running once again, so it may just be that the Pi 3 is the minimum viable configuration. Since I simply moved the card from one machine to the other, it’s not actually running the right version of ExaGear though. There is a separate version for the Pi 3, which may perform even better. I’ll be looking into upgrading that as soon as my backup has completed. I’m a bit out of date, and safety comes first.

Update 3: Everything was going pretty well. Then I walked away to go do some stuff around the house, came back, and it was all gone again. The CrashPlan service was “active (exited)”, So I restarted the service (sudo service crashplan restart), and waited to see if the backup would resume on its own. 10 minutes went by with no connection, so I started the desktop application to see what was going on. 20 minutes later, still no connection, but the service is still up and running, and the desktop application on the Pi says it’s listening for inbound connections. I’m going to try something I wasn’t really expecting, but would not surprise me. I’m trying a better power supply next. Maybe it’s something simple like that. This Pi happens to have a screen on it (the official 7″ screen), and maybe that’s pulling more power than this supply would like. I have seen the lightning bolt in the corner, after all. But first, another round of update/upgrade just in case the move to the Pi 3 requires an update to ExaGear.

Update 4: Everything was going so well. And then it wasn’t. Rebooting both my laptop and the Pi doesn’t get me anywhere. They just don’t seem to want to talk anymore. I’ve changed to a beefier power supply, tried to update/upgrade everything, and there’s just nothing happening. The next thing to try is to boot from the hard drive and increase the size of the swap file. That may free up enough resources, and stop hammering my SD card long enough to get somewhere. That may happen tomorrow. I’ll post about it when it happens.

Update 5: I went ahead and changed it to boot from the hard drive tonight, and expanded the swap file to 2GB. It didn’t look like it had any effect, but once again, after sitting there for a good long time, it was suddenly ready to communicate. It’s been up for 18 minutes straight so far, backing up my stuff. I’m trying not to jinx it, but maybe it might work after all. Not very fast, and not always when you’re watching, but eventually, silently in the background, it might just back some stuff up.

Update 6: Well, that’s it for this weekend. Up next is a clean install following Eltech’s instructions to the letter. Maybe it’ll make a difference. We’ll find out later this week, assuming I find the free time.

Update 7: It took a little longer to make some sort of reportable progress than I’d hoped. There’s a new version of ExaGear desktop out, and supposedly people have had greater success using v2.2 than previous versions. I thought I’d give it a try, and last night I updated the version on the CrashPi to 2.2. It does seem to work better, but it’s still not what I’d call stable. I was able to get CrashPlan working again, and left it going overnight. This morning, the service was still up and running this morning, but the desktop application had died. When I restart it, it says “Unable to connect to the backup engine, retry?”, even though the service still seems to be running.

I have rebooted the CrashPi, and after giving it plenty of time to settle, eventually got the desktop application to connect. It looks like the backup of the CrashPi itself completed, so I’m going to try reconfiguring the CrashPi to boot to the command line in order to conserve resources, and try using it as a backup destination for another computer with more stuff on it.

Posted in Computers and Internet | 11 Comments

Adding an “Emergency Stop” button to the Raspberry Pi

One constant problem I’ve had with my CrashPi in particular is that, since it has no screen or keyboard, when it stops responding, I have no way to tell it to shut down. I watch the activity light, and wait for everything to appear calm, and then pull the power. Then, when I take it over to my desk, hook it up to the monitor, and plug it in to try to figure out what the problem is, I see nothing but a wall of filesystem corruption error messages.

Sometimes, I’ve been able to take the card out, put it in a USB adapter, plug it into another one of my Raspberries, and run fsck on it to recover the filesystem. Not always, though. It could be that the filesystem corruption is what caused it to stop responding in the first place, but there’s not really a good way to know for sure. What I want is a way to shut the Pi down safely, even when it doesn’t have a monitor and keyboard.

There are quite a few articles out there about how to hook a button up to one of the GPIO pins, and use that to trigger a shutdown. Just recently, I saw this article, and liked what I saw. I had not previously known that shorting pin 6 to ground would wake a Pi up from “sleep” when it’s been shut down, but the power is still attached.

I’d seen some other articles with far more complicated scripts such that a momentary press of the button would force a reboot, while holding it longer would cause a full shutdown. The idea is sound, repurposing a single button to perform multiple functions, but this approach seemed much simpler to me. Now, if I want to reboot, all I need to do is wait for the activity light to stop flashing, and push the button again to wake up the Pi.

This is just what I was looking for, but I felt that the scripting portion of it could be reduced to just the Python file rather than requiring a second shell script, so today I decided to see if that was possible.

I’m taking a huge cue from one of the older articles I found, on element14’s site. The author there pulled it off using only a Python script, but the script contained a polling loop in it, and that’s just not good form. The newer articles all use wait_for_edge to trigger the shutdown without burning CPU cycles constantly polling for the button press. This is a cleaner approach in my opinion.

The script provided on the howchoo.com article is fine just the way it is, and I’ll be using it exactly as written in that article with a couple blank lines removed. I’m just interested in ditching the companion shell script.

Create the script using nano:

sudo nano /usr/local/bin/listen-for-shutdown.py

Paste the following into nano.

#!/usr/bin/env python

import RPi.GPIO as GPIO
import subprocess

GPIO.setmode(GPIO.BCM)
GPIO.setup(3, GPIO.IN)
GPIO.wait_for_edge(3, GPIO.FALLING)

subprocess.call(['shutdown', '-h', 'now'], shell=False)

All this script does is listen for GPIO 3 to short to ground before continuing, at which point is shuts down the Pi. It requires importing two libraries, one for interacting with the GPIO pins, and one for interacting with the system.

All I’m going to do is follow the old element14 instructions for running the script automatically at startup with one important difference. The element14 article runs the python script after the section that reports the computer’s IP address. If the problem plaguing your Pi is that the network isn’t working, then the shutdown script supposedly won’t get run. To fix this, I’m going to run the shutdown listener before reporting the IP address.

Edit the rc.local file using nano.

sudo nano /etc/rc.local

Insert the following highlighted section

#!/bin/sh -e
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.

# Listen for the shutdown button
sudo python /usr/local/bin/listen-for-shutdown.py &

# Print the IP address
_IP=$(hostname -I) || true
if [ "$_IP" ]; then
  printf "My IP address is %s\n" "$_IP"
fi

exit 0

The “&” at the end of the call to Python tells the script not to wait around for an answer. It’s “shelling” that command off onto its own thread, essentially. Everything else should continue on as if the call to Python wasn’t even there.

Testing

You can start the script manually by typing the call to Python at the command line, but I’m interested in the whole thing working together automatically, so I’m just going to shut the system down, hook up my wires, and then start it back up again.

sudo shutdown -h now

Get a couple of jumper wires and attach them to pins 5 & 6. In my case, pin 6 is already being used for something else, so I used pins 5 & 9 instead. The important thing is that you’re choosing pin 5 and something that’s a ground. Refer to any freely available pinout like the one on this page to choose the ground pin. You could also use a different GPIO pin, but you won’t get the “wake up” functionality unless you use pin 5 (GPIO 3).

Power up the Pi again, wait for everything to settle in, and then touch the free end of the two jumper wires together briefly. The Pi should shut down immediately. Keep an eye on the activity light, and wait for it to stop blinking. The Pi is now “sleeping”. Don’t unplug the power though. Instead, touch the jumper wires together again and the Pi should reboot itself, ready for action again.

Making it permanent

Find a small switch that will fit into your Raspberry Pi’s case without interfering with any of the internal components, and then pick a suitable location for it. You can get as creative as you like with the placement, and different cases will provide different opportunities for where to put it. Just make sure that the switch isn’t going to bump into anything inside the case. Drill a hole in the case (remove the Pi first, obviously) for the switch.

Solder one jumper wire to each pole of the switch, cover the joint with some heat-shrink tubing for good measure, and then mount the switch into the case. Plug the free ends of the jumpers back into the pins from above. Close everything back up and you’re ready to go.

When you need to shut down a headless Pi, you can just hit the button, wait for the activity light to stop blinking, and then pull the power.

Posted in Computers and Internet, Home Server, Raspberry Pi | 1 Comment

Raspberry Pi Home Server v2: Creating a Home Page

Note: This post is part of a series. Each post builds on the previous ones. If you are just trying to add one thing to an existing system that was not built following this series, then I cannot promise that these instructions will work for you, although they probably will. If you’ve started from something other than a non-NOOBS Raspbian image, then you’ll probably need to adjust for that.

Please refer to the series Introduction for a list of all the different posts in the series.

Self-Promotion: I have recorded this series as a screencast for Pluralsight:
(http://www.pluralsight.com/courses/raspberry-pi-home-server)
If you have a Pluralsight subscription, please consider watching it. Reading the instructions is one thing, but watching it done demystifies the whole process.

Thank you!


Over the course of this series, we’ve installed a number of packages that expose web interfaces on various ports, and it’s starting to get a little hard to keep track of. So far, we have

  1. Webmin: port 10000
  2. Transmission: port 9091
  3. Resilio Sync: port 8888
  4. Syncthing: port 8384

It’s getting hard to keep track of what port does what already. What we need is some kind of index with links to these various services. In this post, we’ll create a simple home page for the Raspberry Pi Home Server with links to the UIs for the various packages.

If you already have a web server running (e.g. Apache, Lighttpd, Nginx), then you can just use those. If you’re running WordPress, then you can just create a post with the links in it and make it sticky.

I’m not actually running WordPress on my Pi, so I’ll need to start from scratch. I’m going to show how to build the home page using Nginx (Engine-X). It’s significantly smaller and faster than Apache, and is very light on its resource usage, which makes it a great choice for the Raspberry Pi. Since it’s going to be serving up a very small and simple static web page, we don’t really need a full-blown Apache instance here.

As with most things, install Nginx using apt-get

sudo apt-get install nginx

That’s it. You now have a web server, and it’s already running. Open a browser, and go to the address of your raspberry pi with no port specified. You should see a “Welcome to nginx” page like this.

01-nginx

All we need to do now is customize that page to say what we want. The file we need to edit lives at /var/www/html/index.nginx-debian.html. This location has changed in the past, so the filename may not be exactly the same, but it should definitely be found in that location. Open it for editing.

sudo nano /var/www/html/index.nginx-debian.html

I’m going to make some simple changes, changing the title to match the name of the computer it lives on, and replacing the body with a simple list of available services. You can get as fancy as you want. Use jQuery and Bootstrap. Go nuts. Build the page of your dreams. For this demo, however, I will go with the simplest thing that works.

<!DOCTYPE html>
<html>
<head>
<title>RPHS</title>
<style>
  body {
    width: 35em;
    margin: 0 auto;
    font-family: Tahoma, Verdana, Arial, sans-serif;
  }
</style>
</head>
<body>
<h1>RPHS</h1>
<p>The following services are available:</p>
<ul>
  <li><a href="https://192.168.1.11:10000">Webmin</a></li>
  <li><a href="http://192.168.1.11:9091">Transmission</a></li>
  <li><a href="http://192.168.1.11:8888">Resilio Sync</a></li>
  <li><a href="http://192.168.1.11:8384">Syncthing</a></li>
</ul>
</body>
</html>

You can just copy and paste this over the existing content if you want. Rather than saving it over the original file though, lets save it as simply “Index.html”. Press Ctrl-O to save the file, and simplify the name when prompted. Nginx will use the new “Index” file instead of the old “index.nginx-debian.html” one. You can delete “index.nginx-debian.html” if you want, or keep it around.

There are a few things to note here. The links are all fully qualified from the static address down to the port. It is not possible through plain old html to create a link that goes to a different protocol (https) or port (10000) under the same root. You can do it through javascript, but since this server has a static IP address, I can safely assume that it’s not going to change on me, and this does the job for now.

I’ll leave it up to you to make your index fancier/prettier, but this accomplishes the job of keeping everything straight, even if it’s not very attractive.

Posted in Computers and Internet, Home Server, Raspberry Pi | Tagged , , , | Leave a comment

Raspberry Pi Home Server v2: Syncthing

Note: This post is part of a series. Each post builds on the previous ones. If you are just trying to add one thing to an existing system that was not built following this series, then I cannot promise that these instructions will work for you, although they probably will. If you’ve started from something other than a non-NOOBS Raspbian image, then you’ll probably need to adjust for that.

Please refer to the series Introduction for a list of all the different posts in the series.

Self-Promotion: I have recorded this series as a screencast for Pluralsight:
(http://www.pluralsight.com/courses/raspberry-pi-home-server)
If you have a Pluralsight subscription, please consider watching it. Reading the instructions is one thing, but watching it done demystifies the whole process.

Thank you!


Sync What?

SyncThing. It’s another cloud synchronization program, very much like Resilio Sync, which I covered earlier in the series. It’s a free and open-source project though, and offers most of the same functionality as Resilio Sync, although not as much a Resilio Sync with a Pro license.

I’m just putting it out there as an option, because it’s easily installed on the Raspberry Pi, and has clients available for just about any platform you’d care to run it on. I’m using it to mirror my public share to my CrashPi.

Note: Make sure you check out the Resilio Sync post in this series as well. Look at both options and decide which you like better. It seems that running both Resilio Sync and Syncthing at the same time works just fine, but you probably won’t need both.

Installing Syncthing

Syncthing has its own repository and signing key, similar to a few of the packages we’ve installed previously. Download and install the signing key like this.

wget -O - https://syncthing.net/release-key.txt | sudo apt-key add -

Here, we’re downloading the key file from Syncthing’s own site, and installing it by piping it directly into apt-get with the “add” keyword. This key is required in order for apt-get to install any packages that it was used to sign.

Next, add the Syncthing repository to APT’s list of known sources by creating a new source list. As with previous examples, I’ll create a separate file rather than just tacking onto the main sources.list file.

sudo nano /etc/apt/sources.list.d/syncthing.list

Paste in the following

deb http://apt.syncthing.net/ syncthing release

Exit nano, saving your changes (Ctrl-X, Y).

Tell apt-get to update its list of repositories, and then install Syncthing

sudo apt-get update
sudo apt-get install syncthing

Now we’re ready to start configuring Syncthing. Start Syncthing from the command line.

syncthing

Configuring from a Local Browser

If your Pi is hooked up to a monitor, and you’re running the desktop, then a browser window should open automatically, and in a few moments, the Syncthing web UI will appear.

01-web-ui

This web UI is very full-featured, and you’ll use it to set up folders that you want to sync between devices. I’ll show an example of that a little later on. The only problem is that the UI is only available from a local browser running on the same machine as Syncthing. If you don’t mind remoting in to make changes, then you can just skip ahead, but if you’re like me, you’d like to connect directly from the browser on your regular computer.

This is easy enough through the web UI. Click the “Actions” drop-down in the upper-right of the page, and then “Settings”.

02-settings

In the right-hand column, change the “GUI Listen Addresses” from “127.0.0.1:8384” to “0.0.0.0:8384”. This will allow Syncthing to be administered from other computers on your home network. We don’t want to leave it wide open to just anyone though, so fill in values for the “GUI Authentication User” and “GUI Authentication Password” as well. You can decide to make up an entirely different user name, or just use “pi” and its password. It’s up to you.

Configuring from a Remote Browser

If your Pi doesn’t boot to the desktop, or you’re doing this installation through a remote SSH session, then obviously the local browser is not going to be an option, but we can still make the required changes.

Now that Syncthing has been run once, it has created a config file where we can make the same adjustments. Press Ctrl-C in the terminal window to shut down Syncthing.

Open Syncthing’s config file using nano.

sudo nano ~/.config/syncthing/config.xml

Find the “gui” section. It should look something like this:

<gui enabled="true" tls="false" debugging="false">
    <address>127.0.0.1:8384</address>
    <apikey></apikey>
    <theme>default</theme>
</gui>

Change the “127.0.0.1:8384”, which refers to the local machine, in this case the Pi itself, to “0.0.0.0:8384”, which means any machine on the local network. Technically, you could forward a port through your firewall and make Syncthing accessible from outside your house as well, but there’s no reason to do that. If you set up a VPN as covered in my OpenVPN post, you can always connect securely to the Pi and make any changes you need from there.

Close nano, saving your changes (Ctrl-X, Y), and restart Syncthing from the command line.

syncthing

Give Syncthing a moment to get started up, and then open a browser from your regular computer. Connect to the Pi, port 8384. The web UI will open up, and after a few moments you can expect to see a warning that the web UI is now configured to allow remote access, but isn’t going to require a password.

nopassword

Click the “Settings” button, and fill in values for the “GUI Authentication User” and “GUI Authentication Password”. The same way as we did in the previous section. Click the Save button, and you’ll soon be prompter to re-authenticate with the name and password you just set up.

Sync some things

For a demonstration, I’ll be syncing a folder between two of my servers, my main Raspberry Pi Home Server, and my CrashPi (although sadly, CrashPlan isn’t working right now). By default, Syncthing already created a folder called “Sync” in the pi user’s home folder. Following the same directions above, I’ve installed Syncthing on my second server already, and it now has a similar default folder already created. All I need to do is introduce these two Syncthings to each other.

Click the Actions drop-down in the upper-right corner of the page again, and then “Show Id”. You’ll get a page with a QR code on it, as well as a text representation of this Syncthing’s “Device Id”. The QR code can be used by the mobile versions of Syncthing so that you can sync things to your phone. For now, copy the text above the QR code. Make sure you get the whole thing.

Switch to the Syncthing web UI running on the second device, and click the “Add Remote Device” button in the lower-left corner of the page. You may have to scroll a bit to see it. Paste the first Syncthing instance’s Device Id in the first field. Fill in a name so that you can keep track of which device is which, and then check the “Default Folder” checkbox at the bottom. Fincally, click “Save” to dismiss the dialog. There are a lot of other options here, but I’ll leave it up to you to explore Syncthing’s documentation on your own. Just click the “Help” button in the upper-right and you’re off.

Return to the web UI of the first Syncthing instance, and you should see a prompt asking you to confirm the addition of the second instance. This ensures that even if someone were to get ahold of your Device Id, they couldn’t just help themselves to your stuff. You need to give the okay first. Click “Add Device”, and you’ll see the same dialog that you were looking at a moment ago, but most of the fields will be filled in for you already. Check off the “Default Folder” checkbox again, and then click the Save button.

Finally, switch back to the other instance, and you’ll see another prompt asking you to confirm sharing the default folder.

Time to Test

Open a second Terminal window, SSH session, or simply use the file browser on the desktop to create a new file in the “Sync” folder that Syncthing created for you. Perhaps name it after the device on which it was created so we can keep track. Do the same on the other computer, and then observe what happens. If you’re lucky, the files will magically sync from one computer to the other. At the time I’m writing this, there’s some kind of bug when adding new folders, and you’ll need to go to the Actions drop-down and click “Restart” on both instances to get things flowing. After that though, the folders on each computer should stay in sync.

As with and file synchronization technology, it’s possible to have conflicts. If you edit the same file on both systems, or try to sync a file that receives regular changes from both places, things are going to get ugly. Rather than just clobbering what the other computer did, Syncthing will create “conflict” files, and leave it up to you to sort things out. This is good in that you won’t have one version simply overwriting another, but it does mean that you’ll have to be involved in fixing it when things go wrong.

Try to use Syncthing only for items that you want available everywhere, but will typically only make changes from one place at a time. Refer to Syncthing’s documentation for help with synchronization strategies.

Stopping Syncthing

Since we started Syncthing from the command line, that command line has been held up waiting for Syncthing to finish. You can press Ctrl-C in the terminal window to shut it down, or go to the Actions drop-down and click “Shutdown”. In either case, your command line will wake back up so you can issue more commands.

If you want to start Syncthing without locking up your command line, you can always append an ampersand (&) to the command to run it in the background.

syncthing &

Since you can’t hit Ctrl-C to stop it anymore, you’ll have to use the menu to shut down this instance if you want. Syncthing will also keep interjecting things into your terminal session, which can be annoying. This approach also means manually starting Syncthing every time you reboot, though.

Starting Syncthing automatically

Logging in and starting Syncthing manually isn’t going to work long-tern. Fortunately, Syncthing’s authors have done most of the hard work for you here. There’s already a service description set up and waiting for you. All you need to do is enable it and reboot.

sudo systemctl enable syncthing@pi.service
reboot

That’s it. When the pi reboots, Syncthing should be up and running as the “pi” user automatically.

At this point, I refer you to Syncthing’s documentation for more information on its configuration, care and feeding. It’s a very capable program, and I’ll be interested to watch its progress.

Posted in Computers and Internet, Home Server, Raspberry Pi | Tagged , , , | 1 Comment