Note: This post is part of a series. Each post builds on the previous ones. If you are just trying to add one thing to an existing system that was not built following this series, then I cannot promise that these instructions will work for you, although they probably will. If you’ve started from something other than a non-NOOBS Raspbian image, then you’ll probably need to adjust for that.
Please refer to the series Introduction for a list of all the different posts in the series.
Self-Promotion: I have recorded this series as a screencast for Pluralsight:
(http://www.pluralsight.com/courses/raspberry-pi-home-server)
If you have a Pluralsight subscription, please consider watching it. Reading the instructions is one thing, but watching it done demystifies the whole process.
Thank you!
Now that the Raspberry Pi Home Server is up and running off of the hard drive, we’re presented with a problem. We can’t just take a backup of the SD card anymore. In fact, what’s on the SD card probably shouldn’t change from this point on. All the important stuff is on the OS partition of the hard drive.
In this post, I’ll present a couple different methods for backing up the hard drive using a secondary computer, or just using the Pi itself and a second SD card.
Backing up from a Windows computer
I’ve been using and advocating HDD Raw Copy over Win32DiskImager in this second edition of the Raspberry Pi Home Server series because of its ability to compress the image files on the fly, and that ability can be put to good use here as well. All you need to do is hook the Pi’s hard drive up to your Windows computer, and use HDD Raw Copy to copy the drive to an image file just like you’ve been doing with the SD card. It’s exactly the same process. If you’re using Win32DiskImager, the process is the same, only the resulting image files will be the same size as the drive you’re backing up, which brings me to the downside of this method.
Both HDD Raw Copy and Win32DiskImager are going to want to back up the entire hard drive. Not just the OS partition, but the data partition as well. If you have the space to keep that kind of backup around, then good for you, you can restore the entire server, including the OS and Data partitions as well as all of their contents in one go. It’s going to be slow, and it’s going to take up space, but you can do it.
This is part of the reason I keep my actual shared stuff on a completely separate drive now. I like the separation between the devices that make up the Pi itself, and the stuff that the Pi is sharing. I only have to back up a 120GB drive, and it’s mostly empty space, so my compressed image files are not that large.
If you can build your setup like this, I highly recommend it. If not, then you’ll need to continue reading and decide which is right for you.
Backing up from the Pi itself
If, for some reason, you don’t have access to a full Windows, Mac, or Linux computer, you can back up your hard drive by using another Pi. If you’re stranded alone on a desert island and only have a single Pi with you, then you’ll need to do a little prep work, the first step of which will be establishing a power grid.
Since this is the worst-case scenario, we’ll get it out of the way now. We’re going to create another SD card specifically for making backups. The Raspbian Lite image will have everything we need on it, or you can use a full Raspbian installation too. Just prepare another SD card, exactly like we did in “Installing the OS“, and then proceed to the “Backing up from Linux” section.
Backing up from Mac OS
Mac OS is built on top of Unix, which Linux is a sort of clone of, so things work pretty much the same there. You’ll just need do the same thing as the Linux users in the next section.
Backing up from Linux
The process for backing up from Linux is exactly like what you did to create the SD card in the first place, except that you’ll be using dd to copy from the Pi’s OS partition to another spot in your filesystem. This could be somewhere on the hard drive of your main computer, but for this demo I’ll be creating an image of the OS partition on the Data partition of the same drive. Just adjust the file paths to match where you want the image and you’re set to go.
Shut down the Pi, and transfer the hard drive to your main computer, or swap SD cards to the new one you created above. If you’re doing this from a single Pi, then I would also take this opportunity to unplug any other drives you might be using, leaving nothing but the system drive attached, and then start up the Pi with the new SD card.
Use blkid to figure out where the Pi’s hard drive got attached (not mounted, mind you).
sudo blkid
In my case, the hard drive shows up at /dev/sda, and the partitions are /dev/sda1 and /dev/sda2, with sda1 being my OS partition, and sda2 being my data partition. If you’re using a separate Linux system, then sda is probably your own hard drive, and the Pi’s drive will be sdb, sdc, sdd, etc.
I’m writing my image to the second partition of the same hard drive, so I’ll need to mount it first so that I can write files to it. Since this is the first time I’m doing this, I’ll also need to create a placeholder directory for the drive to mount at.
sudo mkdir /mnt/data sudo mount /dev/sda2 /mnt/data
Now, we’ll use the dd command to copy the sda1 partition to sda2 (or wherever you’ve decided to store the image). The dd command is part of “coreutils”, and newer versions have the ability to give you a progress report as they go. See which version you are running.
dd --version
If it says 8.24 or above, you’ll be able to get a status report with the “status=progress” switch. Back up the OS partition like this.
sudo dd bs=1M if=/dev/sd?? of=IMAGEPATH.img status=progress
Unfortunately, Raspbian is still on 8.23 at the time I’m writing this, so status=progress won’t work. If your OS partition is still relatively small, then doing without feedback is a perfectly viable option, so you can proceed like this.
sudo dd bs=1M if=/dev/sd?? of=IMAGEPATH.img
If your OS partition is very large, or you opted to have just one partition, then you’ll probably want some kind of feedback. Install the dcfldd package through apt-get.
sudo apt-get install dcfldd
The syntax for dcfldd is very similar to that of dd, but it will report its progress as it goes.
sudo dcfldd bs=1M if=/dev/sd?? of=IMAGEPATH.img
When the copy is complete, you can shut down the Pi, or safely remove the hard drive card from your main computer, plug it back into the Pi or swap your SD cards back to normal, make sure everything’s hooked back up like it was before, and restart the system.
Restoring from a backup image
If you need to restore your system to a previous state, the process is just the same, but with the “if” and “of” parameters flipped like they were when you created the original SD card. You’ll need to read from the image file and write it to the hard drive partition.
Shut down the system and move the hard drive to the computer that will be doing the restore, or swap to the “backup” SD card if you’re doing this directly from the Pi, and run the dd or dcfldd command as appropriate. Don’t forget to mount the data partition if you’re doing this from the same Pi. You might want to edit the /etc/fstab file so that this happens automatically whenever you’re using your “backup” SD card.
sudo dd bs=1M if=IMAGEPATH.img of=/dev/sd??
or, if you’re doing this from the same Pi
sudo mount /dev/sda2 /mnt/os sudo dcfldd bs=1M if=/dev/sd?? of=IMAGEPATH.img
Easy, right?
What’s next?
Everything after this point is purely optional. Return to the Introduction, and follow whatever links you want, mixing and matching whatever additional components you want your server to have.
MelGrubb.ToBlog() 
Doing the OS partition backup using a second RasPi, I think there is a command missing. After:
sudo mkdir /mnt/data
sudo mount /dev/sda2 /mnt/data
Don’t you need to change to the /mnt/data directory before executing the dd command? Otherwise, it will try to write the output file to your current directory, which is on the SD card and not big enough to hold the image.
Similarly, when doing the Restore, don’t you want to be working from the /mnt/data directory where the image file is located?
The output file (of) switch will take a full path, I believe, so as long as you are telling the file where to go, you shouldn’t have to actually change your working directory before doing the backup.
I recommend using a second Pi, or at least a second SD card so that the card you’re backing up isn’t “live” when you do it. For example, assume you have a Pi that has a hard drive attached, and boots from a partition on the hard drive, which is what you end up with if you follow this series. All you’ll need is second SD card with a fresh OS image on it. This can just be Raspbian Lite, there’s no need for the desktop on the secondary card. Shut down the Pi, swap to the secondary card, and start it back up. Because this secondary SD card hasn’t been told to boot from the hard drive, it will boot from the SD card like any regular Raspbian installation. Use parted to examine the hard drive and identify the boot partition. Use the “mount” command to mount up the data partition so that it’s usable by the OS on the secondary SD card. It should be the first partition, and be pretty small. Then use dd to back your boot partition to a location on the data partition (or to a different drive entirely, but I’ll leave that up to you). The key is to properly aim “if” (input file) at the primary partition, and “of” (output file) to a location on the data partition of the hard drive. When the backup is complete, shut down the pi, swap the SD cards back, and carry on as usual. You’ll be able to see your backup images on the data partition, right where you left them.
If you want to keep this “backup” card around in order to use it again, I’d edit fstab to permanently mount the data partition just like with your primary OS, and create a new shell script for performing the backup so that it gets done the same way next time.
I have been using the following to back up my OS partition to the Data partition of the HDD:
sudo dcfldd bs=1m if=/dev/sda1 of=/mnt/data/OS_Backup/IMAGEPATH.img
This has been working well to take a snapshot after every major installation that I get working. I got TVHeadend server installed and saved an image. Then I installed some scripts to get the Electronic Program Guide working, and something went wrong, so I restored the previous image and tried it again. I got the EPG working, and tried to save my usual image, but ran into a problem. It only got about two-thirds the way through and terminated, with the following error:
dcfldd:/dev/sda1: Input/output error
657024+0 records in
657024+0 records out
I am just guessing that it either means that it couldn’t read a block, or it couldn’t write one. I ran fsck on the /dev/sda1 OS partition and got no errors. I have lots of free space on both partitions, so it isn’t out of storage space. What else can I do to get this working again? Since everything seems to be working well with the TVHeadend DVR and EPG, I’d really like to get an image backed up before I bork it up again. Any suggestions?
Just a few more thoughts on this. When I started this project, I used a 32 GB SD card in my Raspberry Pi, because that was what I had on hand. It was much larger capacity than what I needed, but it was a good high-speed card, so I used it.
When it came time to move everything to the HDD, I used a 3TB drive that I had from an old computer, and put it into an external USB enclosure with its own power supply. I created a 32 GB partition for the OS, and all remaining space for the Data partition. That all worked fine.
But now I am thinking that I could save a lot of time when saving the partition image if that SDA1 OS partition were smaller. Is there any reason I couldn’t use the Partition Editor (parted) and just shrink that SDA1 partition down to maybe 8GB, and just leave the rest of the space unallocated (in case I ever want to go back and increase the partition size later)?
And what about that 32GB SD card? It now just points to the HDD OS partition, right? Is there a way that I can replace it with a tiny (1GB?) SD card? What would I need to copy from the 32GB card to the 1GB card for this to work?
I always build on an 8GB card just to keep the images small, and so I don’t have to sacrifice a more expensive card once I move to the hard drive. I believe the only thing that needs to remain on the card is the contents of the FAT partition, so you should be able to either copy that, or just start with a fresh, smaller image, and copy over the /boot/cmdline.txt file, since it’s the one pointing at the hard drive and saying “look over there”.
I just ran the dcfldd command again and got the following error:
656896 blocks (20528Mb) written.dcfldd:/dev/sda1: Input/output error
656979+2 records in
656980+1 records out
Slightly different number of blocks than before. Still no clue why it is stopping.
Yeah, I don’t think there’s quite enough to go on in order to diagnose this easily. What is the reason for using dcfldd instead of plain old dd?
I also tried running:
sudo dd bs=1m if=/dev/sda1 of=IMAGEPATH.img
It gives me the error:
dd: invalid number ‘1m’
Any thoughts?
On a whim, I tried changing “bs=1m” to “bs=1M” and ran the dd command that way. The good news is that it ran. The bad news is that the dd command also stopped just like the dcfldd command did, with this output:
dd: error reading ‘/dev/sda1’: Input/output error
20530+1 records in
20530+1 records out
21527937024 bytes (22 GB) copied, 3264.96 s, 6.6 MB/s
The IMAGEPATH.img file size shown in its Properties is 20.0 GB in both cases. The dd command does seem to be more specific in its error report, saying that was an error reading from the sda1 partition. Does this mean there is an unreadable area on the HDD? Since fsck didn’t show any file system errors, what tool would I use to analyze/fix this problem. I’m a Linux newbie, so any pointers you can give would be great.
Thanks!
Yes, that should be a capital “M”, my mistake. I’ll fix the post right away. I’m not sure what to say about the read error. One thing you might want to consider is creating a bootable Clonezilla USB drive that works on a regular computer, and using that to do the backups. It’s purpose-built for doing exactly that, and may offer some insight as to what the problem actually is. There’s also an SD image writer built into Raspbian, but I haven’t tried to trick it into backing up the hard drive partitions yet. It will probably want to back up the entire physical drive, and that’s probably not what you’re after.
Since I got errors at about the same spot with either the dcfldd or the dd command, and the dd command error actually indicated a read error, I did more checking on the partition. Although the fsck command didn’t seem to show any errors in the filesystem, I guess it doesn’t check the unused areas in the partition. So I ran:
s2fsck -c
The -c option has it run the badblocks command on the whole partition. It started showing read errors right at the 20GB point in the 32GB partition. My actual files were in the first 6GB of that partition, so I decided to go ahead and shrink the /dev/sda1 partition from 32GB down to 8GB. That fixed two problems:
1. No more errors when backing up the image, since the bad area is now in the unused/unallocated area between the sda1 and sda2 partitions.
2. The image backup commands now take a quarter of the time to complete, with resulting file size also a quarter the size.
I think I got a good image backup with TVHeadend server & DVR all working, and using the TV-Guide listings for its EPG. I’m copying that image to file to my main computer for safe keeping, just in case the 3TB HDD that I used on the RPi server does end up failing. I’ll keep an eye on it.
There’s something to be said for keeping the OS partition as small as possible. I used to use 4GB cards for all my builds, but around the time Jessie came out, that stopped working, and I moved to 8GB cards. If it weren’t for Wolfram, LibreOffice, and maybe some of the games like MineCraft, it might still fit under 4GB, but it’s nearly impossible to even find 4GB cards anymore anyway. It’s actually getting to the point where the price difference between 8GB and 16GB is hardly worth it.
Why can’t the Windows imaging apps (like HDD Raw Copy tool) back up by partition rather than just by device? Is there something unique to Windows that prevents them from being able to see individual partitions on a device?
Hi Mell, my usb stick on where I had transfered my PI Os died on me today.
So I bought a new one and restored a backup to it using HDD RAW Copy Tool.
It boots up, but goes to emergency mode.
Do you have any idea what could be the cause. Has some ID changed so FSTAB doesn’t match anymore or something ?
If the stick had “died”, I don’t think I’d probably trust its contents at all. There’s no way to know what was or wasn’t corrupted. I try to stay on top of making backups of my system drives, but that’s harder and harder now that my system drives are often hard drives, or SSDs embedded in the same case as the Pi. They’re not as easy to just “clone” at a moment’s notice.
I misread part of that. You DID restore from a backup, not from the dead drive. Okay, that’s good. I would have thought that using HDD Raw Copy would have maintained the same UUIDs across the board, so I’m not sure what that was about. I’m glad you got it working (based on your other comment)
Hmm. I seem to have fixed it. Even though the faulty USB stick had a lot of bad sectors, I tried backing it up using HDD RAW Copy anyway. I gave a lot of error about bad sectors.
But for some reason when I restored this backup to a new stick it still worked !! What do you know 🙂
I do have one quick question though. If I start the PI without my hard drive (not the above mentioned stick) it won’t boot al the way through. Why is that ?
As long as the drive isn’t actually NEEDED to boot, you can put a “nofail” along with the other options in fstab like “noatime”. Without that, the computer thinks the sky is falling and won’t boot until it sees its drive. This won’t work for the actual boot drive, of course.
Yes this works. I now have this line (got it from the RPi forums) and somehow transfer speeds have doubled from 2MB/s to 4MB/s. Which still isn’t very fast but quite a bit faster.
Next step is to convert from NTFS to EXT4.