Raspberry Pi Home Server: Part 6, Adding a hard drive

Article Obsolete

A new version of this series has been published. Please refer to the new index for updated articles and ordering. This article is kept for historical reference, but should be considered out of date.


Note: This article is part of a series. See the Index for more information.

Self-promotion: I’ve recorded this series as a screencast for Pluralsight:
(http://www.pluralsight.com/courses/raspberry-pi-home-server)
If you have a Pluralsight subscription, please consider watching it. Thanks!

Updates: If you didn’t change the boot option using raspi-config so that the Pi boots to the command-line, the hard drive will auto-mount each time you boot up, and copying the root filesystem to the external drive will fail. Make sure you’ve configured the Pi to boot to the command line, and everything should work as per the original post.


Running the Raspberry Pi off of an SD card is simple, affordable, and very convenient. SD cards are also very small compared to hard drives, though, and they can only be written to a finite number of times. That number is ridiculously large, but some have said that it can still be used up faster than you think by things like virtual memory swap files which are written and re-written constantly.

Whether you believe that or not, one thing is certain; In order to be a useful home server, the Raspberry Pi is going to need access to more storage space than you get from a simple SD card. The Raspberry Pi Home Server is going to need a proper hard drive, and as long as you’re adding a hard drive, you may as well take advantage of its speed and boot the Raspberry Pi from it.

Okay, I’ll level with you. You can’t actually boot entirely from the hard drive, but you can boot mostly from the hard drive. When the Raspberry Pi boots up, it looks to the first partition on the SD card for instructions on what to do next. That boot partition contains just a tiny piece of the boot process, but at least for now, its location is non-negotiable. That’s just the way it is. That’s not to say things couldn’t change in the future. I saw someone on a forum claiming to have done away with the SD card altogether, but I’ll believe it when I see it.

What you can do is to move all the stuff that comes after that first step to a different device, such as the external hard drive. All that stays behind on the SD card is a very small boot partition, everything else moves to the hard drive.

The resulting system should run faster and smoother, and you won’t be using up your SD card’s limited number of write cycles on high-frequency stuff like a virtual memory swap file.

Acknowledgements

This article would not have been possible if not for others that went before me. In particular, Ted Hale’s article at http://raspberrypihobbyist.blogspot.com/2013/07/running-from-external-hard-drive.html is pretty much the blueprint I started from. In turn, that article was based on a forum post by “Rattus” at http://www.raspberrypi.org/phpBB3/viewtopic.php?f=26&t=10914&p=129474&hilit=resizefs#p122476.

I’ve added my own bits, such as consulting the boot/cmdline.txt file rather than simply assuming the root file system’s location, using UUIDs to identify the partitions, and expanding the existing swap file rather than creating a separate swap partition.

I’ve turned these resources into a walkthrough that matches the rest of this series, but I wanted to at least point out where my source material originated.

Gather your tools

This post uses some software packages that are not included by default in the Raspbian distribution. You’re going to need NTFS filesystem support, and the gdisk tool. Install them before continuing.

sudo apt-get install ntfs-3g ntfs-config gdisk

Pick a hard drive

Pretty much any external USB hard drive will do. For my own server at home, I’m using a dual-drive 2TB RAID enclosure (Cavalry CADA-SA2). This might seem like overkill for a Raspberry Pi, but one of my home server’s jobs is to centralize and share my music collection. It took me four solid weeks to rip every CD I own, and I do not plan to repeat that process. My family photos and videos are on that drive as well. Basically, this is all stuff I don’t want to lose due to a hard drive failure.

Note: Yes, I have an offline backup, too, but swapping out a dead drive in a RAID is easier than rebuilding the whole thing by hand.

Linux device names

Before we start, lets take a moment to discuss the way in which Linux refers to devices in general, and more specifically the way it refers to hard drives. Windows users are used to seeing multiple drives show up with different letters. You have your “C” drive, where Windows and all of your programs are installed. Maybe you also have a second hard drive called “D”, or maybe “D” is CD or DVD drive on your computer. “A” and “B” used to refer to floppy drives, and are reserved for historical purposes. Each of these devices is completely separate, and has its own individual root.

Linux has no drive letters. Linux uses what’s called a “unified file system”, meaning that files, folders, and even entire drives appear together under one root. You can see all of these different devices in the “/dev” folder. Take a look:

ls /dev

image

I won’t go into what all of these entries mean, but you can see that there are a lot of them. If I were to plug in an external hard drive, or perhaps a flash drive, I would see new entries pop up in the list. Here’s another screen shot after I’ve plugged in a flash drive.

image

Look about halfway down the third column, and you’ll see new entries called “sda” and “sda1”. These represent the flash drive, and the first partition it contains, respectively.

Storage devices like a flash drive or a hard drive can be divided up into multiple pieces called partitions that can make it appear to be multiple drives. This list of where the partitions begin and end is called the “partition table”, and every drive has one. If there were more partitions on the flash drive, there would be more entries in /dev called “sda2”, “sda3”, etc.

On a computer where drives are connected via SCSI (Small Computer System Interface), “a” would be the first device, “b” would be the second, etc. Drives connected via IDE (Integrated Drive Electronics), or SATA (Serial AT Attachment) have similar concepts of “first”, “second”, etc.

This is not the case with USB. USB devices have no concept of “first” or “second” and so their names are assigned on a first-come, first-served basis. The first drive to be ready gets to be “sda”, the next one “sdb” and so on.

If you only have one drive, then it will obviously always be called “sda”, and there’s no problem, but since you’re building a server, it’s conceivable, and even likely that you will eventually end up with more than one hard drive attached to it. Perhaps one drive will house videos, and another one will have music on it.

Even if you don’t see a need for that yet, you’ll end up filling up a drive eventually, and you’ll want to add another one. As soon as you add more USB drives, however, you can’t count on their names being the same every time the machine starts up. As a result, drives may end up mounted in the wrong places, and programs won’t find their files where they expect them to be because the drive that used to be “sda” was a little slower off the line one morning and got stuck with the name “sdb” instead.

Let’s take care of that particular problem before going any further.

Format the hard drive

If the drive you’re planning to use already has stuff on it, then back its contents up somewhere. and restore them when you’re done. This approach calls for nuking and repaving the whole hard drive.

Caution: Even if you wanted to live dangerously and manipulate the partitions in place, I wouldn’t do it without a backup copy first. There are just too many things that could go wrong. What if the power went out partway through the process? Don’t risk it. Just copy your stuff to another drive, and copy it back when you’re done.

The key to fixing the device naming problem requires us to have a specific kind of partition table called a GUID Partition Table. This is different than the older Master Boot Record which is used by most Windows machines.

In order to create a GPT, you’ll need to use Linux’s Partition Editor (parted) application. Raspbian already includes parted, so there’s nothing to install. Run it with the following command:

sudo parted

You should see something almost, but not quite entirely unlike a command line. To see a list of all the known devices and partitions, type “print all”. Parted should print out a table with two rows similar to this:

image

The rows in the table represent the partitions. The columns will tell you where each partition begins and ends, how large it is, and what kind of filesystem it contains.

If you see more than one table at this point, then you probably have another drive plugged in somewhere. I strongly advise shutting down and removing all drives other than the SD card before continuing.

The SD card in the image above was created using the Raspbian image file, and has two partitions on it, a small “boot” partition, and the root filesystem partition. If you started by using NOOBS, then you probably have five partitions. Future releases could divide things up differently, so don’t worry if your output doesn’t look like the example.

Attach the hard drive

Plug your external drive into a free USB port on the Raspberry Pi, and give the OS a few seconds to notice it. List the known partitions by typing “print all” again.

image

This time, there are two tables. In my example, the second one is the external drive, although the order is not guaranteed here.

Notice the headers above the tables. The header tells you information about the drive in general. In this case, the header tells us that the hard drive was assigned the name “/dev/sda”, that it’s 2TB in size, and that its partition table type is “msdos”. That means it’s using the old MBR-style partition table, and that’s not going to work for this server.

Select the external drive by typing “select /dev/sda”. Type “print” all by itself to make sure. You should see the second table from above again. Read the header carefully, and make sure it’s referring to the external drive, because you’re about to blow away the partition table.

Create a new partition table with the command “mklabel gpt”. Read the warning carefully, and make sure it refers to “/dev/sda”.

image

Answer “y” to the prompt, and in a few seconds, you will have a nice blank slate to work with. Check the results with another “print”. The header should now say that the partition table type is “gpt”, and there should be no partitions on it.

image

Create new partitions

Next, you’ll need to create two new partitions; one to hold the root filesystem, and one to hold general data. You could make one big partition, combining the root filesystem and data storage in one, but there are some benefits to keeping the data partition separate. The root filesystem must be formatted using Linux’s native filesystem, ext4. The data partition, however, can be any type you want, which means you can pick something other, non-Linux computers will understand.

Root Partition

This is where Linux and your programs will go. You can think of it as Windows’ C: drive. The size is up to you, but 16GB is probably fine for the simple server described in this series. If your existing SD card is larger, or you are planning to install a lot of programs, go bigger. Just make sure the partition is at least as large as the SD card you are currently using.

Create a new, 16GB partition starting at the beginning of the drive:

mkpart primary 0GB 16GB

Data Partition

This is where you’ll put your “stuff”, and it should take up the rest of the available drive space. The mkpart command can interpret the beginning and ending parameters in a variety of different formats. Use this ability to create a second partition that begins where the first partition ends, and takes up all the rest of the space on the drive:

mkpart primary 16GB 100%

Finally, use the “print” command again to see the results.

image

That’s it for the partitions, so type “q” and press enter to exit parted. You can ignore the warning about updating /etc/fstab for now. We’ll get to that soon enough. You now have two partitions, but they’re completely blank at this point.

Copy the old root partition to the new drive

The file /boot/cmdline.txt specifies the location of the root filesystem. You’ll need to edit this file later in order to boot from the hard drive, but for now, you just need to know for sure where the root filesystem currently resides. Show the cmdline file’s contents like this:

cat /boot/cmdline.txt

image

The part immediately to the right of “root=” specifies the device and partition that holds the root filesystem, and that’s the partition whose contents you need to move to the hard drive. In my case, it says “mmcblk0p2”. If you started from the raw Raspbian image, then yours probably says the same thing. If you started from NOOBS, then it’s probably “mmcblk0p6”

Copy the existing root partition’s contents to the new 16GB partition (sda1) on the hard drive, changing the highlighted part to match the current root filesystem path from above.

sudo dd if=/dev/mmcblk0p2 of=/dev/sda1 bs=32M conv=noerror,sync

This can run for quite a while, depending on how large your SD card is. Unfortunately, the dd command doesn’t give you any kind of feedback. In this case, no news is good news. As long as the drive appears to be busy, then keep waiting.

Here’s what that all means:

  • “dd” copies things
  • “if” specifies the input file, in this case an entire partitions
  • “of” specifies the output file
  • “bs” specifies how many bytes to copy in one chunk, here it’s 32 MB
  • “conv” specifies how to convert things as they are copied
    • noerror says to continue if anything goes wrong
    • sync does some padding during the copy

When the copy process eventually finishes, check the target partition on the hard drive for errors.

sudo e2fsck -f /dev/sda1

Press enter at the prompts to fix any errors that it finds. You can probably expect one about the free block count, and one about the free inodes count.

Since the image that you just copied over from the SD card is almost definitely smaller than the partition you copied it to, you’ll want to expand it to fill up the available space:

sudo resize2fs /dev/sda1

Format the data partition

That takes care of the root filesystem. Now on to the data partition. Choosing NTFS as the filesystem will allow you to simply plug the drive into any Windows, MAC, or Linux desktop computer and manipulate the drive contents. This can be especially helpful for loading up large amounts of data, such as the family’s music and photo collection, without having to squeeze it all through the network connection.

Format the second partition as follows, filling in whatever name you want the partition to have. I’ve called mine “Data”.

sudo mkfs.ntfs -Q -L Data /dev/sda2

Note: Don’t forget the –Q (Quick) or you’ll be waiting around a while.

Uniquely identifying a drive

Warning: Because the next section involves copying around large strings of unmemorizable data, I recommend performing the steps through a remote SSH window, or from a terminal on the X desktop. The Ids you’ll need to copy need to be copied verbatim, letter for letter, or you’ll find yourself unable to boot, and you’ll have to start over again from your last backup.

Currently, the cmdline.txt file specifies that the root file system is on the second partition of the internal SD card (/dev/mmcblk0p2). We’d like that to say “/dev/sda1” instead, but there’s a problem. As I mentioned earlier, we don’t have any guarantee that this particular drive will be called “sda” in the future. What we need is a way to uniquely refer to this drive no matter what letter it get assigned on any given day.

This was why we built the drive with a GUID Partition Table. Each partition on a GPT device is assigned a universally-unique identifier (UUID).

Note: The difference between GUID and UUID is not important here, they are, for all intents and purposes, the same thing; a very long, randomly-assigned number.

Instead of identifying the root filesystem’s partition by location, we want to refer to it by Id so that it can be found no matter what order the drives were discovered in. Use the “gdisk” package you installed earlier to find out more about /dev/sda1:

sudo gdisk /dev/sda

You should see another not-quite-a-command-prompt like this:

image

Type “i” and press enter to show information about partitons, then type “1” (one) to identify the partition you are interested in. The partition’s details will be displayed like this:

image

The piece of information we’re interested is the “Partition unique GUID” on the second line, not the “Partition GUID code”. Take a picture, write it down very carefully, or select and copy the text if you are doing this through a remote terminal connection like I am.

Type “q” and press enter to exit the gdisk tool.

Boot Configuration

The new root filesystem’s partition is now uniquely identifiable, but the Raspberry Pi doesn’t know to use it. Edit the /boot/cmdline file to change where the bootloader will look for the root filesystem.

sudo nano /boot/cmdline.txt

Find the part that says “root=/dev/mmcblk0p…” and change it to “root=PARTUUID=” and whatever your “Partition unique GUID” was from above. Also add the string “rootdelay=5” at the end. This will give the Raspberry Pi time to discover the USB drive before it tries booting from it.

The result should look like this (sorry for the small picture):

image

Exit Nano, saving your changes (ctrl-x,y,enter)

Reboot (Important, do not skip this step)

Before the next set of changes will “stick”, you’ll need to reboot so that the Raspberry Pi uses the hard drive for the initial load. If you don’t reboot now, nothing you’re about to do will count, and you’ll just have to do it all again. You’ve been warned.

sudo reboot

If everything goes well, you should find yourself back at a login, and you can continue. If something went wrong, go back to your most recent backup and try again.

Mounting filesystems

Partitions and filesystems are not the same thing. Linux now knows what partition to load the OS from, and that’s great, but as things stand right now, it’s still going to mount the root filesystem from the SD card.

To complete the transition to the hard drive, you’ll need to edit the filesystem table. This file controls what gets mounted where, and in what order, and it needs to know where the root filesystem is. Take a look at the current contents.

cat /etc/fstab

image

That third line is the root filesystem, which you can tell by the “/” in the second column. Unfortunately, and it’s still loading from the 2nd partition on the SD card (mmcblk0p2). You could change this to say “/dev/sda1”, but that would only work as long as the drive continues to get the name “sda”.

Fortunately, you can use a very similar UUID-based trick here to uniquely identify the filesystem no matter what letter the device gets. Filesystems have UUIDs too, and you can see them all with this command:

sudo blkid

image

If you look carefully, you may notice a problem. both /dev/mmblk0p2 and /dev/sda1 have the same UUID. So much for being unique, right? This is because of the way we cloned the old root filesystem into a new location. It brought the whole filesystem over, including its UUID. Before you can use a unique Id to identify the drive, you’ll need to make sure it’s actually unique.

You need to give /dev/sda1 a new UUID. You can hand-assign your favorite UUID, or just let the computer pick a random one. Use the following command to assign anew UUID to the first partition on the hard drive:

sudo tune2fs /dev/sda1 -U random

Note: At the time of this edit, there seems to be an issue with the newer “Jessie” release of Raspbian, and it may prevent you from changing the UUID on the partition. If you see an error that says “The UUID may only be changed when the filesystem is unmounted.”, even though it’s NOT mounted, then you’ll need to perform one more tweak. The magic words are

sudo tune2fs -O ^uninit_bg /dev/sda1

I can’t take credit for this one at all. I just found it here. After reading the man page on tune2fs, I still don’t honestly even understand what this has to do with anything, but it does seem to do the trick, so if you get the above error, give this command a shot, then try the “-U random” command again and it should succeed.

Either way, display the device Ids again, just to be sure

sudo blkid

image

Copy down the UUIDs for /dev/sda1 and /dev/sda2, you’ll need them both in a minute. Then open the filesystem table in an editor.

sudo nano /etc/fstab

Change “/dev/mmcblk0p2” on the third line to “/dev/disk/by-uuid/” and the new UUID you just assigned to /dev/sda1. Add another line for /sda2, substituting its UUID, and mounting it as /mnt/data, with a filesystem type of “ntfs”.

The end result should look like this:

image

Don’t worry about the columns lining up, it doesn’t matter, but I’m presenting mine to the public, so I’ve gone ahead and made it pretty

Exit Nano, saving your changes (ctrl-x,y,enter)

The system should be just about ready to boot from the hard drive, but there’s one more thing to take card of before we do that. Currently, the filesystem table says to mount the second partition at /mnt/data. Before that can succeed, you need to create a folder in that location. You can think of it as a placeholder where the actual drive will appear later on.

sudo mkdir /mnt/data

That’s it. you’re ready to reboot again, and this time, everything should be faster.

sudo reboot

You’ll notice that the “ACT” light on the Raspberry Pi will not blink much anymore. That’s because the SD card is no longer being accessed for anything other than the boot partition. The hard drive’s activity light will now blink where the Raspberry Pi’s activity light used to.

When the system has rebooted, double-check your filesystem table to make sure your changes are still there.

cat /etc/fstab

If you see /dev/mmcblk0p2, and no line for your data partition, it’s because you skipped that “Reboot” step above. See? I told you it was important. Redo everything in the “Mounting filesystems” section, and reboot again.

Swap configuration

One thing you may have noticed if you’ve looked at other walkthroughs for booting from the hard drive is that they usually create a “swap” partition on the hard drive to be used as virtual memory.

If you look back at the filesystem table from the beginning of this post, though, you’ll notice that Raspbian never had a swap partition in the first place. That’s because Raspbian is set up to use swap files instead of swap partitions. Raspbian’s swap file lives at /var/swap, and since we just moved the whole root filesystem onto the hard drive, the swap file came along for the ride.

At this point, you’re already running your swap file from the hard drive, and you didn’t even have to do anything. Check it out with the “swapon” command:

sudo swapon –s

This shows the swap summary, which will tell you what swaps are in use. It should have one entry in it (/var/swap). It’s pretty small, though; only 100MB:

image

The partition we created for the root filesystem is 16GB (or more if you so chose). There’s plenty of space left to expand the swap file to something roomier.

Edit the swap file configuration:

sudo nano /etc/dphys-swapfile

The configuration file that came with Raspbian only has a single line in it which says “CONF_SWAPSIZE=100”. This is the size of the swap file in megabytes. Change the value to 2048, which will create a 2GB swap file.

image

Close Nano, saving your work (ctrl-x,y,enter).

Reboot one more time to apply the changes, and check them by typing “swapon –s” again. You should see a table similar to the first time you ran this command, only now the swap file is much larger:

image

Wrapping up

You now have a Raspberry Pi that boots (mostly) from an external USB drive. It also uses this drive for its virtual memory swap file. The whole system should run more smoothly now, and you won’t have to worry about using up your SD card, if that’s the sort of thing you worry about.

Although most of the important stuff is on the hard drive, you should probably make at least one more backup of the SD card for safety. I may cover backing up the root filesystem from the hard drive later on in another post, or update this one with the changes.

What’s next?

In the next post, we’ll put the new drive to work sharing files over the local network.

This entry was posted in Computers and Internet, Home Server, Raspberry Pi and tagged . Bookmark the permalink.

291 Responses to Raspberry Pi Home Server: Part 6, Adding a hard drive

  1. Pingback: Raspberry Pi Home Server: Index | MelGrubb.ToBlog()

  2. Pingback: Raspberry Pi Home Server: Part 5, Remote Desktop | MelGrubb.ToBlog()

  3. Pingback: Raspberry Pi Home Server: Part 7, Sharing Files With Samba | MelGrubb.ToBlog()

  4. Pingback: Raspberry Pi Home Server: Part 10, CrashPlan | MelGrubb.ToBlog()

  5. cheese says:

    Hi,i have an issue here that it cannot detect my external hard drive ,I reformatted it in NTFS system but ntg appear in the pi

    • Mel Grubb says:

      If the drive just never shows up at all, meaning it never appears as sda/sdb/etc, then you may have simply found a drive that’s just not compatible with the Pi. I’d double check that it’s getting power. I have an external 3.5″ drive that has no power cord other than the USB cable. It works just fine on normal computers, but the Pi just doesn’t have enough juice to run it. The drives I’m actually using for my server are in a separately-powered external enclosure.

      • Can I use an external drive and route it through a powered USB hub?

      • Mel Grubb says:

        Absolutely, in fact I thought I suggested that in the article. Not only that, but you can even power the Pi off of its own hub. The micro-USB port that powers the Pi doesn’t use the data lines, only the power lines, so you can plug it into its own hub, and it won’t try to “talk to itself”. I run one of my systems this way. I have a good, strong, powered hub, with the hub’s USB cable plugged into one of the Pi’s regular ports. Then, I have a micro-USB cable running from the Pi to a regular port on the hub along with any external drives. Hitting the switch on the hub powers up the whole thing. This also avoids problems with the hard drive “back-powering” the Pi, and preventing it from shutting down.

  6. cheese says:

    ok thx for your information , can i use a USB drive like a 64 GB capacity would u recommend or the ones using external power

  7. Mel Grubb says:

    Flash drives usually don’t draw much power, so it should be fine. Another approach you might try is to hook up your hard drive through a powered USB hub. They’re pretty cheap, and would take the power drain away from the Pi.

    • cheese says:

      do you mean that the hub is connected to the Pi while connecting to the drive ,or another hub separated from the Pi and power?

      • Mel Grubb says:

        I mean that you get a USB hub that has its own power adapter, plug that hub into one of the full-size USB ports on the Pi, and plug the drive into the hub. If the hub has its own power supply, then it should be able to power the drive.

  8. cheese says:

    So if i have a powered hub thats only provide power for devices ,it won’t work cause no data will be transfered?mine is a powered hub

    • Mel Grubb says:

      No, it’s just like any other hub. If you plug the hub in to one of the full-sized USB ports on the Pi, data will be transferred. The power for the drive will come from the hub instead of the Pi, though. This should allow the drive to run. It’s just a guess, but I’m thinking that the drive simply draws more power than the Pi can provide on its own.

      • cheese says:

        so i need to get another one…cause that only provides power for the Pi

      • Mel Grubb says:

        Actually, you CAN power the Pi off of its own hub. It looks a little funny, but it works. The micro-usb port on the Pi ignores the data lines entirely, and only uses the power lines. If your USB hub has enough power, you can run everything from it. You plug the hub into the Pi’s regular USB port like you would anything else. Next, use a micro-USB cable to plug the Pi’s power line into the hub. Finally, plug the hard drive into the hub. when you power up the hub, everything should come on at once. I haven’t done this myself, because my power strip has USB ports on it, and I use those to power the Pi. I did try it once, though, and out worked fine. Your mileage may vary, of course.

  9. cheese says:

    my hub is this ..think it’ll work ?
    http://www.oricoonline.us/info.aspx?pid=839

  10. Mel Grubb says:

    Aha, I see the problem. That’s not a hub. That’s just a charger. Do you have, or can you borrow a powered hub from someone just to check whether power is the problem?

  11. cheese says:

    All right , i’ll see how i can solve it

  12. cheese says:

    Well..my drive is busy for awhile …after that it stop but ntg happen also

  13. cheese says:

    i mean the copy part to the sda1 ,my drive stops running but ntg appear on screen so i can’r command anything

  14. Mel Grubb says:

    Do you mean the partition copy from the SD card to the drive? Yeah, that can take a while, depending on the size of the partition. My advice is to leave it running way longer than it seems like it should take. Go have dinner and come back. If it just never finishes, then maybe there is something else going on. I’ve done this several times, though, and didn’t have a problem. Remember that booting from the hard drive is optional. It turned out that my Pi actually didn’t like booting from the RAID box because of the drives I used constantly entering power saving mode, so my own personal server currently boots from the SD card for now. If I upgrade storage later on, I’ll go back to booting from the hard drive again.

  15. cheese says:

    I solved it with “sudo reboot” and now working with Samba things..

  16. cheese says:

    So now i have an serious issue my just dropped my HDD so…ya its beyong repair so how to i boot back from the SD card?

    • Mel Grubb says:

      Just restore the last backup of the SD card from before you added the hard drive. If you weren’t making backups (shame), then you can just undo the changes to the fstab and cmdline.txt files. Change the “root=” part in cmdline.txt back to what it originally said. If you don’t have that written down anywhere, then you might be safer starting over than trying to guess.

  17. cheese says:

    i backup after doing it…

  18. cheese says:

    Guess i have to start all over again until i get another HDD

  19. cheese says:

    All right,thx for the advice starting over with the image…

  20. bkonicek says:

    This guide is great. However, I got to the step where I reboot after setting the boot to the UUID of the external hard drive. Once I reboot I can’t connect through SSH anymore. The first time I realized I forgot to resize the partition, but the second time I made sure to follow each step exactly. Any ideas what might be going wrong?

    • Mel Grubb says:

      Are you able to hook the Pi to a monitor to make sure it’s booting? If the UUID gets even a little bit wrong, it will never find the drive, and will fail to boot. The article warns about this. If you can see that it’s booting, then I don’t really have any advice. I don’t see how the drive and ssh could be related apart from the machine not booting at all. If that’s the case, then roll back to the previous backup and try again, but be extra careful about typos.

      • benkonicek says:

        Thanks, I can hook it up to a TV to check but didn’t have the time last night. I reread your post and another search on google and realized that even after doing it twice and reading carefully I left out the PARTUUID after “root=”. I had thought you were saying to replace PARTUUID with the actual UUID rather than enter it after, so this is probably the issue. This is what I get for watching Netflix while I do this kind of stuff.

        I’ll try it again after work.

  21. Alan Cai says:

    a quick question:
    will this kind of settting make the hard drive spin all the time? I want to use the raspberry pi for a server for bt sync and my SQL. I am wondering whether any of the three setting (1) booting from the external drive, (2) using bt sync, and (3) using my sql will make the driver spinning all the time.

    Thanks

    • Mel Grubb says:

      Since this is a server, I think drives spinning all the time is to be expected, but most consumer hard drives will go to sleep on their own when they are not being actively used. With my own server, I found that my raid enclosure didn’t LIKE my hard drives falling asleep to save power. I reverted to using my SD card for the system drive, and the raid for bulk storage. If power is a concern, then maybe you should do the same. That way the drive can spin down, and you’ll use only the SD card when the server is idle.

      • DavidA says:

        Hi Mel
        I am very interested in your instructions but am confused about whether or not to use the SD card for the system drive. Your article seems to recommend that the system be put on the hard drive but in the comment above you seem to suggest it be put on the SD card. Which do you now recommend and, if the answer is the SD drive, what changes would be made to the instructions in your article on this page to achieve this?
        (I am hoping to build my first home server, using a Raspberry Pi 2 and a single 2TB USB hard drive.)
        Best regards

      • Mel Grubb says:

        I think my problems had more to do with the external drive in question being a RAID enclosure with desktop grade drives in it. I’ve had no problems using a regular hard drive as the system drive. Only the RAID box had issues.

  22. nosko says:

    When i use command swapon –s, get error swapon: –s: stat failed: No such file or directory. Do you know what i do wrong or miss something. Sorry for my bed english

    • Mel Grubb says:

      It would have to be complaining about the swap file itself. Did you start from a fresh Raspbian installation, or are you trying to add on to an existing instance? If you are following my posts against an existing installation, I can’t guarantee that it was set up the same way as Raspbian to begin with. If you started from a fresh install, then it’s possible that something has changed in the current distribution of Raspbian, in which case I’d have to look into it before answering.

      • linxiaopi says:

        I had the same error on my second implementation of your tutorial, and the only difference is that I am using the “Jessie” distribution this time instead of “Wheezy.”
        The solution is to use “sudo” prior to “swapon -s”. Apparently, “Jessie” needs elevated permission for the swapon command.

      • Mel Grubb says:

        I’ll incorporate that into the post. Thanks.

      • Mel Grubb says:

        Interestingly, if you look at the screenshot, I was running that through sudo already, probably out of habit. Now the text matches the screenshot. But I have to wonder whether sudo was always required and I just mis-typed the instructions, or whether this is truly new under Jessie. I’m curious, but not quite curious enough to re-run the whole thing under both releases to find out.

      • linxiaopi says:

        Sorry. I was mistaken. Go ahead and delete my last post (and this one). I won’t be offended. The following are some details of what happened. Maybe it will help you pinpoint the problem.
        I’m not sure what happened, but I was getting the same error as nosko was getting, but when I tried “sudo” it worked. After a reboot, “swapon -s” without “sudo” worked again. However, after another reboot, I had numbers filling my screen and had to restart from the image I created just prior to this part of the tutorial. On a whim, after creating my root and data partitions on my USB HDD, I decided to check for errors with the “sudo e2fsck -f /dev/sda1” command, and sure enough, I had errors on around the 5th error check that e2fsck ran through, and numbers filled the screen again and continued to do so for a minute or so. I was given the chance to allow the errors to be fixed, which I allowed, and then numbers filled the screen again, and I was given another chance to fix these (other?) errors, which I allowed, and everything was fine.
        I don’t know why this happened, but I wonder if the “mkpart” command needs a more precise number than 16GB when creating the second partition (perhaps in MB or KB), or if it would be safer to start the second partition at 17GB (shame to waste ~1GB, though). Another possibility is that my HDD is failing/corrupted, but I’m not sure how to check this via Raspbian/Linux.
        I have just finished running through this part until the “Swap Configuration” section, and I can confirm that, now (after having fixed errors with “e2fsck”), “swapon -s” does indeed work without “sudo.”
        I hope that helps.
        I’ve really been enjoying your tutorial and have found it immensely useful. Thanks for all the hard work!

        Oh, and just a reminder: the “sudo tune2fs -O ^unint_bg /dev/sda1” command still needs an “i” added to the “^unint” part.

      • Mel Grubb says:

        I’m not really a Linux expert. I usually work in .Net development on Windows, so the bulk of my Linux experiment has been through the research and building of the Raspberry Pi Home Server. I can tell you what I’ve seen, though. I originally tried booting from my RAID box, but saw the same sort of errors you’re seeing, with numbers filling the screen. It would just do that sometimes. It would run for about a week at a time and then freak out. I finally decided that it just didn’t like booting from that particular RAID box, and I switched back to booting from the SD card, using the RAID just for the mass storage. I figured that maybe the RAID box introduced enough lag to the equation that it wasn’t fast enough to be used for the swap file or something. I ran with the whole system booting and swapping to the SD card for about a year without a problem. SD cards are cheap enough to be replaced when they wear out. Just take backups every now and then, and you should be fine.

        Most recently, I added a small SSD to the server, and set it up to be my boot and swap drive. I tried to find a 32GB SSD like the ones they put in tablets, but found that it was cheaper to just grab an off-the-shelf 120GB drive and use an enclosure I already had around. Anyway, that has been running great for me.

        You may be right about the size of the partition. I know there’s some issue with “aligning” your partitions, but don’t have much more detail than that. All I can say is that phrasing the partitioning in terms of “16GB” and “100%” has worked on 4 different drives I’ve personally tried without complaining. Sorry I don’t have a simple, sure-fire fix for the problem.

  23. Alex says:

    Hi, Think you for your instruction first.
    I met a problem, I can’t find /boot/cmdline.txt file.
    Then will can I find information of current boot device.

    • Mel Grubb says:

      Off the top of my head, I would make sure you’re logging in as the “pi” users when configuring stuff like this. It could possibly be a permissions issue. I can’t imagine any other reason you wouldn’t be able to see the file. It HAS to be there, otherwise the whole thing wouldn’t boot. You are starting from the standard Raspbian image, right?

      • Alex says:

        I use pi. I don’t think it is permission issue.
        I alreay use ‘sudo find / -name “cmdline.txt”‘ to search the file. But nothing.

      • Alex says:

        Maybe the framework of mine is too old. it is berryBoot.
        Finally i find the file in its ‘FAT’ partition.
        But the content is like below
        ‘smsc95xx.turbo_mode=N elevator=deadline quiet bootmenutimeout=10 datadev=mmcblk0p2’.
        Maybe i need to start it from very beginning.
        Thank you all the same

      • Mel Grubb says:

        Yes, that’s probably it. The series is written against a plain, out-of-the-box Raspbian image. Berryboot is a multi-boot front-end, right? It will probably have moved each OS into it’s own partition, and taken over the job of the commandline file. Without knowing more about BerryBoot, I couldn’t tell you where it has moved the information from the Raspbian commandline, though.

        I think I will update the entire series, and draw attention to the base Raspbian assumption at the top, next to the link to the Index. At least it should make readers aware that starting from a different distribution means that the instructions may not work exactly as written.

      • Alex says:

        It is the best series i have seen for pi.

  24. Pingback: The Raspberry Pi Home Server – Part 1 | UnaX.dk

  25. JB Rolland says:

    Hi,
    I am trying to install my old Lacie 250GB external USB but after step “Reboot (Important, do not skip this step)”, I cannot write /etc/fstab and and I am having the following message when I press completion/tab button:

    -bash: cannot create temp file for here-document: Read-only file system

    I have googled a bit but not found any interesting fix so I try to move the sector of the sda1 from 0-16GB to 32-48GB sector but same issue happen again.
    Any idea would be very much appreciated.

    Thanks

    • Mel Grubb says:

      The first thing off the top of my head would be to make sure that you’re logging in as “pi”, and running most of the commands through “sudo”. Other than that, make sure the write tab on the SD card didn’t get flipped.

      • JB Rolland says:

        Hi,
        Thanks but I did make sure I was running as sudo and when I rollbacked /boot/cmdline.txt, it was back to normal i.e no more “-bash:…” message (so no SD card flipped).
        I tried with a WD HDD, but this time the dd command never stop and system is becoming unstable after this.

      • Mel Grubb says:

        By any chance, is the SD card completely full? That can result in the “cannot create temp file” message.

      • JB Rolland says:

        Does not seem to:

        pi@raspberrypi ~ $ df -h
        Filesystem Size Used Avail Use% Mounted on
        rootfs 15G 3.5G 11G 25% /
        /dev/root 15G 3.5G 11G 25% /
        devtmpfs 215M 0 215M 0% /dev
        tmpfs 44M 396K 44M 1% /run
        tmpfs 5.0M 0 5.0M 0% /run/lock
        tmpfs 88M 0 88M 0% /run/shm
        /dev/mmcblk0p1 56M 9.7M 47M 18% /boot

      • Mel Grubb says:

        I’ve done a little basic searching, and it seems that this happens to quite a few people, and it’s always traced back to a corrupted file system. The reasons for that could be anything from not shutting down the Pi properly to using a cheap SD card. You’re at the point where you haven’t yet changed over to booting from the hard drive, right? That probably means that the SD card has gone wonky. I’d suggest rolling beck to the latest “happy” backup you have, and trying again from there.

  26. Kevin says:

    Hi! I love your tutorials – it made setting up my Pi a snap! 🙂 I’m playing with a Pi 2, and I would really love to be able to backup the root filesystem from the hard drive. Did you ever post on this subject yet?

    All the best! – Kevin

    • Mel Grubb says:

      I have not covered that, partly because I’m not really running my system configured that way. It turned out that my server didn’t do every well with its root file system on the external drive, which is a RAID enclosure in my case. It will boot off a single drive just fine, and will even boot from the RAID, but it doesn’t like running from there continuously. I’ve reverted my own server to booting from the SD card, but storing everything else on the RAID. It may be my enclosure, or more likely something about the drives I put in it. I’ll look into it, though.

  27. Kevin says:

    Thanks – that would be awesome! I’ve followed your steps and had an awesome setup since November, but last week, “something” died. I thought it was the SD card, but it seemed to run fine. Turned out it was the hard drive. I was able to save my content files, but I’ve needed to build pretty well from scratch. This time around, I’m using a newer Raspberry Pi 2, and I must say that it is quite peppy!

    Any words of wisdom would be greatly appreciated, and again, thank you for this amazing tutorial series!

  28. Nigel says:

    Every time I try and partition the drive It comes back with File System’ ntfs’, This happens even if I insert ext4 in the ‘mkpart primary ext4 0GB 16GB’ command. Any idea what I’m doing wrong?

    • Mel Grubb says:

      The file system gets established by mkfs.ntfs in the next section, not by mkpart. mkpart just carves out the space where the file system will go.

      • Nigel says:

        Cheers. You wouldn’t believe how long I’ve been on that step!
        Thanks for the help – I appreciate it.

  29. Craig says:

    Hi Mel. I followed your instructions and added an external USB drive to my Pi 2. I can see it listed in /dev as sda and made one large partition. I don’t want to boot from it, simply use it for data storage. I added the external drive to fstab and have it mounted in /mnt/data. I then created a Samba share to the same drive. What I am noticing now is when I copy files to /mnt/data it seems to fill up the SD card and not copy the files to the 1TB external drive.

    What did I miss?

    • Mel Grubb says:

      It may be that the drive isn’t actually mounted, and the files are going to the placeholder directory instead. This is part of why I used Webmin to set up the share in the article. It really takes a lot of the guesswork out of it. Make sure you’ve checked “Mount now”, and see if that helps. Clear out the folder first, or I’m not sure what will happen.

      • Craig says:

        You are correct, it doesn’t seem to mount it when the server boots. I can manually mount it using “sudo mount /dev/sda1 /mnt/data” and when I then run “df -v” I can see my 1TB drive with 1% space used. The boot process doesn’t seem to do auto mount the drive. I tried running through the process again creating a new mount point named “share”, and updated fstab…unfortunately same result. I’m not sure why it won’t mount the USB drive at boot

  30. Craig says:

    Reply to my last post. I added the attribute rootdelay=10 to my cmdline.txt file and the 1TB drive auto mounts now. Someone mentioned some drives need this as the Pi2 boots faster than the original model and the USB ports aren’t ready.

  31. timxjr1300 says:

    I’m having problems getting the Pi to see the boot partition on sda3 (in my case).
    The start up gets to the point where it says the Pi is waiting for the disk which is odd because when I try to start up from SD card everything works and the disk mounts with my 3 partitions. (I have rootdelay=10 and have tried moving it to 20)
    I wonder if anyone has a problem with doing this on a Pi 2?

    • Mel Grubb says:

      I’m going to try to build a server from scratch over the weekend using a Pi2, and see how that goes. I haven’t personally seen the Pi say it’s “waiting” for a drive before, so that’s a new one. I did have problems using a drive that kept entering power saving mode, though. The Pi didn’t like having to wait for its swap file to be ready. I am currently booting from the SD on my own server as a result. ________________________________

    • Mel Grubb says:

      Okay, I got around to verifying things on a Pi2 this morning, and the disk mounting still works according to the directions. I did find one thing, though. I used a smaller, older external drive for this test, and during the file system check, it found more errors that just the counts I mentioned in the article. I switched to a different drive and everything went fine, but that first drive was never quite right in the head, so if you had additional errors, that might explain things. Also, make sure that all the drivers and partitions have unique uuids per the instructions in the article. ________________________________

  32. Hi Mel,
    I followed your instructions and it worked perfectly. You blog post is the one reference which explains all the required steps from start to finish. Kudos!

  33. Reblogged this on electronics – my hobby since childhood and commented:
    The perfectt guide to booting up the Raspberry Pi from a USB stick

  34. Pingback: Booting up the RPi from a USB flash drive | electronics - my hobby since childhood

  35. Hi Mel,

    Great blog. The most thorough I have found so far. Perfect for a Noob like me hoping to learn fast.

    I have a couple of questions about “Part 6: adding a HD”. I am keen to move the root over to my external HD

    1. I will use ext4 for my whole drive as this works for me and I will use my backup/redundant/NAS drive in NTFS anyway, which will help with accessing data on my mac. But, I wonder if there is any advantage to setting 2 ext4 partitions – one for root and one for data? I imagine not (but this is all new to me) so I plan to create just 1 ext4 partition. What do you think?

    2. In your guide above, after copying the root from the SD to the HD you then resize the root/boot image to fill the partition using this command “sudo resize2fs /dev/sda1”. I am unsure what this is actually doing. If I only have 1 partition (see Q1) and I resize the image to fill the available space, will this effectively reduce the available space I have on the HD for my data files?

    3. Is there any advantage to making the swap size more than 2GB? This is the virtual memory, right? I have a new R-Pi 2 and I want to see how far I can push it – NAS/Owncloud/KODI/Retro-Pi all on one device if possible.

    Many thanks for your help

    • Mel Grubb says:

      ext4 works fine, and actually a bit more efficiently than NTFS on a Linux machine, but you’ll need to know a little bit more about Linux permissions than you would with NTFS. chmod-ing your shared folder to 777 will leave it wide open, which is the goal for a home server share. I recommend using Webmin’s Samba module to set up the share if you’re a Linux novice. The only advantage I see to making two partitions is that it makes it easier to nuke and remake the root partition if you want to make large changes later on.

      resize2fs expands the file system to the full size of the partition it’s on. If you had a 2gb, and a 2tb partition, expanding the first one should only make it 2gb, and not take over the space in the other partition. If you only had one partition, and DIDN’T expand the file system, then a bunch ofspace on your hard drive would go to waste.

      Like most things with computers “it depends”. I’m just using the old standby “twice your physical RAM” rule here, which would be 2gb for a 1gb Pi2. Experiment and see if you can detect a difference. A lot of how it performs will have to do with what the computer is busy doing at the time. I have left my original model B doing just “server” tasks, and added a Pi2 to my collection, which I’m using for experiments and Media/RetroPie. They’re cheap like that.

      • @Mel: I had a question regarding the resize2fs. If we do a ‘dd’ from a partition A to a partition B of a large size, where exactly in partition B is it stored that “xx” is the effective size of the partition (equal to that of partition A)? I couldn’t find any reference to this.

      • Mel Grubb says:

        It’s like when you copy the Raspbian image to your SD card. The image is only 2-3GB, even though your SD card may be 16GB. Until you expand the filesystem, it will continue to appear to be only 2-3GB. Here, we’re copying one partition’s filesystem to another partition. Until you expand that partition, it still THINKS it’s whatever size it was on the SD card.

      • Oh I understand the theory. What I’m saying is: where is the apparent size kept? If the OS is not looking at the real size of the new partition by itself, it must be kept in some file right? I cannot find any reference to that. Basically it boils down to what file does the resize2fs modify, to make the change?

      • Mel Grubb says:

        Without getting into some deeper Linux research, I really don’t have an answer to how our where it stores that information. This being Linux, though, it’s almost sure to be a plain text file somewhere, right? What is it you’re trying to do, exactly? ________________________________

      • Oh I was just curious to know that’s all. I am doing experiments on my raspberry pi and regularly transfer data between micro sd cards and a usb drive.

      • I figured out where this info is stored. It has something to do with the “block group descriptor table” of the ext4 FS. You can get this info by doing:

        sudo dumpe2fs /dev/sd(xx)

        resize2fs simply changes the ‘block count’.

  36. Dominic Maratt says:

    Glad I found your blog. I will be adding a USB Drive to Pi2. Is it possible to use NTFS 4.1 ?
    (Saw there was a Kickstarter funded product called Slice, with a nice enclosure, lit by 25 RGB LEDS, Pi Compute module and 1TB internal Drive selling for 189 British Pounds )

    • Mel Grubb says:

      Sorry this comment slipped by me. I’m not sure exactly which versions of NTFS are supported by ntfs-3g, but a drive that was formatted on my Windows 8.1 computer worked just fine with it, so it seems to be completely up-to-date at this time.

      • Dominic Maratt says:

        Last weekend I bought a drive enclosures with just a USB 3 interface and populated it with a spare 120GB SSD and connected it to RPi 2, ls /dev found the drive, but parted could not read the partitions on the disk.
        So I removed SSD and put a 160GB 7500 RPM Drive in the enclosure and connected it to Rpi, no luck – I could hear it clicking continuously.
        When the same 160GB drive was connected to RPi2 using an externally powered APRICORN drive interface, parted had no problem finding the partitions.
        It appears Rpi cannot provide enough power to even smaller capacity drives like the 120GB SSD, (It needs 1.0A at 5V, or the 160GB 7500 RPM spinner (800mA at 5V), 512GB SSD needs 1.7A at 5V. The plan to use a compact drive enclosure with just one USB cable for power and data was not successful even for the smaller drives.

      • Mel Grubb says:

        The capacity of a drive seems to have less to do with its power requirements than its age does. I have a 160Gb drive that the Pi can’t power, and a 500Gb drive that it can. Remember also that the strength of your power supply has a lot to do with it. If you have a strong enough power supply, and a B+ or newer, you can set the max_usb_current value in boot/config.txt to increase the output. (see http://raspberrypi.stackexchange.com/questions/27708/is-setting-max-usb-current-1-to-give-more-power-to-usb-devices-a-bad-idea)

    • just a note: the highest version of the NTFS format is 3.1. (introduced in winXP). it’s driver, ntfs.sys may have a higher version, but that has nothing to do with the version of the format itself.

      • Dominic Maratt says:

        Thank you for the reply. I made a mistake in typing, what I wanted to ask was about NFS 4.1 and not NTFS4.1.

      • Dominic Maratt says:

        Inserting the line –> “max_usb_current=1” in the /boot/config.txt did help.
        (BTW in RPi2 Model B — /boot/config.txt – does not have a line like this for controlling default USB current. It had to be inserted.)
        After this line was inserted, the 160GB 7500RPM drive stopped clicking and started spinning. Parted was able to print all partitions. Thank you very much.

  37. cmd204 says:

    I plugged in a new WD Elements 2TB hard drive (this one: http://www.amazon.com/gp/product/B00DULWSXI/ref=pd_lpo_sbs_dp_ss_1?pf_rd_p=1944687542&pf_rd_s=lpo-top-stripe-1&pf_rd_t=201&pf_rd_i=B002QEBMCI&pf_rd_m=ATVPDKIKX0DER&pf_rd_r=0ABBPX7KACK1RK9RFD54)

    The drive can’t be found in the parted area and the drive spins up for a few seconds, makes a click, and then spins up again. I don’t think it’s getting enough power. I bought a plug that is made for the raspberry pi. I think the drive isn’t getting enough power to spin up though, and that’s why it’s making the clicking noises and is not being recognized by the Pi 2. Any suggestions?

    • Mel Grubb says:

      The USB ports on the Raspberry Pi don’t put out enough power to run a drive on their own, no matter how strong the power supply TO the Pi is. You need a drive with its own power supply, or a powered Usb hub. I recommend the hub, myself, but if the drive has a place for a power adapter, that works fine too.

  38. On the “Change UUID” step you could also use “sudo tune2fs /dev/sda1 -U random” to avoid the use of an external service…

  39. Roel says:

    Thank you Mel for this wonderful article!

  40. zacchaeus says:

    Hi Mel,

    Thank you for an excellent series of tutorials. I have one question that’s left me scratching my head a bit. I’m asking out of interest, because the instructions clearly work as they should.

    Under “Mounting filesystems” you say that “Linux now knows what partition to load the OS from [at this point, but] it’s still going to mount the root filesystem from the SD card.”

    If this is the case, how does it know to access the version of fstab on the HDD in the steps that follow (if, indeed, that is the version of fstab we edit at this stage)?

    Many thanks,

    Gareth

    • Mel Grubb says:

      I’ve never worked under the covers of Linux, so this is a vast oversimplification, but what you’re looking at is the difference between the kernel being allowed to see stuff, and YOU being allowed to see stuff. Cmdline.txt told the kernel where to find the root file system, and the kernel made some assumptions about where things had better be. As the boot process continues, it may re-evaluate some of those assumptions, but at first, those assumptions are set in stone. I suppose you might be able to create a second fstab in another place and Linux would switch to it after processing the first one which you’d better not mess with. Then again, it might end the universe as we know it. Great power, great responsibility, yada yada.

      • It has been listed as a big since 2007, as mentioned here: https://bugs.launchpad.net/ubuntu/+source/partman-partitioning/+bug/148743 AND https://bugs.launchpad.net/ubuntu/+source/gparted/+bug/737387. It certainly goes against the uniqueness of “UUID”s, but that’s just the way it is. I think in actual operation, the behaviour of the OS is undefined, as to which one it chooses. Its possible that it is looking at the ‘fstab’ from the USB drive, but picking the first partition for “/” that matched the duplicated UUID in question, and then continuing with the rest of the ‘fstab’ on the USB drive.

      • Mel Grubb says:

        I don’t think I would call it a problem, really, but it’s definitely something you need to be aware of. If it behaved any other way, things wouldn’t work. For instance, if I just wanted to upgrade a hard drive, I would clone the old drive to the new one. If the uuids didn’t come along for the ride, then the system wouldn’t be able to find the drive it needed when it started up, and would fail to boot. Cloning a drive means just that. The result should be identical to the original.

  41. KVT says:

    Hi Mel,

    Thanks for this great blog.
    I went through parts 1-5 without any problems. However, in this part something goes wrong.
    I’m able to create the ext4 partition on my SSD drive and reboot the Pi with the modified cmdline.txt (« root=PARTUID=… ») I also could change the UUID of /dev/sda1

    When I try to mount the SSD by changing the file /etc/fstab a got the error message »error writing /etc/fstab : read-only file system”. This seems to be the case for a lot of files (when i try to run startx I get ” mktemp: failed to create file via template `/tmp/serverauth.XXXXXXXXXX’: Read-only file system”)

    I went through part 1-5 more than 5 times, so I’m pretty sure that I’ve made no mistakes.
    I’m using a Pi 2 with a 8GB SD card, 60GB SSDnow V300 Kingston and a 3A USB power supply. (usb_max_current set to 1 in config.txt)

    Any ideas what might be going wrong ?

    • Mel Grubb says:

      Further down in the comments, you should find another question about read only file systems. I did some searching the last time someone mentioned this, and all the reports I found traced the problem back to the SD card getting corrupted at some point, usually by not shutting down the system properly. Do a search for “Linux recover read only file system”, and you should find plenty of posts on the topic. Most suggest running “sudo mount -o remount,re /”. YMMV.

      • Mel, I think that command should be: sudo mount -t ext4 -o rw,remount -force /dev/sd /

        (rw not re)

        and do a filesystem check before you re-mount in read-write mode. Temporarily, boot and use the root FS on the SD card to check the USB partition:
        sudo fsck /dev/sd

  42. KVT says:

    Hi,
    Thanks to Mel and electronicsguy for your reactions.
    I bought an new SDcard, formatted it with SD formatter, always shut down the Pi correctly but the problem persists. I’m an absolute beginner, so I tried “dmesg | grep sda” :

    (this time, I made 1 partition (sda1) and only used 1 USB device so I didn’t use the UUID in cmdline.txt)

    ….
    console=ttyAMA0,115200 console=tty1 root=/dev/sda1 rootfstype=ext4 elevator=deadline rootwait
    [ 2.538733] Waiting for root device /dev/sda1…
    [ 7.712886] sd 0:0:0:0: [sda] 117231408 512-byte logical blocks: (60.0 GB/55.8 GiB)
    [ 7.726448] sd 0:0:0:0: [sda] Write Protect is off
    [ 7.736140] sd 0:0:0:0: [sda] Mode Sense: 47 00 10 08
    [ 7.736573] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, supports DPO and FUA
    [ 7.758188] sda: sda1
    [ 7.767973] sd 0:0:0:0: [sda] Attached SCSI disk
    [ 7.832943] EXT4-fs (sda1): INFO: recovery required on readonly filesystem
    [ 7.844944] EXT4-fs (sda1): write access will be enabled during recovery
    …..
    [ 7.936039] blk_update_request: critical target error, dev sda, sector 2623488
    [ 7.948178] Buffer I/O error on dev sda1, logical block 327680, lost sync page write
    [ 7.960934] JBD2: Error -5 detected when updating journal superblock for sda1-8.
    [ 7.973561] EXT4-fs warning (device sda1): ext4_clear_journal_err:4658: Filesystem error recorded from previous mount: IO failure
    [ 7.990429] EXT4-fs warning (device sda1): ext4_clear_journal_err:4659: Marking fs in need of filesystem check.
    ….

    Could it be possible that the SD card has nothing to do with it and that the EXT4-fs(sda1) is read only ? see line [ 7.832943]

    When do “sudo fsck /dev/sda” I got

    fsck from util-linux 2.20.1
    e2fsck 1.42.5 (29-Jul-2012)
    ext2fs_open2: Bad magic number in super-block
    fsck.ext2: Superblock invalid, trying backup blocks…
    fsck.ext2: Bad magic number in super-block while trying to open /dev/sda

    The superblock could not be read or does not describe a correct ext2
    filesystem. If the device is valid and it really contains an ext2
    filesystem (and not swap or ufs or something else), then the superblock
    is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193

    • Mel Grubb says:

      If you’re up to the point that you’re booting from the external drive, then that could absolutely be it. You’d need to check the external drive for errors in that case. If you’re not, then I’m not sure what to tell you. Is the card the same brand as before? Check the raspberry pi verified hardware list (google knows where it is), and see if others have reported problems with that particular brand, maybe?

    • adding to Mel’s reply… It may indeed be a problem with the USB drive. I can think of 2 things – either the filesystem was incorrectly copied onto it. You could check this by mounting the USB drive on another linux computer and see how it mounts, if you can edit files there, etc.

      A second problem could be, that the USB drive has become write-protected. Some flash drives, when they are failing to write correctly due to wear, enable a flag in the controller saying that it’ll be read-only now. This is unfortunately permanent, and is a mechanism for you to recover data from an about-to-fail drive. If that is so, there’s nothing you can do other than buying a new one.

      A third possibility exists, which though is rare. Some flash drives come with a write protection switch. Check that this is not so in your case.

  43. KVT says:

    Hi,
    Something very strange is going on here. I’ve replaced the Kingston SSD by a Toshiba HDD (MQ01 ABF032 250GB) and now I get this (dmesg | grep sda) :

    I use a very cheap enclosure (Olmaster Transition2). Could this perhaps cause these problems ?)

    15200 console=tty1 root=/dev/sda1 rootfstype=ext4 elevator=deadline rootwait roo tdelay=5
    [ 4.783098] sd 0:0:0:0: [sda] 625142448 512-byte logical blocks: (320 GB/298 GiB)
    [ 4.796037] sd 0:0:0:0: [sda] Write Protect is off
    [ 4.805771] sd 0:0:0:0: [sda] Mode Sense: 47 00 10 08
    [ 4.806287] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, supp orts DPO and FUA
    [ 7.056747] sda: sda1
    [ 7.066668] sd 0:0:0:0: [sda] Attached SCSI disk
    [ 7.733563] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)
    [ 12.916868] EXT4-fs (sda1): re-mounted. Opts: (null)
    [ 14.363838] EXT4-fs (sda1): re-mounted. Opts: (null)
    [ 20.222353] sd 0:0:0:0: [sda]
    [ 20.222399] sd 0:0:0:0: [sda]
    [ 20.222444] sd 0:0:0:0: [sda]
    [ 20.222498] sd 0:0:0:0: [sda] CDB:
    [ 20.222581] blk_update_request: critical target error, dev sda, sector 262362 4
    [ 20.222609] blk_update_request: critical target error, dev sda, sector 262362 4
    [ 20.222697] Aborting journal on device sda1-8.
    [ 21.111239] EXT4-fs error (device sda1): ext4_journal_check_start:56: Detecte d aborted journal
    [ 21.125461] EXT4-fs (sda1): Remounting filesystem read-only

    • Mel Grubb says:

      Unless you’re starting from scratch each time, I’d say the problem isn’t the drive, it’s the contents of the SD card you’re copying from. The error exists on the filesystem before you do the copy, so it won’t matter what you copy it to, the damage will still be there.

      If you ARE starting from scratch each time, then there’s something else going on here, and I’m not sure what to tell you. At some step in the process up to this point, the filesystem is getting corrupted. That’s usually from not ejecting it from your main computer before removing it, or from powering off the Pi without letting it shut down properly. Remember to wait for the lights to stop blinking before removing power from the Pi.

      So far you’re the first to report this recently. If more reports like this come along, then perhaps something has changed. I’m about to walk through the entire process again myself in the next few weeks, so I’ll know if it’s a change in any of the software packages causing the problem. Keep an eye on the comments, and see if anyone else says something similar. And again, I recommend checking your SD card against the list of known good/bad brands: http://elinux.org/RPi_VerifiedPeripherals

  44. KVT says:

    Hi Mel,

    I’ve replaced my old HDD enclosure (http://www.thepcharbor.com/shop/en/other-storage-products/2256-olmaster-rambler-transition-2-hdd-docking-enclosure.html ) by a newer one and now everything works fine (http://www.bluestork.eu/products/ehd-enclosures/item/429-external-hard-drive-enclosure-for-25q-hard-drives-sata?lang=en )

    I use the Kingston V300 60 GB SSD (http://www.kingston.com/en/ssd/v#sv300s3 ) and an Integral Ultimapro-X SDHC 16GB Red memory card (not on the verified Peripherals list) (http://www.integralmemory.com/product/ultimapro-x-sdhc-95MB-read-60MB-write-class-10-uhs-i-u3-memory-card )

    I didn’t change anything on the SSD. I reused the copy (dd if=/dev/mmcblk0p2…) made with the old HDD enclosure on the SSD and this copy even works fine with the new enclosure.

    Best regards

  45. Marc-Philip says:

    Hi Mel,
    great recipe, but I have one question: Why are we not supposed to update /boot/cmdline.txt with the generated uuid for the root partition on sda? Only /etc/fstab is supposed to be updated.

    Best regards

    • Mel Grubb says:

      I’m not sure if I have your whole question right, but there are two parts to booting from the drive. One is to identify the physical drive, and the other is to identify the partition containing the root filesystem. If you know a way to do this in fewer steps, please share, but this is what I found in my research. Linux is not my natural habitat, so I’m relying on research rather than years of experience for things like this.

  46. Marc-Philip says:

    Hi Mel,
    when following your step-by-step guide, we end up with root=PARTUUID=5d18be51-3217-4679-9c72-a54e0fc53d6b in /boot/cmdline.txt and
    /dev/disk/by-uuid/bb4231bb-9d5e-47dc-9b94-5dad0743affc in /etc/fstab.

    /boot/cmdline.txt contains the original uuid of the SD card’s partition and /ect/fstab contains the generated one. Hmm, documentation says that /boot/cmdline.txt is for passing parameters to the kernel. I guess we should have bb423… in cmdline.txt as well. That’s my question.

    Best regards
    Marc-Philip

  47. Marc-Philip says:

    Hi Mel,
    I mixed up filesystem uuid and partition uuid. Everything is fine with the recipe here.

    One other remark: if you copy the root partition with dd while it is not mounted, the filesystem check does not report any errors. You can do this e.g. if you halt the raspi and insert the SD card in e.g. a laptop.

    Thank you and best regards
    Marc-Philip

  48. Pingback: Setting Up Raspberry Pi 2 (Including Moving rootfs to USB Drive) | My Scratchpad

  49. i want to leave my comment just to Say Thanks a lot for your help it would be very hard for me to make that work alone without the help of your instructions 🙂

    • Mel Grubb says:

      I couldn’t have written this series without the work of all the people that went before me. I just saw a need to take some rather terse and technical articles and make them a bit more friendly to the reader.

  50. Nicholayli says:

    Hi, I tried partioning the hard drive but came across errors after creating the root and data partion have tried a number of fixes but to no avail. Currently I get “Error: The backup GPT table is corrupt, but the primary appears OK, so that will be used” if I agree. My hard drive shows as being 1000GB and the partion reads
    Partition Table: gpt

    Number Start End Size File system Name Flags
    1 1049kB 16.0GB 16.0GB ntfs primary
    it was a 1TB hard drive.
    Is there anything I can do? I also tried to re-format the HDD on main PC, but was unable to do this either

    • Mel Grubb says:

      Formatting isn’t your problem. It sounds to me like the partition table itself is bad. You’ll need to create s new partition table, I guess. Back up whatever you can first, repeat the mklabel part, and if you’re formatting it as NTFS, maybe skip the -Q parameter. Let it give the drive a good going over just to be sure.

  51. Andrew says:

    Hey Mel,

    Firstly, thanks for writing these guides – very clear and informative.

    I’m having an issue getting the boot files to resize to the partition size. I’m using a RPi2, with a 32GB SDHC, and am linking up to a 320GB ex-laptop hard drive. I’ve copied across the files to the partition sda1, and received the message:

    949+1 records in
    950+0 records out
    31876710400 bytes (32 GB) copied, 1812.66 s, 17.6 MB/s

    However, when attempting commands:

    ~ $ sudo e2fsck -f /dev/sda1
    e2fsck 1.42.5 (29-Jul-2012)
    /dev/sda1: recovering journal
    e2fsck: Bad magic number in super-block while trying to re-open /dev/sda1
    Signal (11) SIGSEGV si_code=SEGV MAPERR fault addr=0x60

    ~ $ sudo resize2fs /dev/sda1
    resize2fs 1.42.5 (29-Jul-2012)
    resize2fs: Bad magic number in super-block while trying to open /dev/sda1
    Couldn’t find valid filesystem superblock.

    I’ve checked a fair few fixes based on the error messages, but most are relating to partition formats other than ext4 File Systems. Do you have any suggestions?

    Many thanks,

    Andy

    • Mel Grubb says:

      I have seen errors like that as well, but they seemed to go away once I started using percentages when partitioning the drive. The drive I’m using now for a demo machine was partitioned 0gb-16gb, 16gb-50%, 50%-100%. Apart from the first partition, everything else was done in percentages, and parted seems to have figured out the best way to begin and end on an even boundary. Unless you were partitioning this drive into multiple pieces, and tried to tightly control the beginning and ending of each partition, though, I’m going to guess that’s not what ‘s going on here. This IS the partition you copied the SD card into, right?

    • Mel Grubb says:

      I should have said this before. Oops. Have you checked the SD card for errors? If there were unresolved errors on the SD card, then they get copied over along with everything else. Make sure your SD card is all clean before making the move, and it may help.

  52. MT455 says:

    Hey Mel, Thanks for the awesome series. I am trying to learn Linux and the raspberry pi has been a really cool introductory system for that. I’m currently working on this step (6) and I keep running into some issue. I have a small laptop hard drive enclosure hooked up to the Pi with one USB cable and there’s a second USB cable coming out of the hard drive directly to power.

    The problem is that whenever I get to the “sudo e2fsck -f /dev/sda1” step to check for errors after copying data to the drive, no matter how many times I run that command I keep getting tons and tons of inode count and free block messages. The messages are like the following:

    Free inodes count wrong for group #3 (0, counted=960).
    Fix?
    Free blocks count wrong for group #33 (1965, counted=4719).
    Fix?

    Unattached inode 278485
    Connect to /lost+found?

    So I have tried several times to go on to setting the boot configuration, but I have never been able to boot from the hard drive. I see kernel panic messages when I look at the screen hooked up to the Pi.

    Could this be the hard drive being bad? (I didn’t think this drive had gone bad)
    Has anyone encountered something like this and might have something to help me out and point me in the right direction? I would really appreciate any help.

    • Mel Grubb says:

      A few people have reported such problems. It could be that the SD card was corrupt before copying it. Try scanning the card itself and see what you find.

  53. Paulo Gomes says:

    Hi Mel
    First of all, Congrats for a magnificent article. This article made me decide to buy a Pi2 and have a go at it. I’m following your article and up to the copy from the SDcard to the hdd is done, but on this particular step, although you mentioned that an error or two was to be expected, I’m getting 50+ errors about inodes and file/cluster sizes being incorrect. I “repair” them all and proceed, but then on first reboot to boot from the hdd it fails and I get nothing. what can be wrong?

    • Mel Grubb says:

      The problem with me answering is that there are so many things that can go wrong, and they’re not really Pi-specific. If you look through the comments, you’ll see this same sort of error from others. Sometimes the solution is simple, like making sure the SD card didn’t have errors on it before cloning it. Turning the power off without shutting down the Pi properly can cause this, and it’s happened to plenty of people. Other than that, it’s time to hit StackExchange and start looking up errors. The weird thing is that as many times as I’ve done this (many times while writing this series), I’ve never had it happen to me, so I haven’t really had the chance to look into it first-hand.

  54. Mel…. WOW! You are the MAN!

    Thank you so much for your efforts and as well the ones who came before you! You should think about putting this up on Udemy or creating a video series as well.. maybe the reach would be greater and your efforts could benefit more people?

    My issue is about RAID 1 and configuring it with this setup. I’ve been looking around for a while and knew this could be done with the Pi and I wasn’t going to spend a bunch of cash on a server that I would have less control over, thanks again for matching me up with exactly what I had in mind btw!

    I got a RAID box from ICYbox ( http://www.amazon.co.uk/dp/B000L105KS/ref=sr_1_1?ie=UTF8&qid=1437670538&sr=8-1&keywords=RD3219StU3-B- ) and a coulple 3TB WD Greens (keeping low power in mind…)

    Although I work in UNIX all day long I am no expert 😦

    Anyway, the problem is that both HDD’s show up even though I have the switches on the box switched to RAID 1 but I see both sda and sdb as the HDDs, and I am just wondering how to get it setup like you with the RAID 1 setup…I’d like to get the transferring of all out pics and vids and stuff soon but Don’t want to start transferring them to sda when at some point I may need to format everyting again to get the RAID 1 working properly.

    Thank you again for all your doing to make the world a better place!!!

    And thank you in advance for your response!
    Jason

    • Mel Grubb says:

      Check that the RAID box can handle 3TB drives. Mine tops out at 2TB, but when I put larger ones in, it just recognized the first 2TB. Maybe yours reacts differently. It’s definitely not mirroring if you can see both drives, that’s for sure. As for setting up the box, that might be more of a question for the manufacturer. One thing I will tell you is that my own Pi didn’t seem to like booting from it. It would use it just fine, but when using it as a system drive, it would run for a week or so and then freak out. I’ve been running off the SD card ever since, although that’s about to change.

      • So the issue was that there were no instructions included (in my box at least..) about how to get the box to function as a RAID array.. had to hold down buttons on the box while plugging it into a machine and praying on one leg while rubbing my belly and patting my head at the same time.. but it worked!

        So now I have one drive (hopefully its config’d correctly as RAID 1.. do you know any way to check?) and am starting to copy stuff over. My question for you is do you even have the swap file on your HDD or do you still have that on the SD card? Any other tips as I’ll be running an identical setup to yours at this point..

        Thx again!

      • Mel Grubb says:

        I think that checking would probably vary by manufacturer, but if you’ve put two drives in it, and it only looks like one, then there should only be two possibilities.

        1) It looks like one drive that’s twice as big, in which case it’s not acting RAID-ey
        2) It looks like one drive that’s the right size, in which case it… probably is working.

        The only way I know to tell for sure would be to take the drives back out, mount them individually, and see if they both contain the same stuff.

        As for the swap file. I think that’s probably what didn’t like running on the RAID box. I’ve been running with the SD card as my system drive ever since discovering that it didn’t like using the RAID that way. Maybe a different brand would act differently. I just know I haven’t worn out the card yet in well over a year of constant uptime, and everything seems fine.

        I’m planning to build up a new server, and this time I’m going to attach an older laptop drive to be the system drive, and just let the RAID box be storage. I’m also considering buying a small SSD, like the kind they use in netbooks, and giving that a shot.

  55. Thanks for your reply Mel.

    So is your RAID box running on its own as far as configuring the array and managing everything, or are you using something like mdadm to do it?

    I’ve just came across an article that has a pretty decent outline on how to accomplish this, but I’m a n00b to using this on my own so any input is appreciated…

    Thanks again!

    • Mel Grubb says:

      The RAID box I’m using handles it all internally. To the Pi, it just looks like a hard drive like any other. It’s completely unaware that it’s a RAID. I did have some problems booting from it, though. It would run for a week or two and then crash. I figured maybe there was some inherent lag there that the Pi just couldn’t tolerate, maybe. I’ve been running off of the SD card ever since then, but I know the swap file is probably tearing it up. On my next rebuild (soon), I’m planning to attach a small leftover hard drive (like 60GB), and let that by the root drive. Then the external RAID will be nothing but data, which it seems to be handling just fine. I’m looking for a cheap, small SSD, like the kind they put in netbooks, and see where that gets me.

  56. So.. after further digging (on page XX of Google Search…) I’ve found some “instructions” on how to get my box into the correct RAID 1 config: http://icyboxforum.de/viewtopic.php?f=60&t=14407&sid=d0a9abf19720c7877a1f8a6dba3d99b4

    I’m pretty sceptical though as it says to hold the rest button, while plugging it in to a pc and so on..

    I guess I don’t have a choice but to give it a shot at this point.. but most of my work will be gone now that I’ve gotten to pg 10 of your amazing guide.

  57. Pingback: Raspberry Pi Home Server: Part 5, Remote Desktop | MelGrubb.ToBlog()

  58. MABinTN says:

    Mel,
    Thank you for this comprehensive guide and all your work (and those before) helping Linux NOOBS and Pi tinkerers. This guide has allowed me to create a CrashPi after an unfortunate HDD crash that almost wiped out a 14 year collection of pictures of my kids, not to mention countless documents, tax returns, etc.
    The issue I’m trying to solve is accessing the GPT NTFS partition created at /mnt/data on a 32-bit Windows XP (I know) desktop PC to perform the initial load (90+ GB) of pics, videos, etc to the share before exposing it via Samba. Have you had any experience in this area?
    Thanks again for your sharing of knowledge.
    -M

    • Mel Grubb says:

      If it’s NTFS, you should just be able to plug it in, and Windows should just see it. What issues are you having?

      • MABinTN says:

        Mel,
        The issue is that Windows does not assign a drive nor does it give the option of assigning one in the Computer Management console. It is recognized as a drive in computer management but with no specified format. I would have at least expected to see NTFS as the format.

        Thanks again.
        -M

      • Mel Grubb says:

        Well, if you haven’t actually put anything on the drive yet, I’d say your best bet is to let Windows format that partition rather than doing it from the Pi. You should have something you KNOW Windows likes then. The Pi should still read and write to it just fine.

      • MABinTN says:

        Nothing on the drive yet. Sound advice.
        Thank you.
        -M

  59. Hi, Thanks for a very good and useful guide.

    I have run in to problems though. I can’t install ntfs-config or ntfsprogs or gdisk since they all throw the same error (se below) and I have no idea how to go about solving this. I am on a Gen 1 model B Raspi and if you could just point me in the right direction I’d be very thankful!

    Setting up libblas3 (1.2.20110419-5) …
    update-alternatives: error: /var/lib/dpkg/alternatives/libblas.so.3 corrupt: line not terminated while trying to read status
    dpkg: error processing libblas3 (–configure):
    subprocess installed post-installation script returned error exit status 2
    .
    .
    .
    dpkg: dependency problems prevent configuration of liblapack3:
    liblapack3 depends on libblas3 | libblas.so.3 | libatlas3-base; however:
    Package libblas3 is not configured yet.
    Package libblas.so.3 is not installed.
    Package libblas3 which provides libblas.so.3 is not configured yet.
    Package libatlas3-base is not installed.
    .
    .
    .
    Errors were encountered while processing:
    libblas3
    liblapack3
    python-numpy
    python-gtk2
    python-glade2
    ntfs-config
    E: Sub-process /usr/bin/dpkg returned an error code (1)
    .. install failed!

    • Mel Grubb says:

      That’s certainly not an error I’ve seen before. I have had the occasional problem with repositories being temporarily down, but this sounds like something wrong in the original repository configuration. I just ran through this whole series in the last month while recording the Pluralsight screencast, and didn’t have any errors, but that was on a 2B. I do have an older B, and a B+. I could try reproducing the situation, but I’m not sure how long before I can make that happen. In the meantime, try feeding the most distinct part of the error message through Google, and see if anything comes up. Sometimes I have found Pi forum posts from someone else who has seen a particular error before.

      • Mel Grubb says:

        This morning, I started from a fresh Raspbian image and a Model B (not B+, not 2B, just “B”), installed nothing but the current updates, and then ran “sudo apt-get install ntfs-3g ntfs-config ntfsprogs gdisk”. I did not see the error you mentioned, so I don’t think it’s some version incompatibility problem. Are you sure you listed all four packages? Apt is supposed to take care of dependencies, but I suppose it’s possible that one package might not be directly dependent on another, but still won’t install right without it being there already.

  60. Kev says:

    Hi Mel!
    Is there an easy process for adding a second powered USB Hard Drive? Is it possible to add a second drive that just “extends” the space on the partition created in this tutorial? If so, can it be done without destroying the existing data?

    • Mel Grubb says:

      There’s no simple way to just add more space, but since you can mount the new drive anywhere you want, you can make the second drive appear to be a folder within the first drive. For instance, if I had my first drive mounted at /mnt/data, I could mount the second one at /mnt/data/videos. It would then appear to be part of the same directory structure, but it won’t increase the space available to the other folders inside /mnt/data. I’m sure there are ways to treat both drives as one big space, like a software-based RAID array, but I haven’t looked into them enough to make a recommendation.

  61. Rich says:

    Well done Mel. Thank you. I just found your video on pluralsight and have been following it. I have an rpi2 and those instructions are not working on raspbian. parted is giving me fits about partitions being busy and not creating partitions with ext4. Sometimes with f16 and other times nothing at all so obviously I can’t mount my 984GB partition without a file system. Any thoughts? I’ve done this plenty on CentOS, Ubuntu, Redhat and other linux/unix distros so I’m good there, but I’ve never run into this problem before.

    • Mel Grubb says:

      This is not something I’ve seen myself. I’ve formatted the little 160gb drive I used during the recording as well as 2-3TB drives, and it worked on all of them. I’d ask if the drive was using a GPT, but for a 1TB drive, that wouldn’t even matter. What do you mean by “f16”? I’m not familiar with the acronym, unfortunately.

      • Rich says:

        Yes I was using GPT as well. F16 for fat16 or DOS. I decided to use gparted instead and that fixed my formatting issues. Then I decided to format it as NTFS and got the libraries for that to work. Another question on this topic; I finished of the Samba portion of your video but for some reason I can only access about 2GB on the drive and then it says its full. Everything is set in fstab and shares in Samba using the Webmin interface. Any thoughts on this one? I changed drives thinking I had some issues with the one I was using but that was not the case. I change the uuid info in fstab and it seems to mount but 2GB is not enough for anything.

      • Mel Grubb says:

        My first guess would be that the drive didn’t mount properly, and you’re filling up the placeholder directory on your SD, but you say you already checked that. I’d double-check that the drive is truly mounted where you expect. Try writing a file to the drive, then shut down the Pi and hook it up to your regular computer and see if the files are there. If something’s funny with the way the drive got formatted, you could always try formatting the NTFS partition from your regular computer. That’s how my current external drive was done. My current personal server is running with an SSD for the system drive, and the 2TB RAID box as an NTFS data store. The RAID was formatted from my main laptop. It might make a difference. In my case it just happened to already have stuff on it that I didn’t want to lose. I have run through the posts beginning to end with a smaller drive several times though, especially while doing the Pluralsight recording.

      • Rich says:

        Thanks Mel. I got it figured out last night. I had some residual garbage that needed cleaning up and then I just started over and got it working except for the uuid mounting part. For some reason I’m having issues getting fstab to mount my drive. I mount it just fine from the command line. Double, triple, quadruple checked the ID value and all is good there. I’ll figure it out eventually. Thank you for your thoughts.

  62. Terry Talton says:

    Thanks Mel for all the info! Everything has gone swimmingly up until the last part of this chapter. When I try to save the /etc/dphys-swapfile after editing it tells me it can’t because it’s a read-only filesystem. This is a Pi 2 with the latest Raspbian image and following your instructions to the letter since chapter 1. How do I get it to save?

  63. David Bell says:

    Great article – worked a treat. I am using a 2 TB drive. Incresed size from 16 GB to 1 TB – to accommodate a larger /var/www for the web.
    Looking forward to your article on back up and restore.
    Currently using this but not sure how to recover to SD card, or to ensure backup from the HD Extension is also backed up and restorable. any help gratefully accepted.

    DJB

    #!/bin/bash

    # Setting up backup directories
    SUBDIR=Backups
    DIR=/mnt/data/$SUBDIR

    # Setting up echo fonts
    red=’\e[0;31m’
    green=’\e[0;32m’
    cyan=’\e[0;36m’
    yellow=’\e[1;33m’
    purple=’\e[0;35m’
    NC=’\e[0m’ #No Color
    bold=`tput bold`
    normal=`tput sgr0`

    #Screen clear
    clear

    echo -e “${green}${bold}Starting RaspberryPI backup process!${NC}${normal}”
    echo “”
    # First check if pv package is installed, if not, install it first
    PACKAGESTATUS=`dpkg -s pv | grep Status`;

    if [[ $PACKAGESTATUS == S* ]]
    then
    echo -e “${cyan}${bold}Package ‘pv’ is installed${NC}${normal}”
    echo “”
    else
    echo -e “${yellow}${bold}Package ‘pv’ is NOT installed${NC}${normal}”
    echo -e “${yellow}${bold}Installing package ‘pv’ + ‘pv dialog’. Please wait…${NC}${normal}”
    echo “”
    sudo apt-get -y install pv && sudo apt-get -y install pv dialog
    fi

    # Check if backup directory exists
    if [ ! -d “$DIR” ];
    then
    echo -e “${yellow}${bold}Backup directory $DIR doesn’t exist, creating it now!${NC}${normal}”
    sudo mkdir $DIR

    fi

    # Create a filename with datestamp for our current backup (without .img suffix)
    OFILE=”$DIR/backup_$(date +%Y%m%d_%H%M%S)”

    # Create final filename, with suffix
    OFILEFINAL=$OFILE.img

    # First sync disks
    sync; sync

    # Shut down some services before starting backup process
    echo “”
    echo -e “${purple}${bold}Stopping services before backup${NC}${normal}”
    sudo pkill deluged
    sudo pkill deluge-web
    sudo service deluge-daemon stop
    sudo service noip stop
    sudo service cron stop
    sudo service proftpd stop
    sudo service webmin stop
    sudo service xrdp stop
    sudo service sendmail stop
    sudo service ssh stop
    sudo ervice btsync stop
    sudo service apache2 stop
    sudo service samba stop
    sudo service motion stop
    sudo service ddclient stop
    sudo service avahi-daemon stop
    sudo service netatalk stop

    # Begin the backup process, should take about 45 minutes hour from 8Gb SD card to HDD
    echo “”
    echo -e “${green}${bold}Backing up SD card to img file on HDD${NC}${normal}”
    SDSIZE=`sudo blockdev –getsize64 /dev/mmcblk0`;
    sudo pv -tpreb /dev/mmcblk0 -s $SDSIZE | dd of=$OFILE bs=1M conv=sync,noerror iflag=fullblock

    # Wait for DD to finish and catch result
    RESULT=$?

    # Start services again that where shutdown before backup process
    echo “”
    echo -e “${purple}${bold}Starting the stopped services${NC}${normal}”
    sudo service deluge-daemon start
    sudo deluged
    sudo deluge-web
    sudo service noip start
    sudo service cron start
    sudo service proftpd start
    sudo service webmin start
    sudo service xrdp start
    sudo service sendmail start
    sudo service ssh start
    sudo ervice btsync start
    sudo service apache2 start
    sudo service samba start
    sudo service motion start
    sudo service ddclient start
    sudo service avahi-daemon start
    sudo service netatalk start

    # If command has completed successfully, if not, delete created files
    if [ $RESULT = 0 ];
    then
    sudo mv $OFILE $OFILEFINAL
    echo “”
    echo -e “${green}${bold}RaspberryPI backup process completed! FILE: $OFILEFINAL${NC}${normal}”
    echo -e “${yellow}Removing backups older than 5 days${NC}”
    sudo find $DIR -maxdepth 1 -name “*.img” -mtime +5 -exec rm {} \;
    echo -e “${cyan}If any backups older than 5 days were found, they were deleted${NC}”
    exit 0
    # Else remove attempted backup file
    else
    echo “”
    echo -e “${red}${bold}Backup failed!${NC}${normal}”
    sudo rm -f $OFILE
    echo “”
    echo -e “${purple}Last backups on HDD:${NC}”
    sudo find $DIR -maxdepth 1 -name “*.img” -exec ls {} \;
    echo “”
    echo -e “${red}${bold}RaspberryPI backup process failed!${NC}${normal}”
    exit 1
    fi

    • Mel Grubb says:

      Restoring the SD card is just burning the .img file back to it, just like the first time you created the card. I have not tried to script out a backup like this. I just shut down the system and use Win32DiskImager in “read” mode to back up my card. I have CrashPlan set up to back up the contents of the data partition, but I’ve always figured if anything went horrible wrong I’d probably rebuild the server anyway. I’ve got it down to about 5-6 hours for a rebuild now, and I could probably script that out to make it even shorter, but I take it as an opportunity to check that the instructions in this series are still up to date. Sometimes packages get updated and I have to adjust my posts to compensate.

  64. Sisk says:

    I would argue that the benefits of using a proper Linux filesystem such as ext4 or btrfs outweigh the ability to plug the drive into a desktop to transfer files to it instead of pushing them through the network. Probably you’d only do that once, when you first set up the server, and then you’d be dealing with the downsides of NTFS (which are not inconsequential, especially for a file server – it is a rather antiquated filesystem after all) for as long as you’re using the server.

    • Mel Grubb says:

      I agree. I have used NTFS on my own server as a matter of convenience. If something happens to my main computer, and I want to restore from backup, I might want to just swipe the drive temporarily hook it up locally, and “adopt” the backup set temporarily to get things done faster. I don’t have any hard number comparisons on the relative performance of ext4 vs NTFS, so convenience became my tie-breaker.

  65. Pingback: Raspberry Pi web server | Stefan Adrian Bukos

  66. Richard says:

    I recently got the new Raspbian Jessie image and wrote it to my sd card. I am now at your step part 6 adding a hard drive. It all goes well until I get to the tune2fs to change the UUID. I am getting this error:
    pi@MyRPi ~ $ sudo tune2fs -U random /dev/sda1
    tune2fs 1.42.12 (29-Aug-2014)
    The UUID may only be changed when the filesystem is unmounted.

    I have searched and can’t find a way to change the UUID.

    As a side note, I have followed your steps previously (wheezy) and it worked just fine.

    Any ideas?

  67. Peter says:

    Hi Mel.
    I’ve followed your Series 1 to 6 and am at this point. It’s been great and easy to follow with only a couple of tiny hiccups.
    I’ve installed using Wheezy, and am hesitating to go further until I can clarify a couple of things.

    My main concern is backup and imaging from here on. Up to and including your article 5, I happily went back to my main computer and made an image of the SD card

    Now, after setting up the file system to be accessed from the USB HDD in this article 6, here are the questions:
    1. If from now on I app-get any application, and/or update and upgrade, does ANYTHING now get written to the SD card, or does everything now go to the HDD
    2. Depending upon the answer to the above, what process(es) should I run to backup and/or make an image?

    It would be a nightmare to go forward installing lots of new apps if I cannot maintain concurrent backups and images. As a noob, I am just a bit confused as to whether from now on I have to
    – back up both the card AND the harddisk-based filesytem, or
    – back up the hard disk alone or
    – even combine the SD and the HDD into a new SD card image?

    Any info and advice welcome
    Thanks

    • Mel Grubb says:

      After moving to the hard drive, everything gets written there, but you don’t HAVE to move to the hard drive at all. You can just mount the drive and store data there. In my Pluralsight course, I moved booting from the hard drive to the very end. I ran my own server for over a year from the SD card, and never used it up. I WOULD recommend moving the swap file to the hard drive, though. That should extend the life a bit.

  68. Peter says:

    OK. Thanks for that. Previously I had tried a usb harddisk set up following another guide elsewhere, and my SD card started to go flaky after about 3 months. Perhaps it was the swapfile. At that time I knew even less than I do now.
    So, I’m quite happy to work with this system for at least the next few months to see how it goes, and using your series, the swapfile is obviously already on the harddisk now.

  69. gilad says:

    I am having trouble in this stage.
    Got a Toshiba 1T USB drive – although I increased the USB power (PI2) it did not manage to operate it, so now the disk is connected using a powered USB hub (note: for some reason it is not able to drive both the disk and the PI, It makes the PI reboot again and again – maybe due to disk power draw during spin-up so PI still gets powered from its own supply)
    Got to the stage of:
    sudo dd if=/dev/mmcblk0p2 of=/dev/sda1 bs=32M conv=noerror,sync

    This process hangs at a certain stage – I see the PI green led blinking and the drive light blinking, but then at a certain stage the PI light stays ON and after a while the disk light goes OFF (actually it sounds like it stops spinning altogether). Tried this twice and happened both times.

    Any idea what could be wrong?

    If I decide to abandon moving the root partition to the disk, can I use it only for data (leaving the sda1 partition) or should I re partition to one data partition?
    In the described procedure do I simply skip any command issued for sda1 and refer only to thse for sda2?

    • gilad says:

      Adding to the above – it seems that the PI stopped responding – I have it also connected to a display and it does not respond from that console (and obviously also does not respond in SSH)

    • Mel Grubb says:

      Some drives just draw more power than others. I have some that the Pi can power, and some that it can’t. The big RAID box hooked to my own server has its own power supply, but there Pi doesn’t like to bit from it, so I have run my system primarily from the SD card for years without a problem. I would recommend moving the swap file to the drive, though.

      • gilad says:

        As I wrote, after connecting through a powered hub it works.
        Nevertheless, I guess for the time being I will skip booting from it – can I leave the partitions as already made by this session procedure (and maybe get to solving the boot issue sometime later)? Do I simply skip anything that deals with sda1 and deals only with sda2?

      • Mel Grubb says:

        You could leave them, or if you haven’t done a lot with it yet you could reclaim the space by repartitioning the drive. If you keep both, then you would adjust anything from sda1 to sda2, but going by the instructions I posted, you only ever use those names when using parted itself. fstab will refer to the partition by UUID, and once mounted, no-one will ever use the name “sda1” or “sda2” again. The drive simply becomes /mnt/data (or whatever you decide to call it).

  70. gilad says:

    OK, tried another disk – same issue so gave up booting from disk. only one data partition.
    What about swap configuration – do I leave it untouched or can I increase it as described (I use a 16GB SD card)

    • Mel Grubb says:

      The swap file is the thing that “uses up” SD cards, supposedly, so I would move it to the hard drive (if you use an ext4 file system, that is). You don’t HAVE to, and I never wore mine out in all the time I was using it off the card, but that may have to do with the brand more than anything.

  71. DavidA says:

    Hi Mel, to format the data partition with ntfs you specify:
    sudo mkfs.ntfs -Q -L Data /dev/sda2
    Please can you suggest how to format the data partition with ext4 instead?

  72. Pingback: CrashPi – An off-Site backup for the whole family | MelGrubb.ToBlog()

  73. John Ehresmann says:

    Hi Mel,
    Love the series. Discovered it looking for a way to make a web server and your guide is the most complete one out there. I have gotten this far and want to get a hard disk on the Pi. Made it to point where I need to change the UUID on the hard disk “sda1″. I get a unique identifier, insert it in the command line and when I try to run the line ” sudo tune2fs /dev/sda1 -U UUID-GOES-HERE” get an error message that says “the UUID may only be changed when the file system is unmounted”. How do I unmount sda1, give it a new UUID and remount sda1?.

    • Mel Grubb says:

      Make sure you use raspi-config to set the system to bit to the command line. This is in the updates at the top of the article. The newer “Jessie” release of Raspbian bits to the desktop and auto-mounts any USB drives. Booting to the command line should prevent this auto-mounting.

  74. John Ehresmann says:

    Thanks for the quick reply. I changed the raspi-config so I now mount to the command line with required log in. This gets me to the command line log in on my TFT screen mounted on the pi. I then log in via SSH start working and the Pi says I still can not change the UUID. Thoughts? A confused old engineer from the age of punch cards and vacuum tubes.

    • Mel Grubb says:

      Hmmm. Well I’m on my phone at the moment so I can’t look up the exact syntax right now, but something like “sudo umount /dev/sda#”, where “#” is there device you’re trying to unmount should do it. I’ll have to run through that part again, but rebooting after setting it to boot to the cli should have done the trick.

  75. John Ehresmann says:

    Gave that a try with the following results:

    pi@RPWSJHE ~ $ sudo blkid
    /dev/mmcblk0p1: LABEL=”PIOPSYS” UUID=”4C36-B52F” TYPE=”vfat” PARTUUID=”000214af-01″
    /dev/mmcblk0p3: LABEL=”SETTINGS” UUID=”d917886f-d8eb-4d7b-8163-02d26e2573f7″ TYPE=”ext4″ PARTUUID=”000214af-03″
    /dev/mmcblk0p5: SEC_TYPE=”msdos” LABEL=”boot” UUID=”0AD3-BA87″ TYPE=”vfat” PARTUUID=”000214af-05″
    /dev/mmcblk0p6: LABEL=”root” UUID=”581ba799-40f0-4b34-8f3a-7da6352dbcd7″ TYPE=”ext4″ PARTUUID=”000214af-06″
    /dev/sda1: LABEL=”root” UUID=”581ba799-40f0-4b34-8f3a-7da6352dbcd7″ TYPE=”ext4″ PARTLABEL=”primary” PARTUUID=”2d6e287b-0138-4e48-a937-9f72bd01a969″
    /dev/sda2: LABEL=”Data” UUID=”7C55FA503A49EE47″ TYPE=”ntfs” PARTLABEL=”primary” PARTUUID=”496dfc08-7428-483f-9035-a9fca1b4b41a”
    /dev/mmcblk0: PTUUID=”000214af” PTTYPE=”dos”
    pi@RPWSJHE ~ $ sudo tune2fs /dev/sda1 -U db5bfd25-d8bb-47b2-af43-633a06daf650
    tune2fs 1.42.12 (29-Aug-2014)
    The UUID may only be changed when the filesystem is unmounted.
    pi@RPWSJHE ~ $ clear
    pi@RPWSJHE ~ $ sudo umount /dev/sda1
    umount: /dev/sda1: not mounted
    pi@RPWSJHE ~ $ sudo tune2fs /dev/sda1 -U db5bfd25-d8bb-47b2-af43-633a06daf650
    tune2fs 1.42.12 (29-Aug-2014)
    The UUID may only be changed when the filesystem is unmounted.
    pi@RPWSJHE ~ $
    Is there any other was to change the hard disk “sda1” partion UUID?

    • Mel Grubb says:

      I’ll have to look into this more closely. I’m pretty sure this worked for me last time I tried it, but perhaps something else has changed since then. I may give this another shot this weekend and see what I can find.

  76. Any updates to the whole changing the UUID on the mounted filesystem? Defintely not working on the latest tune2fs. Giving “The UUID may only be changed when the filesystem is unmounted.”.

    • Mel Grubb says:

      Okay, I won’t pretend to be an expert here, but I was able to find someone smarter than me who knew how to get around this. I found a post at http://www.oarie.net/index.php/features that seems to clear it up. What you need to do is issue the command “sudo tune2fs -O ^uninit_bg /dev/sda1” first, which will unstick some flag that’s making the drive appear mounted when it isn’t. Then you can assign a new guid. By the way, you don’t have to go make one up yourself. You can just say “sudo tune2fs /dev/sda1 -U random” and it’ll make one up itself.

  77. John Ehresmann says:

    You are the man! That work around for the tune2fs UUID did the trick. I can now boot from the hard drive. Yea! I really appreciate you tutorials with the “why” as well as the ‘how”. It really helps me in the learning process.

  78. Paulo Gomes says:

    Hi Mel,

    I finally managed to complete part 6, BUT, something starnge has happened.
    I can’t login! I’m not even prompt to!
    During boot the system performs a fsck to sda1, and then after starts a “job for dev-disk-by\x2duuid…”
    when it finalizes booting, I’m left with what seems to be a “guess” account, and some warning about “sulogin: root account is locked, starting shell” and my promp looks like “root@tugadroid:~#”
    I am able to run normal and sudo commands, but the swapon command does nothing.
    Is this something I should worry about, or can I continue with the install?

    • Paulo Gomes says:

      I just tried to follow step 7, the Samba install, and got a series of EXT4-fs errors.
      It seems something went wrong at some point when changing the boot to the hdd.
      Is this still fixable or am I better of restarting from ground zero?

      • Mel Grubb says:

        I wouldn’t move forward with system errors outstanding. There’s telling what could go wrong. Try to clear up the errors before continuing, or put off booting from the hard drive until later. It’s not even necessary, it’s just a little faster. From the comments I have received, it seems like not all hard drives like to play nice, either.

    • Mel Grubb says:

      Well, you generally don’t want to be doing things AS root. It’s dangerous to stay in “God Mode” when it’s not
      truly needed. What happens when you “logout”? Can you log back in as Pi then?

  79. Paulo Gomes says:

    I did have a hard time finding a HDD that my PI would work with, even the powered hub that I picked up seems to be mocking me… I may go the raid enclosure option, if I find one cheap enough

    • Mel Grubb says:

      If you do, I wouldn’t recommend trying to boot from it or use it for swap space. At least with the RAID box I tried, the Pi didn’t seem to like booting from it. I suspect there’s a little too much lag there to be usable for the swap space. What I’ve done most recently was to hook up a small SSD as my boot and swap drive, and left the RAID as storage space only. The system has been very stable this way. Before the SSD, I booted and swapped to the SD card for over a year and never wore it out. It also makes it far more convenient to make backups of the entire configuration. Booting from the hard drive is purely optional. It does speed the system up, but getting a good quality SD card will do that too. The cheap generic cards do not perform nearly as well. I’ve seen more than a few articles that swear by SanDisk Extreme cards, but haven’t tried one myself.

  80. Paulo Gomes says:

    any suggestions on how to try to FIX without starting all over?
    “logout” is not accepted as command, and exit just leaves me with an un-operational pi

    • Mel Grubb says:

      I haven’t seen this behavior myself. Even the auto-login options in raspi-config log you in as Pi normally. Did you do anything different there? I’ve set mine to boot to the command line, but not to log in automatically. On some of my other Pis, I’ve allowed it to auto-start the desktop as Pi, but never the server.

  81. Jake says:

    Hi, Mel! Great blog!

    I haven’t tested it myself, but do you know if you can remove the SD card once it boots? I know it won’t work if you have to reboot or the server crashes because the SD card is missing… but once it boots would it be safe to remove the SD card? No reason other than curiosity… 🙂

  82. Pingback: Configure a USB on the Raspberry Pi | Weaver Solutions

  83. DavidA says:

    Hi Mel, I’m using a Western Digital 3TB Elements drive and have partitioned it similar to your instructions. It constantly spins, which surprised me. I’ve tried using hdparm to put it to sleep after inactivity. I haven’t got it to work correctly. Have you tried this?
    Best regards, David

  84. DavidA says:

    Hi Mel
    I have the same problem that MABinTN described above. My Windows laptop doesn’t recognise the format of the NTFS partition I made one /dev/sda2 (I followed your instructions exactly).

    You replied:
    “Well, if you haven’t actually put anything on the drive yet, I’d say your best bet is to let Windows format that partition rather than doing it from the Pi. You should have something you KNOW Windows likes then. The Pi should still read and write to it just fine.”

    Do you know exactly how one would do this on WIndows? I’d want to leave the sda1 system partition in tact, and leave the NTFS sda2 partition in a manner that the Pi still recognises.

    I saw a similar issue here:
    http://ubuntuforums.org/showthread.php?t=1476185
    but couldn’t use that info to fix my problem.

    Would be grateful for any help you can provide.

    • Mel Grubb says:

      If the drive is partitioned already, Windows should see it as two drives when you plug it in. It will complain that the ext4 partition is unreadable, and want to format it. Don’t let it. Just use windows to format the OTHER partition like you would any other drive. Ignore the smaller ext4 partition, and just format the larger “data” partition. You should be fine after that.

      • DavidA says:

        Hi Mel, I think I have got to the root of the problem. I am using a 3TB disk and formatted a 30GB ext4 system partition and a 2.7GB NTFS partition. I have now been told:

        “Windows won’t be able to deal with this because the disk must be a GPT disk for Windows to recognize the entire 3TB disk.” and “Note that there is 2.2TB maximum size limit for MBR disk partitions, so I recommend you clean and convert the drive to GPT to access all of the drive space.”

        This is a bit of a blow. I assume my disk is MBR. I’m trying to find a way of converting the disk from MBR to GPT but may have to clean it and start again. Do you know which of the commands you specified in this article determines that MBR is used? I.e. How might I select GPT instead?

        Any ideas would be much appreciated.

      • Mel Grubb says:

        When you run parted and “print all”, each drive entry should tell you what the partition table type is. Look at the first example in this article. It’s right after running “sudo parted”. You’ll see that the partition table type says “msdos”. Now read down further to after the hard drive has been hooked up, and it says “gpt”. As for converting the contents. I don’t know of any way to do that. You’ll need to copy everything onto a different drive while you reformat the main one, and then copy everything back. If someone out there knows a trick for getting around this, I’d be pretty amazed.

  85. DanL says:

    Thank you so much for this guide, Mel! It’s been a HUGE help in learning Linux and using a Raspberry Pi. I’m running into an issue with my external hard drive. I’ve followed your steps to the letter but I’m having trouble accessing the HD from my Windows machines. It shows up in Disk Management with the two partitions, one at 14.90GB (16GB) and the other is the remaining space. It doesn’t recognize it as NTFS but I did format it that way. The hard drive doesn’t show up in Explorer under This PC (I’m using Windows 10.) Like you suggested above, I was going to format the data partition but that is not an option in Disk Management. Do you have any suggestions? I don’t want to have to erase the whole drive and start step 6 again. Any help will be greatly appreciated!

  86. DanL says:

    Thank you so much for this guide! It’s been a HUGE help! I’ve followed your guide to the word but I’m having issues connecting my external HD to my Windows 10 machine. The drive shows up in Disk Management and I see the two partitions that were created, but I can’t access the drive through Explorer. Neither of the partitions show NTFS in Disk Management. I thought I would format the data partition like you suggested above, but that isn’t an option. The only option I have in Disk Management is “Delete Volume…”. Do you have any suggestions on how to fix this? Your help is greatly appreciated!

    • Mel Grubb says:

      I’ve had several reports of this exact behavior lately. I’m not sure what changed, but it seems that to get the NTFS partition to work under windows, you need to let Windows format it these days. Back your stuff up first, and you’ll need to edit fstab again because the partition will have a new uuid afterwards. There are some recent comments about exactly this situation. Give them a read as well.

    • Mel Grubb says:

      You don’t need to use Disk Management to do the formatting. You can just do it through explorer. Right click on the NTFS partition, and then “format”. Just make sure you pick the larger partition (presumably) so you don’t overwrite the os.

      • DanL says:

        Thanks for the quick reply! (Also, sorry for the double post, my first post wasn’t showing up for some reason) I can’t access the drive through explorer at all. It shows up in Disk Management but won’t appear in explorer. Maybe I can delete the data portion of the disk and let Windows partition and format it that way? I’ll try when I get home. Thanks again!

    • Amol P says:

      if your drive is working correctly on say a linux or mac machine (read only is fine), then the partition data itself is fine. You may not have assigned a drive letter, due to which it doesn’t show up in Windows explorer. Follow the procedure here: http://www.tomshardware.com/forum/284807-32-external-hard-drive-device-manager-comp

      • DanL says:

        Amol, thank you for the link. I couldn’t assign the partition a letter, but I was able to delete it and reformat using Windows. I then had to hook the HDMI and USB keyboard back up and find the new UUID using sudo blkid. Then I went back and edited /etc/fstab with nano and I’m back up and running! On to the next step!

    • Amol P says:

      Well the problem has been sidestepped for the time being 🙂 Which is perfectly fine if you don’t intend on doing this again. If you do, it’d be ok to find the root cause of the problem. You would definitely need another drive and another machine, preferably with Linux installed, to compare with.

  87. Jake says:

    Planning for the future… If my SD card fails, even with its limited use since I’ve attached an external HDD, can I back it up from the actual HDD?

    For example,
    sudo dd if=/dev/sda1 of=/NEW/SDCARD bs=32M conv=noerror,sync

    Whenever I run apt-get update && upgrade, it upgrades all of the files located on sda1 (my hdd 1st partition) correct? Should I now save images from that partition of the HDD?

    My example scenario: keeping up with the server with regular updates and upgrades. I still have my old backup .img files of the original install (as you suggest). My SD card craps out (with its limited use)… should I just save an image from my HDD instead of a now outdated SD card? Then restore a new SD card from my HDD, plug and go…

    P.S. thanks so much for this reference guide, I’ve combined it with a few others but this has ultimately served as my go-to guide. I even automated a lot of it with a bash script; yes, even the HDD section – took a lot of guess and checks. I figured out how to store PARTUUIDs as variables and write them to the cmdline.txt file. I could share that with you when I complete the rest of it. 🙂

    • Mel Grubb says:

      The command you show would be trying to back up the hard drive to the SD card. Remember that the first partition on the SD card is still needed to start the boot process, so your backup would have to fit in the remaining space. Assuming you made the OS partition on the hard drive the same size as the OS partition on the SD card, theoretically this would work, yes. What would probably work better would be backing up to another hard drive.

      One thing you can do to avoid problems when doing such backups is to create another SD card to boot the Pi with. This one shouldn’t be loading its OS from the hard drive at all. Boot with the second SD card, attach both hard drives (your OS drive, and your backup drive), and then run the dd command. Just make very sure you’re copying the right direction. Plugging the drives in one at a time and checking which positions they mount in would work. You don’t want to restore when you meant to back up.

      I’ve done some scripting as well, but as more of an integration test to make sure my instructions still work when things get updated. I’d like to see what you’ve done with the UUIDs into variables. That would help automate another piece of the puzzle.

      • Jake says:

        In lieu of your command here:
        sudo dd if=/dev/mmcblk0p2 of=/dev/sda1 bs=32M conv=noerror,sync

        I use (I wanted a little more feedback on the screen):
        (sudo dd bs=32M if=/dev/mmcblk0p2 conv=noerror,sync) | pv | (sudo dd of=/dev/sda1)

        Gathering your Partition unique GUID (instead of copy/paste, store it as a variable):
        puuid=”$(sudo blkid -s PARTUUID -o value /dev/sda1)”
        echo $puuid

        Changing the bootloader to look for your hard drive:
        cmdtxt1=’dwc_otg.lpm_enable=0 console=ttyAMA0,115200 console=tty1 root=PARTUUID=’
        cmdtxt2=’ rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait rootdelay=5′
        cmdtxt=$cmdtxt1$puuid$cmdtxt2
        echo $cmdtxt | sudo tee /boot/cmdline.txt

        This part specifically applies to this project and there may be a better way of doing this, for instance, looking for “root=” inside the cmdline.txt file and inserting the variable there. This is a hardcoded insert for my raspberry pi. This command breaks it into separate strings and combines it with the PARTUUID. Also, I’m not sure if “tee” is the best method, but it works.

      • Jake says:

        For the pv command you’ll need that package: sudo apt-get install pv

      • Jake says:

        I found a better method to insert inside of the cmdline.txt file using “sed”

        According to here: http://askubuntu.com/questions/20414/find-and-replace-text-within-a-file-using-commands

        You can use sed in this format:
        sed -i ‘s/original/new/’ file.txt

        To use a variable:
        puuid=”$(sudo blkid -s PARTUUID -o value /dev/sda1)”
        sudo sed -i “s/\/dev\/mmcblk0p2/PARTUUID=$puuid/” /boot/cmdline.txt

        That will replace /dev/mmcblk0p2 with “PARTUUID=…….”

        Breaking down this command as the variable: sudo blkid -s PARTUUID -o value /dev/sda1

        -s = searches for the term (PARTUUID in this case)
        -o = the output you want (the value of PARTUUID)
        /dev/sda1 = the partition of the UUID we want (can apply to /dev/mmcblk0p2 later on)

        Note: you’ll still need to append rootdelay=5 to the end of the file as well. But this should solve your copy/paste of long strings problem 🙂

    • Jake says:

      Finally, to append to the /boot/cmdline.txt file “rootdelay=5” use this:
      sed -i ‘1 s/$/ rootdelay=5/’ /boot/cmdline.txt

      That command says to search the first line and append rootdelay=5 to the end of it.

      Sorry for all of the replies 🙂

  88. Amol P says:

    Well you only need the initial boot partition backed up somewhere if you mess up the SD-card. Since all the /home and / filesystems are on the USB drive, they are safe. If you’re using NOOBS, you’ll need to backup the /boot and /SETTINGS partitions from your SD card, since they’re used during initial boot.

    • Jake says:

      Thanks! I’ll test this out and report back. I’ll make a separate SD card with only the /boot files (Raspbian Jessie Lite) and see the results.

  89. DavidA says:

    Hi Mel, could you help me please? Having got to the end of Part 6 of your instructions, what should I do to revert to having the system on the SD Card (instead of on the HDD System partition)?

    I want to consider doing this because I am still trying to get the HDD to sleep when the data partition is not being accessed.

    Best regards, David

  90. Davidb says:

    Hi Mel, was wondering if you could answer a question that maybe i’ve glossed over accidentally? I successfully moved my root file system to a 320gb hdd which contains 2 partitions sda1 (rootfs) and sda2 (pure storage ext4). my rootfs is partitioned for 50gb and remaining partition is storage. How would i make a backup image of the rootfs to sda2 being that the rootfs is mounted. Long way around i unplugged the hdd plugged it into another linux machine and used disk utility to backup the partition.. is there an easier way to backup my mounted rootfs to the second partition on the hdd? I already have a backup image of the sdcard . Thanks for any help you can provide.

    • Mel Grubb says:

      That’s the best way I know of. You could use dd to back it up the same way you originally cloned the sd card to the drive, but since it IS mounted, it will be upset when it comes back online after restoration. Shutting it down and using dd from a different system seems safest to me.

  91. Doug says:

    I have been following your directions closely. I’m using a new My Book 3TB. Formatted a 16GB partition and copied the boot files to it, pointed the cmdline.txt file to the partition unique GUID, then created a data partition (ntfs) and got to your reboot step.

    I’m currently waiting at boot up, everything seems to be marked “OK”, and the last item is “Reached target Network is Online” — and nothing seems to be happening. I have rebooted the Pi once as well, current wait time ~20min

    Has anyone else experienced this delay, and is there any indication that I should start over or that this is normal?

    • Mel Grubb says:

      Are you auto-mounting a file system from over the network in fstab? I did a little checking, and most of the people I see reporting this message are reporting it in connection with mounting something from over the network. If that’s the case, I believe raspi-config has an option to wait until the network is online before continuing the boot process. Give that a shot, and see if it makes a difference. In my situation, the Pi isn’t mounting anything over the network, so it’s never been a problem.

  92. Eric Barta says:

    Hi Mel,

    First off, thank you so much for this tutorial; it is absolutely wonderful. I have never used a Linux before, let alone a Raspberry Pi, so the fact that I can take this one step at a time and actually learn as I go is perfect. Again, thank you.

    On to my question: everything (well, except for the VNC, but I don’t really care about that at this point) has worked without an issue. When I go to the Webmin on the next step though, I notice that virtual memory still only says 100mb, when I allocated 4GBs. When I check swapon -s it says 102396 bytes (still 100mb) but when I check the actual file it says 4096. I allocated 40gbs to the boot partition and 600 to the data, just to keep things round, so I know there is space for it…

    I rebooted and checked all the variables again, and still the same results. Any thoughts? Thanks in advance,

    Eric Bartha

    • Eric Barta says:

      Alright, so I may have commented prematurely. I went back into the swap file (sudo nano /etc/dphys-swapfile) and I noticed you mentioned there was only one line. I had 27 lines. Most of them were commented out, but I realized the line I was edited was a max swap file size line.

      I deleted everything from the swap file and then re-wrote CONF_SWAPSIZE=4096. Once I rebooted, swapon -s shows the increased file size, however, it capped out at 2GBs, even though I verified I wrote 4096 to the swap file itself. Webmin is also showing 2GBs of virtual memory now.

      I got it working, so I have no real complaints here, more just curious at this point. There must be a hard cap on the swap file? Have you heard of this? I plan on researching this tomorrow, but for now, I must get some sleep.

      Thanks,

      Eric Bartha

      • Mel Grubb says:

        I’ve never tried setting it over 2gb, so I can’t say for sure, but there may be a cap of some kind. The general rule is that your swap file should be twice your RAM, and on a Pi 2, that’s 1gb. That guideline may come into play automatically. I’m not really sure. Sticking to the general guidance, though, you shouldn’t need more than 2gb. I may try to change the swap size on a Pi with less RAM and see if it follows the pattern. I can’t say when I’ll get to do that. Perhaps this weekend. I’ll put it on my list.

  93. Jules Pitsker says:

    I am surprised it has not been mentioned earlier…
    Why not use the human readable /dev/disk/by-label/ rather than the clunky /dev/disk/by-uuid/?
    You could have easily tweaked your fstab for the already labeled “Data” partition with
    “/dev/disk/by-label/Data”
    If the tutorial were to recommend labeling the new root filesystem with e2label at format time, (like you did for the Data partition) you could have also used by-label for it.
    Much easier, human readable, elegant, smarter-not-harder.

    • Mel Grubb says:

      Labels are far less likely to be unique than uuids, and the odds of someday plugging in a new drive and having a name collision are non-zero, especially with labels like “data”. Since I’m putting this out there as instructions for others, I need to reduce, as much as possible, the number of potential landmines.

  94. Kirk Messinger says:

    Thanks, Mel, for all this great info. Unfortunately, at my age I sometimes don’t follow instructions well, so I’ve been quite a while getting set up to the point where I can boot to the external drive (sda1). However, neither partition on the external drive shows up in file manager, and the /mnt/data/ file is empty. Strangely, the old mmcblk0p2 does show up (unmounted) in Places in the File Manager. And an external drive plugged in to the other usb port on the Pi also shows up and can be mounted.

    Any ideas?

    Kirk

    • Mel Grubb says:

      My first guess would be that the drive isn’t getting enough power. Does it have its own power supply? It sounds like the Pi isn’t even seeing it at all.

      • Kirk Messinger says:

        Thanks for your interest, Mel. I hope you do not tire of advising newbies such as myself. I think part of my problems and questions lie in my inadequate grasp of the Linux file system, so I’m going to start by attempting to get more familiar with it.

        At this point, I have my OS hard drive, which has two partitions (one ext4 and one ntfs) plugged into one of the usb ports on my pi 1. Neither of these drives show up on the File Manager in Jessie’s graphical desktop. In addition, I have a 5-port usb hub plugged into the other port. This hub has three usb external drives, each with it’s own power supply, except for one which is drawing power from the hub via its own second cable. The pi is powered by cable from the remaining port on the hub. The hub also has three charging ports, one of which will power the pi, but not if the external drives are plugged into the same hub.

        What resulted was, in the desktop File Manager, in “Places”, the hard drive containing the os partition and the ntfs are not showing up at all. The original os partition (on the sd card with the boot partition) shows up unmounted. In addition, the other three usb drives also show up in “Places”. None of the four are mounted on power-up, but all can be mounted by a right-click.

        The speed is great, and I am sure I am running the os from the unseen hard drive. As an experiment, I went back and changed the ntfs partition on my os hard drive to the same parameters as the other three drives with respect to discovery and mounting in fstab. Still no go.

        As I say, I think I need a more thorough understanding of the boot process and file system in Linux to solve my problem. Somewhat annoyingly, I had this system running fine with three hard drives until I got the idea of running the os from a hard drive to make Teamviewer run on it. If I can get back to that with the os on an external drive, I’ll be happy.

        If you can see anything obvious, please let me know, but don’t waste too much time or effort on my problem. I’m going to drag out one of my Linux texts and see what I can see.

        Thanks,

        Kirk

      • Mel Grubb says:

        Let’s try the very basics first. I’d try to get the system down to just the one drive, at least temporarily. Try the “lsusb” command and see whether the device is seen at all. Also, if the drive doesn’t appear, then how did you get the positions made? Assuming it’s there, and the Pi sees it, then the problem of it not showing up under /mnt is probably something up with the fstsb file. Double check that it’s referring to the drive properly, and pointing it to a placeholder directory where you want it to appear.

      • IT Guy Kirk says:

        I think I’m going back to the very beginning, with a new installation of Jessie, and no deviations from your scenario at all. I had made a few small changes, such as using Windows Remote Desktop instead of VNC. I may need some advice when it comes to using Webmin to set up Samba shares as well. What folder/file should my os and data (I’m calling it “Current”) partitions be found in?

        Again, many thanks,

        Kirk

      • Mel Grubb says:

        Where you put your share is entirely up to you. I usually divide my hard drive up into two pieces (OS and Data). Partly because it means that you can take a backup of the OS separately from the data, but mostly because I like to keep my data partition as NTFS. It lets me temporarily attach the hard drive to the computer to do mass file moves, and generally simplifies file permission issues. You could just as easily make the whole drive one big ext4 partition and put your files in regular filesystem folders.

        As for where the partitions show up in your filesystem, that’s determined by what you put in fstab. The nice things about a unified filesystem is that you can put things wherever you want. I can add another hard drive to the system and tell it to appear at /usr/pi/stuff if I want to. My convention is to give each drive a name such as “data” or “media”, and then mount it under /mnt/data or /mnt/media. That’s just me, though. If I were to add another drive specifically to hold movies, for instance, I might even mount it so that it appears to be inside of another drive. I might put it in /mnt/data/public/movies, for example. Basically, if you can create a directory somewhere, you can mount a drive there. It’s magic.

      • IT Guy Kirk says:

        So the key is fstab, then. I guess I’d better read that part again until I understand it. I got the part about using UUID to point to a specific os partition. I’m going to chant a mantra and then start again from scratch, trying to understand what it is I am doing at each step.
        BTW, the IT Guy just refers to my own title on a WordPress blog I used to manage for a HOA. By no means a reflection on my skills!

        Kirk

      • Mel Grubb says:

        Yes. Read the fstab part carefully. The partition uuid is used to set up booting from the drive, but there’s a different uuid used for mounting the filesystem that the partition contains, and there’s a very important reboot step in the middle or the changes won’t stick.

  95. IT Guy Kirk says:

    Sorry to keep beating a dead horse, Mel, and maybe I should just forget about it, but after several runs-through, including downloading a new copy of Jessie, and starting with a brand new sd card, I can still not “see” my Data partition nor my system partition on my hard drive in file manager on the desktop. I can see the old sd card system partition (unmounted), and I looked in /mnt/data/ but nothing there. The directory exists, but it’s empty.

    Think I should give up, or am I wrong in thinking that those two partitions should be available in the file manager? Do they somehow have to be mounted at boot? I hate to give up 500 gb of storage just to make the system run off a hard drive. How can I be certain that the system is actually on the hard drive and being booted into?

    Thanks,

    Kirk

    • Mel Grubb says:

      Don’t give up, but the first step really is figuring out where the problem is. You say that the drive is powered from the hub, so I’m assuming that’s not it, but you still need to figure out whether the Pi is seeing the drive at all. “lsusb” (List USB) will list the devices connected to the Pi. I recommend attaching only the problem drive if you can, unless you have some other way to tell them apart from one another. Always start simple. Just the SD card, then one drive, then another. Baby steps.

      If the USB device is seen by the Pi, then move on to checking the partitions with “sudo blkid”. Does the drive even HAVE partitions? If you can see the partitions, then what filesystems are they using. Did you attach an NTFS drive, but not install the NTFS drivers perhaps? Is the drive completely blank, and needs to be formatted?

      Diagnosing the problem isn’t that hard, but it’s best if you start from “is it plugged in”, and work your way forward.

      • IT Guy Kirk says:

        Thanks Amol P and Mel:

        Here’s where I am now. I am using a new from scratch distro, set up according to your instructions. I skipped Webtime for now, and installed xrdp so I can use Windows Remote Desktop, but I stayed with Putty command line until I had everything completed through the end of Part 6. The Pi is powered from a 2a wall wart. The HDD, a new 1TB, is plugged into and powered by a USB hub, which is then plugged into the USB Port of the Pi. Nothing shows up in /mnt/data, either by Putty command line or Remote Desktop.

        df -h:
        Filesystem Size Used Avail Use% Mounted on
        /dev/root 15G 5.5G 8.5G 40% /
        devtmpfs 182M 0 182M 0% /dev
        tmpfs 186M 0 186M 0% /dev/shm
        tmpfs 186M 4.6M 181M 3% /run
        tmpfs 5.0M 4.0K 5.0M 1% /run/lock
        tmpfs 186M 0 186M 0% /sys/fs/cgroup
        /dev/mmcblk0p1 60M 22M 39M 36% /boot
        /dev/sda2 917G 94M 917G 1% /mnt/data
        tmpfs 38M 0 38M 0% /run/user/1000

        lsusb:
        Bus 001 Device 008: ID 0835:8502 Action Star Enterprise Co., Ltd
        Bus 001 Device 007: ID 0835:8500 Action Star Enterprise Co., Ltd
        Bus 001 Device 006: ID 14cd:6116 Super Top M6116 SATA Bridge
        Bus 001 Device 005: ID 0835:8501 Action Star Enterprise Co., Ltd
        Bus 001 Device 004: ID 0835:8500 Action Star Enterprise Co., Ltd
        Bus 001 Device 003: ID 0424:ec00 Standard Microsystems Corp. SMSC9512/9514 Fast Ethernet Adapter
        Bus 001 Device 002: ID 0424:9514 Standard Microsystems Corp.
        Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

        sudo blkid:
        /dev/mmcblk0p1: SEC_TYPE=”msdos” LABEL=”boot” UUID=”04FE-51E9″ TYPE=”vfat” PARTUUID=”8f1eafaf-01″
        /dev/mmcblk0p2: UUID=”e6e7f776-11a4-4cd7-b4fd-c44ecdbfcf90″ TYPE=”ext4″ PARTUUID=”8f1eafaf-02″
        /dev/sda1: UUID=”28597b27-6a8b-4de6-b792-d7d7062ed7ec” TYPE=”ext4″ PARTLABEL=”primary” PARTUUID=”a35fb77c-109d-404c-be89-a7ec252597c1″
        /dev/sda2: LABEL=”Data” UUID=”3BE5956A4787EC8C” TYPE=”ntfs” PARTLABEL=”primary” PARTUUID=”31f5121e-4e7e-4eb7-b19c-46a99d4938e2″
        /dev/mmcblk0: PTUUID=”8f1eafaf” PTTYPE=”dos”

        ls -l /dev/disk/by-id/usb*:
        lrwxrwxrwx 1 root root 9 Apr 11 20:13 /dev/disk/by-id/usb-Mass_Storage_Device_116AC2101219-0:0 -> ../../sda
        lrwxrwxrwx 1 root root 10 Apr 11 20:13 /dev/disk/by-id/usb-Mass_Storage_Device_116AC2101219-0:0-part1 -> ../../sda1
        lrwxrwxrwx 1 root root 10 Apr 11 20:13 /dev/disk/by-id/usb-Mass_Storage_Device_116AC2101219-0:0-part2 -> ../../sda2
        (this is the first time I’ve gotten a response from this
        command)

        (parted) print all:
        Model: Mass Storage Device (scsi)
        Disk /dev/sda: 1000GB
        Sector size (logical/physical): 512B/512B
        Partition Table: gpt
        Disk Flags:

        Number Start End Size File system Name Flags
        1 1049kB 16.0GB 16.0GB ext4 primary
        2 16.0GB 1000GB 984GB ntfs primary

        Model: SD 00000 (sd/mmc)
        Disk /dev/mmcblk0: 7861MB
        Sector size (logical/physical): 512B/512B
        Partition Table: msdos
        Disk Flags:

        Number Start End Size Type File system Flags
        1 4194kB 67.1MB 62.9MB primary fat16 lba
        2 67.1MB 7861MB 7794MB primary ext4

        Nothing in /media
        Nothing in /mnt/data, although the empty directory is there.
        No mounted disks in File Manager/Places. Unmounted 7.8GB Volume, which is the old mmcblk)p2.

      • Mel Grubb says:

        Okay then. The drive is being seen by the Pi. That pretty much leaves the mounting part. What does your /etc/fstab file say?

    • Amol P says:

      @Kirk do this and report the output:

      > sudo mkdir /media/test
      > sudo mount /dev/sda1 /media/test

      > cd /media/test
      > ls -al

      • IT Guy Kirk says:

        Here’s the result. (I did it three times with three different names, “test” and “trial” and “three)

        pi@raspberrypi:/media/three $ ls -al
        total 104
        drwxr-xr-x 23 root root 4096 Apr 12 13:25 .
        drwxr-xr-x 6 root root 4096 Apr 12 13:30 ..
        drwxr-xr-x 2 root root 4096 Apr 11 18:25 bin
        drwxr-xr-x 2 root root 4096 Mar 18 02:09 boot
        drwxr-xr-x 4 root root 4096 Apr 11 18:36 boot.bak
        drwxr-xr-x 4 root root 4096 Mar 18 02:07 dev
        drwxr-xr-x 111 root root 4096 Apr 11 20:15 etc
        drwxr-xr-x 3 root root 4096 Mar 18 02:09 home
        drwxr-xr-x 20 root root 4096 Apr 11 19:01 lib
        drwx—— 2 root root 16384 Mar 18 02:46 lost+found
        drwxr-xr-x 6 root root 4096 Apr 12 13:30 media
        drwxr-xr-x 2 root root 4096 Apr 12 13:25 mkdir
        drwxr-xr-x 3 root root 4096 Apr 11 20:05 mnt
        drwxr-xr-x 6 root root 4096 Mar 18 02:40 opt
        drwxr-xr-x 2 root root 4096 Jan 6 2015 proc
        drwx—— 2 root root 4096 Apr 11 19:03 root
        drwxr-xr-x 5 root root 4096 Mar 18 02:10 run
        drwxr-xr-x 2 root root 4096 Apr 11 19:19 sbin
        drwxr-xr-x 2 root root 4096 Mar 18 02:05 srv
        drwxr-xr-x 2 root root 4096 Apr 12 2015 sys
        drwxrwxrwt 10 root root 4096 Apr 12 13:17 tmp
        drwxr-xr-x 10 root root 4096 Mar 18 02:05 usr
        drwxr-xr-x 11 root root 4096 Apr 11 20:13 var

  96. Amol P says:

    @Kirk Use this:

    > lsusb
    This will list the USB devices connected. I have a westedn digital drive connected to my RPi so I’d see something like “WD” or “WesternDigital”.
    If you don’t see this, your USB device is under-powered

    > sudo blkid
    This will list all partitions in your drive. What’s the output of this command? It’ll also list the device name in the first column (like /dev/sda1)
    See if you can see those devices by doing:

    > ls -l /dev/disk/by-id/usb*

    Now auto mount them by doing:
    > sudo partprobe

    all available partitions should now be mounted. Check them under “/mnt”, “/mount” or “/media”

  97. IT Guy Kirk says:

    Latest report:

    drwxr-xr-x 3 root root 4096 Apr 12 09:36 ..
    drwxr-xr-x 2 root root 4096 Apr 11 18:25 bin
    drwxr-xr-x 2 root root 4096 Mar 18 02:09 boot
    drwxr-xr-x 4 root root 4096 Apr 11 18:36 boot.bak
    drwxr-xr-x 4 root root 4096 Mar 18 02:07 dev
    drwxr-xr-x 111 root root 4096 Apr 11 20:15 etc
    drwxr-xr-x 3 root root 4096 Mar 18 02:09 home
    drwxr-xr-x 20 root root 4096 Apr 11 19:01 lib
    drwx—— 2 root root 16384 Mar 18 02:46 lost+found
    drwxr-xr-x 3 root root 4096 Apr 12 09:36 media
    drwxr-xr-x 3 root root 4096 Apr 11 20:05 mnt
    drwxr-xr-x 6 root root 4096 Mar 18 02:40 opt
    drwxr-xr-x 2 root root 4096 Jan 6 2015 proc
    drwx—— 2 root root 4096 Apr 11 19:03 root
    drwxr-xr-x 5 root root 4096 Mar 18 02:10 run
    drwxr-xr-x 2 root root 4096 Apr 11 19:19 sbin
    drwxr-xr-x 2 root root 4096 Mar 18 02:05 srv
    drwxr-xr-x 2 root root 4096 Apr 12 2015 sys
    drwxrwxrwt 10 root root 4096 Apr 12 09:17 tmp
    drwxr-xr-x 10 root root 4096 Mar 18 02:05 usr
    drwxr-xr-x 11 root root 4096 Apr 11 20:13 var

    pi@raspberrypi:/ $ cat /etc/fstab
    proc /proc proc defaults 0 0
    /dev/mmcblk0p1 /boot vfat defaults 0 2
    /dev/disk/by-uuid/28597b27-6a8b-4de6-b792-d7d7062ed7ec / ext4 defaults,noatime 0 1
    /dev/disk/by-uuid/3BE5956A4787EC8C /mnt/data ntfs defaults 0 0
    #/dev/mmcblk0p2 / ext4 defaults,noatime 0 1
    # a swapfile is not a swap partition, no line here
    # use dphys-swapfile swap[on|off] for that

    Hope this reveals something. When I had this system set up before, with three external drives, it worked perfectly, but of course I wasn’t trying to boot from an external drive. That seems to be the problem.

    Thanks for your help.

    Kirk

    • Mel Grubb says:

      Did you try manually mounting the drive as Amol suggested? What did that look like? Let’s take this one step at a time. Leave booting from the hard drive until after we’ve gotten the hard drive to show up at all. My recommendation these days is to create the partitions, but postpone booting from the hard drive as long as you can so that you can keep taking simple backups of the SD card all the way up until you get the system “just right”, then cut over to the HDD. This will be reflected in an updated series of posts I’m currently putting together.

      When it comes to the actual booting, you need to be careful of a few things.
      1) It requires changes to both cmdline.txt and fstab, and the order is important. So is the reboot. If you skip any of these, it won’t stick.
      2) You have to use the correct UUID in cmdline.txt. There are two. One identifies the partition, the other identifies the filesystem. You have to get the right one in the right spot.
      3) You probably need the extra delay at the end of cmdline.txt to allow the USB device to become ready before the system needs it.

      • Amol P says:

        @Kirk I think we’re getting ahead of ourselves here. I think you are able to detect the drives now and mount them. What you said: “At least I know I’m booting from the HDD, even if I can’t see it” – I don’t see how this could be possible. If you are booting from an external drive, it needs the root filesystem to be visible. You can’t have it any other ways. The other partitions may not be mounted but the main one has to be. Something tells me you’re not booting from it.

      • Mel Grubb says:

        I’m not sure that’s entirely correct. Someone who’s more of a Linux expert can correct me here, but in my experience you can definitely boot from one root filesystem, but mount another one to /. You wouldn’t WANT to, but it happens. That’s why the reboot step I point out in the booting from the hard drive section is so important. cmdline.txt identifies a partition to boot from, but fstab says what to put at /. You can mismatch these. What you see is a difference in what the OS sees vs what YOU see. The OS boots from one filesystem, but “ls /” shows you somewhere else. This can get very confusing, and I don’t know of a legitimate reason to run a system this way, but I have forgotten that all-important reboot step and found myself in this state before.

      • IT Guy Kirk says:

        that sounds like a good explanation. I think I’ll believe it! Anyway, it works. Now to get file sharing working properly. Gotta read the paper and drink my coffee first.

        Thanks for your excellent instruction.

        Kirk

      • Amol P says:

        @Mel sorry I’m having to reply out of order since it’s not allowing me to reply to your replies to my post.
        Unless he used something like pivot_root (explained here: http://serverfault.com/questions/300764/unmounting-root-filesystem-without-rebooting-the-server), I don’t see how he could have booted and switched the filesystem. In RPi, the boot process is explained here: http://raspberrypi.stackexchange.com/questions/10489/how-does-raspberry-pi-boot. The ‘cmdline’ options is used by start.elf to load the kernel image from. Unless you manually dismount this partition, load another as root it won’t work IMHO.

        The user can easily check this by including a simple “touch /tmp/test.txt” in the startup “/etc/init.d/” section of the external root filesystem and check if the file has been created. If not, it definitely has not been mounted.

      • IT Guy Kirk says:

        Amol, I’m getting posts out of order and to the wrong poster. Anyway, I at least could see my partitions last night with File Manager. They just don’t show up in the Places list, or on the desktop, where I have told desktop config to show them. Other external drives do, but at this point I don’t care, since I can get to them via File Manager. Mel suggested that maybe it’s because they are mounted before Desktop opens. I’m thinking maybe the ntfs won’t show unless it is mounted in media, as my other hdd’s are. I’m going to change the mount path from /mnt/data/ to /media/ and see what happens. I guess I’ll have to mkdir a folder.
        So, now I’m trying to get sharing set up with Samba.
        “I ain’t down yet”
        Kirk

    • Amol P says:

      @Kirk well ok so you are able to mount the drive and see the contents. You said you loaded it with the root file system correct? So the directory listing seems right

      • IT Guy Kirk says:

        It appears so, at least to my untutored eye. I’m never sure what I’m seeing in command line. BUT… I Iooked at the partitions in Gparted on the desktop, and both partitions actually were mounted, and accessible at their mount points in File Manager. So apparently all is well as far as accessibility from the desktop is concerned. They just don’t show up in Places, nor on the desktop itself. Since my ultimate plan is to be able to access them via Samba and Teamviewer, I’ll have to see how I can do that. BTW, I plugged in another HDD which had been used for an earlier attempt at this project. It requested to be mounted, and did so, at least the ntfs partition, and showed up in Places and the desktop.

        The saga continues. Next step: Samba and Teamviewer. At least I know I’m booting from the HDD, even if I can’t see it. You guys are command line users, right? Maybe your system drive doesn’t appear in File Manager either.

        Thanks,

        Kirk

    • Mel Grubb says:

      Well that looks mounted to me. So id the problem that you can see the drives from the command line, but not from the desktop file manager? How about navigating to /mnt/data by hand in the file manager? Does it see the drive then?

      • IT Guy Kirk says:

        That is correct. As I said in my reply to Amol above, I think I’ve been talking past you. I’m not very good at interpreting what I see in the command line. When I looked at the partitions in Gparted on the desktop, everything is mounted properly, and in its proper place. Knowing that, I found them in File Manager. So now I am going to install Samba, xrdp, and TeamViewer. I was just being sidetracked by the fact that the hdd containing the system files and my Data partition don’t show up in the Places list in File Manager. As long as I can find them, I think I can share them with Samba and TeamViewer.

        Many thanks to both of you for all your help. I’m sorry to be such an obtuse old fuddy-dud, but I’m 73 years old, and even though I got my first digital computer when I was 14 (google Geniac), I’m by no means a sophisticated user. I know just about enough to help the moldy-oldies in our retirement park get their e-mail.

        Kirk

      • Mel Grubb says:

        I don’t do much from the desktop, since I use my various servers in more of a “fire and forget” mode, but I think (and could be very wrong about this) that perhaps drives and partitions that were mounted before the desktop started up, due to being represented in fstab, don’t show up in Places. Drives that show up after you’re up and running get auto-mounted and show up there. My knowledge in that area is far from complete, though. I remote in to the desktop in order to maintain CrashPlan, and sometimes to use the browser in order to pull files (such as the CrashPlan installer) down directly to the Pi. Occasionally I do it in order to edit multiple files side by side. That’s pretty much the extend of my desktop use, though.

  98. IT Guy Kirk says:

    Finally got it working the way I wanted. I added a folder (/media/NASDRIVE) as suggested by this website:

    http://www.techjawab.com/2013/06/how-to-setup-mount-auto-mount-usb-hard.html

    I think I could have done the same with your instructions, but apparently File Manager only shows drives it finds in /media. No real problem for command-line types, but difficult for GUI junkies such as myself.

    Thanks again,

    Kirk

  99. IT Guy Kirk says:

    Oops, no I didn’t! 😦

    Kirk

  100. IT Guy Kirk says:

    OooooKay. Last post on this subject, I hope. After undoing mistakes involving confused disc labels, mount points, disks that weren’t working that I thought were, and other confusions related to aging brain cells, I finally actually have things working the way I want. All drives except the system partition showing up on the desktop, and in the File Manager Places list. Moving on to automount the drives and Samba sharing, but not ’til tomorrow. My brain needs a rest and a beer.

    BTW, did you ever come up with a way to copy the system files from the hdd, and use them for restoration if necessary?

    Thanks again,

    Kirk

    • Mel Grubb says:

      I haven’t written it up yet, but backing up the HDD is actually dead simple. You just make another SD card, install the NTFS support packages, and set up fstab to mount just the data partition.

      When you want to make a backup, you shut down the system, swap in the new SD card, boot back up, and use dd to copy /dev/sda1 to /mnt/data/backups or some other folder you’ve set up. You could also attach and mount a different drive to catch the backups if you’re worried about the physical drive failing, rather than just having a convenient way to roll back.

      • IT Guy Kirk says:

        I hadn’t thought of that. I’ll try it, before I start working with my file permissions, and installing teamviewer. I got the permissions wrong on the directories I mount my hdd’s in (except the one one my system hdd). But orherwise everything is working well.

        Thanks,

        Kirk

  101. IT Guy Kirk says:

    Just a quick update and a round of applause!
    I now have my system up and running the way I wanted it to, I can even stream movies to my cell phone. And I’ve learned a LOT about Linux in the bargain, All thanks to you and your blog mates.
    Kirk

  102. Jake says:

    Have you noticed a high CPU % by mount.ntfs lately? It’s spikes occasionally and slows the RPI down. Not sure if some update had any effect.

    • Mel Grubb says:

      I haven’t noticed, but I haven’t exactly been looking, either. NTFS does add some additional overhead over what you would see with ext4, but it’s usually not that significant on its own. The increased usage might be a result of something else making use of the drive. I stopped running Transmission long ago because I found that it was the most disruptive influence on my server, and I wasn’t really spending a lot of time on torrent-able things anyway. When I shut it down, my drive usage dropped dramatically.

  103. Gilad Barak says:

    I followed this section several months ago.
    I recently installed Owncloud and tried to move the data folder to the external drive, ut it does not work since the ownership and permission are “stuck” with root and I need them changed to www-data. Seems that NTFS does not support this (or I dd not understand how to do it)
    Is there a simple way to change the data partition to ext4, knowing that all data on it is lost, but without affecting the other partitions and all the work done moving root partition, swap etc to the hard disk?

    • Mel Grubb says:

      You can’t change a partition from one filesystem to another without losing everything it contains, but you can reformat a partition to a different filesystem type without harming the other partitions. You’d have to back up the contents of the partition somewhere, and then copy it all back when you’re done reformatting that partition. Just be very careful about making sure you’re selected the right partition. Follow the directions for setting up the hard drive, but skip the part where you create the partition table and partitions. Just do the part where you do the formatting with mkfs.ext4.

      • Gilad Barak says:

        Reformatted disk to ext4
        Then I also changed the entry in fstab for the line of /mnt/data from ntfs-g3 to ext4 (you did not mention it, but I thought I should).
        After booting could not login again via SSH
        Connected a monitor and I see the following at the end of the boot process:

        Welcome to emergency mode! After logging in type “journalctl -xb” to view system logs “systemctl reboot” to reboot, “systemctl default” to try to boot again into default mode
        syslogin: root accout is locked starting shell
        root@SERVERNAME:~#

        looked into the log and can’t find any errors
        tried “systemctl default” and nothing happened
        tried “systemctl reboot” and system rebooted to same situation

        So now I am in a shell, but really do not know how to proceed to make the system boot correctly.

        Any suggestions?

  104. Gilad Barak says:

    Changed also the UUID of the partition, since I noticed that it changed – but still same problem

  105. Paulo Gomes says:

    Hi Mel

    How can we supervise the HDD usage, like free space, quotas and such?
    How to avoid running out of space (for those of us using smaller HDDs)

  106. Pingback: Web Server - The Log Cabin Blog

  107. Rich D. says:

    First, thanks for this series of articles! It’s been incredibly helpful as a linux newbie to begin learning with something real to work on. I have a question. I fear I’ve done something unrecoverable but am hoping you know something I don’t. I’m successfully booting from the HDD and got the minidlna server up and running nicely. I then decided it was time to secure the pi and changed the password after recording what the new password would be. The problem is now that since I logged off, the new password is not accepted. I’ve tried everything I can think of but no joy. I do have an image I can restore to prior to changing to HDD boot. Once I get the SD card restored to that point, how do I change the password without having to redo the HDD stuff all over again. I do have the data backed up on another drive. FYI, the new password includes the symbols !, $, and +. Are any of those illegal? I typically use strong passwords of 15 characters or so. Any advice you can give would be most welcome.

    Thanks again for this series! It’s been most instructive.

    • Mel Grubb says:

      Sounds like a random typing error when changing the password, but you would have had to type it wrong TWICE to successfully change it at all. If you have another root-level account set up, you could possibly use that the reset the Pi user’s password. Otherwise, I don’t know what to tell you. You’re essentially trying to break into your own system, something that it’s designed to stop people from doing.

      As for the booting from the hard drive part. I think you’re going to have to repeat that part, but the STUFF on the hard drive should survive just fine. What I’d recommend is that you put off the booting from the hard drive part until the very end, that way you can keep taking simple SD card backups all the way up until the server is just about finished. Then, as your last step, you make the leap to booting from the hard drive.

      • Rich D. says:

        Thanks for getting back to me! That’s what I thought you would say. One of my co-workers suggested setting up the SSH keys for this exact reason. I changed my password from the GUI and am wondering if that was an unwise thing to do.

        Do you have any advice for how to shut down since I can’t log on? I do have a ups with apcupsd running. I guess I could unplug and let the battery run down till it shut’s down. That might take a while! BTW, I’m running the 1TB Pi Drive from WD and it works great. I’m powering it and the Pi from a usb hub and they have a nifty three way cable just for the Pi. I think I might buy another one.

        Thanks again!

    • Amol P says:

      The easiest way for you to login into your RPi would be to take out the SD card, put it in a computer running linux and modify a single line in a file as described here:

      https://www.raspberrypi.org/forums/viewtopic.php?p=109612

      After that, put the SD card back, boot up your RPi. You’ll directly goto the prompt with admin permissions. After that you can change the password without knowing what it is currently.

      • Mel Grubb says:

        Well that’s useful to know, and also kind of troubling. It shouldn’t be that easy to just take over a computer like that. Then again, Raspbian doesn’t advertise itself as a hardened, secure OS in the first place.

      • Amol P says:

        Yup it is easy. I wrote about it in the Raspberry Pi forum too. Unfortunately there is no way to disable it.

      • Mel Grubb says:

        Well, as they say. If someone has physical access to your computer, it’s not your computer anymore.

      • Amol P says:

        Well I disagree. That is but an old viewpoint, before the era of complex software, encryption and GPS. Today, your iPhone could be stolen but activation lock renders it inoperable to anyone but Apple and big govt. agencies.

        Similarly, it’s a question of kernel and boot security policies in Linux. We could just disable single user mode in Linux but they choose not to by default. The disk can be encrypted and decrypted upon boot. In that case the stolen RPi in theory would be junk data to anyone else. The problem of course lies, in these techniques not being easy to use.

  108. Rich D. says:

    Thanks for the replies, Amol, and Mel. I tried this last night and while the Pi booted to a hash prompt it didn’t recognize any input from the keyboard and the display had an error like user 0, unable to access tty, job control disabled. That’s not the exact text but the gist. Looks like i’ll have to restore the SD card to my last backup image. I’m going to take Mel’s advice and change the password first, then install all the other utility stuff and then move the root file system back over to the HDD. Live and learn! But then, that’s what the Pi is all about. It’s frustrating, but you can bet I won’t make that mistake again! On to new and different mistakes! 🙂

    Thanks again, guys!

  109. Paulo Gomes says:

    Hi Mel,

    My server ROCKS thanks to your tutorial!
    I have a one-off question I think, that hasn’t yet been posted to you….
    My server is running on a pi2, 16GB sd and a 260GB hdd…
    Now I’d like to replace the 250GB hdd by a 1TB… How can I do this without going throu all the process again and without just adding it as a second hdd to the system?

  110. Paulo Gomes says:

    one last question, then… when on raspi-config, will I be able to tell if the filesystem I’m trying to expand is the hdd and not the sd card?

    • Mel Grubb says:

      It’s whichever filesystem you booted from.

    • Mel Grubb says:

      You’ll need to be careful, though. If you clone the drive, both drives will have the same partition and filesystem UUIDs on them, so you’ll need to swap the drives, not just add the new one. If you have them both hooked up when the system boots, though, it’s going to get confused as to which one to use. Once you’ve booted from the new one, and expanded the filesystem, then you can try hooking up the second drive, and use parted to blow away its partition table and create a new one. Just be careful about which one you’re killing. run “sudo blkid” before and after hooking up the original drive to be sure that it is, in fact, /dev/sdb. Once the old drive has been given a new identity, then you can leave them both hooked up all the time, if that’s what you want.

  111. Rich D. says:

    Hi Mel,
    So, I’ve recovered from the silly mistakes I made earlier and the pi had been running smoothly. Now, every few days, I see ‘Write error on swap device with additional info. Rebooting the Pi seems to clear things up for a few days.

    Do you have any recommendations for diagnostics and remediation? Can I script that? FYI, I’m using a 1TB WD Pi-Drive that has a USB3 connector on the drive end.

    Still a Linux NOOB, but learning!

    Thanks!

    Rich

    • Mel Grubb says:

      If you’re booting and running from the SD card, I would look at getting a better SD card, or moving the swap file to the hard drive, which is better for such things, but will only work on an ext4 partition (as far as I know). Edit the swapfile configuration with “sudo nano /etc/dphys-swapfile”. Increase the size (probably set at 100mb) to twice your physical RAM (so most likely 2048), and add a line to relocate the swap file to somewhere on the hard drive “CONF_SWAPFILE=/mnt/data/swapfile” (replace /mnt/data with wherever you mounted your hard drive). Remember, if your hard drive is formatted NTFS, I don’t think this will work.

      If you’ve moved your whole OS partition over to the drive, and are completely booting and running from there, then there is nothing to move, although you probably still want to increase the size. If that’s the case, then I’d suggest consulting the great oracle of Google. That is not an error I have personally encountered.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s