Article Obsolete
A new version of this series has been published. Please refer to the new index for updated articles and ordering. This article is kept for historical reference, but should be considered out of date.
Note: This article is part of a series. See the Index for more information.
Self-promotion: I’ve recorded this series as a screencast for Pluralsight:
(http://www.pluralsight.com/courses/raspberry-pi-home-server)
If you have a Pluralsight subscription, please consider watching it. Thanks!
Updates: If you didn’t change the boot option using raspi-config so that the Pi boots to the command-line, the hard drive will auto-mount each time you boot up, and copying the root filesystem to the external drive will fail. Make sure you’ve configured the Pi to boot to the command line, and everything should work as per the original post.
Running the Raspberry Pi off of an SD card is simple, affordable, and very convenient. SD cards are also very small compared to hard drives, though, and they can only be written to a finite number of times. That number is ridiculously large, but some have said that it can still be used up faster than you think by things like virtual memory swap files which are written and re-written constantly.
Whether you believe that or not, one thing is certain; In order to be a useful home server, the Raspberry Pi is going to need access to more storage space than you get from a simple SD card. The Raspberry Pi Home Server is going to need a proper hard drive, and as long as you’re adding a hard drive, you may as well take advantage of its speed and boot the Raspberry Pi from it.
Okay, I’ll level with you. You can’t actually boot entirely from the hard drive, but you can boot mostly from the hard drive. When the Raspberry Pi boots up, it looks to the first partition on the SD card for instructions on what to do next. That boot partition contains just a tiny piece of the boot process, but at least for now, its location is non-negotiable. That’s just the way it is. That’s not to say things couldn’t change in the future. I saw someone on a forum claiming to have done away with the SD card altogether, but I’ll believe it when I see it.
What you can do is to move all the stuff that comes after that first step to a different device, such as the external hard drive. All that stays behind on the SD card is a very small boot partition, everything else moves to the hard drive.
The resulting system should run faster and smoother, and you won’t be using up your SD card’s limited number of write cycles on high-frequency stuff like a virtual memory swap file.
Acknowledgements
This article would not have been possible if not for others that went before me. In particular, Ted Hale’s article at http://raspberrypihobbyist.blogspot.com/2013/07/running-from-external-hard-drive.html is pretty much the blueprint I started from. In turn, that article was based on a forum post by “Rattus” at http://www.raspberrypi.org/phpBB3/viewtopic.php?f=26&t=10914&p=129474&hilit=resizefs#p122476.
I’ve added my own bits, such as consulting the boot/cmdline.txt file rather than simply assuming the root file system’s location, using UUIDs to identify the partitions, and expanding the existing swap file rather than creating a separate swap partition.
I’ve turned these resources into a walkthrough that matches the rest of this series, but I wanted to at least point out where my source material originated.
Gather your tools
This post uses some software packages that are not included by default in the Raspbian distribution. You’re going to need NTFS filesystem support, and the gdisk tool. Install them before continuing.
sudo apt-get install ntfs-3g ntfs-config gdisk
Pick a hard drive
Pretty much any external USB hard drive will do. For my own server at home, I’m using a dual-drive 2TB RAID enclosure (Cavalry CADA-SA2). This might seem like overkill for a Raspberry Pi, but one of my home server’s jobs is to centralize and share my music collection. It took me four solid weeks to rip every CD I own, and I do not plan to repeat that process. My family photos and videos are on that drive as well. Basically, this is all stuff I don’t want to lose due to a hard drive failure.
Note: Yes, I have an offline backup, too, but swapping out a dead drive in a RAID is easier than rebuilding the whole thing by hand.
Linux device names
Before we start, lets take a moment to discuss the way in which Linux refers to devices in general, and more specifically the way it refers to hard drives. Windows users are used to seeing multiple drives show up with different letters. You have your “C” drive, where Windows and all of your programs are installed. Maybe you also have a second hard drive called “D”, or maybe “D” is CD or DVD drive on your computer. “A” and “B” used to refer to floppy drives, and are reserved for historical purposes. Each of these devices is completely separate, and has its own individual root.
Linux has no drive letters. Linux uses what’s called a “unified file system”, meaning that files, folders, and even entire drives appear together under one root. You can see all of these different devices in the “/dev” folder. Take a look:
ls /dev
I won’t go into what all of these entries mean, but you can see that there are a lot of them. If I were to plug in an external hard drive, or perhaps a flash drive, I would see new entries pop up in the list. Here’s another screen shot after I’ve plugged in a flash drive.
Look about halfway down the third column, and you’ll see new entries called “sda” and “sda1”. These represent the flash drive, and the first partition it contains, respectively.
Storage devices like a flash drive or a hard drive can be divided up into multiple pieces called partitions that can make it appear to be multiple drives. This list of where the partitions begin and end is called the “partition table”, and every drive has one. If there were more partitions on the flash drive, there would be more entries in /dev called “sda2”, “sda3”, etc.
On a computer where drives are connected via SCSI (Small Computer System Interface), “a” would be the first device, “b” would be the second, etc. Drives connected via IDE (Integrated Drive Electronics), or SATA (Serial AT Attachment) have similar concepts of “first”, “second”, etc.
This is not the case with USB. USB devices have no concept of “first” or “second” and so their names are assigned on a first-come, first-served basis. The first drive to be ready gets to be “sda”, the next one “sdb” and so on.
If you only have one drive, then it will obviously always be called “sda”, and there’s no problem, but since you’re building a server, it’s conceivable, and even likely that you will eventually end up with more than one hard drive attached to it. Perhaps one drive will house videos, and another one will have music on it.
Even if you don’t see a need for that yet, you’ll end up filling up a drive eventually, and you’ll want to add another one. As soon as you add more USB drives, however, you can’t count on their names being the same every time the machine starts up. As a result, drives may end up mounted in the wrong places, and programs won’t find their files where they expect them to be because the drive that used to be “sda” was a little slower off the line one morning and got stuck with the name “sdb” instead.
Let’s take care of that particular problem before going any further.
Format the hard drive
If the drive you’re planning to use already has stuff on it, then back its contents up somewhere. and restore them when you’re done. This approach calls for nuking and repaving the whole hard drive.
Caution: Even if you wanted to live dangerously and manipulate the partitions in place, I wouldn’t do it without a backup copy first. There are just too many things that could go wrong. What if the power went out partway through the process? Don’t risk it. Just copy your stuff to another drive, and copy it back when you’re done.
The key to fixing the device naming problem requires us to have a specific kind of partition table called a GUID Partition Table. This is different than the older Master Boot Record which is used by most Windows machines.
In order to create a GPT, you’ll need to use Linux’s Partition Editor (parted) application. Raspbian already includes parted, so there’s nothing to install. Run it with the following command:
sudo parted
You should see something almost, but not quite entirely unlike a command line. To see a list of all the known devices and partitions, type “print all”. Parted should print out a table with two rows similar to this:
The rows in the table represent the partitions. The columns will tell you where each partition begins and ends, how large it is, and what kind of filesystem it contains.
If you see more than one table at this point, then you probably have another drive plugged in somewhere. I strongly advise shutting down and removing all drives other than the SD card before continuing.
The SD card in the image above was created using the Raspbian image file, and has two partitions on it, a small “boot” partition, and the root filesystem partition. If you started by using NOOBS, then you probably have five partitions. Future releases could divide things up differently, so don’t worry if your output doesn’t look like the example.
Attach the hard drive
Plug your external drive into a free USB port on the Raspberry Pi, and give the OS a few seconds to notice it. List the known partitions by typing “print all” again.
This time, there are two tables. In my example, the second one is the external drive, although the order is not guaranteed here.
Notice the headers above the tables. The header tells you information about the drive in general. In this case, the header tells us that the hard drive was assigned the name “/dev/sda”, that it’s 2TB in size, and that its partition table type is “msdos”. That means it’s using the old MBR-style partition table, and that’s not going to work for this server.
Select the external drive by typing “select /dev/sda”. Type “print” all by itself to make sure. You should see the second table from above again. Read the header carefully, and make sure it’s referring to the external drive, because you’re about to blow away the partition table.
Create a new partition table with the command “mklabel gpt”. Read the warning carefully, and make sure it refers to “/dev/sda”.
Answer “y” to the prompt, and in a few seconds, you will have a nice blank slate to work with. Check the results with another “print”. The header should now say that the partition table type is “gpt”, and there should be no partitions on it.
Create new partitions
Next, you’ll need to create two new partitions; one to hold the root filesystem, and one to hold general data. You could make one big partition, combining the root filesystem and data storage in one, but there are some benefits to keeping the data partition separate. The root filesystem must be formatted using Linux’s native filesystem, ext4. The data partition, however, can be any type you want, which means you can pick something other, non-Linux computers will understand.
Root Partition
This is where Linux and your programs will go. You can think of it as Windows’ C: drive. The size is up to you, but 16GB is probably fine for the simple server described in this series. If your existing SD card is larger, or you are planning to install a lot of programs, go bigger. Just make sure the partition is at least as large as the SD card you are currently using.
Create a new, 16GB partition starting at the beginning of the drive:
mkpart primary 0GB 16GB
Data Partition
This is where you’ll put your “stuff”, and it should take up the rest of the available drive space. The mkpart command can interpret the beginning and ending parameters in a variety of different formats. Use this ability to create a second partition that begins where the first partition ends, and takes up all the rest of the space on the drive:
mkpart primary 16GB 100%
Finally, use the “print” command again to see the results.
That’s it for the partitions, so type “q” and press enter to exit parted. You can ignore the warning about updating /etc/fstab for now. We’ll get to that soon enough. You now have two partitions, but they’re completely blank at this point.
Copy the old root partition to the new drive
The file /boot/cmdline.txt specifies the location of the root filesystem. You’ll need to edit this file later in order to boot from the hard drive, but for now, you just need to know for sure where the root filesystem currently resides. Show the cmdline file’s contents like this:
cat /boot/cmdline.txt
The part immediately to the right of “root=” specifies the device and partition that holds the root filesystem, and that’s the partition whose contents you need to move to the hard drive. In my case, it says “mmcblk0p2”. If you started from the raw Raspbian image, then yours probably says the same thing. If you started from NOOBS, then it’s probably “mmcblk0p6”
Copy the existing root partition’s contents to the new 16GB partition (sda1) on the hard drive, changing the highlighted part to match the current root filesystem path from above.
sudo dd if=/dev/mmcblk0p2 of=/dev/sda1 bs=32M conv=noerror,sync
This can run for quite a while, depending on how large your SD card is. Unfortunately, the dd command doesn’t give you any kind of feedback. In this case, no news is good news. As long as the drive appears to be busy, then keep waiting.
Here’s what that all means:
- “dd” copies things
- “if” specifies the input file, in this case an entire partitions
- “of” specifies the output file
- “bs” specifies how many bytes to copy in one chunk, here it’s 32 MB
- “conv” specifies how to convert things as they are copied
- noerror says to continue if anything goes wrong
- sync does some padding during the copy
When the copy process eventually finishes, check the target partition on the hard drive for errors.
sudo e2fsck -f /dev/sda1
Press enter at the prompts to fix any errors that it finds. You can probably expect one about the free block count, and one about the free inodes count.
Since the image that you just copied over from the SD card is almost definitely smaller than the partition you copied it to, you’ll want to expand it to fill up the available space:
sudo resize2fs /dev/sda1
Format the data partition
That takes care of the root filesystem. Now on to the data partition. Choosing NTFS as the filesystem will allow you to simply plug the drive into any Windows, MAC, or Linux desktop computer and manipulate the drive contents. This can be especially helpful for loading up large amounts of data, such as the family’s music and photo collection, without having to squeeze it all through the network connection.
Format the second partition as follows, filling in whatever name you want the partition to have. I’ve called mine “Data”.
sudo mkfs.ntfs -Q -L Data /dev/sda2
Note: Don’t forget the –Q (Quick) or you’ll be waiting around a while.
Uniquely identifying a drive
Warning: Because the next section involves copying around large strings of unmemorizable data, I recommend performing the steps through a remote SSH window, or from a terminal on the X desktop. The Ids you’ll need to copy need to be copied verbatim, letter for letter, or you’ll find yourself unable to boot, and you’ll have to start over again from your last backup.
Currently, the cmdline.txt file specifies that the root file system is on the second partition of the internal SD card (/dev/mmcblk0p2). We’d like that to say “/dev/sda1” instead, but there’s a problem. As I mentioned earlier, we don’t have any guarantee that this particular drive will be called “sda” in the future. What we need is a way to uniquely refer to this drive no matter what letter it get assigned on any given day.
This was why we built the drive with a GUID Partition Table. Each partition on a GPT device is assigned a universally-unique identifier (UUID).
Note: The difference between GUID and UUID is not important here, they are, for all intents and purposes, the same thing; a very long, randomly-assigned number.
Instead of identifying the root filesystem’s partition by location, we want to refer to it by Id so that it can be found no matter what order the drives were discovered in. Use the “gdisk” package you installed earlier to find out more about /dev/sda1:
sudo gdisk /dev/sda
You should see another not-quite-a-command-prompt like this:
Type “i” and press enter to show information about partitons, then type “1” (one) to identify the partition you are interested in. The partition’s details will be displayed like this:
The piece of information we’re interested is the “Partition unique GUID” on the second line, not the “Partition GUID code”. Take a picture, write it down very carefully, or select and copy the text if you are doing this through a remote terminal connection like I am.
Type “q” and press enter to exit the gdisk tool.
Boot Configuration
The new root filesystem’s partition is now uniquely identifiable, but the Raspberry Pi doesn’t know to use it. Edit the /boot/cmdline file to change where the bootloader will look for the root filesystem.
sudo nano /boot/cmdline.txt
Find the part that says “root=/dev/mmcblk0p…” and change it to “root=PARTUUID=” and whatever your “Partition unique GUID” was from above. Also add the string “rootdelay=5” at the end. This will give the Raspberry Pi time to discover the USB drive before it tries booting from it.
The result should look like this (sorry for the small picture):
Exit Nano, saving your changes (ctrl-x,y,enter)
Reboot (Important, do not skip this step)
Before the next set of changes will “stick”, you’ll need to reboot so that the Raspberry Pi uses the hard drive for the initial load. If you don’t reboot now, nothing you’re about to do will count, and you’ll just have to do it all again. You’ve been warned.
sudo reboot
If everything goes well, you should find yourself back at a login, and you can continue. If something went wrong, go back to your most recent backup and try again.
Mounting filesystems
Partitions and filesystems are not the same thing. Linux now knows what partition to load the OS from, and that’s great, but as things stand right now, it’s still going to mount the root filesystem from the SD card.
To complete the transition to the hard drive, you’ll need to edit the filesystem table. This file controls what gets mounted where, and in what order, and it needs to know where the root filesystem is. Take a look at the current contents.
cat /etc/fstab
That third line is the root filesystem, which you can tell by the “/” in the second column. Unfortunately, and it’s still loading from the 2nd partition on the SD card (mmcblk0p2). You could change this to say “/dev/sda1”, but that would only work as long as the drive continues to get the name “sda”.
Fortunately, you can use a very similar UUID-based trick here to uniquely identify the filesystem no matter what letter the device gets. Filesystems have UUIDs too, and you can see them all with this command:
sudo blkid
If you look carefully, you may notice a problem. both /dev/mmblk0p2 and /dev/sda1 have the same UUID. So much for being unique, right? This is because of the way we cloned the old root filesystem into a new location. It brought the whole filesystem over, including its UUID. Before you can use a unique Id to identify the drive, you’ll need to make sure it’s actually unique.
You need to give /dev/sda1 a new UUID. You can hand-assign your favorite UUID, or just let the computer pick a random one. Use the following command to assign anew UUID to the first partition on the hard drive:
sudo tune2fs /dev/sda1 -U random
Note: At the time of this edit, there seems to be an issue with the newer “Jessie” release of Raspbian, and it may prevent you from changing the UUID on the partition. If you see an error that says “The UUID may only be changed when the filesystem is unmounted.”, even though it’s NOT mounted, then you’ll need to perform one more tweak. The magic words are
sudo tune2fs -O ^uninit_bg /dev/sda1
I can’t take credit for this one at all. I just found it here. After reading the man page on tune2fs, I still don’t honestly even understand what this has to do with anything, but it does seem to do the trick, so if you get the above error, give this command a shot, then try the “-U random” command again and it should succeed.
Either way, display the device Ids again, just to be sure
sudo blkid
Copy down the UUIDs for /dev/sda1 and /dev/sda2, you’ll need them both in a minute. Then open the filesystem table in an editor.
sudo nano /etc/fstab
Change “/dev/mmcblk0p2” on the third line to “/dev/disk/by-uuid/” and the new UUID you just assigned to /dev/sda1. Add another line for /sda2, substituting its UUID, and mounting it as /mnt/data, with a filesystem type of “ntfs”.
The end result should look like this:
Don’t worry about the columns lining up, it doesn’t matter, but I’m presenting mine to the public, so I’ve gone ahead and made it pretty
Exit Nano, saving your changes (ctrl-x,y,enter)
The system should be just about ready to boot from the hard drive, but there’s one more thing to take card of before we do that. Currently, the filesystem table says to mount the second partition at /mnt/data. Before that can succeed, you need to create a folder in that location. You can think of it as a placeholder where the actual drive will appear later on.
sudo mkdir /mnt/data
That’s it. you’re ready to reboot again, and this time, everything should be faster.
sudo reboot
You’ll notice that the “ACT” light on the Raspberry Pi will not blink much anymore. That’s because the SD card is no longer being accessed for anything other than the boot partition. The hard drive’s activity light will now blink where the Raspberry Pi’s activity light used to.
When the system has rebooted, double-check your filesystem table to make sure your changes are still there.
cat /etc/fstab
If you see /dev/mmcblk0p2, and no line for your data partition, it’s because you skipped that “Reboot” step above. See? I told you it was important. Redo everything in the “Mounting filesystems” section, and reboot again.
Swap configuration
One thing you may have noticed if you’ve looked at other walkthroughs for booting from the hard drive is that they usually create a “swap” partition on the hard drive to be used as virtual memory.
If you look back at the filesystem table from the beginning of this post, though, you’ll notice that Raspbian never had a swap partition in the first place. That’s because Raspbian is set up to use swap files instead of swap partitions. Raspbian’s swap file lives at /var/swap, and since we just moved the whole root filesystem onto the hard drive, the swap file came along for the ride.
At this point, you’re already running your swap file from the hard drive, and you didn’t even have to do anything. Check it out with the “swapon” command:
sudo swapon –s
This shows the swap summary, which will tell you what swaps are in use. It should have one entry in it (/var/swap). It’s pretty small, though; only 100MB:
The partition we created for the root filesystem is 16GB (or more if you so chose). There’s plenty of space left to expand the swap file to something roomier.
Edit the swap file configuration:
sudo nano /etc/dphys-swapfile
The configuration file that came with Raspbian only has a single line in it which says “CONF_SWAPSIZE=100”. This is the size of the swap file in megabytes. Change the value to 2048, which will create a 2GB swap file.
Close Nano, saving your work (ctrl-x,y,enter).
Reboot one more time to apply the changes, and check them by typing “swapon –s” again. You should see a table similar to the first time you ran this command, only now the swap file is much larger:
Wrapping up
You now have a Raspberry Pi that boots (mostly) from an external USB drive. It also uses this drive for its virtual memory swap file. The whole system should run more smoothly now, and you won’t have to worry about using up your SD card, if that’s the sort of thing you worry about.
Although most of the important stuff is on the hard drive, you should probably make at least one more backup of the SD card for safety. I may cover backing up the root filesystem from the hard drive later on in another post, or update this one with the changes.
What’s next?
In the next post, we’ll put the new drive to work sharing files over the local network.
MelGrubb.ToBlog() 
Pingback: Raspberry Pi Home Server: Index | MelGrubb.ToBlog()
Pingback: Raspberry Pi Home Server: Part 5, Remote Desktop | MelGrubb.ToBlog()
Pingback: Raspberry Pi Home Server: Part 7, Sharing Files With Samba | MelGrubb.ToBlog()
Pingback: Raspberry Pi Home Server: Part 10, CrashPlan | MelGrubb.ToBlog()
Hi,i have an issue here that it cannot detect my external hard drive ,I reformatted it in NTFS system but ntg appear in the pi
If the drive just never shows up at all, meaning it never appears as sda/sdb/etc, then you may have simply found a drive that’s just not compatible with the Pi. I’d double check that it’s getting power. I have an external 3.5″ drive that has no power cord other than the USB cable. It works just fine on normal computers, but the Pi just doesn’t have enough juice to run it. The drives I’m actually using for my server are in a separately-powered external enclosure.
Can I use an external drive and route it through a powered USB hub?
Absolutely, in fact I thought I suggested that in the article. Not only that, but you can even power the Pi off of its own hub. The micro-USB port that powers the Pi doesn’t use the data lines, only the power lines, so you can plug it into its own hub, and it won’t try to “talk to itself”. I run one of my systems this way. I have a good, strong, powered hub, with the hub’s USB cable plugged into one of the Pi’s regular ports. Then, I have a micro-USB cable running from the Pi to a regular port on the hub along with any external drives. Hitting the switch on the hub powers up the whole thing. This also avoids problems with the hard drive “back-powering” the Pi, and preventing it from shutting down.
ok thx for your information , can i use a USB drive like a 64 GB capacity would u recommend or the ones using external power
Flash drives usually don’t draw much power, so it should be fine. Another approach you might try is to hook up your hard drive through a powered USB hub. They’re pretty cheap, and would take the power drain away from the Pi.
do you mean that the hub is connected to the Pi while connecting to the drive ,or another hub separated from the Pi and power?
I mean that you get a USB hub that has its own power adapter, plug that hub into one of the full-size USB ports on the Pi, and plug the drive into the hub. If the hub has its own power supply, then it should be able to power the drive.
So if i have a powered hub thats only provide power for devices ,it won’t work cause no data will be transfered?mine is a powered hub
No, it’s just like any other hub. If you plug the hub in to one of the full-sized USB ports on the Pi, data will be transferred. The power for the drive will come from the hub instead of the Pi, though. This should allow the drive to run. It’s just a guess, but I’m thinking that the drive simply draws more power than the Pi can provide on its own.
so i need to get another one…cause that only provides power for the Pi
Actually, you CAN power the Pi off of its own hub. It looks a little funny, but it works. The micro-usb port on the Pi ignores the data lines entirely, and only uses the power lines. If your USB hub has enough power, you can run everything from it. You plug the hub into the Pi’s regular USB port like you would anything else. Next, use a micro-USB cable to plug the Pi’s power line into the hub. Finally, plug the hard drive into the hub. when you power up the hub, everything should come on at once. I haven’t done this myself, because my power strip has USB ports on it, and I use those to power the Pi. I did try it once, though, and out worked fine. Your mileage may vary, of course.
my hub is this ..think it’ll work ?
http://www.oricoonline.us/info.aspx?pid=839
Aha, I see the problem. That’s not a hub. That’s just a charger. Do you have, or can you borrow a powered hub from someone just to check whether power is the problem?
All right , i’ll see how i can solve it
Well..my drive is busy for awhile …after that it stop but ntg happen also
i mean the copy part to the sda1 ,my drive stops running but ntg appear on screen so i can’r command anything
Do you mean the partition copy from the SD card to the drive? Yeah, that can take a while, depending on the size of the partition. My advice is to leave it running way longer than it seems like it should take. Go have dinner and come back. If it just never finishes, then maybe there is something else going on. I’ve done this several times, though, and didn’t have a problem. Remember that booting from the hard drive is optional. It turned out that my Pi actually didn’t like booting from the RAID box because of the drives I used constantly entering power saving mode, so my own personal server currently boots from the SD card for now. If I upgrade storage later on, I’ll go back to booting from the hard drive again.
I solved it with “sudo reboot” and now working with Samba things..
So now i have an serious issue my just dropped my HDD so…ya its beyong repair so how to i boot back from the SD card?
Just restore the last backup of the SD card from before you added the hard drive. If you weren’t making backups (shame), then you can just undo the changes to the fstab and cmdline.txt files. Change the “root=” part in cmdline.txt back to what it originally said. If you don’t have that written down anywhere, then you might be safer starting over than trying to guess.
i backup after doing it…
Guess i have to start all over again until i get another HDD
I’d recommend making multiple backups this time through. I kept one from every step until I was finished and satisfied, then I deleted the older ones.
All right,thx for the advice starting over with the image…
This guide is great. However, I got to the step where I reboot after setting the boot to the UUID of the external hard drive. Once I reboot I can’t connect through SSH anymore. The first time I realized I forgot to resize the partition, but the second time I made sure to follow each step exactly. Any ideas what might be going wrong?
Are you able to hook the Pi to a monitor to make sure it’s booting? If the UUID gets even a little bit wrong, it will never find the drive, and will fail to boot. The article warns about this. If you can see that it’s booting, then I don’t really have any advice. I don’t see how the drive and ssh could be related apart from the machine not booting at all. If that’s the case, then roll back to the previous backup and try again, but be extra careful about typos.
Thanks, I can hook it up to a TV to check but didn’t have the time last night. I reread your post and another search on google and realized that even after doing it twice and reading carefully I left out the PARTUUID after “root=”. I had thought you were saying to replace PARTUUID with the actual UUID rather than enter it after, so this is probably the issue. This is what I get for watching Netflix while I do this kind of stuff.
I’ll try it again after work.
a quick question:
will this kind of settting make the hard drive spin all the time? I want to use the raspberry pi for a server for bt sync and my SQL. I am wondering whether any of the three setting (1) booting from the external drive, (2) using bt sync, and (3) using my sql will make the driver spinning all the time.
Thanks
Since this is a server, I think drives spinning all the time is to be expected, but most consumer hard drives will go to sleep on their own when they are not being actively used. With my own server, I found that my raid enclosure didn’t LIKE my hard drives falling asleep to save power. I reverted to using my SD card for the system drive, and the raid for bulk storage. If power is a concern, then maybe you should do the same. That way the drive can spin down, and you’ll use only the SD card when the server is idle.
Hi Mel
I am very interested in your instructions but am confused about whether or not to use the SD card for the system drive. Your article seems to recommend that the system be put on the hard drive but in the comment above you seem to suggest it be put on the SD card. Which do you now recommend and, if the answer is the SD drive, what changes would be made to the instructions in your article on this page to achieve this?
(I am hoping to build my first home server, using a Raspberry Pi 2 and a single 2TB USB hard drive.)
Best regards
I think my problems had more to do with the external drive in question being a RAID enclosure with desktop grade drives in it. I’ve had no problems using a regular hard drive as the system drive. Only the RAID box had issues.
When i use command swapon –s, get error swapon: –s: stat failed: No such file or directory. Do you know what i do wrong or miss something. Sorry for my bed english
It would have to be complaining about the swap file itself. Did you start from a fresh Raspbian installation, or are you trying to add on to an existing instance? If you are following my posts against an existing installation, I can’t guarantee that it was set up the same way as Raspbian to begin with. If you started from a fresh install, then it’s possible that something has changed in the current distribution of Raspbian, in which case I’d have to look into it before answering.
I had the same error on my second implementation of your tutorial, and the only difference is that I am using the “Jessie” distribution this time instead of “Wheezy.”
The solution is to use “sudo” prior to “swapon -s”. Apparently, “Jessie” needs elevated permission for the swapon command.
I’ll incorporate that into the post. Thanks.
Interestingly, if you look at the screenshot, I was running that through sudo already, probably out of habit. Now the text matches the screenshot. But I have to wonder whether sudo was always required and I just mis-typed the instructions, or whether this is truly new under Jessie. I’m curious, but not quite curious enough to re-run the whole thing under both releases to find out.
Sorry. I was mistaken. Go ahead and delete my last post (and this one). I won’t be offended. The following are some details of what happened. Maybe it will help you pinpoint the problem.
I’m not sure what happened, but I was getting the same error as nosko was getting, but when I tried “sudo” it worked. After a reboot, “swapon -s” without “sudo” worked again. However, after another reboot, I had numbers filling my screen and had to restart from the image I created just prior to this part of the tutorial. On a whim, after creating my root and data partitions on my USB HDD, I decided to check for errors with the “sudo e2fsck -f /dev/sda1” command, and sure enough, I had errors on around the 5th error check that e2fsck ran through, and numbers filled the screen again and continued to do so for a minute or so. I was given the chance to allow the errors to be fixed, which I allowed, and then numbers filled the screen again, and I was given another chance to fix these (other?) errors, which I allowed, and everything was fine.
I don’t know why this happened, but I wonder if the “mkpart” command needs a more precise number than 16GB when creating the second partition (perhaps in MB or KB), or if it would be safer to start the second partition at 17GB (shame to waste ~1GB, though). Another possibility is that my HDD is failing/corrupted, but I’m not sure how to check this via Raspbian/Linux.
I have just finished running through this part until the “Swap Configuration” section, and I can confirm that, now (after having fixed errors with “e2fsck”), “swapon -s” does indeed work without “sudo.”
I hope that helps.
I’ve really been enjoying your tutorial and have found it immensely useful. Thanks for all the hard work!
Oh, and just a reminder: the “sudo tune2fs -O ^unint_bg /dev/sda1” command still needs an “i” added to the “^unint” part.
I’m not really a Linux expert. I usually work in .Net development on Windows, so the bulk of my Linux experiment has been through the research and building of the Raspberry Pi Home Server. I can tell you what I’ve seen, though. I originally tried booting from my RAID box, but saw the same sort of errors you’re seeing, with numbers filling the screen. It would just do that sometimes. It would run for about a week at a time and then freak out. I finally decided that it just didn’t like booting from that particular RAID box, and I switched back to booting from the SD card, using the RAID just for the mass storage. I figured that maybe the RAID box introduced enough lag to the equation that it wasn’t fast enough to be used for the swap file or something. I ran with the whole system booting and swapping to the SD card for about a year without a problem. SD cards are cheap enough to be replaced when they wear out. Just take backups every now and then, and you should be fine.
Most recently, I added a small SSD to the server, and set it up to be my boot and swap drive. I tried to find a 32GB SSD like the ones they put in tablets, but found that it was cheaper to just grab an off-the-shelf 120GB drive and use an enclosure I already had around. Anyway, that has been running great for me.
You may be right about the size of the partition. I know there’s some issue with “aligning” your partitions, but don’t have much more detail than that. All I can say is that phrasing the partitioning in terms of “16GB” and “100%” has worked on 4 different drives I’ve personally tried without complaining. Sorry I don’t have a simple, sure-fire fix for the problem.
Hi, Think you for your instruction first.
I met a problem, I can’t find /boot/cmdline.txt file.
Then will can I find information of current boot device.
Off the top of my head, I would make sure you’re logging in as the “pi” users when configuring stuff like this. It could possibly be a permissions issue. I can’t imagine any other reason you wouldn’t be able to see the file. It HAS to be there, otherwise the whole thing wouldn’t boot. You are starting from the standard Raspbian image, right?
I use pi. I don’t think it is permission issue.
I alreay use ‘sudo find / -name “cmdline.txt”‘ to search the file. But nothing.
Maybe the framework of mine is too old. it is berryBoot.
Finally i find the file in its ‘FAT’ partition.
But the content is like below
‘smsc95xx.turbo_mode=N elevator=deadline quiet bootmenutimeout=10 datadev=mmcblk0p2’.
Maybe i need to start it from very beginning.
Thank you all the same
Yes, that’s probably it. The series is written against a plain, out-of-the-box Raspbian image. Berryboot is a multi-boot front-end, right? It will probably have moved each OS into it’s own partition, and taken over the job of the commandline file. Without knowing more about BerryBoot, I couldn’t tell you where it has moved the information from the Raspbian commandline, though.
I think I will update the entire series, and draw attention to the base Raspbian assumption at the top, next to the link to the Index. At least it should make readers aware that starting from a different distribution means that the instructions may not work exactly as written.
It is the best series i have seen for pi.
Pingback: The Raspberry Pi Home Server – Part 1 | UnaX.dk
Hi,
I am trying to install my old Lacie 250GB external USB but after step “Reboot (Important, do not skip this step)”, I cannot write /etc/fstab and and I am having the following message when I press completion/tab button:
-bash: cannot create temp file for here-document: Read-only file system
I have googled a bit but not found any interesting fix so I try to move the sector of the sda1 from 0-16GB to 32-48GB sector but same issue happen again.
Any idea would be very much appreciated.
Thanks
The first thing off the top of my head would be to make sure that you’re logging in as “pi”, and running most of the commands through “sudo”. Other than that, make sure the write tab on the SD card didn’t get flipped.
Hi,
Thanks but I did make sure I was running as sudo and when I rollbacked /boot/cmdline.txt, it was back to normal i.e no more “-bash:…” message (so no SD card flipped).
I tried with a WD HDD, but this time the dd command never stop and system is becoming unstable after this.
By any chance, is the SD card completely full? That can result in the “cannot create temp file” message.
Does not seem to:
pi@raspberrypi ~ $ df -h
Filesystem Size Used Avail Use% Mounted on
rootfs 15G 3.5G 11G 25% /
/dev/root 15G 3.5G 11G 25% /
devtmpfs 215M 0 215M 0% /dev
tmpfs 44M 396K 44M 1% /run
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 88M 0 88M 0% /run/shm
/dev/mmcblk0p1 56M 9.7M 47M 18% /boot
I’ve done a little basic searching, and it seems that this happens to quite a few people, and it’s always traced back to a corrupted file system. The reasons for that could be anything from not shutting down the Pi properly to using a cheap SD card. You’re at the point where you haven’t yet changed over to booting from the hard drive, right? That probably means that the SD card has gone wonky. I’d suggest rolling beck to the latest “happy” backup you have, and trying again from there.
Hi! I love your tutorials – it made setting up my Pi a snap! 🙂 I’m playing with a Pi 2, and I would really love to be able to backup the root filesystem from the hard drive. Did you ever post on this subject yet?
All the best! – Kevin
I have not covered that, partly because I’m not really running my system configured that way. It turned out that my server didn’t do every well with its root file system on the external drive, which is a RAID enclosure in my case. It will boot off a single drive just fine, and will even boot from the RAID, but it doesn’t like running from there continuously. I’ve reverted my own server to booting from the SD card, but storing everything else on the RAID. It may be my enclosure, or more likely something about the drives I put in it. I’ll look into it, though.
Thanks – that would be awesome! I’ve followed your steps and had an awesome setup since November, but last week, “something” died. I thought it was the SD card, but it seemed to run fine. Turned out it was the hard drive. I was able to save my content files, but I’ve needed to build pretty well from scratch. This time around, I’m using a newer Raspberry Pi 2, and I must say that it is quite peppy!
Any words of wisdom would be greatly appreciated, and again, thank you for this amazing tutorial series!
Every time I try and partition the drive It comes back with File System’ ntfs’, This happens even if I insert ext4 in the ‘mkpart primary ext4 0GB 16GB’ command. Any idea what I’m doing wrong?
The file system gets established by mkfs.ntfs in the next section, not by mkpart. mkpart just carves out the space where the file system will go.
Cheers. You wouldn’t believe how long I’ve been on that step!
Thanks for the help – I appreciate it.
Hi Mel. I followed your instructions and added an external USB drive to my Pi 2. I can see it listed in /dev as sda and made one large partition. I don’t want to boot from it, simply use it for data storage. I added the external drive to fstab and have it mounted in /mnt/data. I then created a Samba share to the same drive. What I am noticing now is when I copy files to /mnt/data it seems to fill up the SD card and not copy the files to the 1TB external drive.
What did I miss?
It may be that the drive isn’t actually mounted, and the files are going to the placeholder directory instead. This is part of why I used Webmin to set up the share in the article. It really takes a lot of the guesswork out of it. Make sure you’ve checked “Mount now”, and see if that helps. Clear out the folder first, or I’m not sure what will happen.
You are correct, it doesn’t seem to mount it when the server boots. I can manually mount it using “sudo mount /dev/sda1 /mnt/data” and when I then run “df -v” I can see my 1TB drive with 1% space used. The boot process doesn’t seem to do auto mount the drive. I tried running through the process again creating a new mount point named “share”, and updated fstab…unfortunately same result. I’m not sure why it won’t mount the USB drive at boot
Reply to my last post. I added the attribute rootdelay=10 to my cmdline.txt file and the 1TB drive auto mounts now. Someone mentioned some drives need this as the Pi2 boots faster than the original model and the USB ports aren’t ready.
I’m having problems getting the Pi to see the boot partition on sda3 (in my case).
The start up gets to the point where it says the Pi is waiting for the disk which is odd because when I try to start up from SD card everything works and the disk mounts with my 3 partitions. (I have rootdelay=10 and have tried moving it to 20)
I wonder if anyone has a problem with doing this on a Pi 2?
I’m going to try to build a server from scratch over the weekend using a Pi2, and see how that goes. I haven’t personally seen the Pi say it’s “waiting” for a drive before, so that’s a new one. I did have problems using a drive that kept entering power saving mode, though. The Pi didn’t like having to wait for its swap file to be ready. I am currently booting from the SD on my own server as a result. ________________________________
Okay, I got around to verifying things on a Pi2 this morning, and the disk mounting still works according to the directions. I did find one thing, though. I used a smaller, older external drive for this test, and during the file system check, it found more errors that just the counts I mentioned in the article. I switched to a different drive and everything went fine, but that first drive was never quite right in the head, so if you had additional errors, that might explain things. Also, make sure that all the drivers and partitions have unique uuids per the instructions in the article. ________________________________
Hi Mel,
I followed your instructions and it worked perfectly. You blog post is the one reference which explains all the required steps from start to finish. Kudos!
Reblogged this on electronics – my hobby since childhood and commented:
The perfectt guide to booting up the Raspberry Pi from a USB stick
Pingback: Booting up the RPi from a USB flash drive | electronics - my hobby since childhood
Hi Mel,
Great blog. The most thorough I have found so far. Perfect for a Noob like me hoping to learn fast.
I have a couple of questions about “Part 6: adding a HD”. I am keen to move the root over to my external HD
1. I will use ext4 for my whole drive as this works for me and I will use my backup/redundant/NAS drive in NTFS anyway, which will help with accessing data on my mac. But, I wonder if there is any advantage to setting 2 ext4 partitions – one for root and one for data? I imagine not (but this is all new to me) so I plan to create just 1 ext4 partition. What do you think?
2. In your guide above, after copying the root from the SD to the HD you then resize the root/boot image to fill the partition using this command “sudo resize2fs /dev/sda1”. I am unsure what this is actually doing. If I only have 1 partition (see Q1) and I resize the image to fill the available space, will this effectively reduce the available space I have on the HD for my data files?
3. Is there any advantage to making the swap size more than 2GB? This is the virtual memory, right? I have a new R-Pi 2 and I want to see how far I can push it – NAS/Owncloud/KODI/Retro-Pi all on one device if possible.
Many thanks for your help
ext4 works fine, and actually a bit more efficiently than NTFS on a Linux machine, but you’ll need to know a little bit more about Linux permissions than you would with NTFS. chmod-ing your shared folder to 777 will leave it wide open, which is the goal for a home server share. I recommend using Webmin’s Samba module to set up the share if you’re a Linux novice. The only advantage I see to making two partitions is that it makes it easier to nuke and remake the root partition if you want to make large changes later on.
resize2fs expands the file system to the full size of the partition it’s on. If you had a 2gb, and a 2tb partition, expanding the first one should only make it 2gb, and not take over the space in the other partition. If you only had one partition, and DIDN’T expand the file system, then a bunch ofspace on your hard drive would go to waste.
Like most things with computers “it depends”. I’m just using the old standby “twice your physical RAM” rule here, which would be 2gb for a 1gb Pi2. Experiment and see if you can detect a difference. A lot of how it performs will have to do with what the computer is busy doing at the time. I have left my original model B doing just “server” tasks, and added a Pi2 to my collection, which I’m using for experiments and Media/RetroPie. They’re cheap like that.
@Mel: I had a question regarding the resize2fs. If we do a ‘dd’ from a partition A to a partition B of a large size, where exactly in partition B is it stored that “xx” is the effective size of the partition (equal to that of partition A)? I couldn’t find any reference to this.
It’s like when you copy the Raspbian image to your SD card. The image is only 2-3GB, even though your SD card may be 16GB. Until you expand the filesystem, it will continue to appear to be only 2-3GB. Here, we’re copying one partition’s filesystem to another partition. Until you expand that partition, it still THINKS it’s whatever size it was on the SD card.
Oh I understand the theory. What I’m saying is: where is the apparent size kept? If the OS is not looking at the real size of the new partition by itself, it must be kept in some file right? I cannot find any reference to that. Basically it boils down to what file does the resize2fs modify, to make the change?
Without getting into some deeper Linux research, I really don’t have an answer to how our where it stores that information. This being Linux, though, it’s almost sure to be a plain text file somewhere, right? What is it you’re trying to do, exactly? ________________________________
Oh I was just curious to know that’s all. I am doing experiments on my raspberry pi and regularly transfer data between micro sd cards and a usb drive.
I figured out where this info is stored. It has something to do with the “block group descriptor table” of the ext4 FS. You can get this info by doing:
sudo dumpe2fs /dev/sd(xx)
resize2fs simply changes the ‘block count’.
Glad I found your blog. I will be adding a USB Drive to Pi2. Is it possible to use NTFS 4.1 ?
(Saw there was a Kickstarter funded product called Slice, with a nice enclosure, lit by 25 RGB LEDS, Pi Compute module and 1TB internal Drive selling for 189 British Pounds )
Sorry this comment slipped by me. I’m not sure exactly which versions of NTFS are supported by ntfs-3g, but a drive that was formatted on my Windows 8.1 computer worked just fine with it, so it seems to be completely up-to-date at this time.
Last weekend I bought a drive enclosures with just a USB 3 interface and populated it with a spare 120GB SSD and connected it to RPi 2, ls /dev found the drive, but parted could not read the partitions on the disk.
So I removed SSD and put a 160GB 7500 RPM Drive in the enclosure and connected it to Rpi, no luck – I could hear it clicking continuously.
When the same 160GB drive was connected to RPi2 using an externally powered APRICORN drive interface, parted had no problem finding the partitions.
It appears Rpi cannot provide enough power to even smaller capacity drives like the 120GB SSD, (It needs 1.0A at 5V, or the 160GB 7500 RPM spinner (800mA at 5V), 512GB SSD needs 1.7A at 5V. The plan to use a compact drive enclosure with just one USB cable for power and data was not successful even for the smaller drives.
The capacity of a drive seems to have less to do with its power requirements than its age does. I have a 160Gb drive that the Pi can’t power, and a 500Gb drive that it can. Remember also that the strength of your power supply has a lot to do with it. If you have a strong enough power supply, and a B+ or newer, you can set the max_usb_current value in boot/config.txt to increase the output. (see http://raspberrypi.stackexchange.com/questions/27708/is-setting-max-usb-current-1-to-give-more-power-to-usb-devices-a-bad-idea)
just a note: the highest version of the NTFS format is 3.1. (introduced in winXP). it’s driver, ntfs.sys may have a higher version, but that has nothing to do with the version of the format itself.
Thank you for the reply. I made a mistake in typing, what I wanted to ask was about NFS 4.1 and not NTFS4.1.
Inserting the line –> “max_usb_current=1” in the /boot/config.txt did help.
(BTW in RPi2 Model B — /boot/config.txt – does not have a line like this for controlling default USB current. It had to be inserted.)
After this line was inserted, the 160GB 7500RPM drive stopped clicking and started spinning. Parted was able to print all partitions. Thank you very much.
I plugged in a new WD Elements 2TB hard drive (this one: http://www.amazon.com/gp/product/B00DULWSXI/ref=pd_lpo_sbs_dp_ss_1?pf_rd_p=1944687542&pf_rd_s=lpo-top-stripe-1&pf_rd_t=201&pf_rd_i=B002QEBMCI&pf_rd_m=ATVPDKIKX0DER&pf_rd_r=0ABBPX7KACK1RK9RFD54)
The drive can’t be found in the parted area and the drive spins up for a few seconds, makes a click, and then spins up again. I don’t think it’s getting enough power. I bought a plug that is made for the raspberry pi. I think the drive isn’t getting enough power to spin up though, and that’s why it’s making the clicking noises and is not being recognized by the Pi 2. Any suggestions?
The USB ports on the Raspberry Pi don’t put out enough power to run a drive on their own, no matter how strong the power supply TO the Pi is. You need a drive with its own power supply, or a powered Usb hub. I recommend the hub, myself, but if the drive has a place for a power adapter, that works fine too.
Hi Mel,
actually that is incorrect. You are right that the default settings don’t allow a current of more that 600mA. But if you set “max_usb_current=1″ in the /boot/config.txt, the USB ports can output a current of upto 1.2A.
I did this and am able to easily power and run my WD 2TB my passport external 2.5” portable drive, whose power is directly thru the USB port on the Pi.
check this link for further info:
http://raspberrypi.stackexchange.com/questions/27708/is-setting-max-usb-current-1-to-give-more-power-to-usb-devices-a-bad-idea
Good point, but that setting only works on B+ and newer models. I guess if your pi is new enough, your power supply is strong enough, and your drive is low power enough, it would work. Perhaps I’ll add this to the post, but I think it would be a case by case thing. A powered hub or a self powered drive will work every time. ________________________________
On the “Change UUID” step you could also use “sudo tune2fs /dev/sda1 -U random” to avoid the use of an external service…
Nice. I’ll have to add that to the instructions next time I’m editing.
Thank you Mel for this wonderful article!
Hi Mel,
Thank you for an excellent series of tutorials. I have one question that’s left me scratching my head a bit. I’m asking out of interest, because the instructions clearly work as they should.
Under “Mounting filesystems” you say that “Linux now knows what partition to load the OS from [at this point, but] it’s still going to mount the root filesystem from the SD card.”
If this is the case, how does it know to access the version of fstab on the HDD in the steps that follow (if, indeed, that is the version of fstab we edit at this stage)?
Many thanks,
Gareth
I’ve never worked under the covers of Linux, so this is a vast oversimplification, but what you’re looking at is the difference between the kernel being allowed to see stuff, and YOU being allowed to see stuff. Cmdline.txt told the kernel where to find the root file system, and the kernel made some assumptions about where things had better be. As the boot process continues, it may re-evaluate some of those assumptions, but at first, those assumptions are set in stone. I suppose you might be able to create a second fstab in another place and Linux would switch to it after processing the first one which you’d better not mess with. Then again, it might end the universe as we know it. Great power, great responsibility, yada yada.
A discussion about this problem is posted here: http://ubuntuforums.org/showthread.php?t=1367460
It has been listed as a big since 2007, as mentioned here: https://bugs.launchpad.net/ubuntu/+source/partman-partitioning/+bug/148743 AND https://bugs.launchpad.net/ubuntu/+source/gparted/+bug/737387. It certainly goes against the uniqueness of “UUID”s, but that’s just the way it is. I think in actual operation, the behaviour of the OS is undefined, as to which one it chooses. Its possible that it is looking at the ‘fstab’ from the USB drive, but picking the first partition for “/” that matched the duplicated UUID in question, and then continuing with the rest of the ‘fstab’ on the USB drive.
I don’t think I would call it a problem, really, but it’s definitely something you need to be aware of. If it behaved any other way, things wouldn’t work. For instance, if I just wanted to upgrade a hard drive, I would clone the old drive to the new one. If the uuids didn’t come along for the ride, then the system wouldn’t be able to find the drive it needed when it started up, and would fail to boot. Cloning a drive means just that. The result should be identical to the original.
Hi Mel,
Thanks for this great blog.
I went through parts 1-5 without any problems. However, in this part something goes wrong.
I’m able to create the ext4 partition on my SSD drive and reboot the Pi with the modified cmdline.txt (« root=PARTUID=… ») I also could change the UUID of /dev/sda1
When I try to mount the SSD by changing the file /etc/fstab a got the error message »error writing /etc/fstab : read-only file system”. This seems to be the case for a lot of files (when i try to run startx I get ” mktemp: failed to create file via template `/tmp/serverauth.XXXXXXXXXX’: Read-only file system”)
I went through part 1-5 more than 5 times, so I’m pretty sure that I’ve made no mistakes.
I’m using a Pi 2 with a 8GB SD card, 60GB SSDnow V300 Kingston and a 3A USB power supply. (usb_max_current set to 1 in config.txt)
Any ideas what might be going wrong ?
Further down in the comments, you should find another question about read only file systems. I did some searching the last time someone mentioned this, and all the reports I found traced the problem back to the SD card getting corrupted at some point, usually by not shutting down the system properly. Do a search for “Linux recover read only file system”, and you should find plenty of posts on the topic. Most suggest running “sudo mount -o remount,re /”. YMMV.
Mel, I think that command should be: sudo mount -t ext4 -o rw,remount -force /dev/sd /
(rw not re)
and do a filesystem check before you re-mount in read-write mode. Temporarily, boot and use the root FS on the SD card to check the USB partition:
sudo fsck /dev/sd
Hi,
Thanks to Mel and electronicsguy for your reactions.
I bought an new SDcard, formatted it with SD formatter, always shut down the Pi correctly but the problem persists. I’m an absolute beginner, so I tried “dmesg | grep sda” :
(this time, I made 1 partition (sda1) and only used 1 USB device so I didn’t use the UUID in cmdline.txt)
….
console=ttyAMA0,115200 console=tty1 root=/dev/sda1 rootfstype=ext4 elevator=deadline rootwait
[ 2.538733] Waiting for root device /dev/sda1…
[ 7.712886] sd 0:0:0:0: [sda] 117231408 512-byte logical blocks: (60.0 GB/55.8 GiB)
[ 7.726448] sd 0:0:0:0: [sda] Write Protect is off
[ 7.736140] sd 0:0:0:0: [sda] Mode Sense: 47 00 10 08
[ 7.736573] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 7.758188] sda: sda1
[ 7.767973] sd 0:0:0:0: [sda] Attached SCSI disk
[ 7.832943] EXT4-fs (sda1): INFO: recovery required on readonly filesystem
[ 7.844944] EXT4-fs (sda1): write access will be enabled during recovery
…..
[ 7.936039] blk_update_request: critical target error, dev sda, sector 2623488
[ 7.948178] Buffer I/O error on dev sda1, logical block 327680, lost sync page write
[ 7.960934] JBD2: Error -5 detected when updating journal superblock for sda1-8.
[ 7.973561] EXT4-fs warning (device sda1): ext4_clear_journal_err:4658: Filesystem error recorded from previous mount: IO failure
[ 7.990429] EXT4-fs warning (device sda1): ext4_clear_journal_err:4659: Marking fs in need of filesystem check.
….
Could it be possible that the SD card has nothing to do with it and that the EXT4-fs(sda1) is read only ? see line [ 7.832943]
When do “sudo fsck /dev/sda” I got
fsck from util-linux 2.20.1
e2fsck 1.42.5 (29-Jul-2012)
ext2fs_open2: Bad magic number in super-block
fsck.ext2: Superblock invalid, trying backup blocks…
fsck.ext2: Bad magic number in super-block while trying to open /dev/sda
The superblock could not be read or does not describe a correct ext2
filesystem. If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
e2fsck -b 8193
If you’re up to the point that you’re booting from the external drive, then that could absolutely be it. You’d need to check the external drive for errors in that case. If you’re not, then I’m not sure what to tell you. Is the card the same brand as before? Check the raspberry pi verified hardware list (google knows where it is), and see if others have reported problems with that particular brand, maybe?
adding to Mel’s reply… It may indeed be a problem with the USB drive. I can think of 2 things – either the filesystem was incorrectly copied onto it. You could check this by mounting the USB drive on another linux computer and see how it mounts, if you can edit files there, etc.
A second problem could be, that the USB drive has become write-protected. Some flash drives, when they are failing to write correctly due to wear, enable a flag in the controller saying that it’ll be read-only now. This is unfortunately permanent, and is a mechanism for you to recover data from an about-to-fail drive. If that is so, there’s nothing you can do other than buying a new one.
A third possibility exists, which though is rare. Some flash drives come with a write protection switch. Check that this is not so in your case.
Hi,
Something very strange is going on here. I’ve replaced the Kingston SSD by a Toshiba HDD (MQ01 ABF032 250GB) and now I get this (dmesg | grep sda) :
I use a very cheap enclosure (Olmaster Transition2). Could this perhaps cause these problems ?)
15200 console=tty1 root=/dev/sda1 rootfstype=ext4 elevator=deadline rootwait roo tdelay=5
[ 4.783098] sd 0:0:0:0: [sda] 625142448 512-byte logical blocks: (320 GB/298 GiB)
[ 4.796037] sd 0:0:0:0: [sda] Write Protect is off
[ 4.805771] sd 0:0:0:0: [sda] Mode Sense: 47 00 10 08
[ 4.806287] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, supp orts DPO and FUA
[ 7.056747] sda: sda1
[ 7.066668] sd 0:0:0:0: [sda] Attached SCSI disk
[ 7.733563] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)
[ 12.916868] EXT4-fs (sda1): re-mounted. Opts: (null)
[ 14.363838] EXT4-fs (sda1): re-mounted. Opts: (null)
[ 20.222353] sd 0:0:0:0: [sda]
[ 20.222399] sd 0:0:0:0: [sda]
[ 20.222444] sd 0:0:0:0: [sda]
[ 20.222498] sd 0:0:0:0: [sda] CDB:
[ 20.222581] blk_update_request: critical target error, dev sda, sector 262362 4
[ 20.222609] blk_update_request: critical target error, dev sda, sector 262362 4
[ 20.222697] Aborting journal on device sda1-8.
[ 21.111239] EXT4-fs error (device sda1): ext4_journal_check_start:56: Detecte d aborted journal
[ 21.125461] EXT4-fs (sda1): Remounting filesystem read-only
Unless you’re starting from scratch each time, I’d say the problem isn’t the drive, it’s the contents of the SD card you’re copying from. The error exists on the filesystem before you do the copy, so it won’t matter what you copy it to, the damage will still be there.
If you ARE starting from scratch each time, then there’s something else going on here, and I’m not sure what to tell you. At some step in the process up to this point, the filesystem is getting corrupted. That’s usually from not ejecting it from your main computer before removing it, or from powering off the Pi without letting it shut down properly. Remember to wait for the lights to stop blinking before removing power from the Pi.
So far you’re the first to report this recently. If more reports like this come along, then perhaps something has changed. I’m about to walk through the entire process again myself in the next few weeks, so I’ll know if it’s a change in any of the software packages causing the problem. Keep an eye on the comments, and see if anyone else says something similar. And again, I recommend checking your SD card against the list of known good/bad brands: http://elinux.org/RPi_VerifiedPeripherals
Hi Mel,
I’ve replaced my old HDD enclosure (http://www.thepcharbor.com/shop/en/other-storage-products/2256-olmaster-rambler-transition-2-hdd-docking-enclosure.html ) by a newer one and now everything works fine (http://www.bluestork.eu/products/ehd-enclosures/item/429-external-hard-drive-enclosure-for-25q-hard-drives-sata?lang=en )
I use the Kingston V300 60 GB SSD (http://www.kingston.com/en/ssd/v#sv300s3 ) and an Integral Ultimapro-X SDHC 16GB Red memory card (not on the verified Peripherals list) (http://www.integralmemory.com/product/ultimapro-x-sdhc-95MB-read-60MB-write-class-10-uhs-i-u3-memory-card )
I didn’t change anything on the SSD. I reused the copy (dd if=/dev/mmcblk0p2…) made with the old HDD enclosure on the SSD and this copy even works fine with the new enclosure.
Best regards
I’ll add that to the list of things that can go wrong, I guess. Thanks.
Hi Mel,
great recipe, but I have one question: Why are we not supposed to update /boot/cmdline.txt with the generated uuid for the root partition on sda? Only /etc/fstab is supposed to be updated.
Best regards
I’m not sure if I have your whole question right, but there are two parts to booting from the drive. One is to identify the physical drive, and the other is to identify the partition containing the root filesystem. If you know a way to do this in fewer steps, please share, but this is what I found in my research. Linux is not my natural habitat, so I’m relying on research rather than years of experience for things like this.
Hi Mel,
when following your step-by-step guide, we end up with root=PARTUUID=5d18be51-3217-4679-9c72-a54e0fc53d6b in /boot/cmdline.txt and
/dev/disk/by-uuid/bb4231bb-9d5e-47dc-9b94-5dad0743affc in /etc/fstab.
/boot/cmdline.txt contains the original uuid of the SD card’s partition and /ect/fstab contains the generated one. Hmm, documentation says that /boot/cmdline.txt is for passing parameters to the kernel. I guess we should have bb423… in cmdline.txt as well. That’s my question.
Best regards
Marc-Philip
Hi Mel,
I mixed up filesystem uuid and partition uuid. Everything is fine with the recipe here.
One other remark: if you copy the root partition with dd while it is not mounted, the filesystem check does not report any errors. You can do this e.g. if you halt the raspi and insert the SD card in e.g. a laptop.
Thank you and best regards
Marc-Philip
Pingback: Setting Up Raspberry Pi 2 (Including Moving rootfs to USB Drive) | My Scratchpad
i want to leave my comment just to Say Thanks a lot for your help it would be very hard for me to make that work alone without the help of your instructions 🙂
I couldn’t have written this series without the work of all the people that went before me. I just saw a need to take some rather terse and technical articles and make them a bit more friendly to the reader.
Hi, I tried partioning the hard drive but came across errors after creating the root and data partion have tried a number of fixes but to no avail. Currently I get “Error: The backup GPT table is corrupt, but the primary appears OK, so that will be used” if I agree. My hard drive shows as being 1000GB and the partion reads
Partition Table: gpt
Number Start End Size File system Name Flags
1 1049kB 16.0GB 16.0GB ntfs primary
it was a 1TB hard drive.
Is there anything I can do? I also tried to re-format the HDD on main PC, but was unable to do this either
Formatting isn’t your problem. It sounds to me like the partition table itself is bad. You’ll need to create s new partition table, I guess. Back up whatever you can first, repeat the mklabel part, and if you’re formatting it as NTFS, maybe skip the -Q parameter. Let it give the drive a good going over just to be sure.
Hey Mel,
Firstly, thanks for writing these guides – very clear and informative.
I’m having an issue getting the boot files to resize to the partition size. I’m using a RPi2, with a 32GB SDHC, and am linking up to a 320GB ex-laptop hard drive. I’ve copied across the files to the partition sda1, and received the message:
949+1 records in
950+0 records out
31876710400 bytes (32 GB) copied, 1812.66 s, 17.6 MB/s
However, when attempting commands:
~ $ sudo e2fsck -f /dev/sda1
e2fsck 1.42.5 (29-Jul-2012)
/dev/sda1: recovering journal
e2fsck: Bad magic number in super-block while trying to re-open /dev/sda1
Signal (11) SIGSEGV si_code=SEGV MAPERR fault addr=0x60
~ $ sudo resize2fs /dev/sda1
resize2fs 1.42.5 (29-Jul-2012)
resize2fs: Bad magic number in super-block while trying to open /dev/sda1
Couldn’t find valid filesystem superblock.
I’ve checked a fair few fixes based on the error messages, but most are relating to partition formats other than ext4 File Systems. Do you have any suggestions?
Many thanks,
Andy
I have seen errors like that as well, but they seemed to go away once I started using percentages when partitioning the drive. The drive I’m using now for a demo machine was partitioned 0gb-16gb, 16gb-50%, 50%-100%. Apart from the first partition, everything else was done in percentages, and parted seems to have figured out the best way to begin and end on an even boundary. Unless you were partitioning this drive into multiple pieces, and tried to tightly control the beginning and ending of each partition, though, I’m going to guess that’s not what ‘s going on here. This IS the partition you copied the SD card into, right?
I should have said this before. Oops. Have you checked the SD card for errors? If there were unresolved errors on the SD card, then they get copied over along with everything else. Make sure your SD card is all clean before making the move, and it may help.
Hey Mel, Thanks for the awesome series. I am trying to learn Linux and the raspberry pi has been a really cool introductory system for that. I’m currently working on this step (6) and I keep running into some issue. I have a small laptop hard drive enclosure hooked up to the Pi with one USB cable and there’s a second USB cable coming out of the hard drive directly to power.
The problem is that whenever I get to the “sudo e2fsck -f /dev/sda1” step to check for errors after copying data to the drive, no matter how many times I run that command I keep getting tons and tons of inode count and free block messages. The messages are like the following:
Free inodes count wrong for group #3 (0, counted=960).
Fix?
Free blocks count wrong for group #33 (1965, counted=4719).
Fix?
Unattached inode 278485
Connect to /lost+found?
So I have tried several times to go on to setting the boot configuration, but I have never been able to boot from the hard drive. I see kernel panic messages when I look at the screen hooked up to the Pi.
Could this be the hard drive being bad? (I didn’t think this drive had gone bad)
Has anyone encountered something like this and might have something to help me out and point me in the right direction? I would really appreciate any help.
A few people have reported such problems. It could be that the SD card was corrupt before copying it. Try scanning the card itself and see what you find.
Hi Mel
First of all, Congrats for a magnificent article. This article made me decide to buy a Pi2 and have a go at it. I’m following your article and up to the copy from the SDcard to the hdd is done, but on this particular step, although you mentioned that an error or two was to be expected, I’m getting 50+ errors about inodes and file/cluster sizes being incorrect. I “repair” them all and proceed, but then on first reboot to boot from the hdd it fails and I get nothing. what can be wrong?
The problem with me answering is that there are so many things that can go wrong, and they’re not really Pi-specific. If you look through the comments, you’ll see this same sort of error from others. Sometimes the solution is simple, like making sure the SD card didn’t have errors on it before cloning it. Turning the power off without shutting down the Pi properly can cause this, and it’s happened to plenty of people. Other than that, it’s time to hit StackExchange and start looking up errors. The weird thing is that as many times as I’ve done this (many times while writing this series), I’ve never had it happen to me, so I haven’t really had the chance to look into it first-hand.
Mel…. WOW! You are the MAN!
Thank you so much for your efforts and as well the ones who came before you! You should think about putting this up on Udemy or creating a video series as well.. maybe the reach would be greater and your efforts could benefit more people?
My issue is about RAID 1 and configuring it with this setup. I’ve been looking around for a while and knew this could be done with the Pi and I wasn’t going to spend a bunch of cash on a server that I would have less control over, thanks again for matching me up with exactly what I had in mind btw!
I got a RAID box from ICYbox ( http://www.amazon.co.uk/dp/B000L105KS/ref=sr_1_1?ie=UTF8&qid=1437670538&sr=8-1&keywords=RD3219StU3-B- ) and a coulple 3TB WD Greens (keeping low power in mind…)
Although I work in UNIX all day long I am no expert 😦
Anyway, the problem is that both HDD’s show up even though I have the switches on the box switched to RAID 1 but I see both sda and sdb as the HDDs, and I am just wondering how to get it setup like you with the RAID 1 setup…I’d like to get the transferring of all out pics and vids and stuff soon but Don’t want to start transferring them to sda when at some point I may need to format everyting again to get the RAID 1 working properly.
Thank you again for all your doing to make the world a better place!!!
And thank you in advance for your response!
Jason
Check that the RAID box can handle 3TB drives. Mine tops out at 2TB, but when I put larger ones in, it just recognized the first 2TB. Maybe yours reacts differently. It’s definitely not mirroring if you can see both drives, that’s for sure. As for setting up the box, that might be more of a question for the manufacturer. One thing I will tell you is that my own Pi didn’t seem to like booting from it. It would use it just fine, but when using it as a system drive, it would run for a week or so and then freak out. I’ve been running off the SD card ever since, although that’s about to change.
So the issue was that there were no instructions included (in my box at least..) about how to get the box to function as a RAID array.. had to hold down buttons on the box while plugging it into a machine and praying on one leg while rubbing my belly and patting my head at the same time.. but it worked!
So now I have one drive (hopefully its config’d correctly as RAID 1.. do you know any way to check?) and am starting to copy stuff over. My question for you is do you even have the swap file on your HDD or do you still have that on the SD card? Any other tips as I’ll be running an identical setup to yours at this point..
Thx again!
I think that checking would probably vary by manufacturer, but if you’ve put two drives in it, and it only looks like one, then there should only be two possibilities.
1) It looks like one drive that’s twice as big, in which case it’s not acting RAID-ey
2) It looks like one drive that’s the right size, in which case it… probably is working.
The only way I know to tell for sure would be to take the drives back out, mount them individually, and see if they both contain the same stuff.
As for the swap file. I think that’s probably what didn’t like running on the RAID box. I’ve been running with the SD card as my system drive ever since discovering that it didn’t like using the RAID that way. Maybe a different brand would act differently. I just know I haven’t worn out the card yet in well over a year of constant uptime, and everything seems fine.
I’m planning to build up a new server, and this time I’m going to attach an older laptop drive to be the system drive, and just let the RAID box be storage. I’m also considering buying a small SSD, like the kind they use in netbooks, and giving that a shot.
Thanks for your reply Mel.
So is your RAID box running on its own as far as configuring the array and managing everything, or are you using something like mdadm to do it?
I’ve just came across an article that has a pretty decent outline on how to accomplish this, but I’m a n00b to using this on my own so any input is appreciated…
Thanks again!
The RAID box I’m using handles it all internally. To the Pi, it just looks like a hard drive like any other. It’s completely unaware that it’s a RAID. I did have some problems booting from it, though. It would run for a week or two and then crash. I figured maybe there was some inherent lag there that the Pi just couldn’t tolerate, maybe. I’ve been running off of the SD card ever since then, but I know the swap file is probably tearing it up. On my next rebuild (soon), I’m planning to attach a small leftover hard drive (like 60GB), and let that by the root drive. Then the external RAID will be nothing but data, which it seems to be handling just fine. I’m looking for a cheap, small SSD, like the kind they put in netbooks, and see where that gets me.
well.. here’s the link: http://blog.drewwithers.com/2013/11/raspberry-pi-usb-raid1-root-partition.html
So.. after further digging (on page XX of Google Search…) I’ve found some “instructions” on how to get my box into the correct RAID 1 config: http://icyboxforum.de/viewtopic.php?f=60&t=14407&sid=d0a9abf19720c7877a1f8a6dba3d99b4
I’m pretty sceptical though as it says to hold the rest button, while plugging it in to a pc and so on..
I guess I don’t have a choice but to give it a shot at this point.. but most of my work will be gone now that I’ve gotten to pg 10 of your amazing guide.
Pingback: Raspberry Pi Home Server: Part 5, Remote Desktop | MelGrubb.ToBlog()
Mel,
Thank you for this comprehensive guide and all your work (and those before) helping Linux NOOBS and Pi tinkerers. This guide has allowed me to create a CrashPi after an unfortunate HDD crash that almost wiped out a 14 year collection of pictures of my kids, not to mention countless documents, tax returns, etc.
The issue I’m trying to solve is accessing the GPT NTFS partition created at /mnt/data on a 32-bit Windows XP (I know) desktop PC to perform the initial load (90+ GB) of pics, videos, etc to the share before exposing it via Samba. Have you had any experience in this area?
Thanks again for your sharing of knowledge.
-M
If it’s NTFS, you should just be able to plug it in, and Windows should just see it. What issues are you having?
Mel,
The issue is that Windows does not assign a drive nor does it give the option of assigning one in the Computer Management console. It is recognized as a drive in computer management but with no specified format. I would have at least expected to see NTFS as the format.
Thanks again.
-M
Well, if you haven’t actually put anything on the drive yet, I’d say your best bet is to let Windows format that partition rather than doing it from the Pi. You should have something you KNOW Windows likes then. The Pi should still read and write to it just fine.
Nothing on the drive yet. Sound advice.
Thank you.
-M
Hi, Thanks for a very good and useful guide.
I have run in to problems though. I can’t install ntfs-config or ntfsprogs or gdisk since they all throw the same error (se below) and I have no idea how to go about solving this. I am on a Gen 1 model B Raspi and if you could just point me in the right direction I’d be very thankful!
Setting up libblas3 (1.2.20110419-5) …
update-alternatives: error: /var/lib/dpkg/alternatives/libblas.so.3 corrupt: line not terminated while trying to read status
dpkg: error processing libblas3 (–configure):
subprocess installed post-installation script returned error exit status 2
.
.
.
dpkg: dependency problems prevent configuration of liblapack3:
liblapack3 depends on libblas3 | libblas.so.3 | libatlas3-base; however:
Package libblas3 is not configured yet.
Package libblas.so.3 is not installed.
Package libblas3 which provides libblas.so.3 is not configured yet.
Package libatlas3-base is not installed.
.
.
.
Errors were encountered while processing:
libblas3
liblapack3
python-numpy
python-gtk2
python-glade2
ntfs-config
E: Sub-process /usr/bin/dpkg returned an error code (1)
.. install failed!
That’s certainly not an error I’ve seen before. I have had the occasional problem with repositories being temporarily down, but this sounds like something wrong in the original repository configuration. I just ran through this whole series in the last month while recording the Pluralsight screencast, and didn’t have any errors, but that was on a 2B. I do have an older B, and a B+. I could try reproducing the situation, but I’m not sure how long before I can make that happen. In the meantime, try feeding the most distinct part of the error message through Google, and see if anything comes up. Sometimes I have found Pi forum posts from someone else who has seen a particular error before.
This morning, I started from a fresh Raspbian image and a Model B (not B+, not 2B, just “B”), installed nothing but the current updates, and then ran “sudo apt-get install ntfs-3g ntfs-config ntfsprogs gdisk”. I did not see the error you mentioned, so I don’t think it’s some version incompatibility problem. Are you sure you listed all four packages? Apt is supposed to take care of dependencies, but I suppose it’s possible that one package might not be directly dependent on another, but still won’t install right without it being there already.
Hi Mel!
Is there an easy process for adding a second powered USB Hard Drive? Is it possible to add a second drive that just “extends” the space on the partition created in this tutorial? If so, can it be done without destroying the existing data?
There’s no simple way to just add more space, but since you can mount the new drive anywhere you want, you can make the second drive appear to be a folder within the first drive. For instance, if I had my first drive mounted at /mnt/data, I could mount the second one at /mnt/data/videos. It would then appear to be part of the same directory structure, but it won’t increase the space available to the other folders inside /mnt/data. I’m sure there are ways to treat both drives as one big space, like a software-based RAID array, but I haven’t looked into them enough to make a recommendation.
Well done Mel. Thank you. I just found your video on pluralsight and have been following it. I have an rpi2 and those instructions are not working on raspbian. parted is giving me fits about partitions being busy and not creating partitions with ext4. Sometimes with f16 and other times nothing at all so obviously I can’t mount my 984GB partition without a file system. Any thoughts? I’ve done this plenty on CentOS, Ubuntu, Redhat and other linux/unix distros so I’m good there, but I’ve never run into this problem before.
This is not something I’ve seen myself. I’ve formatted the little 160gb drive I used during the recording as well as 2-3TB drives, and it worked on all of them. I’d ask if the drive was using a GPT, but for a 1TB drive, that wouldn’t even matter. What do you mean by “f16”? I’m not familiar with the acronym, unfortunately.
Yes I was using GPT as well. F16 for fat16 or DOS. I decided to use gparted instead and that fixed my formatting issues. Then I decided to format it as NTFS and got the libraries for that to work. Another question on this topic; I finished of the Samba portion of your video but for some reason I can only access about 2GB on the drive and then it says its full. Everything is set in fstab and shares in Samba using the Webmin interface. Any thoughts on this one? I changed drives thinking I had some issues with the one I was using but that was not the case. I change the uuid info in fstab and it seems to mount but 2GB is not enough for anything.
My first guess would be that the drive didn’t mount properly, and you’re filling up the placeholder directory on your SD, but you say you already checked that. I’d double-check that the drive is truly mounted where you expect. Try writing a file to the drive, then shut down the Pi and hook it up to your regular computer and see if the files are there. If something’s funny with the way the drive got formatted, you could always try formatting the NTFS partition from your regular computer. That’s how my current external drive was done. My current personal server is running with an SSD for the system drive, and the 2TB RAID box as an NTFS data store. The RAID was formatted from my main laptop. It might make a difference. In my case it just happened to already have stuff on it that I didn’t want to lose. I have run through the posts beginning to end with a smaller drive several times though, especially while doing the Pluralsight recording.
Thanks Mel. I got it figured out last night. I had some residual garbage that needed cleaning up and then I just started over and got it working except for the uuid mounting part. For some reason I’m having issues getting fstab to mount my drive. I mount it just fine from the command line. Double, triple, quadruple checked the ID value and all is good there. I’ll figure it out eventually. Thank you for your thoughts.
Thanks Mel for all the info! Everything has gone swimmingly up until the last part of this chapter. When I try to save the /etc/dphys-swapfile after editing it tells me it can’t because it’s a read-only filesystem. This is a Pi 2 with the latest Raspbian image and following your instructions to the letter since chapter 1. How do I get it to save?
Fixed it. I had an extra / in fstab that caused the entire root fs to go RO. Edited and it’s working now. Thanks again for all the great info!!
Great article – worked a treat. I am using a 2 TB drive. Incresed size from 16 GB to 1 TB – to accommodate a larger /var/www for the web.
Looking forward to your article on back up and restore.
Currently using this but not sure how to recover to SD card, or to ensure backup from the HD Extension is also backed up and restorable. any help gratefully accepted.
DJB
#!/bin/bash
# Setting up backup directories
SUBDIR=Backups
DIR=/mnt/data/$SUBDIR
# Setting up echo fonts
red=’\e[0;31m’
green=’\e[0;32m’
cyan=’\e[0;36m’
yellow=’\e[1;33m’
purple=’\e[0;35m’
NC=’\e[0m’ #No Color
bold=`tput bold`
normal=`tput sgr0`
#Screen clear
clear
echo -e “${green}${bold}Starting RaspberryPI backup process!${NC}${normal}”
echo “”
# First check if pv package is installed, if not, install it first
PACKAGESTATUS=`dpkg -s pv | grep Status`;
if [[ $PACKAGESTATUS == S* ]]
then
echo -e “${cyan}${bold}Package ‘pv’ is installed${NC}${normal}”
echo “”
else
echo -e “${yellow}${bold}Package ‘pv’ is NOT installed${NC}${normal}”
echo -e “${yellow}${bold}Installing package ‘pv’ + ‘pv dialog’. Please wait…${NC}${normal}”
echo “”
sudo apt-get -y install pv && sudo apt-get -y install pv dialog
fi
# Check if backup directory exists
if [ ! -d “$DIR” ];
then
echo -e “${yellow}${bold}Backup directory $DIR doesn’t exist, creating it now!${NC}${normal}”
sudo mkdir $DIR
fi
# Create a filename with datestamp for our current backup (without .img suffix)
OFILE=”$DIR/backup_$(date +%Y%m%d_%H%M%S)”
# Create final filename, with suffix
OFILEFINAL=$OFILE.img
# First sync disks
sync; sync
# Shut down some services before starting backup process
echo “”
echo -e “${purple}${bold}Stopping services before backup${NC}${normal}”
sudo pkill deluged
sudo pkill deluge-web
sudo service deluge-daemon stop
sudo service noip stop
sudo service cron stop
sudo service proftpd stop
sudo service webmin stop
sudo service xrdp stop
sudo service sendmail stop
sudo service ssh stop
sudo ervice btsync stop
sudo service apache2 stop
sudo service samba stop
sudo service motion stop
sudo service ddclient stop
sudo service avahi-daemon stop
sudo service netatalk stop
# Begin the backup process, should take about 45 minutes hour from 8Gb SD card to HDD
echo “”
echo -e “${green}${bold}Backing up SD card to img file on HDD${NC}${normal}”
SDSIZE=`sudo blockdev –getsize64 /dev/mmcblk0`;
sudo pv -tpreb /dev/mmcblk0 -s $SDSIZE | dd of=$OFILE bs=1M conv=sync,noerror iflag=fullblock
# Wait for DD to finish and catch result
RESULT=$?
# Start services again that where shutdown before backup process
echo “”
echo -e “${purple}${bold}Starting the stopped services${NC}${normal}”
sudo service deluge-daemon start
sudo deluged
sudo deluge-web
sudo service noip start
sudo service cron start
sudo service proftpd start
sudo service webmin start
sudo service xrdp start
sudo service sendmail start
sudo service ssh start
sudo ervice btsync start
sudo service apache2 start
sudo service samba start
sudo service motion start
sudo service ddclient start
sudo service avahi-daemon start
sudo service netatalk start
# If command has completed successfully, if not, delete created files
if [ $RESULT = 0 ];
then
sudo mv $OFILE $OFILEFINAL
echo “”
echo -e “${green}${bold}RaspberryPI backup process completed! FILE: $OFILEFINAL${NC}${normal}”
echo -e “${yellow}Removing backups older than 5 days${NC}”
sudo find $DIR -maxdepth 1 -name “*.img” -mtime +5 -exec rm {} \;
echo -e “${cyan}If any backups older than 5 days were found, they were deleted${NC}”
exit 0
# Else remove attempted backup file
else
echo “”
echo -e “${red}${bold}Backup failed!${NC}${normal}”
sudo rm -f $OFILE
echo “”
echo -e “${purple}Last backups on HDD:${NC}”
sudo find $DIR -maxdepth 1 -name “*.img” -exec ls {} \;
echo “”
echo -e “${red}${bold}RaspberryPI backup process failed!${NC}${normal}”
exit 1
fi
Restoring the SD card is just burning the .img file back to it, just like the first time you created the card. I have not tried to script out a backup like this. I just shut down the system and use Win32DiskImager in “read” mode to back up my card. I have CrashPlan set up to back up the contents of the data partition, but I’ve always figured if anything went horrible wrong I’d probably rebuild the server anyway. I’ve got it down to about 5-6 hours for a rebuild now, and I could probably script that out to make it even shorter, but I take it as an opportunity to check that the instructions in this series are still up to date. Sometimes packages get updated and I have to adjust my posts to compensate.
I would argue that the benefits of using a proper Linux filesystem such as ext4 or btrfs outweigh the ability to plug the drive into a desktop to transfer files to it instead of pushing them through the network. Probably you’d only do that once, when you first set up the server, and then you’d be dealing with the downsides of NTFS (which are not inconsequential, especially for a file server – it is a rather antiquated filesystem after all) for as long as you’re using the server.
I agree. I have used NTFS on my own server as a matter of convenience. If something happens to my main computer, and I want to restore from backup, I might want to just swipe the drive temporarily hook it up locally, and “adopt” the backup set temporarily to get things done faster. I don’t have any hard number comparisons on the relative performance of ext4 vs NTFS, so convenience became my tie-breaker.
Pingback: Raspberry Pi web server | Stefan Adrian Bukos
I recently got the new Raspbian Jessie image and wrote it to my sd card. I am now at your step part 6 adding a hard drive. It all goes well until I get to the tune2fs to change the UUID. I am getting this error:
pi@MyRPi ~ $ sudo tune2fs -U random /dev/sda1
tune2fs 1.42.12 (29-Aug-2014)
The UUID may only be changed when the filesystem is unmounted.
I have searched and can’t find a way to change the UUID.
As a side note, I have followed your steps previously (wheezy) and it worked just fine.
Any ideas?
I have not run the while series under Jessie yet, personally, but I expect that this would be a widespread error affecting everybody, so it shouldn’t be long before it’s a well known issue with a well known solution. In the meantime, can you manually dismount the drive first using umount? Also, telling it to boot to the command line instead of the desktop might stop the auto mount behavior from happening on the first place.
I found this,
http://askubuntu.com/questions/132079/how-do-i-change-uuid-of-a-disk-to-whatever-i-want
modified it a bit for my needs and it worked!
Hi Mel.
I’ve followed your Series 1 to 6 and am at this point. It’s been great and easy to follow with only a couple of tiny hiccups.
I’ve installed using Wheezy, and am hesitating to go further until I can clarify a couple of things.
My main concern is backup and imaging from here on. Up to and including your article 5, I happily went back to my main computer and made an image of the SD card
Now, after setting up the file system to be accessed from the USB HDD in this article 6, here are the questions:
1. If from now on I app-get any application, and/or update and upgrade, does ANYTHING now get written to the SD card, or does everything now go to the HDD
2. Depending upon the answer to the above, what process(es) should I run to backup and/or make an image?
It would be a nightmare to go forward installing lots of new apps if I cannot maintain concurrent backups and images. As a noob, I am just a bit confused as to whether from now on I have to
– back up both the card AND the harddisk-based filesytem, or
– back up the hard disk alone or
– even combine the SD and the HDD into a new SD card image?
Any info and advice welcome
Thanks
After moving to the hard drive, everything gets written there, but you don’t HAVE to move to the hard drive at all. You can just mount the drive and store data there. In my Pluralsight course, I moved booting from the hard drive to the very end. I ran my own server for over a year from the SD card, and never used it up. I WOULD recommend moving the swap file to the hard drive, though. That should extend the life a bit.
OK. Thanks for that. Previously I had tried a usb harddisk set up following another guide elsewhere, and my SD card started to go flaky after about 3 months. Perhaps it was the swapfile. At that time I knew even less than I do now.
So, I’m quite happy to work with this system for at least the next few months to see how it goes, and using your series, the swapfile is obviously already on the harddisk now.
I am having trouble in this stage.
Got a Toshiba 1T USB drive – although I increased the USB power (PI2) it did not manage to operate it, so now the disk is connected using a powered USB hub (note: for some reason it is not able to drive both the disk and the PI, It makes the PI reboot again and again – maybe due to disk power draw during spin-up so PI still gets powered from its own supply)
Got to the stage of:
sudo dd if=/dev/mmcblk0p2 of=/dev/sda1 bs=32M conv=noerror,sync
This process hangs at a certain stage – I see the PI green led blinking and the drive light blinking, but then at a certain stage the PI light stays ON and after a while the disk light goes OFF (actually it sounds like it stops spinning altogether). Tried this twice and happened both times.
Any idea what could be wrong?
If I decide to abandon moving the root partition to the disk, can I use it only for data (leaving the sda1 partition) or should I re partition to one data partition?
In the described procedure do I simply skip any command issued for sda1 and refer only to thse for sda2?
Adding to the above – it seems that the PI stopped responding – I have it also connected to a display and it does not respond from that console (and obviously also does not respond in SSH)
Some drives just draw more power than others. I have some that the Pi can power, and some that it can’t. The big RAID box hooked to my own server has its own power supply, but there Pi doesn’t like to bit from it, so I have run my system primarily from the SD card for years without a problem. I would recommend moving the swap file to the drive, though.
As I wrote, after connecting through a powered hub it works.
Nevertheless, I guess for the time being I will skip booting from it – can I leave the partitions as already made by this session procedure (and maybe get to solving the boot issue sometime later)? Do I simply skip anything that deals with sda1 and deals only with sda2?
You could leave them, or if you haven’t done a lot with it yet you could reclaim the space by repartitioning the drive. If you keep both, then you would adjust anything from sda1 to sda2, but going by the instructions I posted, you only ever use those names when using parted itself. fstab will refer to the partition by UUID, and once mounted, no-one will ever use the name “sda1” or “sda2” again. The drive simply becomes /mnt/data (or whatever you decide to call it).
OK, tried another disk – same issue so gave up booting from disk. only one data partition.
What about swap configuration – do I leave it untouched or can I increase it as described (I use a 16GB SD card)
The swap file is the thing that “uses up” SD cards, supposedly, so I would move it to the hard drive (if you use an ext4 file system, that is). You don’t HAVE to, and I never wore mine out in all the time I was using it off the card, but that may have to do with the brand more than anything.
Thanks.
I enjoy this series very much – very detailed and great support to questions
Hi Mel, to format the data partition with ntfs you specify:
sudo mkfs.ntfs -Q -L Data /dev/sda2
Please can you suggest how to format the data partition with ext4 instead?
The command would be “sudo mkfs.ext4 /dev/sda2”. Adjust the “sda2” part as needed if you’re partitions aren’t the same as mine.
Thanks
Pingback: CrashPi – An off-Site backup for the whole family | MelGrubb.ToBlog()
Hi Mel,
Love the series. Discovered it looking for a way to make a web server and your guide is the most complete one out there. I have gotten this far and want to get a hard disk on the Pi. Made it to point where I need to change the UUID on the hard disk “sda1″. I get a unique identifier, insert it in the command line and when I try to run the line ” sudo tune2fs /dev/sda1 -U UUID-GOES-HERE” get an error message that says “the UUID may only be changed when the file system is unmounted”. How do I unmount sda1, give it a new UUID and remount sda1?.
Make sure you use raspi-config to set the system to bit to the command line. This is in the updates at the top of the article. The newer “Jessie” release of Raspbian bits to the desktop and auto-mounts any USB drives. Booting to the command line should prevent this auto-mounting.
Thanks for the quick reply. I changed the raspi-config so I now mount to the command line with required log in. This gets me to the command line log in on my TFT screen mounted on the pi. I then log in via SSH start working and the Pi says I still can not change the UUID. Thoughts? A confused old engineer from the age of punch cards and vacuum tubes.
Hmmm. Well I’m on my phone at the moment so I can’t look up the exact syntax right now, but something like “sudo umount /dev/sda#”, where “#” is there device you’re trying to unmount should do it. I’ll have to run through that part again, but rebooting after setting it to boot to the cli should have done the trick.
Gave that a try with the following results:
pi@RPWSJHE ~ $ sudo blkid
/dev/mmcblk0p1: LABEL=”PIOPSYS” UUID=”4C36-B52F” TYPE=”vfat” PARTUUID=”000214af-01″
/dev/mmcblk0p3: LABEL=”SETTINGS” UUID=”d917886f-d8eb-4d7b-8163-02d26e2573f7″ TYPE=”ext4″ PARTUUID=”000214af-03″
/dev/mmcblk0p5: SEC_TYPE=”msdos” LABEL=”boot” UUID=”0AD3-BA87″ TYPE=”vfat” PARTUUID=”000214af-05″
/dev/mmcblk0p6: LABEL=”root” UUID=”581ba799-40f0-4b34-8f3a-7da6352dbcd7″ TYPE=”ext4″ PARTUUID=”000214af-06″
/dev/sda1: LABEL=”root” UUID=”581ba799-40f0-4b34-8f3a-7da6352dbcd7″ TYPE=”ext4″ PARTLABEL=”primary” PARTUUID=”2d6e287b-0138-4e48-a937-9f72bd01a969″
/dev/sda2: LABEL=”Data” UUID=”7C55FA503A49EE47″ TYPE=”ntfs” PARTLABEL=”primary” PARTUUID=”496dfc08-7428-483f-9035-a9fca1b4b41a”
/dev/mmcblk0: PTUUID=”000214af” PTTYPE=”dos”
pi@RPWSJHE ~ $ sudo tune2fs /dev/sda1 -U db5bfd25-d8bb-47b2-af43-633a06daf650
tune2fs 1.42.12 (29-Aug-2014)
The UUID may only be changed when the filesystem is unmounted.
pi@RPWSJHE ~ $ clear
pi@RPWSJHE ~ $ sudo umount /dev/sda1
umount: /dev/sda1: not mounted
pi@RPWSJHE ~ $ sudo tune2fs /dev/sda1 -U db5bfd25-d8bb-47b2-af43-633a06daf650
tune2fs 1.42.12 (29-Aug-2014)
The UUID may only be changed when the filesystem is unmounted.
pi@RPWSJHE ~ $
Is there any other was to change the hard disk “sda1” partion UUID?
I’ll have to look into this more closely. I’m pretty sure this worked for me last time I tried it, but perhaps something else has changed since then. I may give this another shot this weekend and see what I can find.
Any updates to the whole changing the UUID on the mounted filesystem? Defintely not working on the latest tune2fs. Giving “The UUID may only be changed when the filesystem is unmounted.”.
Okay, I won’t pretend to be an expert here, but I was able to find someone smarter than me who knew how to get around this. I found a post at http://www.oarie.net/index.php/features that seems to clear it up. What you need to do is issue the command “sudo tune2fs -O ^uninit_bg /dev/sda1” first, which will unstick some flag that’s making the drive appear mounted when it isn’t. Then you can assign a new guid. By the way, you don’t have to go make one up yourself. You can just say “sudo tune2fs /dev/sda1 -U random” and it’ll make one up itself.
Hi Mel. Just letting you know there’s a typo in your post. You’ve got ^unint_bg instead of ^uninit_bg. Great work on the Pluralsight videos!
Ugh, thanks. I’ll correct it. Good catch.
You are the man! That work around for the tune2fs UUID did the trick. I can now boot from the hard drive. Yea! I really appreciate you tutorials with the “why” as well as the ‘how”. It really helps me in the learning process.
Hi Mel,
I finally managed to complete part 6, BUT, something starnge has happened.
I can’t login! I’m not even prompt to!
During boot the system performs a fsck to sda1, and then after starts a “job for dev-disk-by\x2duuid…”
when it finalizes booting, I’m left with what seems to be a “guess” account, and some warning about “sulogin: root account is locked, starting shell” and my promp looks like “root@tugadroid:~#”
I am able to run normal and sudo commands, but the swapon command does nothing.
Is this something I should worry about, or can I continue with the install?
I just tried to follow step 7, the Samba install, and got a series of EXT4-fs errors.
It seems something went wrong at some point when changing the boot to the hdd.
Is this still fixable or am I better of restarting from ground zero?
I wouldn’t move forward with system errors outstanding. There’s telling what could go wrong. Try to clear up the errors before continuing, or put off booting from the hard drive until later. It’s not even necessary, it’s just a little faster. From the comments I have received, it seems like not all hard drives like to play nice, either.
Well, you generally don’t want to be doing things AS root. It’s dangerous to stay in “God Mode” when it’s not
truly needed. What happens when you “logout”? Can you log back in as Pi then?
I did have a hard time finding a HDD that my PI would work with, even the powered hub that I picked up seems to be mocking me… I may go the raid enclosure option, if I find one cheap enough
If you do, I wouldn’t recommend trying to boot from it or use it for swap space. At least with the RAID box I tried, the Pi didn’t seem to like booting from it. I suspect there’s a little too much lag there to be usable for the swap space. What I’ve done most recently was to hook up a small SSD as my boot and swap drive, and left the RAID as storage space only. The system has been very stable this way. Before the SSD, I booted and swapped to the SD card for over a year and never wore it out. It also makes it far more convenient to make backups of the entire configuration. Booting from the hard drive is purely optional. It does speed the system up, but getting a good quality SD card will do that too. The cheap generic cards do not perform nearly as well. I’ve seen more than a few articles that swear by SanDisk Extreme cards, but haven’t tried one myself.
any suggestions on how to try to FIX without starting all over?
“logout” is not accepted as command, and exit just leaves me with an un-operational pi
I haven’t seen this behavior myself. Even the auto-login options in raspi-config log you in as Pi normally. Did you do anything different there? I’ve set mine to boot to the command line, but not to log in automatically. On some of my other Pis, I’ve allowed it to auto-start the desktop as Pi, but never the server.
Hi, Mel! Great blog!
I haven’t tested it myself, but do you know if you can remove the SD card once it boots? I know it won’t work if you have to reboot or the server crashes because the SD card is missing… but once it boots would it be safe to remove the SD card? No reason other than curiosity… 🙂
After adding the external harddrive of course.
I wouldn’t think so. Even booting from the hard drive, you’ll occasionally see the access light blink, so I presume it still writes SOMETHING to the boot partition now and then. Maybe not, but that’s just my guess.
Pingback: Configure a USB on the Raspberry Pi | Weaver Solutions
Hi Mel, I’m using a Western Digital 3TB Elements drive and have partitioned it similar to your instructions. It constantly spins, which surprised me. I’ve tried using hdparm to put it to sleep after inactivity. I haven’t got it to work correctly. Have you tried this?
Best regards, David
Hi Mel
I have the same problem that MABinTN described above. My Windows laptop doesn’t recognise the format of the NTFS partition I made one /dev/sda2 (I followed your instructions exactly).
You replied:
“Well, if you haven’t actually put anything on the drive yet, I’d say your best bet is to let Windows format that partition rather than doing it from the Pi. You should have something you KNOW Windows likes then. The Pi should still read and write to it just fine.”
Do you know exactly how one would do this on WIndows? I’d want to leave the sda1 system partition in tact, and leave the NTFS sda2 partition in a manner that the Pi still recognises.
I saw a similar issue here:
http://ubuntuforums.org/showthread.php?t=1476185
but couldn’t use that info to fix my problem.
Would be grateful for any help you can provide.
If the drive is partitioned already, Windows should see it as two drives when you plug it in. It will complain that the ext4 partition is unreadable, and want to format it. Don’t let it. Just use windows to format the OTHER partition like you would any other drive. Ignore the smaller ext4 partition, and just format the larger “data” partition. You should be fine after that.
Hi Mel, I think I have got to the root of the problem. I am using a 3TB disk and formatted a 30GB ext4 system partition and a 2.7GB NTFS partition. I have now been told:
“Windows won’t be able to deal with this because the disk must be a GPT disk for Windows to recognize the entire 3TB disk.” and “Note that there is 2.2TB maximum size limit for MBR disk partitions, so I recommend you clean and convert the drive to GPT to access all of the drive space.”
This is a bit of a blow. I assume my disk is MBR. I’m trying to find a way of converting the disk from MBR to GPT but may have to clean it and start again. Do you know which of the commands you specified in this article determines that MBR is used? I.e. How might I select GPT instead?
Any ideas would be much appreciated.
When you run parted and “print all”, each drive entry should tell you what the partition table type is. Look at the first example in this article. It’s right after running “sudo parted”. You’ll see that the partition table type says “msdos”. Now read down further to after the hard drive has been hooked up, and it says “gpt”. As for converting the contents. I don’t know of any way to do that. You’ll need to copy everything onto a different drive while you reformat the main one, and then copy everything back. If someone out there knows a trick for getting around this, I’d be pretty amazed.
Thank you so much for this guide, Mel! It’s been a HUGE help in learning Linux and using a Raspberry Pi. I’m running into an issue with my external hard drive. I’ve followed your steps to the letter but I’m having trouble accessing the HD from my Windows machines. It shows up in Disk Management with the two partitions, one at 14.90GB (16GB) and the other is the remaining space. It doesn’t recognize it as NTFS but I did format it that way. The hard drive doesn’t show up in Explorer under This PC (I’m using Windows 10.) Like you suggested above, I was going to format the data partition but that is not an option in Disk Management. Do you have any suggestions? I don’t want to have to erase the whole drive and start step 6 again. Any help will be greatly appreciated!
Thank you so much for this guide! It’s been a HUGE help! I’ve followed your guide to the word but I’m having issues connecting my external HD to my Windows 10 machine. The drive shows up in Disk Management and I see the two partitions that were created, but I can’t access the drive through Explorer. Neither of the partitions show NTFS in Disk Management. I thought I would format the data partition like you suggested above, but that isn’t an option. The only option I have in Disk Management is “Delete Volume…”. Do you have any suggestions on how to fix this? Your help is greatly appreciated!
I’ve had several reports of this exact behavior lately. I’m not sure what changed, but it seems that to get the NTFS partition to work under windows, you need to let Windows format it these days. Back your stuff up first, and you’ll need to edit fstab again because the partition will have a new uuid afterwards. There are some recent comments about exactly this situation. Give them a read as well.
You don’t need to use Disk Management to do the formatting. You can just do it through explorer. Right click on the NTFS partition, and then “format”. Just make sure you pick the larger partition (presumably) so you don’t overwrite the os.
Thanks for the quick reply! (Also, sorry for the double post, my first post wasn’t showing up for some reason) I can’t access the drive through explorer at all. It shows up in Disk Management but won’t appear in explorer. Maybe I can delete the data portion of the disk and let Windows partition and format it that way? I’ll try when I get home. Thanks again!
if your drive is working correctly on say a linux or mac machine (read only is fine), then the partition data itself is fine. You may not have assigned a drive letter, due to which it doesn’t show up in Windows explorer. Follow the procedure here: http://www.tomshardware.com/forum/284807-32-external-hard-drive-device-manager-comp
Amol, thank you for the link. I couldn’t assign the partition a letter, but I was able to delete it and reformat using Windows. I then had to hook the HDMI and USB keyboard back up and find the new UUID using sudo blkid. Then I went back and edited /etc/fstab with nano and I’m back up and running! On to the next step!
Well the problem has been sidestepped for the time being 🙂 Which is perfectly fine if you don’t intend on doing this again. If you do, it’d be ok to find the root cause of the problem. You would definitely need another drive and another machine, preferably with Linux installed, to compare with.
Planning for the future… If my SD card fails, even with its limited use since I’ve attached an external HDD, can I back it up from the actual HDD?
For example,
sudo dd if=/dev/sda1 of=/NEW/SDCARD bs=32M conv=noerror,sync
Whenever I run apt-get update && upgrade, it upgrades all of the files located on sda1 (my hdd 1st partition) correct? Should I now save images from that partition of the HDD?
My example scenario: keeping up with the server with regular updates and upgrades. I still have my old backup .img files of the original install (as you suggest). My SD card craps out (with its limited use)… should I just save an image from my HDD instead of a now outdated SD card? Then restore a new SD card from my HDD, plug and go…
P.S. thanks so much for this reference guide, I’ve combined it with a few others but this has ultimately served as my go-to guide. I even automated a lot of it with a bash script; yes, even the HDD section – took a lot of guess and checks. I figured out how to store PARTUUIDs as variables and write them to the cmdline.txt file. I could share that with you when I complete the rest of it. 🙂
The command you show would be trying to back up the hard drive to the SD card. Remember that the first partition on the SD card is still needed to start the boot process, so your backup would have to fit in the remaining space. Assuming you made the OS partition on the hard drive the same size as the OS partition on the SD card, theoretically this would work, yes. What would probably work better would be backing up to another hard drive.
One thing you can do to avoid problems when doing such backups is to create another SD card to boot the Pi with. This one shouldn’t be loading its OS from the hard drive at all. Boot with the second SD card, attach both hard drives (your OS drive, and your backup drive), and then run the dd command. Just make very sure you’re copying the right direction. Plugging the drives in one at a time and checking which positions they mount in would work. You don’t want to restore when you meant to back up.
I’ve done some scripting as well, but as more of an integration test to make sure my instructions still work when things get updated. I’d like to see what you’ve done with the UUIDs into variables. That would help automate another piece of the puzzle.
In lieu of your command here:
sudo dd if=/dev/mmcblk0p2 of=/dev/sda1 bs=32M conv=noerror,sync
I use (I wanted a little more feedback on the screen):
(sudo dd bs=32M if=/dev/mmcblk0p2 conv=noerror,sync) | pv | (sudo dd of=/dev/sda1)
Gathering your Partition unique GUID (instead of copy/paste, store it as a variable):
puuid=”$(sudo blkid -s PARTUUID -o value /dev/sda1)”
echo $puuid
Changing the bootloader to look for your hard drive:
cmdtxt1=’dwc_otg.lpm_enable=0 console=ttyAMA0,115200 console=tty1 root=PARTUUID=’
cmdtxt2=’ rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait rootdelay=5′
cmdtxt=$cmdtxt1$puuid$cmdtxt2
echo $cmdtxt | sudo tee /boot/cmdline.txt
This part specifically applies to this project and there may be a better way of doing this, for instance, looking for “root=” inside the cmdline.txt file and inserting the variable there. This is a hardcoded insert for my raspberry pi. This command breaks it into separate strings and combines it with the PARTUUID. Also, I’m not sure if “tee” is the best method, but it works.
For the pv command you’ll need that package: sudo apt-get install pv
I found a better method to insert inside of the cmdline.txt file using “sed”
According to here: http://askubuntu.com/questions/20414/find-and-replace-text-within-a-file-using-commands
You can use sed in this format:
sed -i ‘s/original/new/’ file.txt
To use a variable:
puuid=”$(sudo blkid -s PARTUUID -o value /dev/sda1)”
sudo sed -i “s/\/dev\/mmcblk0p2/PARTUUID=$puuid/” /boot/cmdline.txt
That will replace /dev/mmcblk0p2 with “PARTUUID=…….”
Breaking down this command as the variable: sudo blkid -s PARTUUID -o value /dev/sda1
-s = searches for the term (PARTUUID in this case)
-o = the output you want (the value of PARTUUID)
/dev/sda1 = the partition of the UUID we want (can apply to /dev/mmcblk0p2 later on)
Note: you’ll still need to append rootdelay=5 to the end of the file as well. But this should solve your copy/paste of long strings problem 🙂
Finally, to append to the /boot/cmdline.txt file “rootdelay=5” use this:
sed -i ‘1 s/$/ rootdelay=5/’ /boot/cmdline.txt
That command says to search the first line and append rootdelay=5 to the end of it.
Sorry for all of the replies 🙂
Well you only need the initial boot partition backed up somewhere if you mess up the SD-card. Since all the /home and / filesystems are on the USB drive, they are safe. If you’re using NOOBS, you’ll need to backup the /boot and /SETTINGS partitions from your SD card, since they’re used during initial boot.
Thanks! I’ll test this out and report back. I’ll make a separate SD card with only the /boot files (Raspbian Jessie Lite) and see the results.
Hi Mel, could you help me please? Having got to the end of Part 6 of your instructions, what should I do to revert to having the system on the SD Card (instead of on the HDD System partition)?
I want to consider doing this because I am still trying to get the HDD to sleep when the data partition is not being accessed.
Best regards, David
Hi Mel, was wondering if you could answer a question that maybe i’ve glossed over accidentally? I successfully moved my root file system to a 320gb hdd which contains 2 partitions sda1 (rootfs) and sda2 (pure storage ext4). my rootfs is partitioned for 50gb and remaining partition is storage. How would i make a backup image of the rootfs to sda2 being that the rootfs is mounted. Long way around i unplugged the hdd plugged it into another linux machine and used disk utility to backup the partition.. is there an easier way to backup my mounted rootfs to the second partition on the hdd? I already have a backup image of the sdcard . Thanks for any help you can provide.
That’s the best way I know of. You could use dd to back it up the same way you originally cloned the sd card to the drive, but since it IS mounted, it will be upset when it comes back online after restoration. Shutting it down and using dd from a different system seems safest to me.
I have been following your directions closely. I’m using a new My Book 3TB. Formatted a 16GB partition and copied the boot files to it, pointed the cmdline.txt file to the partition unique GUID, then created a data partition (ntfs) and got to your reboot step.
I’m currently waiting at boot up, everything seems to be marked “OK”, and the last item is “Reached target Network is Online” — and nothing seems to be happening. I have rebooted the Pi once as well, current wait time ~20min
Has anyone else experienced this delay, and is there any indication that I should start over or that this is normal?
Are you auto-mounting a file system from over the network in fstab? I did a little checking, and most of the people I see reporting this message are reporting it in connection with mounting something from over the network. If that’s the case, I believe raspi-config has an option to wait until the network is online before continuing the boot process. Give that a shot, and see if it makes a difference. In my situation, the Pi isn’t mounting anything over the network, so it’s never been a problem.
Hi Mel,
First off, thank you so much for this tutorial; it is absolutely wonderful. I have never used a Linux before, let alone a Raspberry Pi, so the fact that I can take this one step at a time and actually learn as I go is perfect. Again, thank you.
On to my question: everything (well, except for the VNC, but I don’t really care about that at this point) has worked without an issue. When I go to the Webmin on the next step though, I notice that virtual memory still only says 100mb, when I allocated 4GBs. When I check swapon -s it says 102396 bytes (still 100mb) but when I check the actual file it says 4096. I allocated 40gbs to the boot partition and 600 to the data, just to keep things round, so I know there is space for it…
I rebooted and checked all the variables again, and still the same results. Any thoughts? Thanks in advance,
Eric Bartha
Alright, so I may have commented prematurely. I went back into the swap file (sudo nano /etc/dphys-swapfile) and I noticed you mentioned there was only one line. I had 27 lines. Most of them were commented out, but I realized the line I was edited was a max swap file size line.
I deleted everything from the swap file and then re-wrote CONF_SWAPSIZE=4096. Once I rebooted, swapon -s shows the increased file size, however, it capped out at 2GBs, even though I verified I wrote 4096 to the swap file itself. Webmin is also showing 2GBs of virtual memory now.
I got it working, so I have no real complaints here, more just curious at this point. There must be a hard cap on the swap file? Have you heard of this? I plan on researching this tomorrow, but for now, I must get some sleep.
Thanks,
Eric Bartha
I’ve never tried setting it over 2gb, so I can’t say for sure, but there may be a cap of some kind. The general rule is that your swap file should be twice your RAM, and on a Pi 2, that’s 1gb. That guideline may come into play automatically. I’m not really sure. Sticking to the general guidance, though, you shouldn’t need more than 2gb. I may try to change the swap size on a Pi with less RAM and see if it follows the pattern. I can’t say when I’ll get to do that. Perhaps this weekend. I’ll put it on my list.
I am surprised it has not been mentioned earlier…
Why not use the human readable /dev/disk/by-label/ rather than the clunky /dev/disk/by-uuid/?
You could have easily tweaked your fstab for the already labeled “Data” partition with
“/dev/disk/by-label/Data”
If the tutorial were to recommend labeling the new root filesystem with e2label at format time, (like you did for the Data partition) you could have also used by-label for it.
Much easier, human readable, elegant, smarter-not-harder.
Labels are far less likely to be unique than uuids, and the odds of someday plugging in a new drive and having a name collision are non-zero, especially with labels like “data”. Since I’m putting this out there as instructions for others, I need to reduce, as much as possible, the number of potential landmines.
Thanks, Mel, for all this great info. Unfortunately, at my age I sometimes don’t follow instructions well, so I’ve been quite a while getting set up to the point where I can boot to the external drive (sda1). However, neither partition on the external drive shows up in file manager, and the /mnt/data/ file is empty. Strangely, the old mmcblk0p2 does show up (unmounted) in Places in the File Manager. And an external drive plugged in to the other usb port on the Pi also shows up and can be mounted.
Any ideas?
Kirk
My first guess would be that the drive isn’t getting enough power. Does it have its own power supply? It sounds like the Pi isn’t even seeing it at all.
Thanks for your interest, Mel. I hope you do not tire of advising newbies such as myself. I think part of my problems and questions lie in my inadequate grasp of the Linux file system, so I’m going to start by attempting to get more familiar with it.
At this point, I have my OS hard drive, which has two partitions (one ext4 and one ntfs) plugged into one of the usb ports on my pi 1. Neither of these drives show up on the File Manager in Jessie’s graphical desktop. In addition, I have a 5-port usb hub plugged into the other port. This hub has three usb external drives, each with it’s own power supply, except for one which is drawing power from the hub via its own second cable. The pi is powered by cable from the remaining port on the hub. The hub also has three charging ports, one of which will power the pi, but not if the external drives are plugged into the same hub.
What resulted was, in the desktop File Manager, in “Places”, the hard drive containing the os partition and the ntfs are not showing up at all. The original os partition (on the sd card with the boot partition) shows up unmounted. In addition, the other three usb drives also show up in “Places”. None of the four are mounted on power-up, but all can be mounted by a right-click.
The speed is great, and I am sure I am running the os from the unseen hard drive. As an experiment, I went back and changed the ntfs partition on my os hard drive to the same parameters as the other three drives with respect to discovery and mounting in fstab. Still no go.
As I say, I think I need a more thorough understanding of the boot process and file system in Linux to solve my problem. Somewhat annoyingly, I had this system running fine with three hard drives until I got the idea of running the os from a hard drive to make Teamviewer run on it. If I can get back to that with the os on an external drive, I’ll be happy.
If you can see anything obvious, please let me know, but don’t waste too much time or effort on my problem. I’m going to drag out one of my Linux texts and see what I can see.
Thanks,
Kirk
Let’s try the very basics first. I’d try to get the system down to just the one drive, at least temporarily. Try the “lsusb” command and see whether the device is seen at all. Also, if the drive doesn’t appear, then how did you get the positions made? Assuming it’s there, and the Pi sees it, then the problem of it not showing up under /mnt is probably something up with the fstsb file. Double check that it’s referring to the drive properly, and pointing it to a placeholder directory where you want it to appear.
I think I’m going back to the very beginning, with a new installation of Jessie, and no deviations from your scenario at all. I had made a few small changes, such as using Windows Remote Desktop instead of VNC. I may need some advice when it comes to using Webmin to set up Samba shares as well. What folder/file should my os and data (I’m calling it “Current”) partitions be found in?
Again, many thanks,
Kirk
Where you put your share is entirely up to you. I usually divide my hard drive up into two pieces (OS and Data). Partly because it means that you can take a backup of the OS separately from the data, but mostly because I like to keep my data partition as NTFS. It lets me temporarily attach the hard drive to the computer to do mass file moves, and generally simplifies file permission issues. You could just as easily make the whole drive one big ext4 partition and put your files in regular filesystem folders.
As for where the partitions show up in your filesystem, that’s determined by what you put in fstab. The nice things about a unified filesystem is that you can put things wherever you want. I can add another hard drive to the system and tell it to appear at /usr/pi/stuff if I want to. My convention is to give each drive a name such as “data” or “media”, and then mount it under /mnt/data or /mnt/media. That’s just me, though. If I were to add another drive specifically to hold movies, for instance, I might even mount it so that it appears to be inside of another drive. I might put it in /mnt/data/public/movies, for example. Basically, if you can create a directory somewhere, you can mount a drive there. It’s magic.
So the key is fstab, then. I guess I’d better read that part again until I understand it. I got the part about using UUID to point to a specific os partition. I’m going to chant a mantra and then start again from scratch, trying to understand what it is I am doing at each step.
BTW, the IT Guy just refers to my own title on a WordPress blog I used to manage for a HOA. By no means a reflection on my skills!
Kirk
Yes. Read the fstab part carefully. The partition uuid is used to set up booting from the drive, but there’s a different uuid used for mounting the filesystem that the partition contains, and there’s a very important reboot step in the middle or the changes won’t stick.
Sorry to keep beating a dead horse, Mel, and maybe I should just forget about it, but after several runs-through, including downloading a new copy of Jessie, and starting with a brand new sd card, I can still not “see” my Data partition nor my system partition on my hard drive in file manager on the desktop. I can see the old sd card system partition (unmounted), and I looked in /mnt/data/ but nothing there. The directory exists, but it’s empty.
Think I should give up, or am I wrong in thinking that those two partitions should be available in the file manager? Do they somehow have to be mounted at boot? I hate to give up 500 gb of storage just to make the system run off a hard drive. How can I be certain that the system is actually on the hard drive and being booted into?
Thanks,
Kirk
Don’t give up, but the first step really is figuring out where the problem is. You say that the drive is powered from the hub, so I’m assuming that’s not it, but you still need to figure out whether the Pi is seeing the drive at all. “lsusb” (List USB) will list the devices connected to the Pi. I recommend attaching only the problem drive if you can, unless you have some other way to tell them apart from one another. Always start simple. Just the SD card, then one drive, then another. Baby steps.
If the USB device is seen by the Pi, then move on to checking the partitions with “sudo blkid”. Does the drive even HAVE partitions? If you can see the partitions, then what filesystems are they using. Did you attach an NTFS drive, but not install the NTFS drivers perhaps? Is the drive completely blank, and needs to be formatted?
Diagnosing the problem isn’t that hard, but it’s best if you start from “is it plugged in”, and work your way forward.
Thanks Amol P and Mel:
Here’s where I am now. I am using a new from scratch distro, set up according to your instructions. I skipped Webtime for now, and installed xrdp so I can use Windows Remote Desktop, but I stayed with Putty command line until I had everything completed through the end of Part 6. The Pi is powered from a 2a wall wart. The HDD, a new 1TB, is plugged into and powered by a USB hub, which is then plugged into the USB Port of the Pi. Nothing shows up in /mnt/data, either by Putty command line or Remote Desktop.
df -h:
Filesystem Size Used Avail Use% Mounted on
/dev/root 15G 5.5G 8.5G 40% /
devtmpfs 182M 0 182M 0% /dev
tmpfs 186M 0 186M 0% /dev/shm
tmpfs 186M 4.6M 181M 3% /run
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 186M 0 186M 0% /sys/fs/cgroup
/dev/mmcblk0p1 60M 22M 39M 36% /boot
/dev/sda2 917G 94M 917G 1% /mnt/data
tmpfs 38M 0 38M 0% /run/user/1000
lsusb:
Bus 001 Device 008: ID 0835:8502 Action Star Enterprise Co., Ltd
Bus 001 Device 007: ID 0835:8500 Action Star Enterprise Co., Ltd
Bus 001 Device 006: ID 14cd:6116 Super Top M6116 SATA Bridge
Bus 001 Device 005: ID 0835:8501 Action Star Enterprise Co., Ltd
Bus 001 Device 004: ID 0835:8500 Action Star Enterprise Co., Ltd
Bus 001 Device 003: ID 0424:ec00 Standard Microsystems Corp. SMSC9512/9514 Fast Ethernet Adapter
Bus 001 Device 002: ID 0424:9514 Standard Microsystems Corp.
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
sudo blkid:
/dev/mmcblk0p1: SEC_TYPE=”msdos” LABEL=”boot” UUID=”04FE-51E9″ TYPE=”vfat” PARTUUID=”8f1eafaf-01″
/dev/mmcblk0p2: UUID=”e6e7f776-11a4-4cd7-b4fd-c44ecdbfcf90″ TYPE=”ext4″ PARTUUID=”8f1eafaf-02″
/dev/sda1: UUID=”28597b27-6a8b-4de6-b792-d7d7062ed7ec” TYPE=”ext4″ PARTLABEL=”primary” PARTUUID=”a35fb77c-109d-404c-be89-a7ec252597c1″
/dev/sda2: LABEL=”Data” UUID=”3BE5956A4787EC8C” TYPE=”ntfs” PARTLABEL=”primary” PARTUUID=”31f5121e-4e7e-4eb7-b19c-46a99d4938e2″
/dev/mmcblk0: PTUUID=”8f1eafaf” PTTYPE=”dos”
ls -l /dev/disk/by-id/usb*:
lrwxrwxrwx 1 root root 9 Apr 11 20:13 /dev/disk/by-id/usb-Mass_Storage_Device_116AC2101219-0:0 -> ../../sda
lrwxrwxrwx 1 root root 10 Apr 11 20:13 /dev/disk/by-id/usb-Mass_Storage_Device_116AC2101219-0:0-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Apr 11 20:13 /dev/disk/by-id/usb-Mass_Storage_Device_116AC2101219-0:0-part2 -> ../../sda2
(this is the first time I’ve gotten a response from this
command)
(parted) print all:
Model: Mass Storage Device (scsi)
Disk /dev/sda: 1000GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:
Number Start End Size File system Name Flags
1 1049kB 16.0GB 16.0GB ext4 primary
2 16.0GB 1000GB 984GB ntfs primary
Model: SD 00000 (sd/mmc)
Disk /dev/mmcblk0: 7861MB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:
Number Start End Size Type File system Flags
1 4194kB 67.1MB 62.9MB primary fat16 lba
2 67.1MB 7861MB 7794MB primary ext4
Nothing in /media
Nothing in /mnt/data, although the empty directory is there.
No mounted disks in File Manager/Places. Unmounted 7.8GB Volume, which is the old mmcblk)p2.
Okay then. The drive is being seen by the Pi. That pretty much leaves the mounting part. What does your /etc/fstab file say?
@Kirk do this and report the output:
> sudo mkdir /media/test
> sudo mount /dev/sda1 /media/test
> cd /media/test
> ls -al
Here’s the result. (I did it three times with three different names, “test” and “trial” and “three)
pi@raspberrypi:/media/three $ ls -al
total 104
drwxr-xr-x 23 root root 4096 Apr 12 13:25 .
drwxr-xr-x 6 root root 4096 Apr 12 13:30 ..
drwxr-xr-x 2 root root 4096 Apr 11 18:25 bin
drwxr-xr-x 2 root root 4096 Mar 18 02:09 boot
drwxr-xr-x 4 root root 4096 Apr 11 18:36 boot.bak
drwxr-xr-x 4 root root 4096 Mar 18 02:07 dev
drwxr-xr-x 111 root root 4096 Apr 11 20:15 etc
drwxr-xr-x 3 root root 4096 Mar 18 02:09 home
drwxr-xr-x 20 root root 4096 Apr 11 19:01 lib
drwx—— 2 root root 16384 Mar 18 02:46 lost+found
drwxr-xr-x 6 root root 4096 Apr 12 13:30 media
drwxr-xr-x 2 root root 4096 Apr 12 13:25 mkdir
drwxr-xr-x 3 root root 4096 Apr 11 20:05 mnt
drwxr-xr-x 6 root root 4096 Mar 18 02:40 opt
drwxr-xr-x 2 root root 4096 Jan 6 2015 proc
drwx—— 2 root root 4096 Apr 11 19:03 root
drwxr-xr-x 5 root root 4096 Mar 18 02:10 run
drwxr-xr-x 2 root root 4096 Apr 11 19:19 sbin
drwxr-xr-x 2 root root 4096 Mar 18 02:05 srv
drwxr-xr-x 2 root root 4096 Apr 12 2015 sys
drwxrwxrwt 10 root root 4096 Apr 12 13:17 tmp
drwxr-xr-x 10 root root 4096 Mar 18 02:05 usr
drwxr-xr-x 11 root root 4096 Apr 11 20:13 var
@Kirk Use this:
> lsusb
This will list the USB devices connected. I have a westedn digital drive connected to my RPi so I’d see something like “WD” or “WesternDigital”.
If you don’t see this, your USB device is under-powered
> sudo blkid
This will list all partitions in your drive. What’s the output of this command? It’ll also list the device name in the first column (like /dev/sda1)
See if you can see those devices by doing:
> ls -l /dev/disk/by-id/usb*
Now auto mount them by doing:
> sudo partprobe
all available partitions should now be mounted. Check them under “/mnt”, “/mount” or “/media”
Latest report:
drwxr-xr-x 3 root root 4096 Apr 12 09:36 ..
drwxr-xr-x 2 root root 4096 Apr 11 18:25 bin
drwxr-xr-x 2 root root 4096 Mar 18 02:09 boot
drwxr-xr-x 4 root root 4096 Apr 11 18:36 boot.bak
drwxr-xr-x 4 root root 4096 Mar 18 02:07 dev
drwxr-xr-x 111 root root 4096 Apr 11 20:15 etc
drwxr-xr-x 3 root root 4096 Mar 18 02:09 home
drwxr-xr-x 20 root root 4096 Apr 11 19:01 lib
drwx—— 2 root root 16384 Mar 18 02:46 lost+found
drwxr-xr-x 3 root root 4096 Apr 12 09:36 media
drwxr-xr-x 3 root root 4096 Apr 11 20:05 mnt
drwxr-xr-x 6 root root 4096 Mar 18 02:40 opt
drwxr-xr-x 2 root root 4096 Jan 6 2015 proc
drwx—— 2 root root 4096 Apr 11 19:03 root
drwxr-xr-x 5 root root 4096 Mar 18 02:10 run
drwxr-xr-x 2 root root 4096 Apr 11 19:19 sbin
drwxr-xr-x 2 root root 4096 Mar 18 02:05 srv
drwxr-xr-x 2 root root 4096 Apr 12 2015 sys
drwxrwxrwt 10 root root 4096 Apr 12 09:17 tmp
drwxr-xr-x 10 root root 4096 Mar 18 02:05 usr
drwxr-xr-x 11 root root 4096 Apr 11 20:13 var
pi@raspberrypi:/ $ cat /etc/fstab
proc /proc proc defaults 0 0
/dev/mmcblk0p1 /boot vfat defaults 0 2
/dev/disk/by-uuid/28597b27-6a8b-4de6-b792-d7d7062ed7ec / ext4 defaults,noatime 0 1
/dev/disk/by-uuid/3BE5956A4787EC8C /mnt/data ntfs defaults 0 0
#/dev/mmcblk0p2 / ext4 defaults,noatime 0 1
# a swapfile is not a swap partition, no line here
# use dphys-swapfile swap[on|off] for that
Hope this reveals something. When I had this system set up before, with three external drives, it worked perfectly, but of course I wasn’t trying to boot from an external drive. That seems to be the problem.
Thanks for your help.
Kirk
Did you try manually mounting the drive as Amol suggested? What did that look like? Let’s take this one step at a time. Leave booting from the hard drive until after we’ve gotten the hard drive to show up at all. My recommendation these days is to create the partitions, but postpone booting from the hard drive as long as you can so that you can keep taking simple backups of the SD card all the way up until you get the system “just right”, then cut over to the HDD. This will be reflected in an updated series of posts I’m currently putting together.
When it comes to the actual booting, you need to be careful of a few things.
1) It requires changes to both cmdline.txt and fstab, and the order is important. So is the reboot. If you skip any of these, it won’t stick.
2) You have to use the correct UUID in cmdline.txt. There are two. One identifies the partition, the other identifies the filesystem. You have to get the right one in the right spot.
3) You probably need the extra delay at the end of cmdline.txt to allow the USB device to become ready before the system needs it.
@Kirk I think we’re getting ahead of ourselves here. I think you are able to detect the drives now and mount them. What you said: “At least I know I’m booting from the HDD, even if I can’t see it” – I don’t see how this could be possible. If you are booting from an external drive, it needs the root filesystem to be visible. You can’t have it any other ways. The other partitions may not be mounted but the main one has to be. Something tells me you’re not booting from it.
I’m not sure that’s entirely correct. Someone who’s more of a Linux expert can correct me here, but in my experience you can definitely boot from one root filesystem, but mount another one to /. You wouldn’t WANT to, but it happens. That’s why the reboot step I point out in the booting from the hard drive section is so important. cmdline.txt identifies a partition to boot from, but fstab says what to put at /. You can mismatch these. What you see is a difference in what the OS sees vs what YOU see. The OS boots from one filesystem, but “ls /” shows you somewhere else. This can get very confusing, and I don’t know of a legitimate reason to run a system this way, but I have forgotten that all-important reboot step and found myself in this state before.
that sounds like a good explanation. I think I’ll believe it! Anyway, it works. Now to get file sharing working properly. Gotta read the paper and drink my coffee first.
Thanks for your excellent instruction.
Kirk
@Mel sorry I’m having to reply out of order since it’s not allowing me to reply to your replies to my post.
Unless he used something like pivot_root (explained here: http://serverfault.com/questions/300764/unmounting-root-filesystem-without-rebooting-the-server), I don’t see how he could have booted and switched the filesystem. In RPi, the boot process is explained here: http://raspberrypi.stackexchange.com/questions/10489/how-does-raspberry-pi-boot. The ‘cmdline’ options is used by start.elf to load the kernel image from. Unless you manually dismount this partition, load another as root it won’t work IMHO.
The user can easily check this by including a simple “touch /tmp/test.txt” in the startup “/etc/init.d/” section of the external root filesystem and check if the file has been created. If not, it definitely has not been mounted.
Amol, I’m getting posts out of order and to the wrong poster. Anyway, I at least could see my partitions last night with File Manager. They just don’t show up in the Places list, or on the desktop, where I have told desktop config to show them. Other external drives do, but at this point I don’t care, since I can get to them via File Manager. Mel suggested that maybe it’s because they are mounted before Desktop opens. I’m thinking maybe the ntfs won’t show unless it is mounted in media, as my other hdd’s are. I’m going to change the mount path from /mnt/data/ to /media/ and see what happens. I guess I’ll have to mkdir a folder.
So, now I’m trying to get sharing set up with Samba.
“I ain’t down yet”
Kirk
@Kirk well ok so you are able to mount the drive and see the contents. You said you loaded it with the root file system correct? So the directory listing seems right
It appears so, at least to my untutored eye. I’m never sure what I’m seeing in command line. BUT… I Iooked at the partitions in Gparted on the desktop, and both partitions actually were mounted, and accessible at their mount points in File Manager. So apparently all is well as far as accessibility from the desktop is concerned. They just don’t show up in Places, nor on the desktop itself. Since my ultimate plan is to be able to access them via Samba and Teamviewer, I’ll have to see how I can do that. BTW, I plugged in another HDD which had been used for an earlier attempt at this project. It requested to be mounted, and did so, at least the ntfs partition, and showed up in Places and the desktop.
The saga continues. Next step: Samba and Teamviewer. At least I know I’m booting from the HDD, even if I can’t see it. You guys are command line users, right? Maybe your system drive doesn’t appear in File Manager either.
Thanks,
Kirk
Well that looks mounted to me. So id the problem that you can see the drives from the command line, but not from the desktop file manager? How about navigating to /mnt/data by hand in the file manager? Does it see the drive then?
That is correct. As I said in my reply to Amol above, I think I’ve been talking past you. I’m not very good at interpreting what I see in the command line. When I looked at the partitions in Gparted on the desktop, everything is mounted properly, and in its proper place. Knowing that, I found them in File Manager. So now I am going to install Samba, xrdp, and TeamViewer. I was just being sidetracked by the fact that the hdd containing the system files and my Data partition don’t show up in the Places list in File Manager. As long as I can find them, I think I can share them with Samba and TeamViewer.
Many thanks to both of you for all your help. I’m sorry to be such an obtuse old fuddy-dud, but I’m 73 years old, and even though I got my first digital computer when I was 14 (google Geniac), I’m by no means a sophisticated user. I know just about enough to help the moldy-oldies in our retirement park get their e-mail.
Kirk
I don’t do much from the desktop, since I use my various servers in more of a “fire and forget” mode, but I think (and could be very wrong about this) that perhaps drives and partitions that were mounted before the desktop started up, due to being represented in fstab, don’t show up in Places. Drives that show up after you’re up and running get auto-mounted and show up there. My knowledge in that area is far from complete, though. I remote in to the desktop in order to maintain CrashPlan, and sometimes to use the browser in order to pull files (such as the CrashPlan installer) down directly to the Pi. Occasionally I do it in order to edit multiple files side by side. That’s pretty much the extend of my desktop use, though.
Finally got it working the way I wanted. I added a folder (/media/NASDRIVE) as suggested by this website:
http://www.techjawab.com/2013/06/how-to-setup-mount-auto-mount-usb-hard.html
I think I could have done the same with your instructions, but apparently File Manager only shows drives it finds in /media. No real problem for command-line types, but difficult for GUI junkies such as myself.
Thanks again,
Kirk
👍
Oops, no I didn’t! 😦
Kirk
OooooKay. Last post on this subject, I hope. After undoing mistakes involving confused disc labels, mount points, disks that weren’t working that I thought were, and other confusions related to aging brain cells, I finally actually have things working the way I want. All drives except the system partition showing up on the desktop, and in the File Manager Places list. Moving on to automount the drives and Samba sharing, but not ’til tomorrow. My brain needs a rest and a beer.
BTW, did you ever come up with a way to copy the system files from the hdd, and use them for restoration if necessary?
Thanks again,
Kirk
I haven’t written it up yet, but backing up the HDD is actually dead simple. You just make another SD card, install the NTFS support packages, and set up fstab to mount just the data partition.
When you want to make a backup, you shut down the system, swap in the new SD card, boot back up, and use dd to copy /dev/sda1 to /mnt/data/backups or some other folder you’ve set up. You could also attach and mount a different drive to catch the backups if you’re worried about the physical drive failing, rather than just having a convenient way to roll back.
I hadn’t thought of that. I’ll try it, before I start working with my file permissions, and installing teamviewer. I got the permissions wrong on the directories I mount my hdd’s in (except the one one my system hdd). But orherwise everything is working well.
Thanks,
Kirk
Just a quick update and a round of applause!
I now have my system up and running the way I wanted it to, I can even stream movies to my cell phone. And I’ve learned a LOT about Linux in the bargain, All thanks to you and your blog mates.
Kirk
Great to hear it. I’m glad I could help. And I’d like to thank all the commenters that chimed in too.
And I thank them too. You’ve got a great bunch of helpful people.
Kirk
Have you noticed a high CPU % by mount.ntfs lately? It’s spikes occasionally and slows the RPI down. Not sure if some update had any effect.
I haven’t noticed, but I haven’t exactly been looking, either. NTFS does add some additional overhead over what you would see with ext4, but it’s usually not that significant on its own. The increased usage might be a result of something else making use of the drive. I stopped running Transmission long ago because I found that it was the most disruptive influence on my server, and I wasn’t really spending a lot of time on torrent-able things anyway. When I shut it down, my drive usage dropped dramatically.
I followed this section several months ago.
I recently installed Owncloud and tried to move the data folder to the external drive, ut it does not work since the ownership and permission are “stuck” with root and I need them changed to www-data. Seems that NTFS does not support this (or I dd not understand how to do it)
Is there a simple way to change the data partition to ext4, knowing that all data on it is lost, but without affecting the other partitions and all the work done moving root partition, swap etc to the hard disk?
You can’t change a partition from one filesystem to another without losing everything it contains, but you can reformat a partition to a different filesystem type without harming the other partitions. You’d have to back up the contents of the partition somewhere, and then copy it all back when you’re done reformatting that partition. Just be very careful about making sure you’re selected the right partition. Follow the directions for setting up the hard drive, but skip the part where you create the partition table and partitions. Just do the part where you do the formatting with mkfs.ext4.
Reformatted disk to ext4
Then I also changed the entry in fstab for the line of /mnt/data from ntfs-g3 to ext4 (you did not mention it, but I thought I should).
After booting could not login again via SSH
Connected a monitor and I see the following at the end of the boot process:
Welcome to emergency mode! After logging in type “journalctl -xb” to view system logs “systemctl reboot” to reboot, “systemctl default” to try to boot again into default mode
syslogin: root accout is locked starting shell
root@SERVERNAME:~#
looked into the log and can’t find any errors
tried “systemctl default” and nothing happened
tried “systemctl reboot” and system rebooted to same situation
So now I am in a shell, but really do not know how to proceed to make the system boot correctly.
Any suggestions?
Changed also the UUID of the partition, since I noticed that it changed – but still same problem
Hi Mel
How can we supervise the HDD usage, like free space, quotas and such?
How to avoid running out of space (for those of us using smaller HDDs)
df -h
Pingback: Web Server - The Log Cabin Blog
First, thanks for this series of articles! It’s been incredibly helpful as a linux newbie to begin learning with something real to work on. I have a question. I fear I’ve done something unrecoverable but am hoping you know something I don’t. I’m successfully booting from the HDD and got the minidlna server up and running nicely. I then decided it was time to secure the pi and changed the password after recording what the new password would be. The problem is now that since I logged off, the new password is not accepted. I’ve tried everything I can think of but no joy. I do have an image I can restore to prior to changing to HDD boot. Once I get the SD card restored to that point, how do I change the password without having to redo the HDD stuff all over again. I do have the data backed up on another drive. FYI, the new password includes the symbols !, $, and +. Are any of those illegal? I typically use strong passwords of 15 characters or so. Any advice you can give would be most welcome.
Thanks again for this series! It’s been most instructive.
Sounds like a random typing error when changing the password, but you would have had to type it wrong TWICE to successfully change it at all. If you have another root-level account set up, you could possibly use that the reset the Pi user’s password. Otherwise, I don’t know what to tell you. You’re essentially trying to break into your own system, something that it’s designed to stop people from doing.
As for the booting from the hard drive part. I think you’re going to have to repeat that part, but the STUFF on the hard drive should survive just fine. What I’d recommend is that you put off the booting from the hard drive part until the very end, that way you can keep taking simple SD card backups all the way up until the server is just about finished. Then, as your last step, you make the leap to booting from the hard drive.
Thanks for getting back to me! That’s what I thought you would say. One of my co-workers suggested setting up the SSH keys for this exact reason. I changed my password from the GUI and am wondering if that was an unwise thing to do.
Do you have any advice for how to shut down since I can’t log on? I do have a ups with apcupsd running. I guess I could unplug and let the battery run down till it shut’s down. That might take a while! BTW, I’m running the 1TB Pi Drive from WD and it works great. I’m powering it and the Pi from a usb hub and they have a nifty three way cable just for the Pi. I think I might buy another one.
Thanks again!
The easiest way for you to login into your RPi would be to take out the SD card, put it in a computer running linux and modify a single line in a file as described here:
https://www.raspberrypi.org/forums/viewtopic.php?p=109612
After that, put the SD card back, boot up your RPi. You’ll directly goto the prompt with admin permissions. After that you can change the password without knowing what it is currently.
Well that’s useful to know, and also kind of troubling. It shouldn’t be that easy to just take over a computer like that. Then again, Raspbian doesn’t advertise itself as a hardened, secure OS in the first place.
Yup it is easy. I wrote about it in the Raspberry Pi forum too. Unfortunately there is no way to disable it.
Well, as they say. If someone has physical access to your computer, it’s not your computer anymore.
Well I disagree. That is but an old viewpoint, before the era of complex software, encryption and GPS. Today, your iPhone could be stolen but activation lock renders it inoperable to anyone but Apple and big govt. agencies.
Similarly, it’s a question of kernel and boot security policies in Linux. We could just disable single user mode in Linux but they choose not to by default. The disk can be encrypted and decrypted upon boot. In that case the stolen RPi in theory would be junk data to anyone else. The problem of course lies, in these techniques not being easy to use.
Thanks for the replies, Amol, and Mel. I tried this last night and while the Pi booted to a hash prompt it didn’t recognize any input from the keyboard and the display had an error like user 0, unable to access tty, job control disabled. That’s not the exact text but the gist. Looks like i’ll have to restore the SD card to my last backup image. I’m going to take Mel’s advice and change the password first, then install all the other utility stuff and then move the root file system back over to the HDD. Live and learn! But then, that’s what the Pi is all about. It’s frustrating, but you can bet I won’t make that mistake again! On to new and different mistakes! 🙂
Thanks again, guys!
Hi Mel,
My server ROCKS thanks to your tutorial!
I have a one-off question I think, that hasn’t yet been posted to you….
My server is running on a pi2, 16GB sd and a 260GB hdd…
Now I’d like to replace the 250GB hdd by a 1TB… How can I do this without going throu all the process again and without just adding it as a second hdd to the system?
Check out the most recent post I put up (https://melgrubb.com/2016/07/10/backing-up-a-pi-that-boots-from-the-hard-drive/). Back up your hard drive, restore it to the new drive, and then run raspi-config again to expand the filesystem to fill the new drive.
that simple? can I do that from network side, before attaching the new drive to the PI? ( the backup/restore procedure)
You could certainly try, but it’s really best if you can shut the Pi down completely and back up the drive when it’s not being used. It’s just like moving from the SD card to the hard drive in the first place, but you’re dealing with a lot more data, so there’s more that can go wrong.
one last question, then… when on raspi-config, will I be able to tell if the filesystem I’m trying to expand is the hdd and not the sd card?
It’s whichever filesystem you booted from.
You’ll need to be careful, though. If you clone the drive, both drives will have the same partition and filesystem UUIDs on them, so you’ll need to swap the drives, not just add the new one. If you have them both hooked up when the system boots, though, it’s going to get confused as to which one to use. Once you’ve booted from the new one, and expanded the filesystem, then you can try hooking up the second drive, and use parted to blow away its partition table and create a new one. Just be careful about which one you’re killing. run “sudo blkid” before and after hooking up the original drive to be sure that it is, in fact, /dev/sdb. Once the old drive has been given a new identity, then you can leave them both hooked up all the time, if that’s what you want.
Hi Mel,
So, I’ve recovered from the silly mistakes I made earlier and the pi had been running smoothly. Now, every few days, I see ‘Write error on swap device with additional info. Rebooting the Pi seems to clear things up for a few days.
Do you have any recommendations for diagnostics and remediation? Can I script that? FYI, I’m using a 1TB WD Pi-Drive that has a USB3 connector on the drive end.
Still a Linux NOOB, but learning!
Thanks!
Rich
If you’re booting and running from the SD card, I would look at getting a better SD card, or moving the swap file to the hard drive, which is better for such things, but will only work on an ext4 partition (as far as I know). Edit the swapfile configuration with “sudo nano /etc/dphys-swapfile”. Increase the size (probably set at 100mb) to twice your physical RAM (so most likely 2048), and add a line to relocate the swap file to somewhere on the hard drive “CONF_SWAPFILE=/mnt/data/swapfile” (replace /mnt/data with wherever you mounted your hard drive). Remember, if your hard drive is formatted NTFS, I don’t think this will work.
If you’ve moved your whole OS partition over to the drive, and are completely booting and running from there, then there is nothing to move, although you probably still want to increase the size. If that’s the case, then I’d suggest consulting the great oracle of Google. That is not an error I have personally encountered.