Putting a big disk on a rasperry pi

Software related topics

Putting a big disk on a rasperry pi

Postby Doug Coulter » Sun Jan 12, 2014 2:30 pm

Not sure where this fit the best, so I just put it under the main software forum, since it's all of PC, embedded, homebrew, commercial.

Here's the issue and goal. A pi comes with, and uses part of, a single 8gb SD flash card. Not the most reliable or capacious storage you could imagine.
It runs (among other things) raspian linux, which was my choice as it's more familiar to me, with a modified LXDE desktop for the gui, which turns out to be slightly slicker than the LXDE for most machines - they really tuned and trimmed this distro - my approval and props to the guys who did it.

But that's too limiting. And as it turns out, I wanted an always-on server for my off-grid household that used nil power, or even net saved me some by automating some things. The real issue is I wanted mySQL, and apache on it along with the rest, and as things rolled out - it turns out you can't just edit my.conf and move where it keeps data so I could move that (expected to grow large) to another device - it wont' start the daemon if you try that, and bugs related to this are over 6 years old (searching on google) and the response seems to be "tough luck, I'm not going to search all that code and figure out where someone assumed the dirname starts with /var/mysql or something. Well, fsck you very much, but I have to make the thing work, darnit.

Here's how - and the one thing I found with google-fu doesn't actually work and assumes some pre-mods or another version of everything that isn't relevant for this pi that I actually own (thanks BillF). Here's that link just in case it helps you, though, as it was a bear to even find.
http://c-mobberley.com/wordpress/index. ... o-usb-hdd/

He had the right idea, more or less, but assumed all you can use is the pi itself, and dd'ing an active mounted filesystem is a bit dangerous to say the least.
And at least my pi had everything in different places anyway - but his methods didn't work for me even when I changed the paths to the right ones for mine.

Preliminary info:
I wanted low power, so I bought a Samsung Evo 120gb SSD, and put it into an external USB case. The drive itself draws 100mw - yup, which turns out to be 1/5 or 1/10th what the sata-usb converter chip in the external box draws....sigh. But it works.

Note: Many, if not most issues with flakey pi's are due to lousy microUSB power supply. I had them just trying to power up the drive. Even without the drive load, just a used dell keyboard and mouse (keyboard has an extra two ports, so it has a USB hub in it) I was seeing only 4.2v at the pi end of the wire, while measuring 5v out of the supply - it's drop in the too-skinny cable and the crap micro-USB connector (which never deserved to live, so now everything is standardizing on it, of course, the one on my Nexus 10 almost melts if you're using the thing while plugged in). There's no substitute for actual metal and square inches of it.

In my case, I solved this by (Thanks Jane) getting a power supply with two outputs, and hardwiring (with fat wire and you know, solder) 5v to the drive as a separate issue. The SSD just wouldn't run right at 4.x volts, it needs the whole supply. The pi doesn't care that much, since it runs on 3.3, 2.5 aqnd 1.8 regulated down from the input, but 5v things on USB - they care, or at least anything cool does.

///////

Down to it.
Turns out the pi SD card is partitioned into about 8 slots. It's kind of gross - some are fat32, some ext4. You need one of them - /boot, still on it and in fstab to make it boot at all (and that one's fat32 for booting linux, go figure). So you can only mount the SSD over /, then mount the /boot on the SD card over that, having copied the SD card to the SSD first, changing fstab and an obscure file on the SD card so it knows where to find the opsys image. And, strangely, cp or a gui copy won't copy virtual files and so on, and pi cares. So you are reduced to dd for the copy, which royally screws up partitioning and other things on the destination. So, instead of what this guy did (which doesn't work even if you supply the correct paths/partitions), I did all of this on another computer, my laptop (lenovo thinkpad, ancient) which handily has a bunch of USB ports as well as an SD card reader.

Steps:

First, you have to partition the SSD drive (or a hard disk if you can afford the power, and take note of the above for even something that draws 1/100th of a real disk). Turns out DD is going to mess that up somewhat and fill the partition, no matter how big you make it. Seems ext4, which is what you want, saves something like "room remaining" instead of "room used" and my first try with the below steps ate the entire drive - for 3.x gb of files. Stupid.

So, to partition the new, big drive, use a partition of small size, I used 6 gb, which probably wasted a few, since only about 3.x are used on the SD card for the entire root filesystem. I used gparted (sudo apt-get install gparted if you don't have it), to do this, and labeled the partition as root (seems important to keep it the same as it is on the SD card). I just left the other 100+ gb unallocated for this step.

Step 2:
Put the SC card from your configured pi (I'd already put a ton of software stuff on mine) into the reader. It should show up in /dev as mmcblk0 and /devmmcblk0p1 (through 7, it has 8 partitions or so). The / filesystem on it is on p7. So use the command line:

sudo dd if=/dev/mmcblk0p7 of=/dev/sdb1 bs=1M

to copy the root filesystem from the pi SD card to the SSD. Case sensitive. cp won't do it, nor will the gui copy functions, every bit needs to come over, so you use the dd chainsaw. In your case the second /dev might be different, on my laptop sda is the laptop drive and we don't want to wipe that out!

Ok, we now have the root filesystem from the SD card on our SSD as it's root file system. There are a couple other changes to make before it will work, and I was super careful and tested with just the small SSD partition before putting it back on the laptop to use gparted to extend it to full size.

One is in /etc/fstab. You have to change the mountpoint for / from /dev/mmcblk0p7 to /dev/sda1 so it uses the SSD. It'll wind up looking like this:
(I use gedit for everything, but any text editor will do here)
Code: Select all
proc            /proc           proc    defaults          0       0
/dev/mmcblk0p6  /boot           vfat    defaults          0       2
/dev/sda1       /               ext4    defaults,noatime  0       1
/dev/mmcblk0p5  /home/pi/public/SD   ext4   defaults,noatime

# a swapfile is not a swap partition, so no using swapon|off from here on, use  dphys-swapfile swap[on|off]  for that


It used to look like this:
Code: Select all
proc            /proc           proc    defaults          0       0
/dev/mmcblk0p6  /boot           vfat    defaults          0       2
/dev/mmcblk0p7  /               ext4    defaults,noatime  0       1
/dev/mmcblk0p5  /home/pi/public/SD   ext4   defaults,noatime
/dev/sda1   /home/pi/public/SSD   ext4   defaults,noatime
# a swapfile is not a swap partition, so no using swapon|off from here on, use  dphys-swapfile swap[on|off]  for that


Just for reference.

Step 3 - have to change one more thing so the pi knows to look for the SSD for the boot image junk. The link above has this right.
Remeber we kept /boot on the SD card? Have to, it's the only thing the firmware boot loader knows to look for.
We have to change the file /boot/cmdline.txt to:

dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 root=/dev/sda1 rootfstype=ext4 elevator=deadline rootwait

Which again, I just did while it was plugged into my laptop (gedit as usual). The only change is the root= part, the rest stays the same.

Now, "safely remove/remount" all the drives - the SD shows up as a bunch, the SSD just one - and plug them into the pi, and power it all up.
Bingo!

Bonus - here's a screenshot of my pi (using scrot which I had to add there, as it's not builtin)
2014-01-12-133201_1824x984_scrot.png
Pi. after all this.


I've added mySQL, apache, scrot, samba, gedit, padre, the arduino IDE and you-name it already. The pi SD would have been enough for all that - there was over 2 gb left; I'd already removed all the "kid stuff" like the python games, scratch, and anything else not "real". Sorry python fans, my visceral reaction to that language is puke, and I can't paste an entire program into my one line edit boxes like I can with perl, since it's so stupid it uses whitespace and indentation for what real syntax is used for in every real language - and it's slow on top. Screw python. You have to keep the language itself, since that's what the interns who write most of the linux sys scripts use, but you don't have to make it worse by using it yourself. It was more the fact that I knew I'd be using tons of space for file serving and data logging, as well as the known lack of reliability of SD cards, that prompted this rather arduous job - remember, I'm not saying all the things I tried that DID NOT WORK, just what did.

Oh, for what it's worth, making backups of everything you change is never a dumb thing to do first. I backed up the SD card via this:
(while plugged into my laptop where there's room)
dd if=/dev/mmcblk0 of=piclone bs=1m

This took around 6 minutes. If you need to go the other way, just switch the if and of filenames, which takes more like 12 minutes - SD cards are slow, and slower yet when writing. There's no progress bar, just wait till the prompt comes back to indicate it's save to unmount the SD card and go about your business.
Posting as just me, not as the forum owner. Everything I say is "in my opinion" and YMMV -- which should go for everyone without saying.
User avatar
Doug Coulter
 
Posts: 2944
Joined: Wed Jul 14, 2010 8:05 pm
Location: Floyd county, VA, USA

Return to Software

Who is online

Users browsing this forum: No registered users and 1 guest