Stop imaging and frying pi SD cards

For PC type software that runs under some PC opsys.

Stop imaging and frying pi SD cards

Postby Doug Coulter » Fri Mar 11, 2016 1:03 pm

I guess I'm starting to call Raspberry Pis PCs at this point, because they kind of are for a lot of uses. Pre-emptive opsys (linux), pretty general purpose, the newer ones can even be your desktop for light duty - or a NAS, a router, a firewall and so on, some people (not I) even use them for set-top boxes or light gaming.

A fairly major concern with these guys is that they run from a micro SD card, and flash - especially the cheap stuff - is flaky, SD cards being the flakiest since they lack the more sophisticated internal CPU power for bad blocks, wear leveling and so forth (they have some...but not like an SSD or even USB stick). For some good nerdy entertainment on how messed up flash is - note some of the write-life and read-disturb numbers here - enjoy this:
If you know opsys, you know that if there wasn't wear leveling going on (and taking time), a single boot of many opsys will kill a sector dead it gets beat on so hard. That's why there's a noatime option in fstab...helps a little bit.

Imaging an SD card once debian is installed, especially a large one (16 GB for example, though they can get a lot larger) - is a time consuming pain for either backing up or restoring, and you don't get much if any warning when it's time to make a new one. If you're picky like I am, you've spent tons of time configuring the linux to be just so - in my case, ripping out all the "kiddie crap", installing and configuring things like a real text editor with syntax highlighting, an SQL server, a web server, a real package manager (discoverability in apt-get is near zero - synaptic is faster despite how slow it is if you don't know something's exact name) and so forth (it's a long list - modules, libraries, apps). Once done, one doesn't like to have to do this over and over. Imaging a 16 GB card with dd takes 16 GB whether most of it is in use or not - and one heck of a long time. Compressing it, via zip or bz2 takes one heck of a long time even on a pretty fast normal desktop too, and you still have 1.x GB to store if you're lucky and most of the SD card is still "all ones" so it compresses decently (in other words, if you haven't used it much yet and exposed it to wear leveling etc).

I got sick of doing this and the time and space it wasted. I came up with the following procedure (which some helpful soul might script up for us, taking both directions into account). You can do whatever customization either before or after this, but obviously down the road, you're going to be backing up "as configured by you" for possible later restoration.

First, have a working SD card. In the below, substitute x with your actual drive letter. (I use gparted to find this, there are lots of ways)

1. Make a backup just in case. Thank heavens, only one is needed:
Code: Select all
sudo dd if=/dev/sdx of=BackupFilePath.img bs=4M

This is just in case something goes wrong when we rewrite the card, you may not need it and can skip this if you like living dangerously. To restore this image, should you need or want to, use:
Code: Select all
sudo dd if=BackupFilePath.img of=/dev/sdx bs=4M

Note that any time you write to an external storage device from linux, you should type
Code: Select all
, and then use whatever procedure your distro provides to unmount the storage device before actually unplugging it. I'm just saying this once here, but it's a good idea every time.
Both the the above simply copy the physical media bit by bit till there's nothing left to copy. Boot records, partition tables, the works. If it worked before, it'll work again, no other steps required, but this is slower than molasses in sub zero temperatures.
If all you wanted to do was backup your SD and then run from it, you are done. Don't forget that this needs updating regularly to not lose your work...and it takes forever and uses huge amounts of space.

To do my scheme, with your SD card in your desktop (eg not mounted and running in a pi) do:
Code: Select all
sudo dd if=/dev/sdx of=PiMBR.img bs=512 count=1

This just gets the master boot record and the partition table off the card intact.
The card will have (among a bunch of others) a directory called /boot, and this one will be the sole content of that first fat16 partition. This is all the initial bootup junk for the pi. After mounting the card on your desktop, you can simply copy this stuff (fat16 doesn't understand permissions) by any of the usual means - a file browser with ctrl-a ctrl-c and then ctrl-v into some directory on your desktop to hold it all works fine for example, as do any of the command line copies. While the fat16 partition is 60MB, most of it is unused so this isn't slow to do.

You now have on your desktop machine a copy of the MBR and all the boot stuff required to get a pi going "at all", and this is all we will wind up putting back onto either the original or a new SD card for real world use. This is fast, and it seems not to require re-doing for most minor changes and updates to the main linux itself, which is kinda cool. After all, it's just the part that is smart enough to read in the rest and do it. Bonus is you are no longer using or wearing out your SD card other than for half a second at boot time, and it's now wildly over provisioned against bad blocks anyway.
To restore this part, you do:
Code: Select all
sudo dd if=PiMBR.img of=/dev/sdx

which is very fast. Then format the fat16 partition. I use gparted, there's a command line way of doing this too. While I'm at it, I generally delete the other ext4 partition, else it tends to mount itself somewhere like /media/pi/ upon boot and be confusing, waste time, etc. I suppose you could blank it out, reformat the unused space, and mount it somewhere on purpose (via /etc/fstab) and use it for spare storage in a pinch, but that would be counter to the idea here.
After formatting the fat16 partition (mmcblkp01 in pi's fstab) - copy the contents of /boot you saved earlier back into it.
Once you have the images on your desktop to do this, you can regenerate SD cards in much less time than it took to read this, much less type it.

The rest:
mmcblkp02 on the SD card has the rest - including a spurious copy of the contents of /boot. You need the directory to be present, but the contents is an oops on their part, as /etc/fstab immediately mounts /boot from mmcblkp01 over it and hides that stuff - it's just wasted bits and caused us a little confusion at first. You can safely erase it if you like (on the second partition) but the /boot directory - empty at least so the first partition can be mounted over it - has to be there.

I copied "the rest" via command line to a directory on my desktop, lets call it "./piroot" for example.
Here's the line:
Code: Select all
sudo cp -rp /dev/sdx2/* /piroot

Which takes awhile (and that * has to be right or it will forget to structure the directories properly), but still far faster than the dd did, as it only copies stuff that's actually there. Zips down smaller and faster too if you're into saving space on your backup machine.

Now for the tricky parts. Despite /etc/fstab specifying what goes where *after* a boot, there's a file in the /boot (fat16 partition) directory called cmdline.txt.
During the boot process, long before /etc/fstab is read (or it knows how to read ext4, even), this is consulted to find out where to mount root (/) in the first place.
It will currently say something like root=/dev/mmcplkp02, which will need to be changed before you can restore and reboot, or nothing will work.

Let's say you're going to put a huge USB stick or even a laptop hard drive out there on USB. It'll be /dev/sda unless you're getting fancy enough to not need this post. Assuming we're going for simple here, you will format it as one big partition and as ext4, and that will be called /dev/sda1. Change both cmdline.txt to reflect this, and also /etc/fstab on your "new big safe" media (change mmcblkp02 to /dev/sda1) before plugging it all together and rebooting your pi, and bingo - a real machine, reasonably safe storage, fast...

Some notes:
A big USB "extreme" stick from sandisk is faster than any SD card. Yes, the Pi is only USB2 but that's not the issue - it's read-write latency. Even in a regular USB3 stick (Cruzer or Fit) that's the limiting factor, and most USB2 sticks are dog-slow when used for this.

Surprisingly, I've had very fast results with a 2tb samsung/seagate laptop drive, only 5400 rpm, but it has a ginormous cache ram on it that covers that up. I use a Star Tech USB3<>SATA cable and the high power hackaday hack for this, though with this particular drive, you don't even need the latter. I'm using this rig on both the main fusor controller, and on a NAS on my lan. Silent, low enough, great. A 2tb NAS that comes in under 5 watts? I'm onboard (and off-grid).

Either of the above boot a little faster than a class 10 SD card, last longer...easier to back up (rsync is your friend) and so on. Not trusting things like virtual files to sit still or on-drive caching, I tend to do these things with the pi off and on a card reader or USB jack on another machine (in my case, some version of Intel's NUC most often - the silence makes my background music sound better).

The high power hack involves removing r4 and r50 and substituting a single 10k resistor to ground from the USB power chip. That particular SMD resistors I had in stock cover both of the original footprints, which is just fine. One of the original R's was wired to the collector of a transistor that could switch to ground from config.txt to give more, but still not up to spec, power. "Accidentally" shorting this collector to ground doing this mod has no effect whatever as long as that end of the new resistor IS ground. I had to scrape off a little solder mask to make that happen with the larger footprint resistor.
Posting as just me, not as the forum owner. Everything I say is "in my opinion" and YMMV -- which should go for everyone without saying.
User avatar
Doug Coulter
Posts: 3005
Joined: Wed Jul 14, 2010 7:05 pm
Location: Floyd county, VA, USA

Return to PC

Who is online

Users browsing this forum: No registered users and 1 guest