Raspi camera frame skipping with USB disk

Software related topics

Raspi camera frame skipping with USB disk

Postby Doug Coulter » Sat Sep 16, 2017 11:43 am

I've had an issue with one of my pi cameras skipping frames on complex input (a fast 4 channel scope on a big display), when using a big (128gig) USB3 stick mounted over /, and therefore also being where the media goes. Troubleshooting this is somewhat complex as it's hard to get inside the (not all open source) code to figure out who does what and when. There's even code on the camera module that's free running doing mjpegs at whatever frame rate it decides that can bog down regardless of the response time of the pi stuff, for example.

Well, it turns out that even though a big USB stick on a pi is actually faster than any SD card for most things I've measured, it has the occasional latency issue - a bad one. Admittedly, this is worst case, as in 4 very quickly changing traces "full FOV", trying to be compressed in realtime and written down, evidently with little to no buffering..the net result being frame skipping - sometimes one, but more often 1-3 SECONDS of just plain lost video.

This mattered hugely to our data aq, as this isn't the only realtime data source, and we wanted to be able to go N seconds into a run and see the scope, the grid, hear the audio, look at the numbers for the various other things we're acquiring and so on - and by the nature of this beast, there's no timestamp inside the video stream (or the audio) and we have to depend on them staying in sync once started that way - we have a "clapboard" led signal that shows up in all the streams just after they are started so we can sync the begin times.

After trying a new, faster Pi, and a newer camera (going from version 1.3 to 2) and it not getting any better, in desperation we mounted a "real" SSD (only a Samsung 840 evo, and yes, I know about those issues) over the /www/html/media directory to take the videos.

Problem solved. I haven't tested to full frame rates again; I'd cut down the frame rate thinking that was part of the issue but that had zero effect...but this seems like the real deal. We were losing 3-30 seconds worth of a 5-6 minute run and now...none at all.

I am guessing that the USB drive "went off on some demented errand of its own" to load level for awhile, and I kind of doubt it matters what brand, they all do something like that and have a cpu in them just for that: http://www.bunniestudios.com/blog/?p=3554 There could be a difference in manufacturer's code for this and how well they interleave that with data demands, dunno. Both were samsung in this case...

This is somewhat related to the post about adding a big drive to a Pi, in this case, yet another one. The new one won't fit inside the EMI tight box, but I might make a new box if that starts to matter.
Posting as just me, not as the forum owner. Everything I say is "in my opinion" and YMMV -- which should go for everyone without saying.
User avatar
Doug Coulter
 
Posts: 2988
Joined: Wed Jul 14, 2010 8:05 pm
Location: Floyd county, VA, USA

Re: Raspi camera frame skipping with USB disk

Postby Donovan Ready » Sat Sep 16, 2017 4:02 pm

Did you already post that link to bunniestudios here somewhere, or did I find it myself? Either way, a damned good article.
Image Donovan
Donovan Ready
 
Posts: 132
Joined: Thu Apr 17, 2014 2:22 pm
Location: Austin, Texas

Re: Raspi camera frame skipping with USB disk

Postby Doug Coulter » Sun Sep 17, 2017 11:05 am

I probably have posted that classic a time or two - it's worth the bits, as you say.

The implications are far more "interesting" than most take away from that. No usb device whatever can be trusted to be what it says it is. Shortly after that talk, the usb rubber ducky came out...
The protocol is broken and can't be fixed without breaking all reverse comparability, or maybe even "at all, ever".

If the computer has to trust that a USB thing is what it says it is, well...if the thing changes its mind and "re-hot-plugs" itself, all bets are off.
Even if a computer only trusts things that can do some kind of crypto key challenge and response, then it only works till the key is cracked by someone (DeCSS comes to mind).
That whole web of trust thing is showing itself to be chock full of holes, no matter the mechanism that's been tried so far.
Certs have been stolen (Stuxnet). The DHS's luggage keys were scanned, posted online, and 3d printed in days. The government wants a backdoor to crypto because side-channel attacks are "too hard" and obviously learning how to use the pen test tools in Parrot takes a few days and some brains - which they are kind of short on. Their track record of keeping secrets ain't so hot either...(OPM). How do I opt out of equifax, anyway? /rant1

Bunnie and Xobs are very smart guys...but I had guys that smart working for me at C-Lab and it wouldn't have taken us anywhere near that long to do what they did.
Any of us would have recognized almost any assembly language from the bits immediately, for example (OK, there weren't as many architectures then, but 80xx?).
We wouldn't have bothered to make our problem a nail that fit our hammer - we didn't have just one hammer.

This "had to be a state actor" propaganda stuff gets under my skin really badly in the security biz. As if the state had the only smart guys working for them...for starters. Having been one of those guys, let me tell you, the smart ones quit pretty fast. The resources of a couple people for a couple days do not require a "state actor" to put together - unless that lie serves some other agenda.

Intel might promise never to sell my email and other info - but that has no effect on a rogue employee with a backup tape who is short of cash, eh? (This happened to me and I have documentation via "canary mis-spelling of information"). /rant2

Back on topic, I wonder how well the old samsung evo 840 will hold up - that one hadn't been severely beaten on (it was a backup on a wifi share), and the issues with those happened after a lot of use. I did get it to 20 fps yesterday with no issues, that really seems like it was the problem. It's just up there on a stick that holds the camera box and attached with duct tape for now...
Posting as just me, not as the forum owner. Everything I say is "in my opinion" and YMMV -- which should go for everyone without saying.
User avatar
Doug Coulter
 
Posts: 2988
Joined: Wed Jul 14, 2010 8:05 pm
Location: Floyd county, VA, USA


Return to Software

Who is online

Users browsing this forum: No registered users and 1 guest