Arduino UNO data aq

For stand-alone microprocessors

Arduino UNO data aq

Postby Doug Coulter » Sat Dec 07, 2013 8:42 pm

Well, someone just had to go and give me an Arudino UNO. And then I felt compelled to do something useful with it. So, I did.
I'm finding it a little strange, due to the freedom/responsibility tradeoffs they used here vs the total control I have with PICs, but I'm getting by, and finding that worst-case, I can just go brute force it to be more flexible in pure C (or asm), if I'm willing to decode enough of the 660 page manual for the atmega 328.

So, I wrote a sketch (their terminology) that acquires all the a/d inputs and uses the one available counter (they grab the others for themselves, more or less) as a counter. Seems to work well as long as I run it off a wall-wart. Even blinking a LED makes enough noise to mess up the A/D, and the amount of noise on any of my machines on the USB 5v (laptop is by far the worst) is a couple a/d bits worth...so, we run from wall-wart.

Here's what it looks like at the moment:
ArduinoDataAQ.png
Screenshot of the thing running after some time.


In the serial monitor, the first number is milliseconds since last boot, the next 6 are a/d channels, and the last is counts since last report. I have it set to run every 100ms (more or less), and have the counter counting a pin I toggle each time through the main loop - so that number is half the number of times loop() was called in 100 ms (10 hz) - as you can see, it varies, timing isn't real deterministic in this in the same sense as my multi-threaded interrupt-drive PIC opsys, but hey, looks like 60-70 khz worth of times we get called per actual time to do something, so I'm not wasting a ton of cycles, and neither are they.

I have channel 1 digitizing a 1.56v alkaline battery, channel 4 to ground, and channel 5 digitizing the on-board 3.3v regulator. The rest are just floating and showing you noise plus the inter-channel leakage that's normal for a floating pin in cmos.

This is, of course, a "work in progress". I can't make this useful in my fusor setup until I have *at least* two counters. I managed one hardware, with interrupts only when the 16 bit counter overflows - that one will take MHz speeds. To get another (and stay out of the way of what they grab for their "opsys"), I'll just have to count external digital interrupts, which if you drive it too fast - might mess up the rest. Stay tuned, more to come on this, with variations (I also ordered an SD card add on I plan to use for standalone data aq with other speeds and feeds for the homestead, for example).

Here's the code as is, of course you have to zip stuff like this or the board won't take it.
DataAQ.zip
Arduino sketch
(2.15 KiB) Downloaded 274 times
Posting as just me, not as the forum owner. Everything I say is "in my opinion" and YMMV -- which should go for everyone without saying.
User avatar
Doug Coulter
 
Posts: 3515
Joined: Wed Jul 14, 2010 7:05 pm
Location: Floyd county, VA, USA

Re: Arduino UNO data aq

Postby Doug Coulter » Mon Dec 23, 2013 3:52 pm

Here's an updated version that is much more useful. In the zip file attached, the DataAQ folder should be placed under your arduino sketchbook folder, it's the code for the UNO and will then show up in the IDE under file/sketchbook.
adaq is a perl program, at the moment, linux-only, that talks to the arduino, and logs and plots the a/d and counter data in a format that a lot of other programs (including some I'm about to (re)write for data mining.
Sadly, it needs a bunch of non-standard perl modules, you'll have to install those using perhaps synaptic or unbuntu software center. Put adaq in your home/bin directory (make one if you haven't already) so it will be on your path next time you log in, then all you have to do is type adaq into a terminal and it will run, probably complaining about each missing module in turn - you can use those error messages - paste the name of the missing module into one of the software source programs and use them to install them. I'm working on an automatic installer....but it's not here yet. I plan to do that after I do the equivalent of "plotdat" I use for data mining my other data aq - 4d plots(!), and I might extend that to 5. I get x,y,z plus color now, and I might add "dot size" as yet another axis for this.

You will need to install gnuplot as well. The version signed off on by the usual linux distros is fine - you don't need to get and build the very latest one (and in fact, I've not tested with higher than v4.4).

There's also a sort-of manual about adaq and the system, in odt format (open office, libre office).

A little tricky to get going that first time, but after that - the price is right and this stuff works well. Consider this a beta, though, as I plan to start burning more cycles in the arduino to do some DSP and improve the effective a/d resolution and other such things pretty soon. We will also be offering built-up hardware - arduino as a loose board isn't that handy, but in a metal box with some BNC connectors and perhaps an SD card, it becomes a very nice thing to have in the lab. I will of course publish the design for that too - it's not rocket science at all, just some RC lowpass filters on the a/d inputs, and protection diodes and R's for that odd bolt of lightning often encountered around my fusor. Getting more bits out of the a/d will probably change the format and effective full-scale values a bit. We can use the UNO 10 bit a/d as the "one bit a/d" in a delta-sigma a/d converter if it turns out we have the time and cycles to get that done, so I am planning on at least trying - it worked out great in the PICs.

Share and enjoy - it's all GPL V2 license. So if you come up with a change that makes things better, share it back to us all.
adaqsuite.zip
Stuff for the arduino and PC for data aq
(267.2 KiB) Downloaded 277 times

Adaq360.png
Looks like this in action...note rolling window. Eg, it fixes the problem of using up all your computer if you take too much data in one run, trying to plot it all once a second, by taking data off the front of the display (it all goes into a disk log regardless).
Posting as just me, not as the forum owner. Everything I say is "in my opinion" and YMMV -- which should go for everyone without saying.
User avatar
Doug Coulter
 
Posts: 3515
Joined: Wed Jul 14, 2010 7:05 pm
Location: Floyd county, VA, USA

Re: Arduino UNO data aq

Postby Doug Coulter » Sat Dec 28, 2013 10:57 am

Still working on the entire suite, some of which hasn't shown up here at all yet. Right now I'm working on a log file viewer (not truly a data miner, though I have the 4d plotter planned), that will let you visually scroll through log files at any "reasonable" zoom factor to see parts of it in detail. Handy perhaps for teasing apart cause and effect, since this is so much faster-sampling than the PIC version happened to be.
There's also quite a bit of software yet to be written to take advantage of those SD cards and the real time of day clock (largely for non fusor applications I have in mind) but it's all coming along. At least I finally got the right light for a pic of the hardware.
adaq.JPG
Prototype and parts for more

That bit of wire hanging out from the SD card slot in the proto is simply hooked to one of the arduino pins I toggle each time around the background loop in the opsys, so I can see how many cycles I have left to work with. I also used it in the demo plots above as a source of counts (it will count half the speed, since I merely invert pin 12 each pass). Notice there's a difference - counter 0 takes almost zero arduino overhead, only doing anything when the hardware 16 bit counter overflows (one line of code to increment another integer that counts those for 32 bit totals). While the other one, labeled C1 has to take an interrupt per count. It shows in the length of time it takes to re-call the basic loop. If you count C0, and add C1, both slow down since the count signal slows down. If you count with C1 and then wire in C0 - no change at all. A machine with this many registers to save/restore has a crap-ton of interrupt overhead per event.

In the interest of keeping it simple for now, the hardware has a fixed sample rate of 10 hz. However, it does look like we have the cycles to go faster - and the ability to go slower as well. The file format just reports sample times since start in milliseconds so if I do the rest of the PC software right, it just won't matter till we try for faster than one ms - which is quite unlikely, as most PC's wouldn't be able to eat the data that quick over USB for starters - that's really the domain for intermittent scope screen captures, where you can sample at ghz if required (and yes, that's in the master plan too - time synced with this in the end in a database).
But most users will no easily be able to set up that level of infrastructure, so this is modular and won't require it all by a long shot. Personally, I don't find setting up something like MySQL, even on a different server than the one this is lashed to, and making it all work much challenge, but then computers have been my game since....lets not talk about that first one with real ferrite core memory and dual triodes, OK?
Posting as just me, not as the forum owner. Everything I say is "in my opinion" and YMMV -- which should go for everyone without saying.
User avatar
Doug Coulter
 
Posts: 3515
Joined: Wed Jul 14, 2010 7:05 pm
Location: Floyd county, VA, USA

Re: Arduino UNO data aq

Postby Starfire » Sun Dec 29, 2013 9:06 pm

Hi Doug - does the photo mean it's on the rocks? :lol:

believe it or not - the first computer I worked with did have 1k core - true ferrite and 1k was considered a large memory.
Starfire
 
Posts: 143
Joined: Thu Aug 05, 2010 4:26 pm
Location: North Ireland

Re: Arduino UNO data aq

Postby Doug Coulter » Mon Dec 30, 2013 9:38 pm

Tee-hee, nope, just my driveway often has the best light for pix. My first machine had only 256 by 13 (bytes were way into the future then) and was really intended as a smart PHA for nuclear work. I still have a nybbles worth of the core I hold up on vid chats to shut noobs up. You can actually *see* each little ferrite ring, and I remember at the time (late '50s or so) we were still amazed how those Japanese girls could thread these 256 or so in a 2.5" square...x,y and Z sense wires...

Then I got to move up - Dad bought me a PDP-8s (12 bits, but all serial in operation), then a "straight 8" (12 bit parallel) which had real (4kx12) memory and was *solid state* - transistors, diodes and stuff. The early ones even had a thing called a DCD gate (diode cap diode) that was an AND gate the inputs didn't have to be exactly on time with - just inside a cycle, kind of a cool construct. I wound up working for DEC later, and that knowledge of the then-obsolete stuff was solid gold there, and elsewhere afterwards.

This project is coming along nicely, I'll have some more soon. I'm working on a couple programs to data-mine the results of this guy, and an automated installer - getting this going can be a pain (esp for a noob) since it needs a bunch of software modules that aren't already on everyone's machine, and it's not just a copy to install all that.

So, 3 or so pieces of software left to go. I'm doing one that just re-plots a run, but with sliding start point and window size so you can "zoom in" on the raw data, as well as that 4d plotter for this format (we have more channels than in the last project, 10 times the data rate and so on) - that might grow to be 5d with the new gnuplot - I can alter the dot size as well as the color now. The installer is almost done - just needs testing on more platforms. It's all gonna be linux though - you can make this stuff fly on windows, but it's such a pain there it's just about not worth it unless you are a true hardcore sysadmin (done it), and even then - less work to do virtual box and linux and do it there. Funny how all that supposed co-operation around standards works out.
Just for starters - linux, windows and mac use different line terminators (\n, \r\n,\r) - that which is easily human-readable in one is garbage in another or all on one line. And windows even gets confused if its \r\n are in the wrong order...

How do you find out what serial ports there are (on USB) and what's on them is utterly different from linux to windows to BSD. And on and on - even if the language is the same (perl in this case), the support programs are the same (gnuplot) and so on. I know it's not worth my while to maintain 3 branches of code for something that's not going to rush out the door and make me rich, I just need a few, and I'm sure a few others are like that - and that's it. If you've gotta have windows or mac - the changes are obvious, but extremely tedious and tend to change with versions, unlike linux (so far). Since I use linux on all my machines here (well, one is android but I only use it as a consumption device, don't even log on here with it) - that's what this will work best on - and versions don't really matter.
Posting as just me, not as the forum owner. Everything I say is "in my opinion" and YMMV -- which should go for everyone without saying.
User avatar
Doug Coulter
 
Posts: 3515
Joined: Wed Jul 14, 2010 7:05 pm
Location: Floyd county, VA, USA

Re: Arduino UNO data aq

Postby Doug Coulter » Fri Jun 20, 2014 11:41 am

As I get ready to take the old data aq out of the fusor system and replace it with this project, well, the nice plot program that could eat the log files needed an upgrade to handle the new (still human readable, which also means you can edit them with a text editor, concatenate them and so on - we still use a # at the start of a line as a comment character that means "ignore the rest of this line") ascii log files, so I modified plotdat and it's now called plotudat. Yes, it's perl, for linux, and assumes you have or can install a few perl modules, though it should run on windows if you get perl and gnuplot and so forth. Put this in your youruser/bin directory. I have some presets still there from the old version, mainly just to make sure that presets work - you'd have to change them to fit your setup (would have had to anyway). Now you can map any of the 6 channels of a/d and 2 counters onto 3 axes plus color, in a scatter plot, from the arduino basic data aq. This is what I use to analyze the data myself, the other stuff helps you just acquire good data, but doesn't and can't know how many a/d counts mean what...this is where you can write a line (or few) of perl code to do that kind of mapping of chnl X to kilovolts and so forth.
You can pretty much put any chunk of perl code into the edit boxes for each axis, since unlike more-modern fad languages, a semicolon ends a line of perl, not a linefeed and tabs, so multiline mapping functions still fit in a single edit box, though you might want to cut/paste them from a "real" editor to get them in there.

ploudat.zip
The plotting/analyzing program
(9.46 KiB) Downloaded 240 times


It looks like this when you get it running. This is fake data, just putting some inputs in and using the old presets - meaningless, but it shows how it should look.
plotudat.gif
How it looks with sparse and fake data
Posting as just me, not as the forum owner. Everything I say is "in my opinion" and YMMV -- which should go for everyone without saying.
User avatar
Doug Coulter
 
Posts: 3515
Joined: Wed Jul 14, 2010 7:05 pm
Location: Floyd county, VA, USA

Re: Arduino UNO data aq

Postby Doug Coulter » Fri Jan 23, 2015 5:58 pm

FWIW, this also does run on a Raspberry Pi...a bit slow, but it does work fine. If anyone hits on me hard enough, I'll come up with an install program that will work on the pi (or most any other linux, I don't use the tweaky stuff in this) that will put in all the dependencies it doesn't already have (all linux have perl itself it seems, so I've done other installs via perl scripts that have all the junk "inside the installer itself" insofar as possible, at least for .deb based linii). Makes for a fairly inexpensive data aq/analysis setup that is at least semi-portable if you add a cheap Chinese buck switcher and a gel cel...I have Adafruit's 10" LED/LCD hdmi display on one of mine, and it's a pretty sweet setup. I am right now putting that in a fancy box for use in homestead automation along with more arduinos - stay tuned.

I guess one of the next steps, that goes a bit past what the normal hobbyist will be willing to setup and admin is shoving this into a database and making a web interface for it all. I just might, as it will be running some things on my homestead, and I'd like historical data as well as realtime control - but with next to zero electricity use.
Posting as just me, not as the forum owner. Everything I say is "in my opinion" and YMMV -- which should go for everyone without saying.
User avatar
Doug Coulter
 
Posts: 3515
Joined: Wed Jul 14, 2010 7:05 pm
Location: Floyd county, VA, USA


Return to Embedded software

Who is online

Users browsing this forum: No registered users and 2 guests

cron