A nice little open-source data acquisition package

This is bound to get mixed up with things in Electronics, check both. Physics-specific stuff here, mostly.

A nice little open-source data acquisition package

Postby Doug Coulter » Fri Apr 11, 2014 12:44 pm

It's been important to take decently real-time and reasonably accurate data from the fusor here, for later analysis - it's yielded a lot of great insights. We've used a couple of systems, one based on our standard counter board, and now one based on an arduino - so even the hardware is open source!

Well, there are a few little things you should add to a stock arduino for this to work best, like a box, some BNC connectors, a 2.2k resistor and diodes to the rails for the counter inputs to protect them, and a lowpass filter consisting of a 47k series R and a .1 or so uf ceramic cap inside the box for the A/D inputs.

This outputs data from all 6 A/D channels and the two counters at 10 hz, along with the Arduino's idea of time since start in milliseconds. The perl script knows how to talk to the Arduino, and plot and log the results (in real time, and with a sliding window so a relatively weak PC or even raspberry pi can keep up, though for the pi the window has to be much shorter - 3-5 seconds), with A/D normalized to volts and counts normalized to CPM. The log file will show up named "$date-time".log wherever adaq (the perl program) was run from, more or less guaranteeing unique log file names and saving you some work. The plots aren't further normalized as we do that in another plot program. This is just so you can see if you're getting good data in real time during a run, and so all the traces fit on the same plot - if we had scaled to kilovolts and milliamps...you'd have to have a bit excessive resolution in your monitor to see both on-screen, eh?

For the perl script to run, you'll have to ensure it's executable, and add a few perl modules (I use CPAN as the source of them). You'll also have to install Gnuplot.
For linux machines, this is all relatively easy to do, for windows you have to add the GTK runtime and a few other things as well. I reccomend just running adaq from a terminal - it will give errors till you have all the modules you need, and tell you which one it was missing when it aborted interpretation/compilation into perl's intermediate code. As often as not, pasting that name into some "software center" in linux will get you a list of similar things - get the perl module and all is well.

What we are actually doing internally:
The A/D is actually sampling all 6 channels every 6 ms or so, but simply adding all the results, so you get a 14 bit number from a 10 bit a/d. Experience has shown it's pretty close to really being that good, but maybe 12-13 bits depending somewhat on the signal. These grabs are time-spaced out in the 100 ms recording interval.

C0 is a hardware counter, it's fast, no issues. C1 is an interrupt driven software counter so don't try to put mhz signals into that one - it eats up the arduino's CPU.
Both overflow at 2^32 counts, and are never reset unless an explicit reset command is sent. This way we never can lose a count while reading the counter...and 4 billion counts roughly "ought to be enough for anyone". Anyone who lives through 4 billion geiger/neutron counts at their location in a single run, I'd like to meet.

The arduino communicates over USB and pretends to be a 115200 baud serial port. A serial port emulator works fine as 8n1-115200 with this. Sending the arduino a '!' (without the quotes) resets it. Sending it an 'i' tells it to ID itself.

We're using any line with a # starting it as a non-data line. The PC might want to add data, filenames, constant etc and can do so in the log file by using the # at the first column as a "comment" indicator, and you can later parse that kind of thing anyway you like. My other plotting software does that already. Other than that, the format is pretty simple - time in milliseconds since boot, A0-A5, C0-C1 and a linefeed (this IS linux, you windows guys can make that \r\n in the perl, but won't have to - perl is smart enough to do that). On the other hand, the arduino isn't, so if you want windows, you have to add the ridiculous extra character in the output print.
And they have to be in the right order for windows. Funny, linux works either way, but Microsoft claims they want to be compatable with everyone. Apple used to, but they use just the /r for a line ending just so no one is compatable with anyone.

I'll assume for now you know how to read code some in the arduino IDE, or use some nice syntax highlighting editor for the perl part (I use either gedit or Padre depending on my mood there). If there are questions, fire away. I built mine off a cheap Seeduino copy ($8 shipped from China) and put it in a die-cast box, cutting out some holes and slots on my mill. Remember, around a fusor there can be plenty of noise, and it can get well past the point of a/d errors - it can fry things. A good box is paramount for this work. A big arc can fry a PC from 10 feet away, not even connected up - the mouse or keyboard cables can be a good enough antenna to let the smoke out of the mobo. I'm on #3 for PC's next to the fusor, about 1 a year, from that, and I route cables carefully and use ferrite common mode noise chokes on them. Once I learned to shield 100% of the HV wiring, I didn't fry PC's anymore, but you can't ever manage that perfectly if there's say, a window to look at the fusion, probes etc, and various leakage right through copper screen.

Note the perl script finds *any* arduino connected to the box. You can make that more specific if you like, I'm just looking (linux) in /dev/serial/by-id/ for anything that has "arduino" in the name for that.

Here are some pix. As usual, click the pic for a large version.
adaqTop.JPG
Here's how I made mine look.

adaqFront.JPG
Yes, this implies I might someday write code to use the SD card/clock shield. But it might be too slow even for this data rate...


adaq.zip
The software, arduino sketch and perl script as well as a serial port config file you should probably erase (it will make another anyway)
(12.32 KiB) Downloaded 413 times


Here's a screenshot with me fooling around with the inputs. I connect the counters (sloppy at first with a wire off a clip lead) to a signal that counts once per two passes of arduino's "loop()" function - note that C1 doesn't count quite as fast - it steals interrupt cycles when a count happens. I later connected a channel to a 1.588v (fairly fresh) AA cell to calibrate things...I tested that voltage with a really good couple of voltmeters. Turns out my particular Arduino's 5v is more like 5.0x volts. That's running off a wall wart. If you run off the PC 5v from USB, you'll have tons of errors as the PC 5v isn't that great as an A/D reference...
adaq.png
Step after 80 secs is connecting an AD channel to an alkaline battery for voltage ref.
Posting as just me, not as the forum owner. Everything I say is "in my opinion" and YMMV -- which should go for everyone without saying.
User avatar
Doug Coulter
 
Posts: 3515
Joined: Wed Jul 14, 2010 7:05 pm
Location: Floyd county, VA, USA

Re: A nice little open-source data acquisition package

Postby Doug Coulter » Wed May 07, 2014 4:17 pm

Software upgrades for the PC side of things (or windows if you're a masochist and want to install perl, GTK2 and a ton of other junk linux comes with).
UdataAcq.zip
Newer versions for the PC - works on 64 bit and latest kernal
(18.45 KiB) Downloaded 415 times



I renamed adaq and plotdat to udaq and plotudat to reflect the arduino Uno name and to keep it separate from my pic stuff that has the old names.

I've not tested this on every computer I own yet - that will take awhile, but the two that matter it works on just fine - ubuntu 10.04/32 and Mint/mate/petra 64 on an intel NUC.

I found out that in the 64 bit versions of linux, the Glade UI designer will not open the glade file we use to create the GUIs here (it's embedded in the perl script, but you can copy-paste it out to a text editor and save it for editing with Glade, then reverse the process, to keep the perl stuff self-contained as possible).

The new GTK2 version of Glade for 64 segfaults instantly when our file is opened. The GTK3 version, well, I have some machines that don't have GTK3 (and liikely never will) and it's a totally different format - and one that the perl module, GladeXML, won't eat as of yet. (You'll probably need that module to run either of these).

Included this time is my really cool 4d plot software for this log file format as well, with some example presets. Just unzip this into a directory under home called /bin, and the next time you log in, /bin will be on your path and you can type udaq at a terminal to get data aq going, or plotudat to make those cool multi-dimensional plots. There's a hidden file for presets - it comes in the .zip with a couple of examples already - change them as you please, and you can save them under their current names or new ones. You will of course need Gnuplot (the program) no matter which of these you are using - I didn't write that. I have tested these with the last few versions of Gnuplot (distros differ on what they think is stable), no issues yet. We depend on it a lot for things like autoscaling and general plot stuff - this wheel is round and finely polished, so no need to roll our own on this one. We just change the data into a form that Gnuplot likes to eat.

The arduino code hasn't changed, do I didn't include it this time. It appears to be rock-solid and there's just no reason to change it ever again. Like most linux tools, it just does one thing, and tries to do it perfectly. In this case, that's way-oversample 6 a/d inputs (0-5v) and two counters, one hardware and one software, either of which can take fast count rates. That's all it does, and spits out the answers every 100 ms. In the case of the a/d, since I take quite a few samples per channel per 100ms, I simply add them all up to output, kind of a sloppy FIR filter for oversampling, delta-sigma style. It seems to give us a couple more bits of net accuracy, with the noise that's always present acting like a "dither" source for those in the DSP know.

There is a window in the data plotting software for you to type in volts/count - the number was typical for my arduino with its wall wart. Since the arduino uses its 5v as the a/d reference, this is the suggested configuration - it will be less than 5v off most USB supplies, and worse - it'll wander around with contact resistance, load on your computer, and so on. Rather than just waste a channel on a/d converting a reference voltage, I decided to go this way this time. Once you know the right number for YOUR arduino's regulator, type it in and save the preset...it'll remember it for you from then on in that preset in plotudat. It's "close enough" for udaq, since all we're really doing there with plotting is ensuring we're not at zero or clipping on the a/d, so I didn't provide for changing the number outside of editing the code (which is easy and obvious anyway, at least for me). That's all that really matter when the neutrons are flying - "am I getting good data?".
Obiouvsly, this is all TTL type levels, so whatever you feed in needs to stay within 0-5v more or less, or you risk damaging things. There's a little input protection...

I hope I can come up with a better solution on the glade issue - the xml that defines the GUI is rather a large pain to edit with a normal editor rather than one that actually draws the windows and buttons and so forth, and lets you edit all manner of properties. It appears that at present you'd better keep a 32 bit machine with glade-gtk2 around if you want to use this otherwise pretty cool lashup. Kind of reminds me of JAVA - write once, debug everywhere. So far there isn't a perl library that can do this stuff with GTK3 (uptake of gnome 3 has been abysmal due to well, it sucks) and I'm not holding my breath. We'll just have to wait and see, in the meanwhile, this GTK2+gnome2 stuff seems to work on everything (even, ugh, unity).

Just a reminder to those who aren't already experts ...
Run these from a terminal first. Look at any error messages like "can't find xxxxx" and then go get xxxxx and try again till it works. Nope, there's no easy way for me to give you an installer without either spending a couple weeks writing perl module installers and so on (there's cpan for that) or paying for a real expensive install program bundle. I don't have time for that. Anyone who can't do this, probably doesn't know what to do with data aq in the first place. Once all dependencies are happy - you can in nearly all linux flavors create a gui launcher, and in the better ones, put it on the menubar...from then on, no issues, though you can use the "run in terminal" if you like to see any other errors that may pop up from time to time. That's how I do it.
Posting as just me, not as the forum owner. Everything I say is "in my opinion" and YMMV -- which should go for everyone without saying.
User avatar
Doug Coulter
 
Posts: 3515
Joined: Wed Jul 14, 2010 7:05 pm
Location: Floyd county, VA, USA

Re: A nice little open-source data acquisition package

Postby Donovan Ready » Wed May 07, 2014 10:03 pm

Damn!

Kind of reminds me of JAVA - write once, debug everywhere


I actually choked on that. (Red beans and rice..) :mrgreen:
Donovan Ready
 
Posts: 239
Joined: Thu Apr 17, 2014 1:22 pm
Location: Austin, Texas

Re: A nice little open-source data acquisition package

Postby Doug Coulter » Fri May 09, 2014 5:16 pm

As far as I can tell, there's no such thing as platform independent code if any significant and good job is to be done - at/for anything at all. Our motto has always been "code into the platform strengths so you can make the customer go "wow, I didn't know my computer could DO this!". They paid for the cycles, wasted (adaptation layers or browser when a native app is more efficient?), used usefully, or just not used. Why not give them good value?

Thing is, I quit doing windows a long time ago, roughly when I stopped getting paid to do it - it's just too expensive for a full MSDN subscription now (and less, is, well, less in this case), compared to free everything on linux. Sure, I don't have tools as cool as pre-.NET devstudio here, but they work and are solid.
Posting as just me, not as the forum owner. Everything I say is "in my opinion" and YMMV -- which should go for everyone without saying.
User avatar
Doug Coulter
 
Posts: 3515
Joined: Wed Jul 14, 2010 7:05 pm
Location: Floyd county, VA, USA

Re: A nice little open-source data acquisition package

Postby Donovan Ready » Fri May 09, 2014 9:57 pm

You know, I probably have all my old CDs from IBM OS/2, BeOS, System V, Microsoft and Novell disks of various flavors and license seats... Gathering dust. I threw away more stuff last move than I kept, but there's still pounds of media. Is that a unit? :mrgreen:
Donovan Ready
 
Posts: 239
Joined: Thu Apr 17, 2014 1:22 pm
Location: Austin, Texas

Re: A nice little open-source data acquisition package

Postby solar_dave » Sat May 10, 2014 11:22 am

The closet thing I have actually ran into as fully portable is ANSI 'C' code. Truly written in ANSI it should port seamlessly.

I wrote some stuff a few years ago that sniffed a UDP port for a set of streaming data (Tibco buss) and it ported pretty easily to various OS, The only one that would keep up however was the Linux kernel 2.6 implementation.
Dave Shiels

My TED 5000 power monitoring
http://phx-solar.no-ip.info:8081/Footprints.html
solar_dave
 
Posts: 108
Joined: Tue Aug 21, 2012 12:33 pm
Location: Phoenix, AZ

Re: A nice little open-source data acquisition package

Postby Doug Coulter » Sat May 10, 2014 3:21 pm

You just got lucky. That's because most opsys use the old BSD TCP/IP stack verbatim, and therefore use the same C calling conventions etc.
( We could have an interesting debate about people all grabbing free code because it's free and thereby producing a mono-culture that is dangerous, as HeartBleed demonstrates)

Now, try and draw on the screen (not a terminal, the GUI). We using MFC or win32.h api (or whatever they've changed it to since my last time on the metal there) straight up, or X11 or what? How am I informed about events like keystrokes and mouse clicks?

Try talking to a serial port. In wndows we have com1 and so on. In linux we have /dev/ttyS0 and so on (and a lot of other much cooler stuff like I'm using here -
I can find arduinos by looking under /dev/serial/by-id/ and looking for virtual file names that have "arduino" in them. Plug one into a windows box and it will be randomly assigned a com port - not the same one if you unplug it and plug it right back in, and there's no easy way to tell what is where. Sometimes you can munge around in the registry (which isn't even compatible between windows versions - and obviously not in linux or macs at all) and figure it out - often having to search the entire thing (that's a multimegabyte search), as the device you're looking for is a value, not a key...this one came up when we tried to make the first data aq support windows as well as linux (assuming the user is a masochist and wants to install some version of perl - with some non standard modules, GTK, and you name it, Just about nothing I/O related works the same.

In short, different GUI libs, or almost any i/o beyond POSIX (which windows supports, but only kinda-sorta) - it's not compatible. Java and Perl seem to get closest, but often require a lot of help when a GUI or real hard I/O is involved.

None of them seem to have the same threading/process modes, quite, either, for straight C.

While opsys vendors sometimes make noises about compatibility and working together (remember the lawsuit in EU over Microsofts SMB file sharing?) in fact, they could try harder to not inter-operate. **IX uses /n for a newline, Mac uses /r, and windows uses /r/n and they have to be in that order, for just one trivial example (at least perl does this automatically, but even then, can get you in trouble - no issues with my system talking to my scope, which itself uses /n for newline, but if my perl was running on windows, I'd explicitly have to send hex 0A, else perl would translate my /n to the scope to /r/n on windows, and to /r on Macs.

Which is where we got the "write once, debug everywhere" joke re Java as they were the loudest in saying "write once, runs everywhere" compared to the rest. Not that others didn't make the claim, or somewhat of an attempt (per'ls not bad in that regard) - but...there's still nothing worthwhile that's truly portable at this time. sure, "hello world" ports fine most of the time.
Now add I/O, threads, talk to hardware and the screen, mouse events - compatibility is essentially zero. And then there are the libraries, without which you can't do a heck of a lot unless you like re-inventing the wheel...Java or perl win - but it's an ugly win.

Personally, I like perl as a better glue language, as what I'm mostly doing on PC's these days amounts to duct-taping other big programs together, for which perl rules the roost. It's also faster than Java (no, really, even still), and can be far closer to hard real-time due to the way its garbage collection works vs Javas, which can go off on its own and none of your code get to run for quite some time while it works. In perl, you have control over when it happens, though that's not obvious.

Personally, I dislike distrusting the programmer to handle memory allocation/pointers. It was quite the belly laugh when Java first came out - no pointers, no new/delete or malloc/free. Because in java, everything is a reference (eg, a pointer). Per'ls pretty close there (but at least references are strong-typed, the only typing almost there is in perl) and you can't cast them.
Both of those languages use a ton more memory at runtime, so never really can satisfy my goal of "wow, I didn't realize my computer could DO this" fully - a scalar in perl takes 28 or more bytes...Java's objects (which include things like integers) also have at least a pointer indirection to point to the integer "class" vtable - wasting cycles.

Now this is just philosophy in most cases. But mine is, stick with simpler languages the more important and fast and lean the problem dictates it is. Languages written so monkeys can drag-drop code into existence so far mainly create monkey-code - it's gotten really bad on windows since .NET, but it's everywhere. It's my opinion (rarely humble) that if you're not willing to learn how to code, you shouldn't code. Or, knowing the entire dictionary doesn't make one Shakespeare.

Now, when it comes to pure number crunching, C wins over just about everything, despite the current debate on Ars and Slashdot re fortran - C's had an "can this be aliased" compiler optimization switch for as long as I can remember, on as many platforms as I've ever used (lots). Heck, you can even program something that looks a heck of a lot like C for graphics cards.

At any rate, I hope this project has been useful to people who like to grab data and put it into a PC. The Arduino format is dead-simple so you could, of course, write your own code to eat it with any opsys or language that will let you have serial port access. Heck, I even use a serial port terminal emulator sometimes (many will let you save the whole session to a file too and it comes out in the same format our other software wants when you do) - it's all ascii, human readable, tab and /n (at EOL) delimited.

KISS all the way. The arduino has exactly one command you can send. ! resets it. That's it. You can also toggle DTR and get the same result with no commands at all - this happens every time you connect to it anyway.

I haven't yet done anything with the SD card shield on the Arduino hardware - it's totally not used so far. Any thoughts on what might be useful there? The clock module (Dallas) on the Chinese ones is so far out of time (2 min a day +/-) I don't think there's a use for it as it sits - maybe buying a good american-chinese xtal for it would fix it...they probably used a parallel resonant one for a series resonant circuit or vice versa there - it's just that far off (the ones I got are all too fast).
Posting as just me, not as the forum owner. Everything I say is "in my opinion" and YMMV -- which should go for everyone without saying.
User avatar
Doug Coulter
 
Posts: 3515
Joined: Wed Jul 14, 2010 7:05 pm
Location: Floyd county, VA, USA

Re: A nice little open-source data acquisition package

Postby solar_dave » Sat May 10, 2014 8:57 pm

Oh I agree, one you touch the outside world it is almost impossible to get a smooth port. But then again, it won't be in an ANSI 'c' condition if you do! LOL

Yes my application was a trend analysis for application date that had a "standard" for the data format. (not well adhered to BTW causing all sorts of exception branched in the filter/parse code).
ANSI C was the only way to attempt the performance needed. I don't remember the exact specs now but is was doing well over a million transactions a minute across 6 quad core machines. It built a hashed array with about 250K records that could be updated in a maximum of 2 steps on 99+% of the hits. It recorded transaction start and end, status and several other values in the records. It reported issues in very real near time, so fast that before the sessions were closed some of the notifications were already sent. It is really reminiscent of wall street types of data feeds.
Dave Shiels

My TED 5000 power monitoring
http://phx-solar.no-ip.info:8081/Footprints.html
solar_dave
 
Posts: 108
Joined: Tue Aug 21, 2012 12:33 pm
Location: Phoenix, AZ

Re: A nice little open-source data acquisition package

Postby Doug Coulter » Sun May 11, 2014 11:39 am

Or fusor data feeds...PCs kind of suck at fast realtime things with deadlines. I remember when an entire company I consulted for had to standardize on no more then 19200 baud for serial because anything faster would overflow the ability of a pentium 3 to handle interrupts fast enough to keep the 16 byte fifos in the uarts from overflowing - sure, they'd handle short bursts of higher speeds, but not if you overflowed that hardware fifo. That's why I use the uP stuff to do that, buffer some data (with internal time-stamping so I still know when it happened), so when the PC's next time-slice comes along, I won't have lost anything. And yes, sockets probably work far better than USB, since they tend to have larger buffers in the drivers, but there's this tradeoff - it takes a lot more of a uP to do ethernet (~$70 vs ~$8 if you just buy the boards), and often you lose the time-determinism for things not interrupt-driven in the uP when you add the TCP/IP stack, which gets run in the "endless loop" of the "opsys" - those in quotes because neither is quite right, but it's what they get called.
Posting as just me, not as the forum owner. Everything I say is "in my opinion" and YMMV -- which should go for everyone without saying.
User avatar
Doug Coulter
 
Posts: 3515
Joined: Wed Jul 14, 2010 7:05 pm
Location: Floyd county, VA, USA

Re: A nice little open-source data acquisition package

Postby solar_dave » Sun May 11, 2014 2:20 pm

Well at least the time source of the transactions came from the source of the data, several thousand machines all time sync'd to the same source. I know there is a fair amount of drift from real time but they were all within a second or 2 of each other. I know cause we had monitoring on the time statistics on each source machine to limit the drift to 2 sec.

BTW the professional coders told me it could not be done, that is why I had to do it myself. Of course they wanted to use C++ or Java for the processing. The sample code they came up with leaked so much memory the data machines would grind to a halt in a couple hours of running. They then decided to use a sampling methodology to get the data.

Funny part of it was we put some monitoring software in to look at memory consumption on all the major code in the business. There is some good IBM code (acquired from Rational) called Purify that is great at finding such issues, of course they refused to buy such tools, instead they just bought more iron, ran them under load-balancers and expanded horizontally until the leaks were less noticeable and they could add a reasonably long restart rate into the code, also hidden by the load-balancers. (a single restart out of dozens of machines was insignificant). No one seemed to care about the cost of doing this. Actually my vocal pushing of these issues may have been my downfall there by stepping on some big toes. Their line was that hardware is cheap. My line was that hardware may be cheap but data center space, power and maintenance are not. I suspect if the whole operations code base was cleaned of the leaks and unused code the DC requirements could be cut by 30 to 50%, another fiefdom that didn't want to hear that. Ah but I was let go because my job was no longer needed, the pat answer for hammering another departments management.

BTW that trend analysis software also showed where transactions across multiple machines were spending the most time, I guess the coders management didn't really want to know that either. I heard later they scrapped the whole program of quality improvement as not needed.
Dave Shiels

My TED 5000 power monitoring
http://phx-solar.no-ip.info:8081/Footprints.html
solar_dave
 
Posts: 108
Joined: Tue Aug 21, 2012 12:33 pm
Location: Phoenix, AZ

Next

Return to Metrology

Who is online

Users browsing this forum: No registered users and 3 guests

cron