fusorpi master control code

Well, it appears there's no really good way to do this here. This will contain various bits of code used to support the fusor, which presently includes everything from embedded 8 bit uP code to stuff to run under Linux in some HLL (often perl) to interface-specific stuff for commercial hardware, sysadmin tips and setups, and running on whatever machines I have in the tree here. So any one piece might fit some other category too, but...there's no pigeon-holing even computer science without losing the app-specific stuff and inter-relations of it all in a specific usage.
Forum rules
Code sharing for fusor data aq and control software

fusorpi master control code

Postby Doug Coulter » Sun Mar 05, 2017 2:26 pm

This is, of course, a tiny fraction of what's actually running on this pi we call fusor pi. It's running a MySQL database that has replication set up to this "space heater" machine at the operations position I'm posting from, among other things.
And yes, without the database schema, this isn't going to be terribly useful except maybe as an example of how I write perl so at least I can understand it later....
But it's here, it works, and this is a backup if nothing else. It's now named fmb as for some of the major changes that have happened along the way, it was handy to change the name from FusorMaster, to fm, to...fmb and keep the older versions around to scare the new one.
This is also two files, one for the GUI description and these are expected to be together at runtime (I run them in my bin directory).
fmb.zip
fusor master control with some of the dependencies...but unlikely it's all of them.
(13.55 KiB) Downloaded 228 times


In addition to using the database, this has a status window type of thing I can send debug and info messages to during a run. This is also logged in a file with the name being the current timestamp.log at the start of a run. Sometimes handy, this records whatever the dumb operator did too - in case I forget, being dumb and all.

I'm so dumb I even used some python inline (from Adafruit) as the perl lib for i/o, used here to create the clapboard signal mentioned elsewhere, was a pain to get working right, so...punt.
Here's some gratuitous screenshot action from padre:
fmb_padre1.png
Nice highlighting and navigation in padre

fmb_poll.png
The routine where the action is during a run

fusorpi_VNC.png
What I see via VNC at the op position - this is fusorpi's screen.


So, the above is my master control panel, so to speak, for the raspberry pi nearest the fusor (there are a couple more that are used as video cameras). Fast enough, and cheap enough not to care too much if I happen to fry one.
This one has a 256 gb SSD disk for the root partition, and that's where the database lives. Frankly, a regular spinner would do fine, this isn't heavily loaded during a run and the extra speed isn't needed here. But...I like fast things.

Yes, when set up right, it really is this simple to stuff formatted data into a database which then automatically replicates on another machine just in case. This "poll" routine is actually called by the GTK runtime at 250ms intervals and it merely scoops up data from the arduinos and parses it in order to stuff in the database, using prepared statement handles. But perl is so hard to read (chuckle) no one would ever figure that out, right?

I will try to get the rest of this up here, including the plot stuff (real simple) database schema, and oops, forgot the custom module I wrote to handle pi cameras, which is not real useful without changes for most of you, as it assumes various naming and share conventions.
Posting as just me, not as the forum owner. Everything I say is "in my opinion" and YMMV -- which should go for everyone without saying.
User avatar
Doug Coulter
 
Posts: 3515
Joined: Wed Jul 14, 2010 7:05 pm
Location: Floyd county, VA, USA

Re: fusorpi master control code database schema

Postby Doug Coulter » Sun Mar 05, 2017 2:45 pm

For those who even know what this sort of thing is, here's the SQL schema for our database.
Fsr.sql.zip
Just the layout here
(1.14 KiB) Downloaded 229 times


I didn't include the actual data, not because any of it is a secret, but because it's simply too huge already and all we're really done is shakedown the system so far - even that isn't really done yet.
Anyone who wants it can get in touch with some large storage media for me to copy it all too - those audio and video files get big fast.

Since MySQL doesn't like huge blobs, I created a directory structure under /var/FusorFiles to hold the videos, audio, and log files, and the database simply holds the pathnames for those for each run. The rest is right in the database. This is a pain and requires other scripts at the ops position to keep it all together, but...we're trying to get 'er done here, not be a shining example of the magic on the hill. For that matter, this might even be that, since it's kind stupid to put stuff that huge into a MySQL database especially on a pi, making it near-impossible to back up using the simple tools, when an rsync command does this fine for the big blobs. Also, this limits the required bandwidth during a run to avoid overloading the pi or the network with all that - I can always bulk copy later on. Camera->pi->intel, after a run.

Thanks to David Knight, a good friend and even older neckbeard sysadmin for helping me set up the replication stuff - I don't turn away help like that!
Maybe someday I can get all the config files that make that go posted up here too, it's really cool how data just appears over here from the remote pi like magic - and is then safe even if the fusor shop catches on fire or that pi should crash - we'll have the data right up to that instant.

The cameras also hold local copies of the videos they're taking until I erase them manually for space reasons, though they have 64 gb USB sticks in them for that, sooner or later, even that will fill up. Redundancy is a good thing. If I'd had this level the first time our breakthrough manifested, we'd be done now. Once bitten...it was worth the effort to do this right from then on.
Posting as just me, not as the forum owner. Everything I say is "in my opinion" and YMMV -- which should go for everyone without saying.
User avatar
Doug Coulter
 
Posts: 3515
Joined: Wed Jul 14, 2010 7:05 pm
Location: Floyd county, VA, USA

Re: fusorpi master control code

Postby Doug Coulter » Mon Mar 06, 2017 11:36 am

Here is the code for the perl module that takes care of handling remote pi cameras via Samba (which is somewhat flaky, remember what outfit created windows file sharing).
I won't brag on this code...it's somewhat nasty to say the least, making assumptions not only about how the machine it runs on is set up, but others whose names are passed into it so it can connect with their assumed cameras and shares.
I'll post the smb.conf files for those machines where it belongs here at some point.

One extremely "evil" trick this code does is call back into the using program to take advantage of its logging function, named loggit. I'm not sure any purist of any sort would think this a good idea, but it does work and allows unified handling of debug messages for the control program. I suppose I could have created a logging daemon as a separate piece of code and had everyone talk to that, but then realtime display issues would make that interesting, and when you're working with this much $$ of equipment, if something goes wrong...you want to know now. Despite considerable work to make it otherwise, this fly-by-wire stuff just isn't the same as being there for noticing problems.

This module allows the using code to start and stop cameras in remote raspberry pis, by kind of hacking into the mechanism by which those work - sending commands to a fifo file. Further, when they are done "boxing" which in their terminology means that after the recording is stopped, converting a .h264 file into an mp4 file - we assume they will call a shell routine that I wrote to create a file in the media directory that contains the name of the latest video (there might be a lot of videos in that directory). This is also used to know when that "boxing" stuff is done, as we erase the cap.txt file when we begin, so it's appearance is a done flag.

This code also (re)taught me a lesson I knew long ago but had forgotten. Perl modules in general can't just print(), they have to use something like the mechanism I mention above. Well, what if in rare cases (for example, some of the error reporting) you mis-code the call to the thing that reports the error? I assure you, hilarity does NOT ensue for that case - things just hang or flat die. Code coverage is important! Luckiliy like any good coder, once I saw one instance of the error I searched the code and found/fixed all the other ones, kind of like the proactive stance some opsys took to finding about strcpy() induced buffer overflows and simply searching all their source and replacing where possible with strncpy() and a size parameter.
In this case, I forgot to always pass in "$self" to the loggit alias routine - which it needed to find the real one in the main code.

On my machine, this code lives in /etc/perl/. It could be (I believe) in the same directory as the main program or another place on perl's @INC list.

Rcam.pm.zip
A rather dirty module to hide some fairly rotten issues
(3.8 KiB) Downloaded 239 times
Posting as just me, not as the forum owner. Everything I say is "in my opinion" and YMMV -- which should go for everyone without saying.
User avatar
Doug Coulter
 
Posts: 3515
Joined: Wed Jul 14, 2010 7:05 pm
Location: Floyd county, VA, USA


Return to Fusor support

Who is online

Users browsing this forum: No registered users and 2 guests