Science/Engineering/Technical forums

Once I installed mySQL, NGINX, and various other goodies (camera support and streaming over NGINX) - I wanted to get on with the prime mission here. While I'm not adverse to command line stuff, this thing they call phpmyadmin is pretty cool for setting up and generally messing around with databases. Here's how we put it in when we have an NGINX server: https://www.digitalocean.com/community/ ... emp-server

A caveat is that since the cam streaming stuff uses /var/www for its files, so do we - so we change the default root in the NGINX configuration.

This puts pretty much all the frequent writing of files etc into /var - mysql's databases live there too. Which was the point - all of that is on a USB stick in my setup, easy to copy/erase/edit - whatever - in this machine or another. No funny boot junk, just a buncha files. So when it inevitably wears out... (and I'd bet a buck that the controller on a stick for wear leveling is better than one on a microSD card) - it's easy to handle, I don't even have to get into the box, burn a bootable image, reload software and so on.

The level of customization here goes WAY PAST the old "just backup the user's home directories" mantra. We've put so much custom software, edited so many config files and so on - to back this up for real requires backing up the entire thing.
Which I do - I make images of that little SD card now and again as check points, when this is deployed I will have the latest backed up on other machines for that too.

FWIW, it seems like a Sandisk USB3-extreme USB stick plugged into a pi is, if anything, faster than the SD card...yes, the pi only uses USB2, but this can easily saturate that - cool. Might be a reasonably big deal at some point.

Just a quick progress report. Now that I have the base software in the Pi - and I added tightvnc so I can work on it remotely from the comfort of my couch and on a fast machine; I still have to wait for the pi to do things, but a lot can be done on the big box while waiting. Eventually, the pi will be in a rather inconvienient place to do real work on it, so this is a very nice preparation for that eventuality. Now I'm doing the fun parts. We're almost there with stuffing good information into the database (MySQL running on the pi) once a minute - I only need to make the data aq/database insert program into a daemon and have it autostart at boot for that part, and write a CGI or two to allow the pi to generate plots of the data, and manually control things where I didn't think the automatic control did the right thing (eg, nothing's ever perfect - may as well plan for that).
http://youtu.be/alACCsT_HvE is a little movie of what it looks like from where I'm sitting. Could be worse, nice cold drink, comfy couch, no spiders like in the basement over there...


So now all I have to do is write some nice CGI's to make plots and serve them on the webserver, do automatic overrides of things like water valves, and so on - the easy part (at least for me, I program like breathing, it's that sysadmin stuff that is a pain for me). The sysadmin stuff is largely done - all the hard parts already I believe. So now we're down to the fun stuff, and displaying all this nice data, using it to train neural nets (obviously, doing that on a faster machine) to control some things automatically, alarm me when the outdoors is cold and the woodstove is going out, things like that - the reason for this is to (eventually) save me work, and excess temperature swings in my living space, which will be over there from here out, due to the fusor not being safe to be around in high output mode - this is more or less a useful dry run for the job of remote controlling the fusor from a safe distance. A mistake here - no big deal, with the fusor, could be a problem. That will have a few hard backups - relays and actual wire pairs for them for things like hard-shutdown - you never know when a computer might crash, even good ones, and I'm not going to trust them for the truly crucial/safety functions on that project

Well then, after only some bit of struggle, here's a nice plot of temperatures since roughly 9pm last night (it's now about 8:15 here)...work to do on X axis and so on...and basement temp is actually a sensor about a foot from my woodstove, looks like it's about time to start a fire again. What's new here is that this was from a MySQL database query and run remotely. The real plan is to have another daemon updating some plots as .png in a directory the web server has access to (when the pi is not too busy, it can check for that), and have a little html serve them up as part of a page...a submit button on that page might re-set the time interval and other options. Then with the auto-pull trick above, we can just have the nice 10.1" display on the pi updating all the time and see conditions over however long at a glance when walking by it.

A little more progress here. I now have the pi serving all this up as plots from a database via a perl CGI, and other nice things. I have two setups planned, one per active building on campus (the others are just storage), and they will cross-replicate their databases so that decisions that require information ffrom both are possible. One case of that is the water system - no point opening the inlet valve in the pictured building unless there's something worthy of being put in the cistern, for example. That info will come from another box nearer the water collection rig, for one example. I've managed to design this in such a way that even the slow model pi B+ has enough "grunt" to keep up with web serving, database running, and data aq from itself and a slaved arduino uno. Even though a few operations take seconds at 100% cpu, linux gets 'er done without issues, other than having to wait - and the real time stuff stays realtime. Could be worse, and now I get to keep more pi model 2's for more fun stuff, since I'm using the older ones for things like this.
The obligatory youtube link: http://youtu.be/OrnXi2u5W6s


I believe the issue I initially had with showing just how good the pi camera is could be a problem with the most recent Chrome update - they've been coming fast and furious since so many zero-days have been revealed (and so have firefox ones).
Evidently, they have enough spagethii that they can't fix this without breaking that...beginners!

I will be posting all the nice code as soon as I'm done modifying it daily, so others can build on this. It's kind of unlikely you'll want just the same stuff in a database, have the same sensors and so on, but it should be a nice jump start for your own project(s). Any day now. I plan to build this into a wall that doesn't exist yet, as I mentioned in the video. But that will go up pretty fast.

Good stuff, Doug. Thanks!

I just ordered a pan/tilt servo set for the camera , and am still deciding if I should use the NoIR filter version (with optional IR LEDS for night) or the regular one. This would be independent (I think, so far) controlled off a spare arduino.
I could go "all the way" and add it to the cam's website, I suppose, but I also got a fairly cool old-timey joystick to control it. Somehow, doing that over the web with a mouse doesn't seem as slick...

This is very nearly ready to go into real use. Just a couple details to finish up - drivers for the water solenoids (software done for that already), including a charger for the 6v gel cell that will power them, switched down off the 12v UPS system that will drive the pi, and the 6v system, being diode-isolated from the wimpy power supply that won't do the solenoids without help - has a 7.67v output too - which will power the arduino, making the entire project UPS'd with around 3 days battery power if my AC fails. Not that it has in decades, but that's because I'm careful. This system is no different, and again, is a dry-run for the fusor remote, which involves my safety - good to make my mistakes (plenty) here first...

There will be another version of this for the man-cave building. That one will get data on collection tank water level and turbidity, wind speed, the usual temperatures (pretty much the same set for over here), solar insolation (so I know how much I can use once my controllers just start throwing it away) and so on - with its own database, each building syncing a database to the other so both have all the info to make decisions.

Any solar system that is reasonably provisioned for "the dark months" has one other issue - spring/summer (fall isn't as great due to still being hot, but less sun, so the refrigerators use plenty). I have TONS of spare kwh, currently being thrown away when the house batteries are nearly or fully charged, and no way to know how much and when (even if I sniff the proprietary CAN bus "Xanbus" system). But I can predict if I know how high the batteries are, and how much sun there is - and then turn things on based on some sort of dynamic priority list I can have.

EG - car need charging? Do I want a shower? Should I burn electric heat instead of wood or propane? This list might be dynamic as heck, as I might not need to go anywhere tomorrow, but haven't had a shower in awhile...and so on. Or the car might already be charged up (I can get its charge state off the web, but I'm reluctant to pass my username/password over the net repeatedly). Barring that, I can simply turn it on and see if it draws current (and that state of the arduino-based open EVSE is readily available). Stuff like that is where this really begins to pay off. I might even do some web screen-scraping of weather forecasts, or make my own based on all the data. After all, if it's been cold a long time, and will get well below freezing tonight - I might want to turn on a heater in my cistern *now*, while I have sunshine to power that - not later on when it'd be running the house batteries down. Water is a great heat-battery. But I don't want hot cold water in the summer, either.
Good chance to apply the simple mind of a computer to advantage.

Coming along very nicely -- good stuff!

I'm very much looking forward to seeing it all linked together to have a glimpse at its predictive capabilities.
Especially this prediction stuff I'd love to see in action and try myself one day!

Philipp

At the moment. the main snag is wifi connectivity. Seems all the WAPs I've tried have "issues", other than the dirt cheap slow single band ASUS I started with.
I set the WAP at a fixed IP...because most of them have shared storage I'd like to have - it's one of the few always-on things for that. (I'm on batteries at night or on dark days, so I can't have always on things that draw any real power)

Everything works 100% if wired. But make a Pi or anything else wireless - and the trouble begins. They can all still access the internet, but can't find each other (at least, without my LANDNS software, which only works on linux, I've not got an android version going now, since I don't "do" Java/Dalvik and don't really have time for it).

I don't *have* to have wireless for the homestead stuff. That can all be wired (it would be nice if I could control it from my portable Nexus 10, though). But what about controlling servos on the fusor that float at -50kv? Not only does that have to work, it has to be really reliable and fail into a safe mode. A possible workaround for that is fiber optics to the servos, but I've found that all computers (PC's anyway, Pi's don't seem to have problems there) within about 5 meters of the fusor or connected to any part of it have to be better isolated than the theoretical 500v common mode limit of wired ethernet or they get fried if there's an arc. I'd rather just stick one of these low power IOT boards on there and run the mess on batteries for fusor runs.

Often the wireless stuff will start working if I run a snoop program, ping stuff from a wired machine - pings work, but still, machines can't use tightvnc etc, browse to one another's web servers consistently if any machine is on wireless.
Network scans sometimes start working if I just wait an hour...so many variables I've not tracked this bugger down yet.
Amped and TP-link Archer WAP's are giving me fits - and I've got hundreds of bucks in them, they might as well go in front of this picture and let's get out the real dynamite and make a movie for all they've done for me (other than waste days of my time). I'm sure the youtube subscribers would get a kick out of it. At least Archer support has a nice Chinese lady helping me, but I have the feeling that I know more about networking than she does - from before when she was born. Sigh. At least she's patient and will probably wind up escalating/translating for the real engineers. It seems no one worries about big networks with intermittent machines going and coming being able to talk to one another as long as they can all see the internet - I seem to be hitting all the edge cases in all the new stuff.
Ports blocked, machines not recognized across 2.4-5 ghz on the same WAP...all kinds of nutty stuff. Probing them hard from wired machines sometimes wakes them up, but at night it's not like I want to turn on 5 computers and do sneaker-net running around with avahi and snoop and ping. Which doesn't even always work - it's the only thing that ever works, though. Sneaker net is making my feet hurt...

As to predictions...there are multiple interesting classes. Weather - screen-scrape for large area data (but not their always-wrong predictions), but also have detailed info on "here". Should be a little better than the weather guys who have to make predictions for their entire listening area/viewership - the weather here in the mountains can be very different in half a mile - or 10 feet. You can literally drive from bright sun to torrential rainstorm and there's a well defined line between wet and dry on the road some days. Of course, one of the main things other than temperature I'm interested in (primarily in winter for that) is insolation - do I have power to spare? Most times I do, now how much, how long will it last (half a heated shower isn't too much fun, maybe I should put charge in a car or something that doesn't mind being interrupted, or do welding or run the mill?) - you see the issues, I can tell. To really get full advantage, I can't necessarily wait till the house batteries are full before using any extra...they have a long "absorb" cycle during which the solar controllers are not sending them all there is. I could be using that too - but only if I know I'm going to make it to full charge...

Other predictions, or perhaps a better word is guided decisions involve things like I mentioned above. There's a lot of times I have spare power - I don't know how much - gotta measure that (I can dedicate a spare solar cell loaded to the hilt to an a/d input on something for example, to measure effective insolation - and its variability to know about "partly cloudy" - I'm still trying to think of a good sensor for "generally white sky with no blue" since that's an entirely different indication), but even then, don't know for how long. Long enough for heating a shower (only 1.5kw, but takes two hours)? Should I just put some charge in the car (if it needs any - more screen scraping if I use its web interface, which is a bit insecure (GM's OnStar is kind of a bad joke), else I have to keep track manually, or just try and see if it eats power - have to put on a sensor for that - might be the better plan, but I don't even turn that inverter on most of the time - the tare load is not worth it if I'm not going to use it, and no, there's no remote control for it other than their proprietary can bus junk). Run the AC if it's too hot? Heat the cistern? For those, a little foreknowledge is very nice - no point heating bulk water in the summer, but in winter - if it's going to be cold later and it's sunny now, go for it. And so on, this can get pretty complex in figuring out what the priorities should be at any moment - the prime directive being ending the day with full homestead main batteries, but secondly, using anything spare to do something useful - that's a pretty common case and there's almost always "something" that would be good to do - if it's smart enough to turn off when the sun goes away for whatever reason (or can tolerate that).

FWIW, I've moved all this to the new Pi-2 model. Whoa! This thing is FAST! And it's even faster with a very fast microSD card (class 10 U3) and fast USB3 "extreme" stick (see this for why it's all sandisk). (I've compared with USB2 stick and class 4 SD, and you get a huge factor with the faster bulk storages) Yes, I know the pi-2 is only USB2, but it takes a USB3 stick to saturate USB2 speeds. I'm using the stick for the big data and logs and other things that get written a lot - and the SD card only for code and things I can download again. Makes it all more robust and easier to back up.

I've added Adafruit's mini pan-tilt for the camera, it's pretty cool but has some issues I'll document once I've fixed them. Some play in the gears combined with some inertia can make the thing oscillate which is kinda a disaster on picture quality. I'm pretty sure adding mass will just change the frequency - it needs damping....with luck I can find my old super viscous silicone fluid we used to use to make record player tone arms drop so gracefully. Or just put in something that rubs, but from experience, that will become a maintenance issue over time.

You'd better believe that when remoting the fusor there will be some good old twisted pair phone company burial cable driving a couple relays in series for "main power off" control - no way I depend on a computer for that - what if it fails on, and I can't get to it to turn it off without getting one of those big rad doses? Once was enough (or too much) for that one.

Re: Gross prediction techniques, I'm pretty sure I can use something like ARIMA + multilayer feedforward neural networks to get weather pretty decently for about 1 day, which is all I really need most of the time. It would be nice to predict rain further ahead for water usage...but usually I get enough if I just get all I can, which I'm not doing yet because it's not yet automated. I expect to get that far in a week or two, maybe less. All the stuff is working for it, it's just not installed over there yet, it's here in the shop where it's easier to work on for the moment - and that nasty wireless problem makes me want to keep it here till it's all polished. We haven't buried the CAT-6 to there yet...

Huff, puff. Turns out if a *wired* machine pings the wireless one (already knowing its IP address), then after 16 (interesting number) failures, it begins to work. So, I have to add something like another Pi, wired, with a script to just do that all the time?
If I do that, why not ditch every single WAP I have and run hostapd on the pi instead? Parasites (power and money) - I better not get started.

Moving dirt for the new add-on, making cables for the install of this where it's really going to go (the "office trailer" which I'll be moving back in to) and so on. In the meanwhile, it's been running on the bench here in the shop for over a week, against a "testing" table in the database, and making plots like these. In this case, outdoor temp really is - the sensor is hanging out the window, indoor is on the bench, and "basement" is hanging by my woodstove near an open window. I edited out a couple times of fairly extreme temperature excursions on that one so the rest would be readable on a decent auto-scale.
The thermocouple CJ (cold junction) is also on the bench, and at times I've had the TC itself hovering over my soldering iron. Box temp is what's reported by the BMP180 in the pi box. At some point I added a homebrew heatsink to the pi chip which *seemed* to drop its temperature differential a little (3/4" round stock with fins cut into it, about 1" long, epoxied to the cpu). But it's obviously in no danger, even in summer. Seems most of the heat is coming from the LCD and its driver stuff, FWIW, it's a nice little 10.1" display. I've not figured out how to either make the pi never go into "screen share/idle mode" or how to get the darn backlight of this LCD display (the main power eater I suspect) to go off when the pi is showing a blank, black, screen.
Yes, I googled and applied the supposed "fix" for going idle - and it didn't work.

Here's some pretty plots. I can't easily show you the webpage it emits, as it's not on a public network - which IS THE WHOLE POINT here. Water stuff wasn't hooked up...but will be soon, it's installed and tested in the building it belongs in. As you might guess from this last plot, the system is designed to be "not stupid" and not lock up if a sensor is missing or fails.....Just show blank stuff. I'm looking for lots of 9's reliability here.

Well, it's almost done (for that building) and now is sitting in the right building, hooked up to the actual sensors (well, almost, still have the water flow rate sensors to hook up and check). I encountered one bad DHT-22 (Adafruit) that I'd made a nice little "house" for bolted to the back wall of the place for outdoor temperatures, which will in general be cooler back there in the shade on the north side of that building, which sits at the top of a full forested north facing hill. That's good - I want pessimism in winter when I'm worrying about freezing my water, and optimism in summer...We get a nice breeze almost all summer from the north as south of this building, there's a field that gets hot in the sun, creating an updraft, which draws the breeze right through that building (which is one reason I put it there). This idea of "put buildings everywhere and overcome nature with raw wasted energy" (Where I suppose most readers of this live) is silly - if I sound green, well the type of green I'm thinking about is mostly the buck$ kind. When you can get the free - do it. And yeah, it's nice to walk lightly on the earth, too.

That was a lot more crawling around in dust and dirt in the basement than I like to do in one day, yet there are two more flow rate sensors to hook up and test. Might make it today, but probably not. I'm bushed after all the rest.
Other than a slight slowdown (hey, it's a lot further away on the wireless) and obviously, some different readings, not much new other than yes, the water sensor design (AC couple 50khz 12v pk-pk sq wave into water via Ti wire, sense said AC via other Ti wires or the brass overflow fitting with AC coupled volt doublers with 100k ohm loads, feeding arduino A/D - but logic levels would have worked fine) works well, at least at the two tested levels. It'll get low enough to test the others, I'm sure, at some point.

I will have a little overview video here soon. RIght now, my i5 NUC is spinning its fans as fast as it can rendering two HD clips and one from Kazamm into something I can upload, which will in turn take awhile. Making this post lets me put a link to here on youtube for those who want to see more - there's quite a viewership on my channel there who aren't members here (nor should be, we like real scientists here). But that is not to belittle the rest - if they want to learn, that's our kinda people, totally. Here's the vid, at least when youtube gets done processing it:
http://youtu.be/Qp_ocDwoH6Y



Bill commented it would take God to reverse engineer this stuff. Nah, anyone with a little skill and perhaps a voltmeter - no problem. But why do you think I document things up here anyway? Heck, *I* might forget! Sometimes these things work for years, by which time my memory is pretty lacking on the details. It's yet another good reason to open source all this, which I'm doing as it becomes worth doing (you really don't want all my alphas and most of my betas, trust me).

The major chunks of this system are (electrically):
A bulk ~~ 18v 10 amp power supply. On the same plate is a temperature compensated LM350 type regulator to get to nominal 13.5v for a gel cell mounted on the plate - that's the power for the Raspberry Pi HDMI display and the pi (which has it's own switcher in the box to run off ~12v down to 5.1 for the pi and the USB stuff). This bulk supply also has a switcher on it that makes 7.65v to run the arduino and a nominal 6.3v gel cell (via a diode and resistor) that in turn provides the amp or so that the water valves require - those little National simple switchers get kind of stressed around an amp - no need to push it here. This setup also provided uninterruptable power to the control system (raspberry pi model 2) via that 12v gel cell. I don't have power failures (not one since 1979 other than deliberate inverter shutdowns for changing wiring safely) but you never know - and I may for some reason want to cut AC to that bulk supply if say, the water pump fails funky, yet not lose this other stuff.
This lashup give me that.

The aluminum angle bracket has the arudino uno, the 6.3v gel cell, and the power distribution and switching fets for driving the water solenoids. I'm using DHT-22's for most of the temperatures, but also have the max 8155 termocouple breakout board which will eventually have a thermocouple stuck in a woodstove flue (can't use silicon sensors for that!) and provide another reading on the basement temperature via it's cold-junction readout. The DHT-22 that is plotted as "basement" is actually fastened to the cistern, so it might more-accurately be called "cistern temperature" - maybe I'll fix that next time I'm in the CGI that serves that page. Then the thermo CJ temp can be say, basement ceiling temperature or something similar.
I'm using flow sensors from Adafruit for the water flow into the cistern and out from the spot sprayer pump that drives my plumbing (hey, cheap is good). Haven't hooked those up yet, but they work fine on blown air. I'll get to that soon, and try to figure some decent scaling for those so their plots can live on the same scale as the main water level with gnuplot still in autoscale mode. I'm not so worried about these giving me accurate gallons when integrated - I've got water level sensors as is.

For the software system design, the pi is the master, the arduino uno a slave. It only speaks when spoken to, but it's sampling all the sensors roughly once per 2.5 seconds. The pi asks for data once per 5 seconds. The pi lowpasses this with the old .1/.9 first order LPF (jamming the first value after boot so we don't have to see an exponential tracking to the true value) and stuffs the result into a mysql database once per minute on the 0 second. This is done by a program called "StuffData", while the Uno program is "ArdT1" as it's mostly about temperatures (or was when I was sweating freezing things this last winter). I'll make copies of those available. The pi is running NGINX and MySQL, and using perl for CGI's, one of which in turn uses Gnuplot for making the plots as .png's that are then served, with some nice auto-regenerate when it makes sense - a short plot should update on every minute, but when looking at tons of points (say, 30 days worth of 1/minute) there's little point of wasting cpu effort (and watts) updating once/minute since the plots are only 800x400 pixels anyway. I will do quite a few more CGI's as the whole point of this is automation, and of course, being able to override things manually - yeah, I saw those star trek episodes where they lost the main computer too...Worst case I can go yank out wires...

Of course, this is a LAN of things, not "the internet of things" which is an idiotic idea for anyone not selling advertising or exploits (malware). Why would I want to let anyone else have access to either this data or be able to control any of it? That's just ignorant. As this gains the ability to control more...as El Reg mentioned - do you really want some hacker to be able to turn on the gas, wait awhile, then light the fires? Does Google (even though I don't hate them) really need to know when I take a dump? Does a burgler need to know when I'm not home? He might want to, but I don't want to make it easy. There's so much wrong with the internet of things it's amazing that the sales droids haven't been hung already - as soon as it gets any traction, you can bet all these things and more are going to happen. Where I live, you'd need one heck of a high gain antenna to intercept my wifi - and this is going to be wired just as soon as we build that add-on...so no, this is private stuff.