LOT thing...my living quarters, new version OTA updates

This is somewhat of an admission of failure. You can't easily pigeon-hole everything, and most real projects use commercial software, homebrew, and hardware all at once. So, for you makers out there (including me) - this is where to put whole projects that don't fit well in the other forums.

LOT thing...my living quarters, new version OTA updates

Postby Doug Coulter » Sat Nov 04, 2017 2:46 pm

I've kind of re-architected my systems design for all this. Instead of having a raspberry pi at each node, I decided to use the somewhat more powerful ESP8266s as nodes, and aggregate all of them with a single pi. Eventually this single pi will be an access point (working that one separately) and all these nodes won't even show up on my lan - just the master pi...somewhat reduced insecurity, but also just plain cleaner.

Since nodes will often be going into places I'd rather avoid having to go into frequently myself (nasty crawl spaces with spiders and snakes, oh my), it'd be super nice not to have to go there even if I discover the likely need for the odd update. So I manage to get OTA updates going on with these ESPs along with my usual WiFi use for data and control exchanges.

The one I'm doing first is located in my living quarters, which used to be called "the office trailer" - hence the name of the sketch - OTLotOTA.ino.
For initial testing I'm using a program called Packet Sender which is really nice, and available for all the usual platforms.
I'm using UDP to talk to these as the usual operation isn't super critical, and in general things will either try again...error or not...and it's faster and stateless, which is an advantage when things might be going up and down with a power surge or lightning strike. Even in my super reliable world with a whole-campus UPS...things that are on 24/7/365 really want those 9's of reliability to save me work.

So, this guy does a few things. It looks at inside, outside, basement weather stuff, and also looks at water flow, water amount in the cistern, and can be commanded to control inlet and outlet valves for the cistern (in from rain barrel, out to dump water on the ground in case the stuff in the rain barrel is nasty). I wanted basement weather as you can tell if things are going to freeze by the temperature, and if there's a plumbing leak by the humidity. Those used to be more of an issue when I lived next door and this place wasn't heated...but still nice.

Every time I do this I save a pi worth of power, and that can be more than the usual 5w if it has a lot of USB storage on it, a display, whatever...
The other advantage is that for some of what I'm planning to do - it is kind of necessary for the program making automation decisions to know things from more than one node at a time. The old way I was doing that was just not going to scale well.

In general the info from each node is going into an SQL database table in the master pi, for later plotting by a webserver on that pi. That pi also runs some CGI programs to allow manual setting of things and various other control functions. But really, even a pi is loafing here, and even with a few nodes on it it's really not working up a sweat, so using several (with more sysadmin maintenance required) didn't make a lot of sense. It's just easier to back up one than a multitude, and without adding a few different generations of pi hardware to the confusion.

Hence some work on this while I ruminate on how to make the next step on fusor.
Here's the node hardware, more or less. This runs off the same DC supply that runs the house water pump, and uses some external logic to drive solenoid valves, and obviously some external sensors, but this is the fun part.
20171019-1555-lot-2.jpg
The node hardware


And here's the code as it now sits:
OTLoTOTA.zip
Code and other dox
(1.33 MiB) Downloaded 302 times


There is also substantial pi code associated with this, a work in progress to be sure. There's a lot of sysadmin involved there, from having a SQL server and db setup, to a web server, fastcgi and so on - and it's very specific to this use. If anyone wants more info on that, ping me...it'd be a really long treatise to put here right now.

Here's a screenshot of one of the web pages the pi emits as a result of the data from this thing. You can sure see the propane heater that's right under this test jig cycling...And that there's no sensor hooked up to the water level pins, so random data there.
OTLoT.png
What I see as a result of all this.


At any rate, the current milestone is getting the over-the air update going and verifying that it doesn't interfere with my other wifi usage on this thing. It's about ready to install, but certainly there will be some changes required - I already know my back-of-envelope water valve control stuff won't cut it, and will likely become a state machine vs a few if() statements. There are issues in the real world that any control system has to deal with. For example, if there's no water flow monitored after a valve is open for awhile, it needs to be closed as the solenoid is wasting power and perhaps getting too hot. But what if the water flow sensor eats an insect and is temporarily jammed? I can't get water without an override, or perhaps I could just create a water-hammer with the two valves opening and closing a few times at the right rate, then try again? But not forever, remembering solenoid temperatures or the likelyhood that there's simply no water in the rain barrel?

Obviously the easiest way to get this really working is going to be trying some things and making the appropriate changes as required..
Here's the tutorial I used for OTA updates to an ESP sketch. https://randomnerdtutorials.com/esp8266 ... r-the-air/

That last is why all this is happening. There WILL be a sensor in the rain barrel, but it'll be 60 yards from here...so there has to be a way to aggregate the information to make a smart control system.
Posting as just me, not as the forum owner. Everything I say is "in my opinion" and YMMV -- which should go for everyone without saying.
User avatar
Doug Coulter
 
Posts: 3515
Joined: Wed Jul 14, 2010 7:05 pm
Location: Floyd county, VA, USA

Re: LOT thing...my living quarters, new version

Postby Doug Coulter » Sat Nov 04, 2017 3:07 pm

Here's some of the pi stuff. I've worked up some scripts to install mysql, phpmyadmin, NGINX, FASTCGI, perl modules...it's a long list, and they're not all perfect yet. I'll post them when...
For example, I didn't know you couldn't run gnuplot in a CGI unless it could write the web root directory as www-data, even though it isn't told to and never does...and it gives some really cryptic errors if it can't. Fiddly stuff like that took me hours to sort out, and publishing scripts purported to save you time that have that kind of issue...not responsible.
OTStuff.bz2
Some of the relevant pi code
(8.17 KiB) Downloaded 265 times

This stuff all works...as far as it goes.
Posting as just me, not as the forum owner. Everything I say is "in my opinion" and YMMV -- which should go for everyone without saying.
User avatar
Doug Coulter
 
Posts: 3515
Joined: Wed Jul 14, 2010 7:05 pm
Location: Floyd county, VA, USA

Re: LOT thing...my living quarters, pi pipe

Postby Doug Coulter » Thu Nov 09, 2017 11:43 am

I'm just kinda feeling this out as I go, but also trying to generate what will be good "steal me" examples for other uses.
There was a particular issue in the design, which I just fixed up nicely. I was wanting to do some fairly time-consuming things, like creating a deliberate water-hammer to clear up stuck junk in valves. The ESP code is really a "do it right now" oriented thing, and needs to be mostly. I could have implemented this kind of thing as a repeatedly called state machine there, which would mostly just sit in a state waiting for some time to elapse - well, actually just returning to the "opsys/scheduler" if the time hadn't - but this pushes some complexity - which will likely want some tuning, down to a place where it's the hardest to tune. Even though I now have OTA updates...it's just not the "right way".

The pi master for this has a few pieces of code, none of which were a great fit for the timing either, but at least the pi can run more complex stuff, some in the background, and it's easy to change and generally manage there - heck, it even runs its own code editors and such. But the once per minute piece that is a daemon stuffing the database from the ESP data wasn't right for this - it spends most of the rest of a minute sleeping and thus not consuming pi resources, nor was the cgi, which only runs on user demand, and wants to return quickly so the web page it produces won't seem to be hung.

After a bit of thought, I decided that using a linux named pipe would serve to adapt between our source of commands - the cgi - and doing those commands - via UDP packets sent to the ESP. There's no reason why more than one piece of code can't talk to that node, and in fact, no reason all the communication even needs to come from a single other host. Basically what I wanted was a way to spool up some commands, with about second resolution timescale, and allow queuing to handle the issues of fumble fingers or more than one client (well, thinking ahead here, there's only one of me).

So, commands can be pushed into a named pipe, and a background daemon can read this pipe, add the timing via sleeping some defined amount between commands, and sending the timed commands out to the ESP via a socket. I used a super-simple protocol. The ESP takes single letter commands - a,i,o,c (autofill, intake open, outlet open, close all) - to manipulate the valves.
It will ignore anything it doesn't understand. I added a layer with this, so that I can add numbers to specify a delay between commands. For example, for testing right now I'm using: echo 'a4co2c' > /tmp/valvecmd
to send a test command to the pipe - this interprets as "open the intake in autofill mode, sleep 4 seconds, close it, open the output valve, sleep two seconds, close it". The little bit of pipe->socket code just parses out those numbers and feeds them to a sleep() command, during which time it doesn't use up pi cpu cycles, nor does it use pi cycles while waiting for some command to arrive, which is the desired behavior. Anything can write to that pipe that has access to the path, and it'll all be serialized to the ESP node. Nice!
It's so little code I'm tempted to put it here, and I often give into temptation (other than, yup, still no smoking since spring!).

Code: Select all
#!/usr/bin/perl
use Modern::Perl;
use Socket;
use POSIX;

my $pipewrench; # handle for pipe
my $pipepath = "/tmp/valvecmd";
my $data;
my @commands; # things to do
my $op;
my $msgsock;
my $slavename = "ESP_B0F1E3";
my $slavedest;
my $pid;

#initialize stuff
unless (-p $pipepath) { # in case first run ever or since boot
# say "creating pipe";
unlink $pipepath;
system('mknod', $pipepath, 'p')
&& die "can't mknod $pipepath: $!";
}
open ($pipewrench, "+<$pipepath") # we block on this waiting for something to do
  or die "Couldn't open $pipepath: $!\n";
# get a socket
socket($msgsock,PF_INET,SOCK_DGRAM,getprotobyname("udp")); # or loggit ("socket:");
$slavedest = pack_sockaddr_in(42042,scalar gethostbyname($slavename)); # @@@

##################################################
############## main ##############################
##################################################
$pid = fork(); # we're going to detach from whoever started us
exit if $pid; # parent dies
POSIX::setsid();  #or say ("Can't start new session");
close (STDIN); # never use it anyway
close (STDOUT);
close (STDERR);
while (1)
{
$data = <$pipewrench>;
# say "incoming:$data";
chomp $data;
push @commands,split //,$data;
# say "commands:@commands";
foreach $op (@commands)
{ # do the operation
  if ($op =~ /\d/)  # it's a number
  {
     sleep $op; # digits in stream specify a sleep time
     next; # we're done with this one
  }
  # it's non numeric, pass to ESP...(hope it can handle crazy stuff)
  say "op:$op"; # debug
  send($msgsock,"$op",0,$slavedest); #or die("send:$op\n"); # send request
} # do each op
@commands = (); # but clean up after
} # end while one
close($pipewrench); # probably don't need these if exiting anyway
shutdown ($msgsock,2);
############


Hopefully even perl haters can read this fine...you don't have to make it hard to understand, and I rarely do, as I might have to understand it again later.
Of course it looks nicer in sublime-text, but you can't cut and paste from a screen shot. The syntax coloring does help see what's going on.
editor_view.png
What I see "at work".
Posting as just me, not as the forum owner. Everything I say is "in my opinion" and YMMV -- which should go for everyone without saying.
User avatar
Doug Coulter
 
Posts: 3515
Joined: Wed Jul 14, 2010 7:05 pm
Location: Floyd county, VA, USA

Re: LOT thing...my living quarters, new version OTA updates

Postby Doug Coulter » Thu Nov 09, 2017 3:37 pm

Note to self - "the travails of documenting something before it's mature in use" #howevermany:
Line 20 there works a lot better as: system('mkfifo', '-m=0666', $pipepath)
once this is running as root - you have to let others write to the pipe for it to be useful! An alternate form of mknod would also work, slightly different syntax. In researching the mode input, I saw where mknod is now deprecated so I went to the approved function for this instead, since I was in there editing anyway.
Image
Posting as just me, not as the forum owner. Everything I say is "in my opinion" and YMMV -- which should go for everyone without saying.
User avatar
Doug Coulter
 
Posts: 3515
Joined: Wed Jul 14, 2010 7:05 pm
Location: Floyd county, VA, USA


Return to Combined projects

Who is online

Users browsing this forum: No registered users and 2 guests

cron