LAN iof things for the homestead

This is somewhat of an admission of failure. You can't easily pigeon-hole everything, and most real projects use commercial software, homebrew, and hardware all at once. So, for you makers out there (including me) - this is where to put whole projects that don't fit well in the other forums.

Re: LAN iof things for the homestead

Postby Doug Coulter » Tue Apr 21, 2015 9:19 am

As promised, here is some of the more-stable code for this system. This is the arduino sketch part. Most of this is tested, but the counter stuff was only tested as part of another data aq project, I've not yet seen how it handles water-flow sensors, just geiger counters and the like (works well for those). I'll be testing that today I hope, nice weather for this just now. So consider this a beta. We are doing some interesting stuff I think I documented on an arduino thread elsewhere. We talk via the pi's UART port, with some fancy level converters/boosters such that the pi drives 0-5v levels out - HARD (2 amps!) to the "slaves" which all listen to that line. Coming back the other way, the arduino unos use a 2 transistor inverter and open collector with pullup to drive the pi's UART input line, which means I can put a bunch of arduinos out there as long as they follow our super-simple protocol - speak only when spoken to at your address (which is hard-coded in each slave, this one is ascii '1').
While we've used (invented!) other protocols, it's really doubtful this needs all the fancy space and cycle-eating features those had. We don't need even a full byte of addressing for example, ascii (human readable for debug) will do fine, and we don't need most of that here. We can retry on an error if we like, it doesn't have to be built in. When sampling essentially weather data at once/minute - we can afford to drop the odd sample, not that it's happening (probably would during a direct hit from lightning, though).

The arduino code is designed (and tested) to not lock up almost no matter what - missing or dead sensors for example, but I've not yet implemented a watchdog timer - we'll see if that's needed and put out another version of this if so.
Everything here is static, no possible issue with say, memory leaks. You learn that doing high-9's stuff like I used to do for a living, designing stuff where having to reset it meant a service call, and that didn't even have a power switch (nor does this).
Enjoy! The sketch folder also has the pinouts and command list in a text file.
ArdT1.zip
Arduino unit one for office trailer - code and doc
(5.07 KiB) Downloaded 467 times


I'll get the pi stuff up here too - but it's really changing a lot just now, and I'm a little at wits end to describe things like how all the configuration files for common stuff - NGINX, MySQL, and modules required for perl (most of this, it's my favorite glue language) and gnuplot should/can be documented, as it's important, along with some info on the toolsets I'm using that save me time/effort and don't come with the pi as is. For example, the pi comes with a bunch of toys and games - now gone.
It comes assuming it's all going on that fragile and slow SD card. Not here - we have a 64 gb USB3-extreme USB drive partitioned to hold the stuff that gets written a lot (the database for example), have customized those filesystems for less write amplification, and heck, use a ram drive for the things that really do get lots of traffic. It's a system, not just a jumble of parts...and I want it to live decades, not just for a demo. And the USB card doesn't have any magic junk for booting - you can just make a backup with a regular copy. No "imaging" required there. I'll call that a feature, perhaps a major one.

FWIW, on both pi models, faster SD and USB do matter (more on model 2 of course) - even though the pi is only USB2. Many USB2 sticks can't even saturate the USB2 speeds (even though the pi can't either - but most USB2 sticks can't even keep up with the pi, much less the spec), hence the use of USB3 (and the high end of that) for this. Ditto the SD card which is class 10/U3, even faster than the usual class 10 card - and it's a noticeable boost. I'd estimate this pi somewhere between an i3 and i5 intel NUC in performance (at a lot lower price!). This matters because due to the cheap - I can have a hot spare ready to go no matter what and how far into the future it might fail. This would be bucks-wasteful on something like a NUC.
Posting as just me, not as the forum owner. Everything I say is "in my opinion" and YMMV -- which should go for everyone without saying.
User avatar
Doug Coulter
 
Posts: 3515
Joined: Wed Jul 14, 2010 7:05 pm
Location: Floyd county, VA, USA

Re: LAN iof things for the homestead

Postby Doug Coulter » Tue Apr 21, 2015 11:48 am

I got those cables made for the water flow counters. They're no better than expected...lots of jitter when I know there is none, but good enough for this use - and they told me (heh) the flap valve on the toilet I used to generate some fake demand was leaking, causing some water waste - perfect (for the monitor system, now I have to fix that leaky flap valve, though). They count pretty fast. Estimated peak output rate is around 6 gpm but shows a lot higher on the plot's arbitrary units even after dividing the raw counts/minute by 25 as the legend now shows. I could use a little more fancy in the cgi that makes these plots, to speed up the auto update meta line for short durations such as this plot (one hour, which is a lot of pixels/point at 1 point/minute).
So, I will, but now - from the comfort of my dev cockpit, not while standing shivering in some odd spot or crawling in a basement. #Winning.
WaterTestPlot.png
Testing the final (for now) sensors - water flow.


Interpreting this - I only have 4 discrete levels in the cistern I monitor, and they are (very) approximately 4ths of the total 225 gallon capacity. As time goes on and I eyeball when they actually switch - for certain, water will get much lower sometime this summer - I'll make the reported values more accurate, but they will always be steps, unless I find a way to get tricky and actually use the a/d inputs in some fashion to show values between the discrete levels via how hard the highest "true" sensor is being driven. Might happen, but no point holding your breath, it's just not that important for managing when I use a lot, or a little.

So, I was a little under full - not really down to 175 gallons as reported. I turned on the inlet via the cgi for that - the collection system is full - and waited to see it hit the overflow state, then turned it back off. That's the green line at the beginning around 15:52 (before that I think was a glitch, the vane sitting right on the edge of a count or something). When I saw it hit the overflow level, I used the water control cgi to turn the intake back off - and you see the green line go to zero again. That's how it's supposed to work. Not being able to resist, I then went and used up some water gratuitously, just to see the other sensor work. I'm guessing that peak (blue) is on the rough order of 6-7 gpm (pump rated at 5 gpm, but running on 18v, not 12v rating here). That took the water level out of overflow, so I came back here and let in some more - you can see the green line rise again around 16:19 (these are all UTC, which gnuplot understands and lack the complications of daylight savings time).

And oops, already showing me some value for my work - the blue peak just before and just after I started letting water back in are that leak in the toilet - I wouldn't normally have known until it ate a lot of water, it's not in the building I currently live in so I wouldn't hear the pump. (I fixed it temporarily by shoving it closed tighter with a stick...here in the sticks, we just get 'er done sometimes). And again, used the cgi for water control to shut off the inlet once it was going to go on the ground anyway - who knows, with this weather the water in the collection barrel might stay good a few days, and I can use more of it. When it's hot, or the water has pollen in it (there was a real gully washer last night and this is super clean stuff) - things grow in there and that's why there's a dump valve too...sometimes the better part of valor is just to dump the stuff if it isn't highest quality - and that's why I'm mentally modeling a design for seriously sensitive water turbidity detection that will work in the conditions that obtain out there - sun sometimes, freezing sometimes, wet sometimes...high winds, you name it. No, the cheapo ones for dishwashers are not even in the same order of magnitude sensitivity as what I want here - they can tell filthy water from merely unacceptable water - several orders of magnitude worse than the gov specs for drinking. I get the good stuff often enough...I just need to know what I've got without hoofing it over there and checking...it'll happen at some point.

Edit: FWIW, I did manage to "save" two pins on the arduino. The led pin, while hooked to a led, is now not used, but the command to toggle it is still there to be used - perhaps to control a solid state relay and a cistern heater. Pin 8 which shows ? in the document above is free, but no arduino code to use it exists yet - I don't know whether I'll want an input or an output anyway at this point. A little room for expansion (outputs are easier since I need not change the reporting to the database and add columns to it). And due to my slick serial protocol, worst case, I can add arduinos till it gets silly should I want to. I don't see the need yet, though I may add the one I have programmed to work the camera tilt/pan servo from yet another cgi page.
I am still undecided about keeping Silvan Melchoir's camera software - it's really easy to crash the stuff - just change a setting while the camera is stopped, or one that doesn't work, and down she goes. Then it takes a sudo command to restart raspimjpeg or you get errors. I keep the camera stopped for now as it's a big cpu heat generator when you're not even viewing it - yet another thing I'd have to change if I even keep that software. I'll be trying alternate ways in the near future, now that it basically works and it's easy to do. One good thing about the way it is - super low latency, under 100ms. Using things like VLC to do streaming takes that to seconds latency, not as useful for motion detection, seeing who is at the door or driving by on the road and so forth. It's a work in progress. Adding tilt/pan control to his spaghetti code doesn't look like fun either.
Posting as just me, not as the forum owner. Everything I say is "in my opinion" and YMMV -- which should go for everyone without saying.
User avatar
Doug Coulter
 
Posts: 3515
Joined: Wed Jul 14, 2010 7:05 pm
Location: Floyd county, VA, USA

Re: LAN iof things for the homestead

Postby Doug Coulter » Tue Apr 21, 2015 5:44 pm

Some more stuff - this is alpha, intended as a backup for moi, but feel free to play with it if you realize it's probably not going to be the same "in production". This database has two tables - one for testing, which is the populated one at the moment. Another identical one will be used "for real" - which will require changing the table name in various other pieces of software to fill and pull from the "production" table instead of "testing".
OTEnv.sql.zip
Most of this is from before I moved the unit - only about the last day is the data from the actual use location.
(1.07 MiB) Downloaded 459 times

I used phpmyadmin to create this backup. I *hope* it's enough to recreate the mysql completely, it hasn't always been. I did add the "add create table" etc stuff in the options, but it may fail to create the database itself. Nothing's perfect.

And here is the program that talks to the arduino and stuffs things into the db (all this runs on the pi).
StuffData.zip
Always-running program to stuff data into DB
(6.28 KiB) Downloaded 462 times

This also makes a log file when it starts, or has an error, in whatever directory it's in. I have it in pi/bin (which gets added to the path automatically). I run it via adding the line:
Code: Select all
/home/pi/bin/StuffData &
near the end of the file /etc/rc.local. The & at the end is because I haven't had it fork and become a daemon (yet) with a more-standard init. You might find some more tips on how to do things like this here. With the changes going on in the linux init system (huge flamewar going on now, largely justified IMO as the guy forcing the changes has never done anything else that worked right...)...it's hard to tell what will work a few versions from now.
I also have:
Code: Select all
/usr/bin/tellem
/usrbin/tellme

in the same file (just before
Code: Select all
exit 0
at the end) - you can get those elsewhere on this board (search for LANDNS), but this time I didn't use the installer I provided - I just copied them to /usr/bin and made them executable, since due to upstart and systemd fighting, the old way of starting them doesn't work everywhere. Idiots, sigh. You can get hints how to change their permissions etc by looking at the installer I provided in an editor. Running tellem and tellme on all the computers on my lan these days makes things nicer - these update your /etc/hosts file for address resolution, so you can do most things (surf, vnc, ssh) with just a name, and things resolve FAST since /etc/hosts is checked before any dns name resolution stuff external to the machine. These update automatically as machines are turned on and off...which is nice too.

In my home, I put this line:
Code: Select all
vncserver :1

in the .profile file (it's hidden until you hit ctrl-h) file to start the vnc server so I can login from this nicer machine to work on the pi with its gui. Handy.

This is the content of /var/www, which is where I put most of the web stuff. In copying, I noticed that gnuplot is making short "error" files every time it is run. That's stupid and an artifact of the perl module for gnuplot. I will fix that before production.
For reference, I'm using a big USB3 64 gb sandisk "extreme" disk split into two ext2 partitions (no journaling to make for write amplification) with 10 gb mounted over my /home/pi/bin and the rest mounted over /var for this. That way, it takes the beating instead of the little SD card which doesn't have as nice a wear-leveling controller and is harder to copy - the USB stick can be just a file copy...if/when it starts to have errors. This way, the SD card only has rarely written stuff like software on it, it's almost read-only. I also created a ramdisk of 8 gb that mounts on top of /var/tmp for making little files for state in cgi's and other uses like updatedb/locate. Again, all to reduce the writing on solid state "disks". More info on that on request, it's changing as I learn more and get this more smoothed out.
www.zip
This is /var/www content
(361.2 KiB) Downloaded 768 times


All this should be considered GPLV2 - share and enjoy, but hey - if you come up with an improvement - likely possible at this point - please do share it back. This is far from perfect right now. Most of that weird code in js and php isn't mine, it's the truly nutter camera stuff, which I plan to redo in fewer, clearer languages, and perhaps without raspimjpeg being used at all. To get that part in here, while using NGINX instead of the apache2 that Silvan's installer wants to put in, I just opened his shell installer file in an editor and did most of the steps manually - except for the apache stuff - and then put in NGINX and configured that manually as well. I just used a web tutorial for that latter which I think I've linked here somewhere already.
I modified the setup for php5-fpm (only needed for the cam stuff) to "ondemand" else it eats tons of ram for no reason. This isn't a big server serving the web for many hits - just me, no need to pre-fork and eat ram for tons of spare servers.

I'll see about documenting this more fully as time goes on. This isn't everything by far (for example, not much doc on what accessories I've downloaded and added, except in passing, config files I've modified or optimized and so on). Most sysadmins qualified to do this kind of work will already know that stuff. I had to learn it again, as that's not my main gig - I write code and design things.
Posting as just me, not as the forum owner. Everything I say is "in my opinion" and YMMV -- which should go for everyone without saying.
User avatar
Doug Coulter
 
Posts: 3515
Joined: Wed Jul 14, 2010 7:05 pm
Location: Floyd county, VA, USA

Re: LAN iof things for the homestead

Postby Doug Coulter » Wed May 06, 2015 1:07 pm

Here's some fun plots that this can and does serve on the web. This is all the data I've gotten since the beginning, including bench testing in another building, it's pretty obvious when I moved it - and also when the thermocouple amp failed the other day - I will be fixing/replacing that for the eventual woodstove installation here where I plan to move and have what most people would consider minimal living standards, compared to living in the shop as I have been.
You know, a real stove, plumbing...and so on.
Temp.png
Month duration, more or less

Baro.png
Baro.png (7.12 KiB) Viewed 7342 times

Humid.png

Water.png
Water.png (4.87 KiB) Viewed 7342 times

Stove.png
Stove.png (10.08 KiB) Viewed 7342 times


It's also obvious that when I moved it to here - the water level sensor got hooked up. Works quite nicely. I added an auto-fill option to one of the CGI's that once triggered (that's still manual after I check on water quality) - does the job nicely with zero waste and overflow. Ah, more showers for me now possible!
Posting as just me, not as the forum owner. Everything I say is "in my opinion" and YMMV -- which should go for everyone without saying.
User avatar
Doug Coulter
 
Posts: 3515
Joined: Wed Jul 14, 2010 7:05 pm
Location: Floyd county, VA, USA

Re: LAN iof things for the homestead

Postby Doug Coulter » Sat Aug 15, 2015 3:18 pm

Of course, this thing and I both need a place to live. The pi here will wind up in this new add-on to the "office" trailer. That will also be the living room, where the woodstove, comfy chairs and musical instruments, and I live. The existing living room is in the process of being converted to the remote fusor control room and for software development in comfort with some really nice gear (Pix later when it looks nicer and more pro, of course).
https://youtu.be/4taqqPHHWg8


For now, just getting ready. When I rebuilt the kitchen so that would be nice, the old wood heater had to go in favor of a really nice gas range/oven, so until I get this done - no (cheap) heat. Not a problem this week, but hey, this is the mountains and yes, we have winter, hardcore. From earlier experience (again pix once they make sense), I've laid this out so as to be able to get to the woodpile all while under roof, kick open the door with no hands (fancy latch), and be right at the wood crib, which is right inside the window that lets in fresh air to dry the wood and be heated by the stove on the way to the rest of the place. We call this planning where I live - I almost got there last time, only 6 feet or so not under roof, and a door I couldn't open without using hands...and that 6 feet was often ice-covered or being rained on furiously. Now...should be better over here.

The guys at the hardware store tell me that with this vinyl flooring all your issues are at the edges. Well then. All the edges except an unfortunate splice that has to be in the middle will have whole walls sitting on them. I bet they stay flat and down just fine. Most of this will have carpet over it, other than the area proximate to the front door, wood, and stove anyway. A little extra insulation never hurts. But you don't want carpet near a woodstove or wood - holds the splinters pointy end up for your bare feet, and the dirt...well, I don't like cleaning too much.
Posting as just me, not as the forum owner. Everything I say is "in my opinion" and YMMV -- which should go for everyone without saying.
User avatar
Doug Coulter
 
Posts: 3515
Joined: Wed Jul 14, 2010 7:05 pm
Location: Floyd county, VA, USA

Previous

Return to Combined projects

Who is online

Users browsing this forum: No registered users and 1 guest