Speaking of ’stewardship of stuff’, several energy machines at the shop are sick.
our generator seems to have gotten bit by the load of bad BioEnergy Systems fuel that went around earlier in the summer (in fact I helped figure out that it was BioEnergy’s fuel rather than World Energy which we all thought we were buying). We got clogged injectors and crap had settled into the bottom of the tank after just 50-100 gallons of that stuff, and it’s costing big bucks to deal with.
and our batteries are sulfated. That part is obscene. Many people who go offgrid end up ruining their first set of batteries, and they learn a lesson from it and are careful with their replacement set. We’ve sulfated our replacement set.
…a week ago I was walking around school thinking about people who get excited about going offgrid with a generator. I hear from them a lot. There’s a false sense that generators on biodiesel or SVO seem like a limitless source of offgrid energy- you dont have the dramatic charging limits of PV, so folks feel like it’d be less of a change to their lifestyle. This is the sort of offgrid system we built at the Shop- so much charging capacity and so much battery power and so much inverter that in theory ‘you don’t have to conserve’.
I always want to warn would-be generator users that it’s a massive hassle and that the maintenance isn’t something to take lightly. In many cases the folks wanting to mess with a generator are gearheads like us who think of ‘maintenance’ as changing the occasional oil or fuel filter. IN our case, the fuel filters and the occasional contaminated 300 gallon fuel tank has COST and cost and cost. When we bought some of Jess’ crappy fuel (it developed a crystalisation of monoglycerides problem) last winter, we estimated that it eventually ended up costing us $15 a gallon to deal with the fallout- by the time we were done draining fuel and refilling tanks and purging air and pulling injectors and so forth. And that was with the older crappier generator. The newer, cleaner one seems to be more sensitive to fuel fluctuations.
So when I got on the computer that night after thinking about the ‘false economy’ of generators versus conservation and PV, I saw that the Shop email list was overrun by panicked generator-related email.
Behold- I present to you, the weirdness that is the Shop and it’s list and its energy issues. I actually think this will be funny to those who’ve tried to run a cobbled-togetherpower system, and I"m trying to dissuade any of the rest of you would-be offgrid hippies from trying it.
Don’t buy a generator: The Machine Shop Generator Chronicles of October 2005
note: All this frantic keyboard-strangling took place over 2 hours (mostly in the first half hour), and there are a few extra emails I’ve even deleted:
The Investor writes:
The whisperwatt is belching smoke for some unknown reason. please don’t start it unless your name is P. and you need to for testing.
Until the smoking is figured out, all battery charging should use the top grey generator.
There is now a fuel valve that will direct fuel up there. push 3 way ball valve to the
right of the xfer switch box to up position and fuel will go up there. make sure you hear
pump going in Biodiesel room. Not sure if pump power line got moved to the center
power buss on the xfer switch so it works off of either generator. Liam?
Also make sure xfer switch is in the up position for the top generator.
Brian may or may not have landed the power wires top and bottom to the grey generator.
Look inside the xfer box to make sure. If he has not and you need power, please call him
at 510.xxx-xxxx and ask him very nicely if he will come over and finish the wiring.
then they argue about whether the thing is actually smoking and what is the exact state of the ‘hack’ that we’ve contrived (ie the ‘gen control’ , a custom circuit board P. built):
TheInvestor:
I understand the batteries cut out after hours of charging yesterday. the inverters were in float but the gen did not shut off. I don’t know what the issue is with the gen
control. Maybe P. does. I do know the batteries desperately need an equalization but
that can’t happen until the gen control is done and the generator is not smoking. But I
can’t come over and there and deal with this stuff this week. and liam has school stuff.
So not sure what to do.
Also, the whisperwatt continues to smoke terribly. The injectors need to be pulled as
they are likely not seating well.
Until then, battery charging should use the grey generator. But I’m not sure if it is
wired in. brian? Did you finish the wiring?
If all else fails, I think liam has some kerosene lanterns. And wood is easy to work with
hand saws and hammers.
Sometimes it feels like this just never ends. And some times I simply cannot come over
there to deal with it. such is the case this week.
Not having made his point sufficiently strongly, the same caffeinnated guy argues in a second email to no one in particular three minutes later:
i cannot start the EQ on a questionable generator. if it starts smoking again in the middle, 100s of dollars of fuel get wasted. this is what happened last time and remember the mess and multiple cycles that resulted. i am not starting it again under similar fucked up conditions. also, i’m not terribly interested in sooting the neighborhood for 5 days either.
the injectors need to get pulled and taken to the shop. liam cannot do it due to school.
nesdon, can you pull the injectors tomorrow possibly? crisp money can be paid injectors pulled, shopped and reinstalled to correct torque settings tomorrow.
then i will EQ. but i can’t until this is done. it is easy, but i can’t get there during a weekday to do it.
The EQ thing he’s referring to is an equalizing charge, when they run the genny for days on end. This should happen every few weeks. In our case we’ve had to do it due to torching our batteries, which should have never happened… the second time.
P. writes and suggests we can switch to our really crappy, loud, less-reliable old generator:
It’s absolutely NOT smoking at all right now. I could see “hot air distortion” at the top, but nothing else. Liam saw it burning clean too.
Even so, you can always switch to the gray genny (or even run it instead).
The smoke a couple days ago was thick black sooty stuff that made your eyes burn. Then when I was testing the auto-start, I noticed it was nowhere near as bad, and now, it’s better than most diesels.
The Investor, who has an unwisely abusive relationship with P. (who’s volunteering through all this and isnt’ even a part of our community), proceeds to insult his volunteer, as usual:
Phil, don’t enable a half ass solution that is likely going to bite us later. i do not have time to ignore recurring problems that are going to cost money and hassle later. fix the problem when realized.
this smoking problem is recurring problem. it has come and gone several times. that it is not smoking now in no way changes taht this is a recurring problem. this is an intermitently sealing injector seat or the like. i would to get them looked at by the injector shop.
the grey generator burns vastly more fuel. it is also noisy and bubbles loses her beauty sleep and gets cranky. next time i have to switch to the grey one for a stupid reason, remind me to give you the bill for the extra fuel.
the EQ can’t start until the whisperwatt known issues are fixed.
P. points out:
I would say it’s air in the fuel or other “goodies” in the fuel. The acrid smoke makes me lean toward the latter. I don’t think it’s an intermittent injector, but either way, it’s your call.
The Investor says:
it has done this on many fuels, delivered both from the native tank and the tote. all the filters have been changed. the air filter has been pulled to see if any difference was made. none.
when we first installed the new injectors it smoked like crap. i figured the old poorly atomizing injectors had carboned up the motor. so i gave it an italian tune up and floored it. i pegged all the inverters on max charge, turned on every electrical device in the yard, which loaded the gen well over 30k, and ran it as such for two hours. afterwards it ran much cleaner, but never really right.
some weeks later, cycles of heavy smoking and non smoking started again. now i could this time give it a northern, italian tune up and beat it even harder, but i’d really rather take it to a shop in rome and have the proper rites administered by trained experts.
as it is, white smoke means there is no new pope. and it is about time we got a new pope.
It comes up that we dont’ know actually the name of the shop that we use, because we keep really crappy records:
the Investor explains:
i believe it is the place you were sent to with the dt466 injectors. i do not konw the name. the receipt is lost int he annals of Machine Shop virtual accounting.
but talking will likely work. make it sound bad, not good. claim art. we bought them there 3 months ago. explain we have gone through everything multiple times and are getting intermittent smoking that seems like injectors. stress the intermittent part so if there tests show things ok, still request an exchange. they just rebuild these things so it is not much skin off their fingers.
start and stop the engine cold several times before you pull them to make them in the worst shape possible. though no, smoke is not normal on this engine when cold. maybe a litle black on initial fire, but not while running. this engine should run clear exhaust. this is a CARB compliant engine. clean clean.
i have figured out how to be int he east bay tomorrow afternoon from about 3 onward. so if you can get the injectors out and to hte place and back, i can get them back in later int he day. or i can go pick them up if they need to stay there for a bit. i want to check compression too. and whack the thing with a large pipe several times just for good measure.
someone brings up an entirely other new problem we’re having. Yes, it really never ends around here. And it all needs to be discussed via email list, and in many emails. There’s a joke that in a post-earthquake, Katrina-style disaster, the place would still have the capability to make electricity, but since the Internet will go down there’ll be no way for those stranded at the Shop to know how to run the generator syste, because it’s in such a constant (yet changing) state of ‘hack’ that always requires detailed emails to explain. And the Gremlins that occupy such a hacked-together system will absolutely thrive in an absense of debugging emails…
Well, it auto started some time in the night, and is running, and is not
smoking at all but two of the gauges are fritzing. First time I’ve seem
this, the RPM’s are oscillating at about 8 hz between 1800 and 2000, (it
looks kinda like a bad cable, but I assume its fully electrical) and the
ammeter is jumping but with a little jerky randomness between 15 and 25.
hmmm
Then- here’s the unique moment at the intersection of nerd and gearhead:
It started at 8:24 this morning. It emailed me:
Message from phil-O-Matic Generator Control System: Startup Sequence
Message from phil-O-Matic Generator Control System: Glow Plugs On
Message from phil-O-Matic Generator Control System: Cranking Engine
Message from phil-O-Matic Generator Control System: Engine Warm-Up
Message from phil-O-Matic Generator Control System: Generator Online
Then at 9:49 this morning:
Message from phil-O-Matic Generator Control System: Engine Cool Down
Message from phil-O-Matic Generator Control System: Generator Offline
The gauges are normal. The Tacho started doing that after Jim got water in it. I drilled a hole and let the water out, and it was ok for awhile, but now it’s moving wild. It clearly isn’t varying that much. According to the service manual, it’s got an electric tachometer, but it’s actually driven off a cable. I know this because I built the auto-start originally to use the pulse from that pickup that doesn’t exist.
The ammeter jumps because of the regulator. It tends to “hunt” when there is little load. But 15 to 25 sounds like a little much. I’ll check it today.
-Phil
the Investor writes:
Unfortunately, I must take a moment out from stewing about the unrelenting toil and annoyance of all things Shop Infrastructure to note how, ummm, unbelievably cool (and geeky) it is that the generator emailed phil and told him what it was doing. I do not believe that is a feature that comes stock with the unit.
Can you add me to the cc list?
P continues:
Since the Genny start is set just above the LVCO, this means power would have been lost this morning sometime had it not started. The absorption time is set at 1 hour, so that means it only took 25 minutes of bulk charging. I can only assume BAD sulfation!
I didn’t really look at the voltage trips in there. Do you recall what they are set at?
-Phil
Investor talks about sulfation (which he’s 100% responsible for, since he;s the one who decided to tweak the inverter settings too low). The ‘EQ routine’ he’s describing is a matter of running the generator for 6 days straight to do certain types of charging to help desulfate the sick batteries:
Yes, this means the batteries are nearly as unhappy as the rabbit. Or maybe I should say the hamsters unhappy. the first time I showed darryl the batteries, he said, “oh, so this is where you keep the hamsters".
But yes, the batteries should charge for at least 2 hours in bulk. They are heavily
sulfated. You can probably see it crystallized on the negative plates. This is very bad
we keep doing this. we can NOT run them into the ground like this and expect much life
out of them.
I got a new EQ scenario from the factory for heavy desulfation. It takes 6 days. Be
warned . . .
I’m actually sparing you guys a few of the emails. It turns out there’s also no emergency-disconnect at the generator, so if the inverters (across the property) decide to start it while you’re trying to work on it, you’re screwed.
The Investor writes:
Phil,, there MUST be an autostart disable at the generator. Like the old twist lock we
had. Something simple and obvious must be at the generator. I suggest a toggle switch
mounted next to the circuit breaker. There are nice toggle switches up in the electrical
hardware store by the hamsters. Above the box with the circuit breakers.
aah, it turns out the safety disconnect is a ‘work in progress’. Phil writes:
This is designed into the system, but not yet wired up. It is simple: If the big genny selector switch is “off", it will not start. This is a needed interlock, and also functions in 2 ways for safety.
-Phil
P. then replies to the point about the generator’s ability to email him. As you can imagine, our crowd itself creates an immense amount of email- it’s quite scary to me that the inanimate objects can now start to send us email:
Keep in mind it’s in a debugging mode, so you may get about 20 emails during each event (startup, shutdown, etc).
As I refine the software, this will stop and be reduced to only emails with “alert” status, like start errors. However there will soon be a web page that you can peruse to check the current status of the generator, and the log of events. This will also be shown on the display of the power control center. I also have the “alert” emails sent as text messages to my cellphone.
I am still working the kinks out of the genny control software, so expect to see some strange messages if you join.
-Phil
someone else points out:
Phil, was this all your idea? There must be, by now, something protectable
(patentable) here. If you developed it to this level recently, you may have
been working for The Investor, in which case ownership would be dubious, but without
wanting to start a fight (please guys) it’s friggin’ cool, and should be out
there.
Jake.
The Investor takes his opportunity to rib his volunteer P, which is kinda a dumb habit since, as I said, P does all this crazy electrical engineering stuff for free. Last year there was a minor rent rebellion when he was particularly nasty to Phil while Phil was headed into uncharted territory hacking the inverter synchronisation
Unfortunately phil is constitutionally incapable (or just terribly resistant) to
documenting anything, so it is very difficult to submit patent papers, or even
manufacturing drawing to a producer where more might be made and sold. It is all only
lodged in his mind, but usually gets pushed out by the next impossible (yet successful)
hack.
phil can’t even remember what he did in the 3phase Phil-o-Matic as he made it at 5 am and it
has 200 soldered connections and a weird pulsing heart. It is the absolute core of this
whole system and if it ever went down we are fucked and no one knows how to build another
one. Smart people would have another one there waiting to go. But us? Not even the
software loaded on the eprom is backed up. Duh!
Phil would be a very rich man if only for the spoils that documentation would bring . . .
P defends himself (I would have told The Investor to go fuck himself long before, personally).
More importantly, check out some of the electrical weirdness we’ve been building back there.
the background ( I think I mentioned this in a blog a year and a half ago):
We got a really good deal on some Xantrex inverters and we devised a system whereby we can get both 208 three-phase and split phase 120/240 out of four of them. Those inverters are certainly available in 208/three phase configurations but that would have cost us $6k extra.
We bought the ’singles’ and then sat around with a huge whiteboard designing how we would hack them to talk to each other so as to synchronise into three-phase power. I feel really bad about Phil’s couple of years of Investor hell as I helped make all this happen.
Of course we first asked Xantrex nicely to just sell us the chip that’s installed in their three-phase power centers. They refused.
We at one point joked about photographing our whiteboard schematic and mailing it to the inverter manufacturer as blackmail- “if you dont’ sell us the firmware we’ll publish this diagram! “
Phil says he’s working against buggy software in the inverters themselves, without knowing what exactly it does. The result, um, works. He’s a self-taught electronics genius who somehow manages to do all this while claiming to know no math cause he didn’t go to school. Damn!
Cool as this all is, it’s been like 2 years of constant hacking and refining. We would have been best off paying the 6 grand and gotten an off-the-shelf solution.
Phil writes:
First off, you are wrong on the inverter phase synchronizer. There is no documentation, but there are only 3 active parts in there, the main one is a microcontroller, then there is a quartz crystal, and finally and LED. (which has already entered into the most bizarre failure mode I’ve ever seen!)
So barring a fire or someone smashing it with a hammer, all the needs to be done in the event of a failure is burn a new microcontroller with code that IS backed up in 3 states. It is even mounted with a socket for ease of servicing, so it will take all of 5 minutes to fix. Since this is a software-driven system, the software is really all that’s needed for documentation. Which I have. In triplicate.
The genny auto-start is a different story, as it has much more hardware, but you don’t need documentation to troubleshoot and repair such a simple system anyway. The problem with documenting this is not me, even though I don’t enjoy doing it, it’s the fact that we are HACKING on UNDOCUMENTED systems to begin with! The inverter ports are totally undocumented with software full of bugs, and the genny documentation is sparse, only a hook-up diagram with no full schematics, and on top of that, it’s incorrect by a number of points!
On top of this, The Investor only paid me $500 (He initially only paid me $300) for the entire genny controller which is insane for all the hours that went into it, and are still going into it.
Also: I have built much more marketable products than a generator that emails. This is a limited market thing, and it will cost more to market than you could make selling it.
-Phil
another drudge volunteer:
So the injectors were judged as “weak” by Diamond, they are lapping and
honing (hopefully at no charge) them as we “speak", and I will reinstall
them this afternoon. but Liam did provide me with receipts and docs, but it
has been less than 90 days (approx 70) which is likely the warranty period.
He remarked that the only thing that he knew that could cause such a rapid
deterioration was contaminated fuel, and as lousy as that last batch was,
when we saw all sorts of crap plug up the filters for the upper genny, I
kept my mouth shut.
My auto start disable technique was to pull the + terminals. I did set all
the inverters to gen-off, but just to make 110% certain than I didn’t get my
fingers ripped off, I pulled the terminals, a typical mechanics safety tip.
On to equalization!
So far, I think that what happened with the bad fuel from June/July affecting us so badly in August, is that I’d used a really fast new pump to blast-fill some fuel into the generator’s fuel tank, and the visible layer of ‘crap’ that was left in the fuel tank from the BioEnergy Systems fuel got re-stirred up and got past our filters and promptly clogged injectors.
I’d told the Investor two months before that we’d developed a layer of crap in the bottom of the fuel tank, where it came from, that it had happened to others who had the same fuel, and that it should be drained or filtered out. I guess he ignored that. I just now found out that injector damage had started right then, as well.
the injectors had damage that indicates bad fuel. They are needing new nozzles and a cylinder honing (the injector cylinders, not the motor). They were only in there
two months, but yes, we ran lots of crap through it. but we have previously run lots of
crap through it everything here without problem. Maybe the isuzu in the whisperwatt is
more sensitive than the grey gen.
Anyways, if we are messing up injectors, we might also be messing up injection pumps,
which are deeply expensive.
Given the ambiguities of hippie fuel, we should maybe install a more bomber water oil
separator and finer filter. the napa ones are not likely that fine. Maybe 50 microns at
best. It think you really need a 10 micron one to get all things of relevance.
Girlmark, what do you recommend?
here’s a choice quote that explains how our many systems are hacked together,and why it’s been really difficult to lure non-gearheads here to work on the biodiesel project- this is typical Shop engineering of systems:
Wonderful news. I will be over at the yard about 8pm tonight to start the EQ. please do NOT run the gen tons before I get there. if charging is needed, continue to use the grey one. I want to load it heavily from the beginning cause the motor is likely carbonized
again and I want to get that out as quickly as possible. The only way to load it heavily
enough is with a bunch of inverter setting foo which I cannot explain in email.
he’s discussing how to select between generators for the fuel system. We practically have an SVO type setup in there in terms of fuel tanks and such.Just make sure you plug the BX cable into the right place. And remember a dozen other things at the same time, which change from week to week as the system evolves…
Remember, to run it, you need to switch the 3 way fuel valve to down and make sure you
plug the tote pump BX line into something. When running the whisperwatt, you can plug it
into the side of the generator. When running the grey one, you need to plug it into the
local grid power. This line still needs to go to one of the center power terminals in the
xfer switch so that the pump is powered with either the top or bottom generator is
running.