Originally posted May 17, 2014:
Don’t believe it when the Electric Company tells you that the reason your town lost electricity for an hour was because a squirrel climbed onto a transformer and shorted it out. The real reason just may be more bizarre than that and the company doesn’t want you to know all the different creative ways that power can be shut off. This is a tale of just one of those ways. So, get out your pencil and paper and take notes.
Power Plant Notepad
One spring day in 1993 while sitting at the Precipitator computer for Unit one at the Coal-fired Power Plant in North Central Oklahoma, while I was checking the controls to make sure all the cabinets were operating correctly, suddenly there was a distant boom, and the lights in the control room went out. The computer stayed on because it was connected to an electric panel called the VSP or Vital Services Panel, which in turn was supplied by the UPS system (Uninterruptible Power Supply). That was one of those moments where you may pause for a moment to make sure you aren’t still at home dreaming before you fly into a panic.
The Precipitator cabinets all indicated on the computer that they had just shutdown. I rose from the chair and walked around to the front of the Alarm Panel for Unit one, and found that the fluorescent lights were only out on Unit 1. The lights were still on for Unit 2. The Control Panel was lit up like a Christmas Tree with Green, Red, Blue and Yellow Lights. The Alarm Printer was spewing out paper at high speed. As the large sheets of paper were pouring out onto the floor, I watched as Pat Quiring and other brave Power Plant Control Room operators were scurrying back and forth turning switch handles, pushing buttons, and checking pressure gauges.
Just this site alone gave me confidence that everything was going to be all right. These Control Room operators were all well trained for emergencies just like this, and each person knew what their job was. No one was panicking. Everyone was concentrating on the task at hand.
Someone told me that we lost Unit 1, and the Auxiliary Power to Unit 1 at the same time. So, Unit 1 was dead in the water. This meant, no fans, no pumps, no lights, no vending machines, no cold water at the water fountain and most importantly, no hot coffee!!! I could hear steam valves on the T-G floor banging open and the loud sound of steam escaping.
I turned quickly to go to the electric shop to see what I could do there in case I was needed. I bolted out the door and down the six flights of stairs to the Turbine-Generator (T-G) basement. Exiting the stairway, and entering the T-G basement the sound was deafening. I grabbed the earplugs that were dangling around my neck and crammed them into my ears. Steam was pouring out of various pop-off valves. I ducked into the electric shop where across the room Andy Tubbs, one of the electric foreman was pulling large sheets of electric blueprints from the print cabinet and laying them across the work table that doubled as the lunch table.
Andy Tubbs – True Power Plant Electrician
When I asked Andy what happened, I learned that somehow when a crew was flushing out a fire hydrant the water somehow shot up and into the bus work in the Auxiliary Substation (that supplies backup power to the Power Plant) and it shorted out the 189,000 volt substation directly to ground. When that happened it tripped unit 1 and the auxiliary substation at the same time leaving it without power.
An example of part of an Auxiliary Substation
I will explain how a fire hydrant could possibly spray the bus work in a substation in a little while, but first let me tell you what this meant at the moment to not have any power for a Power Plant Boiler and Turbine Generator that has just tripped when it was at full load which was around 515 Megawatts of power at the time.
Normally when a unit trips, the boiler cools down as the large Force Draft (FD) Fans blow air through the boiler while the even larger Induced Draft (ID) fans suck the air from the boiler on the other end and blow the hot air up the smoke stack. This causes the steam in the boiler tubes to condense back into water. Steam valves open on the boiler that allow excessive steam to escape.
When the boiler is running there is a large orange fireball hovering in space in the middle of the boiler. The boiler water is being circulated through the boiler and the Boiler Feed Pump Turbines are pumping steam back and forth between the turbine generator and the boiler reheating the steam until every bit of heat from the boiler that can be safely harnessed is used.
When all this stop suddenly, then it is important that the large fans keep running to cool down the steam, since it is no longer losing energy in the generator as it was when it was busy supplying electricity to 1/2 million people in Oklahoma City. The power is fed to the fans from the Auxiliary substation located right outside the Main Switchgear where all the breakers reside that supply the power to the fans. Unfortunately, in this case, the Auxiliary substation was shutdown as well, leaving the boiler without any fans.
Without fans for cooling, and pumps to circulate the water, the walls of the boiler began heating up to dangerous temperatures. Steam was whistling out of pop off valves, but if the steam drum on the top of the boiler were to run dry, then the entire boiler structure could be compromised and begin melting down. — So, this was serious. Something had to be done right away. It wouldn’t be as bad as the China Syndrome since we were burning coal instead of nuclear power, but it would have caused a lot of damage nonetheless.
From the movie “The China Syndrome” where a similar emergency existed only in the movie, it was a Nuclear Plant
I have a side story about this picture, but I think I’ll save it for another post because I don’t want to digress from the main story at this point (Ok. Let me just say “Jack Maloy and Merl Wright” for those who can’t wait) See the post: “Power Plant Conspiracy Theory“.
With the prospect that the boiler might melt to the ground in a pile of rubble, it would seem that the main priority was to turn the Auxiliary Substation back on so the fans could be turned back on and prevent the boiler from collapsing. So, we walked out to the substation and looked at the switches that would have to be operated in order to first power up the main bus and then to close to supply power to the two big transformers and the six smaller transformers that supplied the Unit 1 Main Switchgear.
While inspecting the switches where the electricity had gone to ground we found that one of the main insulators was cracked.
A High Voltage Insulator like this
Since this insulator was cracked, we didn’t really want to operate the switch to test if another 189,000 volts would go straight to ground again, especially since one of us would be standing right underneath it cranking the switch. So, we went back to the shop to find an alternative.
By this time the Plant Manager, Ron Kilman arrived in the shop, and understanding the urgency to find a solution asked us what were the alternatives. He was relying on our expertise to make the decision.
Ron Kilman is the one on the left in the plaid shirt
The other solution would be to cut the power over from Unit 2 which was still humming away pushing electricity to Oklahoma City out of the 345,000 volt substation. The cut over would be very simple because the switchgear was designed with this in mind. We analyzed the power rating on the auxiliary transformers on Unit 2 and thought that we might be cutting it close to have them running both sets of fans at the same time, especially since the full load amps of a huge fan starting up was about 10 times the normal rate.
The transformer was rated to handle the load, but consider this. What if this caused Unit 2 to trip as well. With the Auxiliary substation offline, if Unit 2 tripped, we would be in twice the amount of trouble we were currently in. What a day it would have been if that had happened and two 250 foot boilers had come crashing to the ground in a pile of rubble. After reading the power ratings on the auxiliary transformers I was thinking, “Yeah, let’s do it! These transformers can handle it.” Andy was not so eager.
So, we were left with one alternative. That was to shut the switch in the Auxiliary substation that had the cracked insulator and take our chances that it wasn’t going to short to ground and blow up over our heads. I think I was eager to close the switch for Andy, but if I remember correctly, he didn’t want me to be the one to suffer the consequences and decided to close the switch himself. Needless to say. Andy closed the switch, and nothing blew up.
High Voltage Switch similar to the one we were closing
As soon as the power was restored to the switchgear, the fans were powered up and the temperature in the boiler was quickly reduced. The coffee pot in the Electric Shop began heating the coffee again. The power plant was saved from a major catastrophe. That was delayed for another day… of which I will talk about later (see the post “Destruction of a Power Plant God).”
So, how exactly does a fire hydrant shoot water up into the bus work of a substation like the picture of the switch directly above? The culprit fire hydrant wasn’t in the substation, it sat alongside it outside the fence a good 50 feet from the high voltage switch. No hose was attached to the fire hydrant. It was only being flushed out as part of a yearly activity to go around and make sure the fire hydrants are all operating correctly.
Here is the story about how the squirrel climbed into the transformer this time….
George Alley, Dale Mitchell and Mickey Postman were going around to the 30,000 fire hydrants on the plant ground (ok. maybe not that many, but we did have a lot of them), and they were opening up the valves and flushing them out. That means, they were letting them run for a while to clear them out from any contaminates that may have built up over the year of not being used.
Throughout their adventure they had opened a multitude of Hydrants situated out in the fields along the long belt conveyor from the coalyard and around the two one-million gallon #2 Diesel tanks.
Large Oil Tank about the size of the two that are at the Power Plant
The brave Power Plant Men, learned that when opening a fire hydrant wide open in the middle of field had unintended consequences. It tended to wash out the ground in front of the flow of the water shooting out of the hydrant. So the team of experts devised a plan to place a board in front of the hydrant when it would be in danger of tearing a hole in the terrain. The board would divert the water into the air where it would fan out and not cause damage to the surrounding area.
This was working fine, and when they arrived at the fire hydrant next to the substation, since the stream from the hydrant was pointing directly into the substation (hmm. a design flaw, I think), they decided to prop the board up against the fence to keep from washing away the gravel in the substation. Well. When a fire hydrant is opened that hasn’t been used for a year, the first flow of water to shoot out is dark brown.
You may think that this is because the water has somehow become dirty over the past year, but that isn’t quite the case. What has happened is that the pipe has been rusting little by little and the water has become saturated with the rust. So, the water shooting out of the hydrant was full of rust (hence the need to flush them out).
Well. Rust is made of metal. Metal is conductive, especially when it is mixed with water. When the water hit the board, it was deflected into the air and happened to direct itself directly into the high voltage switch in the substation. This caused a circuit to the ground which, once it created an arc pumped all the electricity directly into the ground.
Normally when something like this happens it doesn’t trip the Main Power Transformer to a Power Plant.
A Main Power Transformer
This time it did. I know there was a few heads scratching trying to figure it out. I think I figured out what happened a little while later. You see… here is the rest of the story….
Once the unit was back online and the emergency was over, someone finally noticed that the telephone system couldn’t call outside of the plant. Well. I was the main telephone person at the time, so the control room called me and asked me to look into the problem.
I checked the telephone computer and it was up and running just fine. Internal calls could be made. Only any call outside just concluded with a funny humming sound. After checking the circuit in the Logic Room next to the Rolm Telephone Computer I headed for…. guess where….. the Main Switchgear….
A ROLM Phone Computer
In the middle of the main switchgear in the back of the room right next to the Auxiliary Substation beyond the back wall, the outside telephone line came into the plant. The first thing it did was go through a special Telephone Surge Protector.
Telephone Grounding Panel
In this picture above, the silver circular buttons on the left side are really an old style surge protector. whenever there was a power surge, the carbon connection in the surge protector would quickly melt causing the circuit to go straight to ground. Thus protecting the rest of the telephone circuit. So, if some kid in their house decides to connect the 120 volts circuit to the telephone for fun to see what would happen, this circuit would protect the rest of the phone circuits. Keep in mind that this was during the early 1990 when “Surge Protection” still was basically all “mechanical”.
Anyway, when I arrived at this panel and I checked the surge protector to the main line going out of the plant, guess what I found…. Yep. Shorted to ground. Luckily there were some spares that were not wired to anything in the panel and I was able to swap them out for the ones that had been destroyed. — These were a one time use. Which meant, if they ever had to short to ground, they had to be replaced.
Ok. Fine. After a little while, we were able to call back out of the plant, though there was still some residual noise on the line. It was like this… when you called out of the plant, the person on the other end sounded like they were buried in a grave somewhere and they were trying to talk to someone living just like in an episode of the Twilight Episode where a phone line landed on a grave and the dead person tried to call his long lost love from the past.
Gladys Cooper in the Twilight Zone Episode “Night Call”
I didn’t give it much thought other than that I figured the 189,000 volt arc to ground must have shorted out the telephone line since the phone line ran directly under the auxiliary substation ground grid.
It wasn’t until the next morning when the Southwestern Bell repairman showed up at the plant. I knew him well, since he had been working on our phone lines since before the AT&T breakup in 1984. When I met him in the front of the electric shop, he said that he needed to check our telephone circuits. I told him that I knew that we had a problem because we had a high voltage short to ground yesterday and I found our surge protectors melted away.
He explained to me that not only was our circuit affected, but that every relay house from here to Ponca City was blown out. That’s when I realized that the problem was the reverse of the usual situation. What had happened was that the Ground Grid in the substation and the surrounding area (including the Unit 1 Main Power Transformer) had become hot. What do you do when the ground grid becomes charged?
The Ground Grid is what is supposed to protect you when a surge happens, but what happens when the ground grid itself is the problem? In this case, when the high voltage line about 60 feet from the telephone cable surge protector, arced to ground, it fed a tremendous amount of power back through the ground grid. when equipment detected the surge in voltage, they automatically defaulted their circuits to ground. That’s why the telephone circuit died. That’s what tripped the Main Power Transformer.
When the telephone circuit detected the high voltage surge, it shorted to ground (which was the problem), causing the high voltage to feed directly into the phone line and down the line to the next Southwestern Bell relay switch, which also defaulted to ground, trying to bleed off the surge as it went from relay switch to switch until enough of the power was able to be diverted to ground.
That day sure turned out to be a learning experience. I learned that when all the lights go out in the control room, that it is almost assured that the coffee pot in the electric shop is going to stop working. I also learned that in order to coax the plant manager to the electric shop, a major electrical tragedy is one good way. I learned that when shooting rusty water into the air don’t point it at a high voltage auxiliary substation switch. — I’m sure Mickey Postman learned that lesson too. I also learned that just like in Star Trek… whenever there is a dangerous job to do, the Captain is always the one that wants to do it. Does that make sense? Send a Peon like me in there…
I also learned something else about Power Plant Men…. You see…. People like Dale Mitchell, George Alley and Mickey Postman all are examples of incredibly wonderful Power Plant Men. When they were out there doing their duty and something tragic like this, all the Power Plant Men felt their pain. They knew that they all felt guilty for tripping the unit. It didn’t matter that a million dollars every so many minutes was walking out the door in revenue. The only thing that mattered was that these three men were safe.
Mickey Postman
Since I have left the Power Plant, I have found that the idea that the employee is the greatest asset that a company can possess is not a universal idea. You see, there was never the thought that any of these people should be fired for their mistake. On the contrary. The true Power Plant Men did whatever they could to let them know that they knew exactly how they felt. It could have happened to any of them.
Besides the friendship between Power Plant Men, one of the things I miss most about working at the Power Plant is that the employees are held in high esteem as a real asset to the company. Many could learn from their example.
Comments from the Original post
-
-
I like the mix of storytelling and information sharing you deliver here. Thanks again
-
-
-
Aug 8, 2011 I lost all reserve in and lost both units in 2 separate storms and A1 would not start for a few minutes and A2 was leaking antifreeze terribly not a scratch and back online in 34 hours both units and no one ever asked me one question even with a Safety Dept not one question asked of me the SS on duty. Not many men on the planet have ever experienced an uncontrolled total plant outage- you would of thought a learning opportunity would of took place. Feb 2011 worst winter temps in years 50 Units or more tripped in Texas let alone the trusty units of Redbud and McClain were fighting Sooner rolled right along with storm warnings for two weeks ahead – the ICS still went to Detroit for just tours of other facilities once again the fall guy Tarver McArthur stood alone. I had authored a Freeze Protection Plan for the plant and that seemed to save the day and explain to the regulatory bodies how we were online and everyone else in Texas was off and enjoying rolling blackouts in a terrible winter weather situation not to mention our powers that be were all stranded in Detroit and very few people could get to the plant without getting stuck trying to get there as well – but a few health heart issues later I am still here to tell about it all you would think folks would want to take advantage of someone that had went through the fire and Ice but thats ok I want them someday distant to get all the credit they deserve when the trumpets sound as that is truly what matters most. DT
-
-
Great story, and yes, you took exactly the proper lesson from it, and it is too bad that many of our bosses haven’t learned it.
And yup, I’ve buried a lot of electrocuted squirrels over the years.
Power Plant Men 
That was an exciting day! Another great story. Thanks for the memories.