Runaway Fire Hydrant Leaves Power Plant in the Dark
Favorites Post #13 (posted in no particular order)
Originally posted May 17, 2014:
Don’t believe it when the Electric Company tells you that the reason your town lost electricity for an hour was because a squirrel climbed onto a transformer and shorted it out. The real reason just may be more bizarre than that and the company doesn’t want you to know all the different creative ways that power can be shut off. This is a tale of just one of those ways. So, get out your pencil and paper and take notes.
One spring day in 1993 while sitting at the Precipitator computer for Unit one at the Coal-fired Power Plant in North Central Oklahoma, while I was checking the controls to make sure all the cabinets were operating correctly, suddenly there was a distant boom, and the lights in the control room went out. The computer stayed on because it was connected to an electric panel called the VSP or Vital Services Panel, which in turn was supplied by the UPS system (Uninterruptible Power Supply). That was one of those moments where you may pause for a moment to make sure you aren’t still at home dreaming before you fly into a panic.
The Precipitator cabinets all indicated on the computer that they had just shutdown. I rose from the chair and walked around to the front of the Alarm Panel for Unit one, and found that the fluorescent lights were only out on Unit 1. The lights were still on for Unit 2. The Control Panel was lit up like a Christmas Tree with Green, Red, Blue and Yellow Lights. The Alarm Printer was spewing out paper at high speed. As the large sheets of paper were pouring out onto the floor, I watched as Pat Quiring and other brave Power Plant Control Room operators were scurrying back and forth turning switch handles, pushing buttons, and checking pressure gauges.
Just this site alone gave me confidence that everything was going to be all right. These Control Room operators were all well trained for emergencies just like this, and each person knew what their job was. No one was panicking. Everyone was concentrating on the task at hand.
Someone told me that we lost Unit 1, and the Auxiliary Power to Unit 1 at the same time. So, Unit 1 was dead in the water. This meant, no fans, no pumps, no lights, no vending machines, no cold water at the water fountain and most importantly, no hot coffee!!! I could hear steam valves on the T-G floor banging open and the loud sound of steam escaping.
I turned quickly to go to the electric shop to see what I could do there in case I was needed. I bolted out the door and down the six flights of stairs to the Turbine-Generator (T-G) basement. Exiting the stairway, and entering the T-G basement the sound was deafening. I grabbed the earplugs that were dangling around my neck and crammed them into my ears. Steam was pouring out of various pop-off valves. I ducked into the electric shop where across the room Andy Tubbs, one of the electric foreman was pulling large sheets of electric blueprints from the print cabinet and laying them across the work table that doubled as the lunch table.
When I asked Andy what happened, I learned that somehow when a crew was flushing out a fire hydrant the water somehow shot up and into the bus work in the Auxiliary Substation (that supplies backup power to the Power Plant) and it shorted out the 189,000 volt substation directly to ground. When that happened it tripped unit 1 and the auxiliary substation at the same time leaving it without power.
I will explain how a fire hydrant could possibly spray the bus work in a substation in a little while, but first let me tell you what this meant at the moment to not have any power for a Power Plant Boiler and Turbine Generator that has just tripped when it was at full load which was around 515 Megawatts of power at the time.
Normally when a unit trips, the boiler cools down as the large Force Draft (FD) Fans blow air through the boiler while the even larger Induced Draft (ID) fans suck the air from the boiler on the other end and blow the hot air up the smoke stack. This causes the steam in the boiler tubes to condense back into water. Steam valves open on the boiler that allow excessive steam to escape.
When the boiler is running there is a large orange fireball hovering in space in the middle of the boiler. The boiler water is being circulated through the boiler and the Boiler Feed Pump Turbines are pumping steam back and forth between the turbine generator and the boiler reheating the steam until every bit of heat from the boiler that can be safely harnessed is used.
When all this stop suddenly, then it is important that the large fans keep running to cool down the steam, since it is no longer losing energy in the generator as it was when it was busy supplying electricity to 1/2 million people in Oklahoma City. The power is fed to the fans from the Auxiliary substation located right outside the Main Switchgear where all the breakers reside that supply the power to the fans. Unfortunately, in this case, the Auxiliary substation was shutdown as well, leaving the boiler without any fans.
Without fans for cooling, and pumps to circulate the water, the walls of the boiler began heating up to dangerous temperatures. Steam was whistling out of pop off valves, but if the steam drum on the top of the boiler were to run dry, then the entire boiler structure could be compromised and begin melting down. — So, this was serious. Something had to be done right away. It wouldn’t be as bad as the China Syndrome since we were burning coal instead of nuclear power, but it would have caused a lot of damage nonetheless.

From the movie “The China Syndrome” where a similar emergency existed only in the movie, it was a Nuclear Plant
I have a side story about this picture, but I think I’ll save it for another post because I don’t want to digress from the main story at this point (Ok. Let me just say “Jack Maloy and Merl Wright” for those who can’t wait. See the post: “Power Plant Conspiracy Theory“).
With the prospect that the boiler might melt to the ground in a pile of rubble, it would seem that the main priority was to turn the Auxiliary Substation back on so the fans could be turned back on and prevent the boiler from collapsing. So, we walked out to the substation and looked at the switches that would have to be operated in order to first power up the main bus and then to close to supply power to the two big transformers and the six smaller transformers that supplied the Unit 1 Main Switchgear.
While inspecting the switches where the electricity had gone to ground we found that one of the main insulators was cracked.
Since this insulator was cracked, we didn’t really want to operate the switch to test if another 189,000 volts would go straight to ground again, especially since one of us would be standing right underneath it cranking the switch. So, we went back to the shop to find an alternative.
By this time the Plant Manager, Ron Kilman arrived in the shop, and understanding the urgency to find a solution asked us what were the alternatives. He was relying on our expertise to make the decision.
The other solution would be to cut the power over from Unit 2 which was still humming away pushing electricity to Oklahoma City out of the 345,000 volt substation. The cut over would be very simple because the switchgear was designed with this in mind. We analyzed the power rating on the auxiliary transformers on Unit 2 and thought that we might be cutting it close to have them running both sets of fans at the same time, especially since the full load amps of a huge fan starting up was about 10 times the normal rate.
The transformer was rated to handle the load, but consider this. What if this caused Unit 2 to trip as well. With the Auxiliary substation offline, if Unit 2 tripped, we would be in twice the amount of trouble we were currently in. What a day it would have been if that had happened and two 250 foot boilers had come crashing to the ground in a pile of rubble. After reading the power ratings on the auxiliary transformers I was thinking, “Yeah, let’s do it! These transformers can handle it.” Andy was not so eager.
So, we were left with one alternative. That was to shut the switch in the Auxiliary substation that had the cracked insulator and take our chances that it wasn’t going to short to ground and blow up over our heads. I think I was eager to close the switch for Andy, but if I remember correctly, he didn’t want me to be the one to suffer the consequences and decided to close the switch himself. Needless to say. Andy closed the switch, and nothing blew up.
As soon as the power was restored to the switchgear, the fans were powered up and the temperature in the boiler was quickly reduced. The coffee pot in the Electric Shop began heating the coffee again. The power plant was saved from a major catastrophe. That was delayed for another day… of which I will talk about later (see the post “Destruction of a Power Plant God).”
So, how exactly does a fire hydrant shoot water up into the bus work of a substation like the picture of the switch directly above? The culprit fire hydrant wasn’t in the substation, it sat alongside it outside the fence a good 50 feet from the high voltage switch. No hose was attached to the fire hydrant. It was only being flushed out as part of a yearly activity to go around and make sure the fire hydrants are all operating correctly.
Here is the story about how the squirrel climbed into the transformer this time….
George Alley, Dale Mitchell and Mickey Postman were going around to the 30,000 fire hydrants on the plant ground (ok. maybe not that many, but we did have a lot of them), and they were opening up the valves and flushing them out. That means, they were letting them run for a while to clear them out from any contaminates that may have built up over the year of not being used.
Throughout their adventure they had opened a multitude of Hydrants situated out in the fields along the long belt conveyor from the coalyard and around the two one-million gallon #2 Diesel tanks.
The brave Power Plant Men, learned that when opening a fire hydrant wide open in the middle of field had unintended consequences. It tended to wash out the ground in front of the flow of the water shooting out of the hydrant. So the team of experts devised a plan to place a board in front of the hydrant when it would be in danger of tearing a hole in the terrain. The board would divert the water into the air where it would fan out and not cause damage to the surrounding area.
This was working fine, and when they arrived at the fire hydrant next to the substation, since the stream from the hydrant was pointing directly into the substation (hmm. a design flaw, I think), they decided to prop the board up against the fence to keep from washing away the gravel in the substation. Well. When a fire hydrant is opened that hasn’t been used for a year, the first flow of water to shoot out is dark brown.
You may think that this is because the water has somehow become dirty over the past year, but that isn’t quite the case. What has happened is that the pipe has been rusting little by little and the water has become saturated with the rust. So, the water shooting out of the hydrant was full of rust (hence the need to flush them out).
Well. Rust is made of metal. Metal is conductive, especially when it is mixed with water. When the water hit the board, it was deflected into the air and happened to direct itself directly into the high voltage switch in the substation. This caused a circuit to the ground which, once it created an arc pumped all the electricity directly into the ground.
Normally when something like this happens it doesn’t trip the Main Power Transformer to a Power Plant.
This time it did. I know there was a few heads scratching trying to figure it out. I think I figured out what happened a little while later. You see… here is the rest of the story….
Once the unit was back online and the emergency was over, someone finally noticed that the telephone system couldn’t call outside of the plant. Well. I was the main telephone person at the time, so the control room called me and asked me to look into the problem.
I checked the telephone computer and it was up and running just fine. Internal calls could be made. Only any call outside just concluded with a funny humming sound. After checking the circuit in the Logic Room next to the Rolm Telephone Computer I headed for…. guess where….. the Main Switchgear….
In the middle of the main switchgear in the back of the room right next to the Auxiliary Substation beyond the back wall, the outside telephone line came into the plant. The first thing it did was go through a special Telephone Surge Protector.
In this picture above, the silver circular buttons on the left side are really an old style surge protector. whenever there was a power surge, the carbon connection in the surge protector would quickly melt causing the circuit to go straight to ground. Thus protecting the rest of the telephone circuit. So, if some kid in their house decides to connect the 120 volts circuit to the telephone for fun to see what would happen, this circuit would protect the rest of the phone circuits. Keep in mind that this was during the early 1990 when “Surge Protection” still was basically all “mechanical”.
Anyway, when I arrived at this panel and I checked the surge protector to the main line going out of the plant, guess what I found…. Yep. Shorted to ground. Luckily there were some spares that were not wired to anything in the panel and I was able to swap them out for the ones that had been destroyed. — These were a one time use. Which meant, if they ever had to short to ground, they had to be replaced.
Ok. Fine. After a little while, we were able to call back out of the plant, though there was still some residual noise on the line. It was like this… when you called out of the plant, the person on the other end sounded like they were buried in a grave somewhere and they were trying to talk to someone living just like in an episode of the Twilight Episode where a phone line landed on a grave and the dead person tried to call his long lost love from the past.
I didn’t give it much thought other than that I figured the 189,000 volt arc to ground must have shorted out the telephone line since the phone line ran directly under the auxiliary substation ground grid.
It wasn’t until the next morning when the Southwestern Bell repairman showed up at the plant. I knew him well, since he had been working on our phone lines since before the AT&T breakup in 1984. When I met him in the front of the electric shop, he said that he needed to check our telephone circuits. I told him that I knew that we had a problem because we had a high voltage short to ground yesterday and I found our surge protectors melted away.
He explained to me that not only was our circuit affected, but that every relay house from here to Ponca City was blown out. That’s when I realized that the problem was the reverse of the usual situation. What had happened was that the Ground Grid in the substation and the surrounding area (including the Unit 1 Main Power Transformer) had become hot. What do you do when the ground grid becomes charged?
The Ground Grid is what is supposed to protect you when a surge happens, but what happens when the ground grid itself is the problem? In this case, when the high voltage line about 60 feet from the telephone cable surge protector, arced to ground, it fed a tremendous amount of power back through the ground grid. when equipment detected the surge in voltage, they automatically defaulted their circuits to ground. That’s why the telephone circuit died. That’s what tripped the Main Power Transformer.
When the telephone circuit detected the high voltage surge, it shorted to ground (which was the problem), causing the high voltage to feed directly into the phone line and down the line to the next Southwestern Bell relay switch, which also defaulted to ground, trying to bleed off the surge as it went from relay switch to switch until enough of the power was able to be diverted to ground.
That day sure turned out to be a learning experience. I learned that when all the lights go out in the control room, that it is almost assured that the coffee pot in the electric shop is going to stop working. I also learned that in order to coax the plant manager to the electric shop, a major electrical tragedy is one good way. I learned that when shooting rusty water into the air don’t point it at a high voltage auxiliary substation switch. — I’m sure Mickey Postman learned that lesson too. I also learned that just like in Star Trek… whenever there is a dangerous job to do, the Captain is always the one that wants to do it. Does that make sense? Send a Peon like me in there…
I also learned something else about Power Plant Men…. You see…. People like Dale Mitchell, George Alley and Mickey Postman all are examples of incredibly wonderful Power Plant Men. When they were out there doing their duty and something tragic like this, all the Power Plant Men felt their pain. They knew that they all felt guilty for tripping the unit. It didn’t matter that a million dollars every so many minutes was walking out the door in revenue. The only thing that mattered was that these three men were safe.
Since I have left the Power Plant, I have found that the idea that the employee is the greatest asset that a company can possess is not a universal idea. You see, there was never the thought that any of these people should be fired for their mistake. On the contrary. The true Power Plant Men did whatever they could to let them know that they knew exactly how they felt. It could have happened to any of them.
Besides the friendship between Power Plant Men, one of the things I miss most about working at the Power Plant is that the employees are held in high esteem as a real asset to the company. Many could learn from their example.
Comments from the Original post
-
-
I like the mix of storytelling and information sharing you deliver here. Thanks again
-
-
-
Aug 8, 2011 I lost all reserve in and lost both units in 2 separate storms and A1 would not start for a few minutes and A2 was leaking antifreeze terribly not a scratch and back online in 34 hours both units and no one ever asked me one question even with a Safety Dept not one question asked of me the SS on duty. Not many men on the planet have ever experienced an uncontrolled total plant outage- you would of thought a learning opportunity would of took place. Feb 2011 worst winter temps in years 50 Units or more tripped in Texas let alone the trusty units of Redbud and McClain were fighting Sooner rolled right along with storm warnings for two weeks ahead – the ICS still went to Detroit for just tours of other facilities once again the fall guy Tarver McArthur stood alone. I had authored a Freeze Protection Plan for the plant and that seemed to save the day and explain to the regulatory bodies how we were online and everyone else in Texas was off and enjoying rolling blackouts in a terrible winter weather situation not to mention our powers that be were all stranded in Detroit and very few people could get to the plant without getting stuck trying to get there as well – but a few health heart issues later I am still here to tell about it all you would think folks would want to take advantage of someone that had went through the fire and Ice but thats ok I want them someday distant to get all the credit they deserve when the trumpets sound as that is truly what matters most. DT
-
-
Great story, and yes, you took exactly the proper lesson from it, and it is too bad that many of our bosses haven’t learned it.
And yup, I’ve buried a lot of electrocuted squirrels over the years.
Resistance in a Coal-Fired Power Plant
Originally posted April 19, 2013:
Resistance is Futile! You may have heard that before. Especially if you are a Star Trek Fan. If not, then you know that there is always some form of resistance wherever you are.

Captain Picard as Locutus trying to convince you that “Resistance is Futile” Like that is ever going to happen
I learned a lot about resistance when I first joined the electric shop at the coal-fired power plant in North Central Oklahoma in 1984. I was assigned to work with Sonny Kendrick and Bill Rivers on the Precipitator during overhauls and when I wasn’t working on the manhole pumps and there wasn’t any other emergencies going on. Actually, from 1984 on, for the next 17 years I continued to work on the precipitator… (if I had only known my fate….).
Not only did I learn a lot about resistance, I also learned about capacitance, reactance, transformers, rectifiers, power supplies, diodes, transistors, op amps, and pots (also known as potentiometers). Bill Rivers was the brains of the outfit. Sonny was the Electric Specialist banished to the Precipitator by Leroy Godfrey (See Singing Along with Sonny Kendrick). Bill thought up the ideas and Sonny went to work to implement them. I just jumped in where I was needed.
The Precipitator is the large box between the boiler and the smokestack (maybe you can see this in the Power plant picture). The purpose of the electrostatic precipitator is to take the smoke (or fly ash) out of the exhaust before it went out of the smokestack.
The controls for the Precipitator were all electronic at that time. That meant that there were circuit boards full of resistors, capacitors, transistors, operational amplifiers, diodes and potentiometers. These circuit boards controlled the way the power was distributed throughout the precipitator wires and plates through high powered transformers, and how the rappers and vibrators operated that dropped the collected ash into the hoppers.
Bill had me take an electronics course at the Indian Meridian Vo-tech so I would know the basics. Then he taught me all the shortcuts. I had to be able to look at a resistor and tell right away what the value of resistance it was. Resistors are color-coded and you had to learn what each of the colors represented…
I was expected to know this by sight. Bill would test me. There was a mnemonic device that I was taught to remember what each color represented, but it is not appropriate to repeat it, so I won’t. It is enough to say that the colors go like this: Black, Brown, Red, Orange, Yellow, Green, Blue, Violet, Gray, White (I will never forget this my entire life). These represent the numbers zero through 9. Here is a full explanation of how to read a resistor….. just in case you are curious, or you are such a boring person that you really need some material to bring up when you are at a party and don’t know what to say:
I found that having just the correct amount of resistance was very important. Too much or too less, and everything stops working.
Isn’t it that way with management also? If the management is too resistant to change, then things come to a halt. If they have too little resistance, they lose control of the situation. Depending on the circuit (or managerial decision) and what you are trying to do, it helps to have a manager that has a variable resistance to meet the needs of each situation. Resistance to change is always a balancing act.
During the first two years I was an electrician, the main control panels that controlled the operation of the precipitators were electronic. We spent a lot of time in the lab troubleshooting electric circuits looking for blown (or bad) parts that needed to be replaced. Then we would solder new components on the circuit boards and then put them back in operation. I learned how to be an electronics junky. I became addicted to fixing electronic circuit boards. It was like a game to me.
Later, the precipitator controls were changed to digital controls. That is, they were more like little computers controlling the precipitator. Instead of a bunch of circuit boards dumbly, but cleverly, doing their job, (how many commas can I use in one sentence?), little brains were added that made decisions and reacted to conditions in a much more dynamic way.
What was interesting was that one day Bill Rivers was describing how technology was going to be in the future. He said that some day, we will be able to sit in the lab and look on a computer and see what all the controls in the precipitator were doing (this was 1984). If something isn’t working right, we could just reach over, type a few keys on the computer and adjust the controls. Drink our sweet tea (a necessary staple in Oklahoma at the time), and then wait for the next crisis…. Then he would giggle at the look of disbelief on my face.
When he was telling me this, I was thinking in my head…. Well, that would be nice, but this sounds more like a pipe dream to me than reality. What does an older guy with six kids from a tool and die company in Columbia Missouri (where I grew up, by chance) know about the future of anything….. well…. anything…uh… new age…. If that is what you might call it… I found out you just don’t really know when you are sitting in front of a true “visionary” with tremendous insight.
Bill Rivers had this incredible knack for telling the future. In 1984 he was predicting computer controls in the control room where you ran the entire plant from a computer on a desk instead of using the “Big Board”.

I love this picture! It makes me feel at home! This was not our plant, but is a Power Plant control room
He said you would be able to call someone on a phone you kept in your pocket or your watch like Dick Tracey.
I don’t know what journals Bill was reading or if he just dreamed all this stuff up in his head, or maybe he was a Star Trek Fan that believed that if you can dream it up you can do it. I do know that he picked up on subtle queues and made great inferences from them that seemed astronomically unlikely. However, I have to admit that he caught me off guard a number of times with predictions that definitely came true.
I will talk about this more in a future post, but for now I will say that we did upgrade the precipitator to where you could sit in the control room and monitor and adjust the precipitator controls (all 84 on each unit), and even each of the rappers (672 rappers) and vibrators (168 vibrators) on the roof of each precipitator. With one key on the computer I could send a plume of ash out of the smokestack that looked like the unit had just tripped, and a moment later, clean it up again. This meant that I could send smoke signals to the Osage Indian tribe 20 miles north up the Arkansas (pronounced “Are Kansas”) river, telling them that the Pow Wow would begin at sunset.
Today, I understand that the “Big Board” at the plant is just a large junction box and the plant is controlled almost (if not) completely by computers sitting on the desk. Before I left the plant in 2001, this was being transitioned slowly to computer controls. I have another story to tell some day about this, and how an operator named Jim Cave, a Power Plant Genius and true Power Plant Man of the highest integrity, was snubbed by upper management for speeding this technology along. — Another example of Power Plant Resistance….
But for now…. back to my electronic days… before I began re-programming the Eeprom chips in the precipitator controls….
Bill Rivers confided with me one day that when the new Instrument and Controls department had been formed from the “Results” department that his dream had been to become a part of this team. It meant the world to him. It was where he believed he belonged. It was one of his major goals in life.
There used to be two electrical specialists in the Power Plant. Sonny Kendrick was not always the only one. The other specialist was chosen to go to the Instrument and Controls shop. Bill Rivers wanted to move there also. He definitely had the experience and the knowledge to be a superb instrument and controls person. But Bill had this one problem.
He loved to joke around. He loved to pull strings and push buttons. I have mentioned in a previous posts that Bill would play a new joke on Sonny Kendrick every single day. As I have unfortunately found out in my own life… this tends to make them…. well….. it tends to make enemies out of those who have a chip on their shoulder. Those people who naturally feel inadequate in their abilities or their position in life. To go one step further…. anyone who feels “unloved”….. these people definitely do not like being joked with. They seem to never forgive you. My greatest regret in life is joking around with these individuals.
So, when it came time to choose who would be a part of the new Instrument and Controls shop, Bill Rivers was turned down. It was explained to him that the reason he was not given the job was because he cut off the leads of a transistor when he replaced them. — I’m not kidding. Bill Rivers had the habit of cutting off the leads of each resistor, transistor, diode or capacitor that he replaced…. this is why Monty Adams turned down his request for joining the “elite” Instrument and Controls shop (as he told Bill to his face).
Someone had told the Instrument and Controls Supervisor Monty Adams that Bill Rivers cut the leads off of transistors and resistors when he replaced them so that you couldn’t test them to see if they were all right. Implying that he didn’t want you to know whether he had replaced the transistor or resistor by mistake.
Bill Rivers took several transistors, cut the leads off of each of them and handed them to me and asked me to test them to see if they were still good or if they were bad. I took out my voltmeter, set it to ohms, and proceeded to test them as Bill Rivers had taught me. I told him…. this transistor is good….. this one is bad….
You see…. there is no way to cut the leads off of a transistor in such a way to make it impossible to tell if a transistor is good or bad…. In reality…. you cut the leads off of a bad transistor so that the person working on the circuit board knows that this is a bad transistor and doesn’t use it again by accident. This was electronics 101.
When Bill told me this story, he literally had tears in his eyes. This was because being part of the Instrument and Controls team was part of his dream. The future of his family and the entire rest of his life was decided the day he was told that he was not going to be a part of a team that he believed was his true lot in life.
I remember his exact words as he sat there in the lab alone and told me this story. He said, “… and Monty didn’t know… He didn’t know that you cut the leads…. that is standard procedure….” In Bill’s giggly way, he was crying out loud as he told me this.
From that point on….I knew that the decisions Bill made in his life were driven by that one decision to exclude him from this team. Unlike many of us that could say to ourselves…. “That is their loss”…. Bill kept this pain in his heart each day…. Every decision from that day further was effected by this event.
I calculated it out one day that I spent 414 hours driving back and forth from Stillwater, Oklahoma to the plant and back each day with Bill Rivers (along with Yvonne Taylor and Rich Litzer and occasionally others that needed a ride), and over that time, I became very close to Bill, even to the point of tutoring his son in Algebra (see post: How Many Power Plant Men Can You Put in a 1982 Honda Civic?).
I say this because I know about the pain that inflicted Bill River by a rash decision based on the hearsay of someone that held a grudge. I know how his entire life was changed and how it ended for Bill Rivers as a power plant employee. I know that every decision by Bill after this date was made in response to this one decision. Anyone who experienced Bill after 1983 knows what I am talking about.
I realized that today my own decisions in life help spell out my future. How some little remark may be misinterpreted, or even properly so. I realize as I write this post that how I accept or reject these events in my life, determines the future of my family. After seeing how every event in Bill’s life after that day at the power company was determined by his experience was to his detriment, I am determined not to let the same thing happened to me…..
That is why I have taken on the philosophy in my life that no matter how my actions are misinterpreted, I am determined to remain true to myself. I know what I mean, and I mean what I say, and I say what I mean, and an Elephant is Faithful 100 %.
Comment from the Original Post:
-
It’s amazing how many decisions are made based on incorrect / incomplete information (at all levels).
Runaway Fire Hydrant Leaves Power Plant in the Dark
Originally posted May 17, 2014:
Don’t believe it when the Electric Company tells you that the reason your town lost electricity for an hour was because a squirrel climbed onto a transformer and shorted it out. The real reason just may be more bizarre than that and the company doesn’t want you to know all the different creative ways that power can be shut off. This is a tale of just one of those ways. So, get out your pencil and paper and take notes.
One spring day in 1993 while sitting at the Precipitator computer for Unit one at the Coal-fired Power Plant in North Central Oklahoma, while I was checking the controls to make sure all the cabinets were operating correctly, suddenly there was a distant boom, and the lights in the control room went out. The computer stayed on because it was connected to an electric panel called the VSP or Vital Services Panel, which in turn was supplied by the UPS system (Uninterruptible Power Supply). That was one of those moments where you may pause for a moment to make sure you aren’t still at home dreaming before you fly into a panic.
The Precipitator cabinets all indicated on the computer that they had just shutdown. I rose from the chair and walked around to the front of the Alarm Panel for Unit one, and found that the fluorescent lights were only out on Unit 1. The lights were still on for Unit 2. The Control Panel was lit up like a Christmas Tree with Green, Red, Blue and Yellow Lights. The Alarm Printer was spewing out paper at high speed. As the large sheets of paper were pouring out onto the floor, I watched as Pat Quiring and other brave Power Plant Control Room operators were scurrying back and forth turning switch handles, pushing buttons, and checking pressure gauges.
Just this site alone gave me confidence that everything was going to be all right. These Control Room operators were all well trained for emergencies just like this, and each person knew what their job was. No one was panicking. Everyone was concentrating on the task at hand.
Someone told me that we lost Unit 1, and the Auxiliary Power to Unit 1 at the same time. So, Unit 1 was dead in the water. This meant, no fans, no pumps, no lights, no vending machines, no cold water at the water fountain and most importantly, no hot coffee!!! I could hear steam valves on the T-G floor banging open and the loud sound of steam escaping.
I turned quickly to go to the electric shop to see what I could do there in case I was needed. I bolted out the door and down the six flights of stairs to the Turbine-Generator (T-G) basement. Exiting the stairway, and entering the T-G basement the sound was deafening. I grabbed the earplugs that were dangling around my neck and crammed them into my ears. Steam was pouring out of various pop-off valves. I ducked into the electric shop where across the room Andy Tubbs, one of the electric foreman was pulling large sheets of electric blueprints from the print cabinet and laying them across the work table that doubled as the lunch table.
When I asked Andy what happened, I learned that somehow when a crew was flushing out a fire hydrant the water somehow shot up and into the bus work in the Auxiliary Substation (that supplies backup power to the Power Plant) and it shorted out the 189,000 volt substation directly to ground. When that happened it tripped unit 1 and the auxiliary substation at the same time leaving it without power.
I will explain how a fire hydrant could possibly spray the bus work in a substation in a little while, but first let me tell you what this meant at the moment to not have any power for a Power Plant Boiler and Turbine Generator that has just tripped when it was at full load which was around 515 Megawatts of power at the time.
Normally when a unit trips, the boiler cools down as the large Force Draft (FD) Fans blow air through the boiler while the even larger Induced Draft (ID) fans suck the air from the boiler on the other end and blow the hot air up the smoke stack. This causes the steam in the boiler tubes to condense back into water. Steam valves open on the boiler that allow excessive steam to escape.
When the boiler is running there is a large orange fireball hovering in space in the middle of the boiler. The boiler water is being circulated through the boiler and the Boiler Feed Pump Turbines are pumping steam back and forth between the turbine generator and the boiler reheating the steam until every bit of heat from the boiler that can be safely harnessed is used.
When all this stop suddenly, then it is important that the large fans keep running to cool down the steam, since it is no longer losing energy in the generator as it was when it was busy supplying electricity to 1/2 million people in Oklahoma City. The power is fed to the fans from the Auxiliary substation located right outside the Main Switchgear where all the breakers reside that supply the power to the fans. Unfortunately, in this case, the Auxiliary substation was shutdown as well, leaving the boiler without any fans.
Without fans for cooling, and pumps to circulate the water, the walls of the boiler began heating up to dangerous temperatures. Steam was whistling out of pop off valves, but if the steam drum on the top of the boiler were to run dry, then the entire boiler structure could be compromised and begin melting down. — So, this was serious. Something had to be done right away. It wouldn’t be as bad as the China Syndrome since we were burning coal instead of nuclear power, but it would have caused a lot of damage nonetheless.

From the movie “The China Syndrome” where a similar emergency existed only in the movie, it was a Nuclear Plant
I have a side story about this picture, but I think I’ll save it for another post because I don’t want to digress from the main story at this point (Ok. Let me just say “Jack Maloy and Merl Wright” for those who can’t wait) See the post: “Power Plant Conspiracy Theory“.
With the prospect that the boiler might melt to the ground in a pile of rubble, it would seem that the main priority was to turn the Auxiliary Substation back on so the fans could be turned back on and prevent the boiler from collapsing. So, we walked out to the substation and looked at the switches that would have to be operated in order to first power up the main bus and then to close to supply power to the two big transformers and the six smaller transformers that supplied the Unit 1 Main Switchgear.
While inspecting the switches where the electricity had gone to ground we found that one of the main insulators was cracked.
Since this insulator was cracked, we didn’t really want to operate the switch to test if another 189,000 volts would go straight to ground again, especially since one of us would be standing right underneath it cranking the switch. So, we went back to the shop to find an alternative.
By this time the Plant Manager, Ron Kilman arrived in the shop, and understanding the urgency to find a solution asked us what were the alternatives. He was relying on our expertise to make the decision.
The other solution would be to cut the power over from Unit 2 which was still humming away pushing electricity to Oklahoma City out of the 345,000 volt substation. The cut over would be very simple because the switchgear was designed with this in mind. We analyzed the power rating on the auxiliary transformers on Unit 2 and thought that we might be cutting it close to have them running both sets of fans at the same time, especially since the full load amps of a huge fan starting up was about 10 times the normal rate.
The transformer was rated to handle the load, but consider this. What if this caused Unit 2 to trip as well. With the Auxiliary substation offline, if Unit 2 tripped, we would be in twice the amount of trouble we were currently in. What a day it would have been if that had happened and two 250 foot boilers had come crashing to the ground in a pile of rubble. After reading the power ratings on the auxiliary transformers I was thinking, “Yeah, let’s do it! These transformers can handle it.” Andy was not so eager.
So, we were left with one alternative. That was to shut the switch in the Auxiliary substation that had the cracked insulator and take our chances that it wasn’t going to short to ground and blow up over our heads. I think I was eager to close the switch for Andy, but if I remember correctly, he didn’t want me to be the one to suffer the consequences and decided to close the switch himself. Needless to say. Andy closed the switch, and nothing blew up.
As soon as the power was restored to the switchgear, the fans were powered up and the temperature in the boiler was quickly reduced. The coffee pot in the Electric Shop began heating the coffee again. The power plant was saved from a major catastrophe. That was delayed for another day… of which I will talk about later (see the post “Destruction of a Power Plant God).”
So, how exactly does a fire hydrant shoot water up into the bus work of a substation like the picture of the switch directly above? The culprit fire hydrant wasn’t in the substation, it sat alongside it outside the fence a good 50 feet from the high voltage switch. No hose was attached to the fire hydrant. It was only being flushed out as part of a yearly activity to go around and make sure the fire hydrants are all operating correctly.
Here is the story about how the squirrel climbed into the transformer this time….
George Alley, Dale Mitchell and Mickey Postman were going around to the 30,000 fire hydrants on the plant ground (ok. maybe not that many, but we did have a lot of them), and they were opening up the valves and flushing them out. That means, they were letting them run for a while to clear them out from any contaminates that may have built up over the year of not being used.
Throughout their adventure they had opened a multitude of Hydrants situated out in the fields along the long belt conveyor from the coalyard and around the two one-million gallon #2 Diesel tanks.
The brave Power Plant Men, learned that when opening a fire hydrant wide open in the middle of field had unintended consequences. It tended to wash out the ground in front of the flow of the water shooting out of the hydrant. So the team of experts devised a plan to place a board in front of the hydrant when it would be in danger of tearing a hole in the terrain. The board would divert the water into the air where it would fan out and not cause damage to the surrounding area.
This was working fine, and when they arrived at the fire hydrant next to the substation, since the stream from the hydrant was pointing directly into the substation (hmm. a design flaw, I think), they decided to prop the board up against the fence to keep from washing away the gravel in the substation. Well. When a fire hydrant is opened that hasn’t been used for a year, the first flow of water to shoot out is dark brown.
You may think that this is because the water has somehow become dirty over the past year, but that isn’t quite the case. What has happened is that the pipe has been rusting little by little and the water has become saturated with the rust. So, the water shooting out of the hydrant was full of rust (hence the need to flush them out).
Well. Rust is made of metal. Metal is conductive, especially when it is mixed with water. When the water hit the board, it was deflected into the air and happened to direct itself directly into the high voltage switch in the substation. This caused a circuit to the ground which, once it created an arc pumped all the electricity directly into the ground.
Normally when something like this happens it doesn’t trip the Main Power Transformer to a Power Plant.
This time it did. I know there was a few heads scratching trying to figure it out. I think I figured out what happened a little while later. You see… here is the rest of the story….
Once the unit was back online and the emergency was over, someone finally noticed that the telephone system couldn’t call outside of the plant. Well. I was the main telephone person at the time, so the control room called me and asked me to look into the problem.
I checked the telephone computer and it was up and running just fine. Internal calls could be made. Only any call outside just concluded with a funny humming sound. After checking the circuit in the Logic Room next to the Rolm Telephone Computer I headed for…. guess where….. the Main Switchgear….
In the middle of the main switchgear in the back of the room right next to the Auxiliary Substation beyond the back wall, the outside telephone line came into the plant. The first thing it did was go through a special Telephone Surge Protector.
In this picture above, the silver circular buttons on the left side are really an old style surge protector. whenever there was a power surge, the carbon connection in the surge protector would quickly melt causing the circuit to go straight to ground. Thus protecting the rest of the telephone circuit. So, if some kid in their house decides to connect the 120 volts circuit to the telephone for fun to see what would happen, this circuit would protect the rest of the phone circuits. Keep in mind that this was during the early 1990 when “Surge Protection” still was basically all “mechanical”.
Anyway, when I arrived at this panel and I checked the surge protector to the main line going out of the plant, guess what I found…. Yep. Shorted to ground. Luckily there were some spares that were not wired to anything in the panel and I was able to swap them out for the ones that had been destroyed. — These were a one time use. Which meant, if they ever had to short to ground, they had to be replaced.
Ok. Fine. After a little while, we were able to call back out of the plant, though there was still some residual noise on the line. It was like this… when you called out of the plant, the person on the other end sounded like they were buried in a grave somewhere and they were trying to talk to someone living just like in an episode of the Twilight Episode where a phone line landed on a grave and the dead person tried to call his long lost love from the past.
I didn’t give it much thought other than that I figured the 189,000 volt arc to ground must have shorted out the telephone line since the phone line ran directly under the auxiliary substation ground grid.
It wasn’t until the next morning when the Southwestern Bell repairman showed up at the plant. I knew him well, since he had been working on our phone lines since before the AT&T breakup in 1984. When I met him in the front of the electric shop, he said that he needed to check our telephone circuits. I told him that I knew that we had a problem because we had a high voltage short to ground yesterday and I found our surge protectors melted away.
He explained to me that not only was our circuit affected, but that every relay house from here to Ponca City was blown out. That’s when I realized that the problem was the reverse of the usual situation. What had happened was that the Ground Grid in the substation and the surrounding area (including the Unit 1 Main Power Transformer) had become hot. What do you do when the ground grid becomes charged?
The Ground Grid is what is supposed to protect you when a surge happens, but what happens when the ground grid itself is the problem? In this case, when the high voltage line about 60 feet from the telephone cable surge protector, arced to ground, it fed a tremendous amount of power back through the ground grid. when equipment detected the surge in voltage, they automatically defaulted their circuits to ground. That’s why the telephone circuit died. That’s what tripped the Main Power Transformer.
When the telephone circuit detected the high voltage surge, it shorted to ground (which was the problem), causing the high voltage to feed directly into the phone line and down the line to the next Southwestern Bell relay switch, which also defaulted to ground, trying to bleed off the surge as it went from relay switch to switch until enough of the power was able to be diverted to ground.
That day sure turned out to be a learning experience. I learned that when all the lights go out in the control room, that it is almost assured that the coffee pot in the electric shop is going to stop working. I also learned that in order to coax the plant manager to the electric shop, a major electrical tragedy is one good way. I learned that when shooting rusty water into the air don’t point it at a high voltage auxiliary substation switch. — I’m sure Mickey Postman learned that lesson too. I also learned that just like in Star Trek… whenever there is a dangerous job to do, the Captain is always the one that wants to do it. Does that make sense? Send a Peon like me in there…
I also learned something else about Power Plant Men…. You see…. People like Dale Mitchell, George Alley and Mickey Postman all are examples of incredibly wonderful Power Plant Men. When they were out there doing their duty and something tragic like this, all the Power Plant Men felt their pain. They knew that they all felt guilty for tripping the unit. It didn’t matter that a million dollars every so many minutes was walking out the door in revenue. The only thing that mattered was that these three men were safe.
Since I have left the Power Plant, I have found that the idea that the employee is the greatest asset that a company can possess is not a universal idea. You see, there was never the thought that any of these people should be fired for their mistake. On the contrary. The true Power Plant Men did whatever they could to let them know that they knew exactly how they felt. It could have happened to any of them.
Besides the friendship between Power Plant Men, one of the things I miss most about working at the Power Plant is that the employees are held in high esteem as a real asset to the company. Many could learn from their example.
Comments from the Original post
-
-
That was an exciting day! Another great story. Thanks for the memories.
-
-
-
I like the mix of storytelling and information sharing you deliver here. Thanks again
-
-
-
Aug 8, 2011 I lost all reserve in and lost both units in 2 separate storms and A1 would not start for a few minutes and A2 was leaking antifreeze terribly not a scratch and back online in 34 hours both units and no one ever asked me one question even with a Safety Dept not one question asked of me the SS on duty. Not many men on the planet have ever experienced an uncontrolled total plant outage- you would of thought a learning opportunity would of took place. Feb 2011 worst winter temps in years 50 Units or more tripped in Texas let alone the trusty units of Redbud and McClain were fighting Sooner rolled right along with storm warnings for two weeks ahead – the ICS still went to Detroit for just tours of other facilities once again the fall guy Tarver McArthur stood alone. I had authored a Freeze Protection Plan for the plant and that seemed to save the day and explain to the regulatory bodies how we were online and everyone else in Texas was off and enjoying rolling blackouts in a terrible winter weather situation not to mention our powers that be were all stranded in Detroit and very few people could get to the plant without getting stuck trying to get there as well – but a few health heart issues later I am still here to tell about it all you would think folks would want to take advantage of someone that had went through the fire and Ice but thats ok I want them someday distant to get all the credit they deserve when the trumpets sound as that is truly what matters most. DT
-
-
Great story, and yes, you took exactly the proper lesson from it, and it is too bad that many of our bosses haven’t learned it.
And yup, I’ve buried a lot of electrocuted squirrels over the years.
Resistance in a Coal-Fired Power Plant
Originally posted April 19, 2013:
Resistance is Futile! You may have heard that before. Especially if you are a Star Trek Fan. If not, then you know that there is always some form of resistance wherever you are.

Captain Picard as Locutus trying to convince you that “Resistance is Futile” Like that is ever going to happen
I learned a lot about resistance when I first joined the electric shop at the coal-fired power plant in North Central Oklahoma in 1984. I was assigned to work with Sonny Kendrick and Bill Rivers on the Precipitator during overhauls and when I wasn’t working on the manhole pumps and there wasn’t any other emergencies going on. Actually, from 1984 on, on the Precipitator for the next 17 years I continued to work on the precipitator… (if I had only known my fate….).
Not only did I learn a lot about resistance, I also learned about capacitance, reactance, transformers, rectifiers, power supplies, diodes, transistors, op amps, and pots (also known as potentiometers). Bill Rivers was the brains of the outfit. Sonny was the Electric Specialist banished to the Precipitator by Leroy Godfrey (See Singing Along with Sonny Kendrick). Bill thought up the ideas and Sonny went to work to implement them. I just jumped in where I was needed.
The Precipitator is the large box between the boiler and the smokestack (maybe you can see this in the Power plant picture). The purpose of the electrostatic precipitator is to take the smoke (or fly ash) out of the exhaust before it went out of the smokestack.
The controls for the Precipitator were all electronic at that time. That meant that there were circuit boards full of resistors, capacitors, transistors, operational amplifiers, diodes and potentiometers. These circuit boards controlled the way the power was distributed throughout the precipitator wires and plates through high powered transformers, and how the rappers and vibrators operated that dropped the collected ash into the hoppers.
Bill had me take an electronics course at the Indian Meridian Vo-tech so I would know the basics. Then he taught me all the shortcuts. I had to be able to look at a resistor and tell right away what the value of resistance it was. Resistors are color-coded and you had to learn what each of the colors represented…
I was expected to know this by sight. Bill would test me. There was a mnemonic device that I was taught to remember what each color represented, but it is not appropriate to repeat it, so I won’t. It is enough to say that the colors go like this: Black, Brown, Red, Orange, Yellow, Green, Blue, Violet, Gray, White (I will never forget this my entire life). These represent the numbers zero through 9. Here is a full explanation of how to read a resistor….. just in case you are curious, or you are such a boring person that you really need some material to bring up when you are at a party and don’t know what to say:
I found that having just the correct amount of resistance was very important. Too much or too less, and everything stops working.
Isn’t it that way with management also? If the management is too resistant to change, then things come to a halt. If they have too little resistance, they lose control of the situation. Depending on the circuit (or managerial decision) and what you are trying to do, it helps to have a manager that has a variable resistance to meet the needs of each situation. Resistance to change is always a balancing act.
During the first two years I was an electrician, the main control panels that controlled the operation of the precipitators were electronic. We spent a lot of time in the lab troubleshooting electric circuits looking for blown (or bad) parts that needed to be replaced. Then we would solder new components on the circuit boards and then put them back in operation. I learned how to be an electronics junky. I became addicted to fixing electronic circuit boards. It was like a game to me.
Later, the precipitator controls were changed to digital controls. That is, they were more like little computers controlling the precipitator. Instead of a bunch of circuit boards dumbly, but cleverly, doing their job, (how many commas can I use in one sentence?), little brains were added that made decisions and reacted to conditions in a much more dynamic way.
What was interesting was that one day Bill Rivers was describing how technology was going to be in the future. He said that some day, we will be able to sit in the lab and look on a computer and see what all the controls in the precipitator were doing (this was 1984). If something isn’t working right, we could just reach over, type a few keys on the computer and adjust the controls. Drink our sweet tea (a necessary staple in Oklahoma at the time), and then wait for the next crisis…. Then he would giggle at the look of disbelief on my face.
When he was telling me this, I was thinking in my head…. Well, that would be nice, but this sounds more like a pipe dream to me than reality. What does an older guy with six kids from a tool and die company in Columbia Missouri (where I grew up, by chance) know about the future of anything….. well…. anything…uh… new age…. If that is what you might call it… I found out you just don’t really know when you are sitting in front of a true “visionary” with tremendous insight.
Bill Rivers had this incredible knack for telling the future. In 1984 he was predicting computer controls in the control room where you ran the entire plant from a computer on a desk instead of using the “Big Board”.

I love this picture! It makes me feel at home! This was not our plant, but is a Power Plant control room
He said you would be able to call someone on a phone you kept in your pocket or your watch like Dick Tracey.
I don’t know what journals Bill was reading or if he just dreamed all this stuff up in his head, or maybe he was a Star Trek Fan that believed that if you can dream it up you can do it. I do know that he picked up on subtle queues and made great inferences from them that seemed astronomically unlikely. However, I have to admit that he caught me off guard a number of times with predictions that definitely came true.
I will talk about this more in a future post, but for now I will say that we did upgrade the precipitator to where you could sit in the control room and monitor and adjust the precipitator controls (all 84 on each unit), and even each of the rappers (672 rappers) and vibrators (168 vibrators) on the roof of each precipitator. With one key on the computer I could send a plume of ash out of the smokestack that looked like the unit had just tripped, and a moment later, clean it up again. This meant that I could send smoke signals to the Osage Indian tribe 20 miles north up the Arkansas (pronounced “Are Kansas”) river, telling them that the Pow Wow would begin at sunset.
Today, I understand that the “Big Board” at the plant is just a large junction box and the plant is controlled almost (if not) completely by computers sitting on the desk. Before I left the plant in 2001, this was being transitioned slowly to computer controls. I have another story to tell some day about this, and how an operator named Jim Cave, a Power Plant Genius and true Power Plant Man of the highest integrity, was snubbed by upper management for speeding this technology along. — Another example of Power Plant Resistance….
But for now…. back to my electronic days… before I began re-programming the Eeprom chips in the precipitator controls….
Bill Rivers confided with me one day that when the new Instrument and Controls department had been formed from the “Results” department that his dream had been to become a part of this team. It meant the world to him. It was where he believed he belonged. It was one of his major goals in life.
There used to be two electrical specialists in the Power Plant. Sonny Kendrick was not always the only one. The other specialist was chosen to go to the Instrument and Controls shop. Bill Rivers wanted to move there also. He definitely had the experience and the knowledge to be a superb instrument and controls person. But Bill had this one problem.
He loved to joke around. He loved to pull strings and push buttons. I have mentioned in a previous posts that Bill would play a new joke on Sonny Kendrick every single day. As I have unfortunately found out in my own life… this tends to make them…. well….. it tends to make enemies out of those who have a chip on their shoulder. Those people who naturally feel inadequate in their abilities or their position in life. To go one step further…. anyone who feels “unloved”….. these people definitely do not like being joked with. They seem to never forgive you. My greatest regret in life is joking around with these individuals.
So, when it came time to choose who would be a part of the new Instrument and Controls shop, Bill Rivers was turned down. It was explained to him that the reason he was not given the job was because he cut off the leads of a transistor when he replaced them. — I’m not kidding. Bill Rivers had the habit of cutting off the leads of each resistor, transistor, diode or capacitor that he replaced…. this is why Monty Adams turned down his request for joining the “elite” Instrument and Controls shop (as he told Bill to his face).
Someone had told the Instrument and Controls Supervisor Monty Adams that Bill Rivers cut the leads off of transistors and resistors when he replaced them so that you couldn’t test them to see if they were all right. Implying that he didn’t want you to know whether he had replaced the transistor or resistor by mistake.
Bill Rivers took several transistors, cut the leads off of each of them and handed them to me and asked me to test them to see if they were still good or if they were bad. I took out my voltmeter, set it to ohms, and proceeded to test them as Bill Rivers had taught me. I told him…. this transistor is good….. this one is bad….
You see…. there is no way to cut the leads off of a transistor in such a way to make it impossible to tell if a transistor is good or bad…. In reality…. you cut the leads off of a bad transistor so that the person working on the circuit board knows that this is a bad transistor and doesn’t use it again by accident. This was electronics 101.
When Bill told me this story, he literally had tears in his eyes. This was because being part of the Instrument and Controls team was part of his dream. His family and the entire rest of his life was decided the day he was told that he was not going to be a part of a team that he believed was his true lot in life.
I remember his exact words as he sat there in the lab alone and told me this story. He said, “… and Monty didn’t know… He didn’t know that you cut the leads…. that is standard procedure….” In Bill’s giggly way, he was crying out loud as he told me this.
From that point on….I knew that the decisions Bill made in his life were driven by that one decision to exclude him from this team. Unlike many of us that could say to ourselves…. “That is their loss”…. Bill kept this pain in his heart each day…. Every decision from that day further was effected by this event.
I calculated it out one day that I spent 414 hours driving back and forth from Stillwater, Oklahoma to the plant and back each day with Bill Rivers (along with Yvonne Taylor and Rich Litzer and occasionally others that needed a ride), and over that time, I became very close to Bill, even to the point of tutoring his son in Algebra (see post: How Many Power Plant Men Can You Put in a 1982 Honda Civic?).
I say this because I know about the pain that inflicted Bill River by a rash decision based on the hearsay of someone that held a grudge. I know how his entire life was changed and how it ended for Bill Rivers as a power plant employee. I know that every decision by Bill after this date was made in response to this one decision. Anyone who experienced Bill after 1983 knows what I am talking about.
I realized that today my own decisions in life help spell out my future. How some little remark may be misinterpreted, or even properly so. I realize as I write this post that how I accept or reject these events in my life, determines the future of my family. After seeing how every event in Bill’s life after that day at the power company was determined by his experience was to his detriment, I am determined not to let the same thing happened to me…..
That is why I have taken on the philosophy in my life that no matter how my actions are misinterpreted, I am determined to remain true to myself. I know what I mean, and I mean what I say, and I say what I mean, and an Elephant is Faithful 100 %.
Comment from the Original Post:
-
It’s amazing how many decisions are made based on incorrect / incomplete information (at all levels).
Runaway Fire Hydrant Leaves Power Plant in the Dark
Originally posted May 17, 2014:
Don’t believe it when the Electric Company tells you that the reason your town lost electricity for an hour was because a squirrel climbed onto a transformer and shorted it out. The real reason just may be more bizarre than that and the company doesn’t want you to know all the different creative ways that power can be shut off. This is a tale of just one of those ways. So, get out your pencil and paper and take notes.
One spring day in 1993 while sitting at the Precipitator computer for Unit one at the Coal-fired Power Plant in North Central Oklahoma, while I was checking the controls to make sure all the cabinets were operating correctly, suddenly there was a distant boom, and the lights in the control room went out. The computer stayed on because it was connected to an electric panel called the VSP or Vital Services Panel, which in turn was supplied by the UPS system (Uninterruptible Power Supply). That was one of those moments where you may pause for a moment to make sure you aren’t still at home dreaming before you fly into a panic.
The Precipitator cabinets all indicated on the computer that they had just shutdown. I rose from the chair and walked around to the front of the Alarm Panel for Unit one, and found that the fluorescent lights were only out on Unit 1. The lights were still on for Unit 2. The Control Panel was lit up like a Christmas Tree with Green, Red, Blue and Yellow Lights. The Alarm Printer was spewing out paper at high speed. As the large sheets of paper were pouring out onto the floor, I watched as Pat Quiring and other brave Power Plant Control Room operators were scurrying back and forth turning switch handles, pushing buttons, and checking pressure gauges.
Just this site alone gave me confidence that everything was going to be all right. These Control Room operators were all well trained for emergencies just like this, and each person knew what their job was. No one was panicking. Everyone was concentrating on the task at hand.
Someone told me that we lost Unit 1, and the Auxiliary Power to Unit 1 at the same time. So, Unit 1 was dead in the water. This meant, no fans, no pumps, no lights, no vending machines, no cold water at the water fountain and most importantly, no hot coffee!!! I could hear steam valves on the T-G floor banging open and the loud sound of steam escaping.
I turned quickly to go to the electric shop to see what I could do there in case I was needed. I bolted out the door and down the six flights of stairs to the Turbine-Generator (T-G) basement. Exiting the stairway, and entering the T-G basement the sound was deafening. I grabbed the earplugs that were dangling around my neck and crammed them into my ears. Steam was pouring out of various pop-off valves. I ducked into the electric shop where across the room Andy Tubbs, one of the electric foreman was pulling large sheets of electric blueprints from the print cabinet and laying them across the work table that doubled as the lunch table.
When I asked Andy what happened, I learned that somehow when a crew was flushing out a fire hydrant the water somehow shot up and into the bus work in the Auxiliary Substation (that supplies backup power to the Power Plant) and it shorted out the 189,000 volt substation directly to ground. When that happened it tripped unit 1 and the auxiliary substation at the same time leaving it without power.
I will explain how a fire hydrant could possibly spray the bus work in a substation in a little while, but first let me tell you what this meant at the moment to not have any power for a Power Plant Boiler and Turbine Generator that has just tripped when it was at full load which was around 515 Megawatts of power at the time.
Normally when a unit trips, the boiler cools down as the large Force Draft (FD) Fans blow air through the boiler while the even larger Induced Draft (ID) fans suck the air from the boiler on the other end and blow the hot air up the smoke stack. This causes the steam in the boiler tubes to condense back into water. Steam valves open on the boiler that allow excessive steam to escape.
When the boiler is running there is a large orange fireball hovering in space in the middle of the boiler. The boiler water is being circulated through the boiler and the Boiler Feed Pump Turbines are pumping steam back and forth between the turbine generator and the boiler reheating the steam until every bit of heat from the boiler that can be safely harnessed is used.
When all this stop suddenly, then it is important that the large fans keep running to cool down the steam, since it is no longer losing energy in the generator as it was when it was busy supplying electricity to 1/2 million people in Oklahoma City. The power is fed to the fans from the Auxiliary substation located right outside the Main Switchgear where all the breakers reside that supply the power to the fans. Unfortunately, in this case, the Auxiliary substation was shutdown as well, leaving the boiler without any fans.
Without fans for cooling, and pumps to circulate the water, the walls of the boiler began heating up to dangerous temperatures. Steam was whistling out of pop off valves, but if the steam drum on the top of the boiler were to run dry, then the entire boiler structure could be compromised and begin melting down. — So, this was serious. Something had to be done right away. It wouldn’t be as bad as the China Syndrome since we were burning coal instead of nuclear power, but it would have caused a lot of damage nonetheless.

From the movie “The China Syndrome” where a similar emergency existed only in the movie, it was a Nuclear Plant
I have a side story about this picture, but I think I’ll save it for another post because I don’t want to digress from the main story at this point (Ok. Let me just say “Jack Maloy and Merl Wright” for those who can’t wait) See the post: “Power Plant Conspiracy Theory“.
With the prospect that the boiler might melt to the ground in a pile of rubble, it would seem that the main priority was to turn the Auxiliary Substation back on so the fans could be turned back on and prevent the boiler from collapsing. So, we walked out to the substation and looked at the switches that would have to be operated in order to first power up the main bus and then to close to supply power to the two big transformers and the six smaller transformers that supplied the Unit 1 Main Switchgear.
While inspecting the switches where the electricity had gone to ground we found that one of the main insulators was cracked.
Since this insulator was cracked, we didn’t really want to operate the switch to test if another 189,000 volts would go straight to ground again, especially since one of us would be standing right underneath it cranking the switch. So, we went back to the shop to find an alternative.
By this time the Plant Manager, Ron Kilman arrived in the shop, and understanding the urgency to find a solution asked us what were the alternatives. He was relying on our expertise to make the decision.
The other solution would be to cut the power over from Unit 2 which was still humming away pushing electricity to Oklahoma City out of the 345,000 volt substation. The cut over would be very simple because the switchgear was designed with this in mind. We analyzed the power rating on the auxiliary transformers on Unit 2 and thought that we might be cutting it close to have them running both sets of fans at the same time, especially since the full load amps of a huge fan starting up was about 10 times the normal rate.
The transformer was rated to handle the load, but consider this. What if this caused Unit 2 to trip as well. With the Auxiliary substation offline, if Unit 2 tripped, we would be in twice the amount of trouble we were currently in. What a day it would have been if that had happened and two 250 foot boilers had come crashing to the ground in a pile of rubble. After reading the power ratings on the auxiliary transformers I was thinking, “Yeah, let’s do it! These transformers can handle it.” Andy was not so eager.
So, we were left with one alternative. That was to shut the switch in the Auxiliary substation that had the cracked insulator and take our chances that it wasn’t going to short to ground and blow up over our heads. I think I was eager to close the switch for Andy, but if I remember correctly, he didn’t want me to be the one to suffer the consequences and decided to close the switch himself. Needless to say. Andy closed the switch, and nothing blew up.
As soon as the power was restored to the switchgear, the fans were powered up and the temperature in the boiler was quickly reduced. The coffee pot in the Electric Shop began heating the coffee again. The power plant was saved from a major catastrophe. That was delayed for another day… of which I will talk about later (see the post “Destruction of a Power Plant God).”
So, how exactly does a fire hydrant shoot water up into the bus work of a substation like the picture of the switch directly above? The culprit fire hydrant wasn’t in the substation, it sat alongside it outside the fence a good 50 feet from the high voltage switch. No hose was attached to the fire hydrant. It was only being flushed out as part of a yearly activity to go around and make sure the fire hydrants are all operating correctly.
Here is the story about how the squirrel climbed into the transformer this time….
George Alley, Dale Mitchell and Mickey Postman were going around to the 30,000 fire hydrants on the plant ground (ok. maybe not that many, but we did have a lot of them), and they were opening up the valves and flushing them out. That means, they were letting them run for a while to clear them out from any contaminates that may have built up over the year of not being used.
Throughout their adventure they had opened a multitude of Hydrants situated out in the fields along the long belt conveyor from the coalyard and around the two one-million gallon #2 Diesel tanks.
The brave Power Plant Men, learned that when opening a fire hydrant wide open in the middle of field had unintended consequences. It tended to wash out the ground in front of the flow of the water shooting out of the hydrant. So the team of experts devised a plan to place a board in front of the hydrant when it would be in danger of tearing a hole in the terrain. The board would divert the water into the air where it would fan out and not cause damage to the surrounding area.
This was working fine, and when they arrived at the fire hydrant next to the substation, since the stream from the hydrant was pointing directly into the substation (hmm. a design flaw, I think), they decided to prop the board up against the fence to keep from washing away the gravel in the substation. Well. When a fire hydrant is opened that hasn’t been used for a year, the first flow of water to shoot out is dark brown.
You may think that this is because the water has somehow become dirty over the past year, but that isn’t quite the case. What has happened is that the pipe has been rusting little by little and the water has become saturated with the rust. So, the water shooting out of the hydrant was full of rust (hence the need to flush them out).
Well. Rust is made of metal. Metal is conductive, especially when it is mixed with water. When the water hit the board, it was deflected into the air and happened to direct itself directly into the high voltage switch in the substation. This caused a circuit to the ground which, once it created an arc pumped all the electricity directly into the ground.
Normally when something like this happens it doesn’t trip the Main Power Transformer to a Power Plant.
This time it did. I know there was a few heads scratching trying to figure it out. I think I figured out what happened a little while later. You see… here is the rest of the story….
Once the unit was back online and the emergency was over, someone finally noticed that the telephone system couldn’t call outside of the plant. Well. I was the main telephone person at the time, so the control room called me and asked me to look into the problem.
I checked the telephone computer and it was up and running just fine. Internal calls could be made. Only any call outside just concluded with a funny humming sound. After checking the circuit in the Logic Room next to the Rolm Telephone Computer I headed for…. guess where….. the Main Switchgear….
In the middle of the main switchgear in the back of the room right next to the Auxiliary Substation beyond the back wall, the outside telephone line came into the plant. The first thing it did was go through a special Telephone Surge Protector.
In this picture above, the silver circular buttons on the left side are really an old style surge protector. whenever there was a power surge, the carbon connection in the surge protector would quickly melt causing the circuit to go straight to ground. Thus protecting the rest of the telephone circuit. So, if some kid in their house decides to connect the 120 volts circuit to the telephone for fun to see what would happen, this circuit would protect the rest of the phone circuits. Keep in mind that this was during the early 1990 when “Surge Protection” still was basically all “mechanical”.
Anyway, when I arrived at this panel and I checked the surge protector to the main line going out of the plant, guess what I found…. Yep. Shorted to ground. Luckily there were some spares that were not wired to anything in the panel and I was able to swap them out for the ones that had been destroyed. — These were a one time use. Which meant, if they ever had to short to ground, they had to be replaced.
Ok. Fine. After a little while, we were able to call back out of the plant, though there was still some residual noise on the line. It was like this… when you called out of the plant, the person on the other end sounded like they were buried in a grave somewhere and they were trying to talk to someone living just like in an episode of the Twilight Episode where a phone line landed on a grave and the dead person tried to call his long lost love from the past.
I didn’t give it much thought other than that I figured the 189,000 volt arc to ground must have shorted out the telephone line since the phone line ran directly under the auxiliary substation ground grid.
It wasn’t until the next morning when the Southwestern Bell repairman showed up at the plant. I knew him well, since he had been working on our phone lines since before the AT&T breakup in 1984. When I met him in the front of the electric shop, he said that he needed to check our telephone circuits. I told him that I knew that we had a problem because we had a high voltage short to ground yesterday and I found our surge protectors melted away.
He explained to me that not only was our circuit affected, but that every relay house from here to Ponca City was blown out. That’s when I realized that the problem was the reverse of the usual situation. What had happened was that the Ground Grid in the substation and the surrounding area (including the Unit 1 Main Power Transformer) had become hot. What do you do when the ground grid becomes charged?
The Ground Grid is what is supposed to protect you when a surge happens, but what happens when the ground grid itself is the problem? In this case, when the high voltage line about 60 feet from the telephone cable surge protector, arced to ground, it fed a tremendous amount of power back through the ground grid. when equipment detected the surge in voltage, they automatically defaulted their circuits to ground. That’s why the telephone circuit died. That’s what tripped the Main Power Transformer.
When the telephone circuit detected the high voltage surge, it shorted to ground (which was the problem), causing the high voltage to feed directly into the phone line and down the line to the next Southwestern Bell relay switch, which also defaulted to ground, trying to bleed off the surge as it went from relay switch to switch until enough of the power was able to be diverted to ground.
That day sure turned out to be a learning experience. I learned that when all the lights go out in the control room, that it is almost assured that the coffee pot in the electric shop is going to stop working. I also learned that in order to coax the plant manager to the electric shop, a major electrical tragedy is one good way. I learned that when shooting rusty water into the air don’t point it at a high voltage auxiliary substation switch. — I’m sure Mickey Postman learned that lesson too. I also learned that just like in Star Trek… whenever there is a dangerous job to do, the Captain is always the one that wants to do it. Does that make sense? Send a Peon like me in there…
I also learned something else about Power Plant Men…. You see…. People like Dale Mitchell, George Alley and Mickey Postman all are examples of incredibly wonderful Power Plant Men. When they were out there doing their duty and something tragic like this, all the Power Plant Men felt their pain. They knew that they all felt guilty for tripping the unit. It didn’t matter that a million dollars every so many minutes was walking out the door in revenue. The only thing that mattered was that these three men were safe.
Since I have left the Power Plant, I have found that the idea that the employee is the greatest asset that a company can possess is not a universal idea. You see, there was never the thought that any of these people should be fired for their mistake. On the contrary. The true Power Plant Men did whatever they could to let them know that they knew exactly how they felt. It could have happened to any of them.
Besides the friendship between Power Plant Men, one of the things I miss most about working at the Power Plant is that the employees are held in high esteem as a real asset to the company. Many could learn from their example.
Comments from the Original post
-
-
That was an exciting day! Another great story. Thanks for the memories.
-
-
-
I like the mix of storytelling and information sharing you deliver here. Thanks again
-
-
-
Aug 8, 2011 I lost all reserve in and lost both units in 2 separate storms and A1 would not start for a few minutes and A2 was leaking antifreeze terribly not a scratch and back online in 34 hours both units and no one ever asked me one question even with a Safety Dept not one question asked of me the SS on duty. Not many men on the planet have ever experienced an uncontrolled total plant outage- you would of thought a learning opportunity would of took place. Feb 2011 worst winter temps in years 50 Units or more tripped in Texas let alone the trusty units of Redbud and McClain were fighting Sooner rolled right along with storm warnings for two weeks ahead – the ICS still went to Detroit for just tours of other facilities once again the fall guy Tarver McArthur stood alone. I had authored a Freeze Protection Plan for the plant and that seemed to save the day and explain to the regulatory bodies how we were online and everyone else in Texas was off and enjoying rolling blackouts in a terrible winter weather situation not to mention our powers that be were all stranded in Detroit and very few people could get to the plant without getting stuck trying to get there as well – but a few health heart issues later I am still here to tell about it all you would think folks would want to take advantage of someone that had went through the fire and Ice but thats ok I want them someday distant to get all the credit they deserve when the trumpets sound as that is truly what matters most. DT
-
-
Great story, and yes, you took exactly the proper lesson from it, and it is too bad that many of our bosses haven’t learned it.
And yup, I’ve buried a lot of electrocuted squirrels over the years.
Runaway Fire Hydrant Leaves Power Plant in the Dark
Originally posted May 17, 2014:
Don’t believe it when the Electric Company tells you that the reason your town lost electricity for an hour was because a squirrel climbed onto a transformer and shorted it out. The real reason just may be more bizarre than that and the company doesn’t want you to know all the different creative ways that power can be shut off. This is a tale of just one of those ways. So, get out your pencil and paper and take notes.
One spring day in 1993 while sitting at the Precipitator computer for Unit one at the Coal-fired Power Plant in North Central Oklahoma, while I was checking the controls to make sure all the cabinets were operating correctly, suddenly there was a distant boom, and the lights in the control room went out. The computer stayed on because it was connected to an electric panel called the VSP or Vital Services Panel, which in turn was supplied by the UPS system (Uninterruptible Power Supply). That was one of those moments where you may pause for a moment to make sure you aren’t still at home dreaming before you fly into a panic.
The Precipitator cabinets all indicated on the computer that they had just shutdown. I rose from the chair and walked around to the front of the Alarm Panel for Unit one, and found that the fluorescent lights were only out on Unit 1. The lights were still on for Unit 2. The Control Panel was lit up like a Christmas Tree with Green, Red, Blue and Yellow Lights. The Alarm Printer was spewing out paper at high speed. As the large sheets of paper were pouring out onto the floor, I watched as Pat Quiring and other brave Power Plant Control Room operators were scurrying back and forth turning switch handles, pushing buttons, and checking pressure gauges.
Just this site alone gave me confidence that everything was going to be all right. These Control Room operators were all well trained for emergencies just like this, and each person knew what their job was. No one was panicking. Everyone was concentrating on the task at hand.
Someone told me that we lost Unit 1, and the Auxiliary Power to Unit 1 at the same time. So, Unit 1 was dead in the water. This meant, no fans, no pumps, no lights, no vending machines, no cold water at the water fountain and most importantly, no hot coffee!!! I could hear steam valves on the T-G floor banging open and the loud sound of steam escaping.
I turned quickly to go to the electric shop to see what I could do there in case I was needed. I bolted out the door and down the six flights of stairs to the Turbine-Generator (T-G) basement. Exiting the stairway, and entering the T-G basement the sound was deafening. I grabbed the earplugs that were dangling around my neck and crammed them into my ears. Steam was pouring out of various pop-off valves. I ducked into the electric shop where across the room Andy Tubbs, one of the electric foreman was pulling large sheets of electric blueprints from the print cabinet and laying them across the work table that doubled as the lunch table.
When I asked Andy what happened, I learned that somehow when a crew was flushing out a fire hydrant the water somehow shot up and into the bus work in the Auxiliary Substation (that supplies backup power to the Power Plant) and it shorted out the 189,000 volt substation directly to ground. When that happened it tripped unit 1 and the auxiliary substation at the same time leaving it without power.
I will explain how a fire hydrant could possibly spray the bus work in a substation in a little while, but first let me tell you what this meant at the moment to not have any power for a Power Plant Boiler and Turbine Generator that has just tripped when it was at full load which was around 515 Megawatts of power at the time.
Normally when a unit trips, the boiler cools down as the large Force Draft (FD) Fans blow air through the boiler while the even larger Induced Draft (ID) fans suck the air from the boiler on the other end and blow the hot air up the smoke stack. This causes the steam in the boiler tubes to condense back into water. Steam valves open on the boiler that allow excessive steam to escape.
When the boiler is running there is a large orange fireball hovering in space in the middle of the boiler. The boiler water is being circulated through the boiler and the Boiler Feed Pump Turbines are pumping steam back and forth between the turbine generator and the boiler reheating the steam until every bit of heat from the boiler that can be safely harnessed is used.
When all this stop suddenly, then it is important that the large fans keep running to cool down the steam, since it is no longer losing energy in the generator as it was when it was busy supplying electricity to 1/2 million people in Oklahoma City. The power is fed to the fans from the Auxiliary substation located right outside the Main Switchgear where all the breakers reside that supply the power to the fans. Unfortunately, in this case, the Auxiliary substation was shutdown as well, leaving the boiler without any fans.
Without fans for cooling, and pumps to circulate the water, the walls of the boiler began heating up to dangerous temperatures. Steam was whistling out of pop off valves, but if the steam drum on the top of the boiler were to run dry, then the entire boiler structure could be compromised and begin melting down. — So, this was serious. Something had to be done right away. It wouldn’t be as bad as the China Syndrome since we were burning coal instead of nuclear power, but it would have caused a lot of damage nonetheless.

From the movie “The China Syndrome” where a similar emergency existed only in the movie, it was a Nuclear Plant
I have a side story about this picture, but I think I’ll save it for another post because I don’t want to digress from the main story at this point (Ok. Let me just say “Jack Maloy and Merl Wright” for those who can’t wait) See the post: “Power Plant Conspiracy Theory“.
With the prospect that the boiler might melt to the ground in a pile of rubble, it would seem that the main priority was to turn the Auxiliary Substation back on so the fans could be turned back on and prevent the boiler from collapsing. So, we walked out to the substation and looked at the switches that would have to be operated in order to first power up the main bus and then to close to supply power to the two big transformers and the six smaller transformers that supplied the Unit 1 Main Switchgear.
While inspecting the switches where the electricity had gone to ground we found that one of the main insulators was cracked.
Since this insulator was cracked, we didn’t really want to operate the switch to test if another 189,000 volts would go straight to ground again, especially since one of us would be standing right underneath it cranking the switch. So, we went back to the shop to find an alternative.
By this time the Plant Manager, Ron Kilman arrived in the shop, and understanding the urgency to find a solution asked us what were the alternatives. He was relying on our expertise to make the decision.
The other solution would be to cut the power over from Unit 2 which was still humming away pushing electricity to Oklahoma City out of the 345,000 volt substation. The cut over would be very simple because the switchgear was designed with this in mind. We analyzed the power rating on the auxiliary transformers on Unit 2 and thought that we might be cutting it close to have them running both sets of fans at the same time, especially since the full load amps of a huge fan starting up was about 10 times the normal rate.
The transformer was rated to handle the load, but consider this. What if this caused Unit 2 to trip as well. With the Auxiliary substation offline, if Unit 2 tripped, we would be in twice the amount of trouble we were currently in. What a day it would have been if that had happened and two 250 foot boilers had come crashing to the ground in a pile of rubble. After reading the power ratings on the auxiliary transformers I was thinking, “Yeah, let’s do it! These transformers can handle it.” Andy was not so eager.
So, we were left with one alternative. That was to shut the switch in the Auxiliary substation that had the cracked insulator and take our chances that it wasn’t going to short to ground and blow up over our heads. I think I was eager to close the switch for Andy, but if I remember correctly, he didn’t want me to be the one to suffer the consequences and decided to close the switch himself. Needless to say. Andy closed the switch, and nothing blew up.
As soon as the power was restored to the switchgear, the fans were powered up and the temperature in the boiler was quickly reduced. The coffee pot in the Electric Shop began heating the coffee again. The power plant was saved from a major catastrophe. That was delayed for another day… of which I will talk about later (see the post “Destruction of a Power Plant God).”
So, how exactly does a fire hydrant shoot water up into the bus work of a substation like the picture of the switch directly above? The culprit fire hydrant wasn’t in the substation, it sat alongside it outside the fence a good 50 feet from the high voltage switch. No hose was attached to the fire hydrant. It was only being flushed out as part of a yearly activity to go around and make sure the fire hydrants are all operating correctly.
Here is the story about how the squirrel climbed into the transformer this time….
George Alley, Dale Mitchell and Mickey Postman were going around to the 30,000 fire hydrants on the plant ground (ok. maybe not that many, but we did have a lot of them), and they were opening up the valves and flushing them out. That means, they were letting them run for a while to clear them out from any contaminates that may have built up over the year of not being used.
Throughout their adventure they had opened a multitude of Hydrants situated out in the fields along the long belt conveyor from the coalyard and around the two one-million gallon #2 Diesel tanks.
The brave Power Plant Men, learned that when opening a fire hydrant wide open in the middle of field had unintended consequences. It tended to wash out the ground in front of the flow of the water shooting out of the hydrant. So the team of experts devised a plan to place a board in front of the hydrant when it would be in danger of tearing a hole in the terrain. The board would divert the water into the air where it would fan out and not cause damage to the surrounding area.
This was working fine, and when they arrived at the fire hydrant next to the substation, since the stream from the hydrant was pointing directly into the substation (hmm. a design flaw, I think), they decided to prop the board up against the fence to keep from washing away the gravel in the substation. Well. When a fire hydrant is opened that hasn’t been used for a year, the first flow of water to shoot out is dark brown.
You may think that this is because the water has somehow become dirty over the past year, but that isn’t quite the case. What has happened is that the pipe has been rusting little by little and the water has become saturated with the rust. So, the water shooting out of the hydrant was full of rust (hence the need to flush them out).
Well. Rust is made of metal. Metal is conductive, especially when it is mixed with water. When the water hit the board, it was deflected into the air and happened to direct itself directly into the high voltage switch in the substation. This caused a circuit to the ground which, once it created an arc pumped all the electricity directly into the ground.
Normally when something like this happens it doesn’t trip the Main Power Transformer to a Power Plant.
This time it did. I know there was a few heads scratching trying to figure it out. I think I figured out what happened a little while later. You see… here is the rest of the story….
Once the unit was back online and the emergency was over, someone finally noticed that the telephone system couldn’t call outside of the plant. Well. I was the main telephone person at the time, so the control room called me and asked me to look into the problem.
I checked the telephone computer and it was up and running just fine. Internal calls could be made. Only any call outside just concluded with a funny humming sound. After checking the circuit in the Logic Room next to the Rolm Telephone Computer I headed for…. guess where….. the Main Switchgear….
In the middle of the main switchgear in the back of the room right next to the Auxiliary Substation beyond the back wall, the outside telephone line came into the plant. The first thing it did was go through a special Telephone Surge Protector.
In this picture above, the silver circular buttons on the left side are really an old style surge protector. whenever there was a power surge, the carbon connection in the surge protector would quickly melt causing the circuit to go straight to ground. Thus protecting the rest of the telephone circuit. So, if some kid in their house decides to connect the 120 volts circuit to the telephone for fun to see what would happen, this circuit would protect the rest of the phone circuits. Keep in mind that this was during the early 1990 when “Surge Protection” still was basically was all “mechanical”.
Anyway, when I arrived at this panel and I checked the surge protector to the main line going out of the plant, guess what I found…. Yep. Shorted to ground. Luckily there were some spares that were not wired to anything in the panel and I was able to swap them out for the ones that had been destroyed. — These were a one time use. Which meant, if they ever had to short to ground, they had to be replaced.
Ok. Fine. After a little while, we were able to call back out of the plant, though there was still some residual noise on the line. It was like this… when you called out of the plant, the person on the other end sounded like they were buried in a grave somewhere and they were trying to talk to someone living just like in an episode of the Twilight Episode where a phone line landed on a grave and the dead person tried to call his long lost love from the past.
I didn’t give it much thought other than that I figured the 189,000 volt arc to ground must have shorted out the telephone line since the phone line ran directly under the auxiliary substation ground grid.
It wasn’t until the next morning when the Southwestern Bell repairman showed up at the plant. I knew him well, since he had been working on our phone lines since before the AT&T breakup in 1984. When I met him in the front of the electric shop, he said that he needed to check our telephone circuits. I told him that I knew that we had a problem because we had a high voltage short to ground yesterday and I found our surge protectors melted away.
He explained to me that not only was our circuit affected, but that every relay house from here to Ponca City was blown out. That’s when I realized that the problem was the reverse of the usual situation. What had happened was that the Ground Grid in the substation and the surrounding area (including the Unit 1 Main Power Transformer) had become hot. What do you do when the ground grid becomes charged?
The Ground Grid is what is supposed to protect you when a surge happens, but what happens when the ground grid itself is the problem? In this case, when the high voltage line about 60 feet from the telephone cable surge protector, arced to ground, it fed a tremendous amount of power back through the ground grid. when equipment detected the surge in voltage, they automatically defaulted their circuits to ground. That’s why the telephone circuit died. That’s what tripped the Main Power Transformer.
When the telephone circuit detected the high voltage surge, it shorted to ground (which was the problem), causing the high voltage to feed directly into the phone line and down the line to the next Southwestern Bell relay switch, which also defaulted to ground, trying to bleed off the surge as it went from relay switch to switch until enough of the power was able to be diverted to ground.
That day sure turned out to be a learning experience. I learned that when all the lights go out in the control room, that it is almost assured that the coffee pot in the electric shop is going to stop working. I also learned that in order to coax the plant manager to the electric shop, a major electrical tragedy is one good way. I learned that when shooting rusty water into the air don’t point it at a high voltage auxiliary substation switch. — I’m sure Mickey Postman learned that lesson too. I also learned that just like in Star Trek… whenever there is a dangerous job to do, the Captain is always the one that wants to do it. Does that make sense? Send a Peon like me in there…
I also learned something else about Power Plant Men…. You see…. People like Dale Mitchell, George Alley and Mickey Postman all are examples of incredibly wonderful Power Plant Men. When they were out there doing their duty and something tragic like this, all the Power Plant Men felt their pain. They knew that they all felt guilty for tripping the unit. It didn’t matter that a million dollars every so many minutes was walking out the door in revenue. The only thing that mattered was that these three men were safe.
Since I have left the Power Plant, I have found that the idea that the employee is the greatest asset that a company can possess is not a universal idea. You see, there was never the thought that any of these people should be fired for their mistake. On the contrary. The true Power Plant Men did whatever they could to let them know that they knew exactly how they felt. It could have happened to any of them.
Besides the friendship between Power Plant Men, one of the things I miss most about working at the Power Plant is that the employees are held in high esteem as a real asset to the company. Many could learn from their example.
Comments from the Original post
-
That was an exciting day! Another great story. Thanks for the memories.
-
I like the mix of storytelling and information sharing you deliver here. Thanks again
-
Aug 8, 2011 I lost all reserve in and lost both units in 2 separate storms and A1 would not start for a few minutes and A2 was leaking antifreeze terribly not a scratch and back online in 34 hours both units and no one ever asked me one question even with a Safety Dept not one question asked of me the SS on duty. Not many men on the planet have ever experienced an uncontrolled total plant outage- you would of thought a learning opportunity would of took place. Feb 2011 worst winter temps in years 50 Units or more tripped in Texas let alone the trusty units of Redbud and McClain were fighting Sooner rolled right along with storm warnings for two weeks ahead – the ICS still went to Detroit for just tours of other facilities once again the fall guy Tarver McArthur stood alone. I had authored a Freeze Protection Plan for the plant and that seemed to save the day and explain to the regulatory bodies how we were online and everyone else in Texas was off and enjoying rolling blackouts in a terrible winter weather situation not to mention our powers that be were all stranded in Detroit and very few people could get to the plant without getting stuck trying to get there as well – but a few health heart issues later I am still here to tell about it all you would think folks would want to take advantage of someone that had went through the fire and Ice but thats ok I want them someday distant to get all the credit they deserve when the trumpets sound as that is truly what matters most. DT
-
Great story, and yes, you took exactly the proper lesson from it, and it is too bad that many of our bosses haven’t learned it.
And yup, I’ve buried a lot of electrocuted squirrels over the years.
That was an exciting day! Another great story. Thanks for the memories.