Runaway Fire Hydrant Leaves Power Plant in the Dark
Favorites Post #13 (posted in no particular order)
Originally posted May 17, 2014:
Don’t believe it when the Electric Company tells you that the reason your town lost electricity for an hour was because a squirrel climbed onto a transformer and shorted it out. The real reason just may be more bizarre than that and the company doesn’t want you to know all the different creative ways that power can be shut off. This is a tale of just one of those ways. So, get out your pencil and paper and take notes.
Power Plant Notepad
One spring day in 1993 while sitting at the Precipitator computer for Unit one at the Coal-fired Power Plant in North Central Oklahoma, while I was checking the controls to make sure all the cabinets were operating correctly, suddenly there was a distant boom, and the lights in the control room went out. The computer stayed on because it was connected to an electric panel called the VSP or Vital Services Panel, which in turn was supplied by the UPS system (Uninterruptible Power Supply). That was one of those moments where you may pause for a moment to make sure you aren’t still at home dreaming before you fly into a panic.
The Precipitator cabinets all indicated on the computer that they had just shutdown. I rose from the chair and walked around to the front of the Alarm Panel for Unit one, and found that the fluorescent lights were only out on Unit 1. The lights were still on for Unit 2. The Control Panel was lit up like a Christmas Tree with Green, Red, Blue and Yellow Lights. The Alarm Printer was spewing out paper at high speed. As the large sheets of paper were pouring out onto the floor, I watched as Pat Quiring and other brave Power Plant Control Room operators were scurrying back and forth turning switch handles, pushing buttons, and checking pressure gauges.
Just this site alone gave me confidence that everything was going to be all right. These Control Room operators were all well trained for emergencies just like this, and each person knew what their job was. No one was panicking. Everyone was concentrating on the task at hand.
Someone told me that we lost Unit 1, and the Auxiliary Power to Unit 1 at the same time. So, Unit 1 was dead in the water. This meant, no fans, no pumps, no lights, no vending machines, no cold water at the water fountain and most importantly, no hot coffee!!! I could hear steam valves on the T-G floor banging open and the loud sound of steam escaping.
I turned quickly to go to the electric shop to see what I could do there in case I was needed. I bolted out the door and down the six flights of stairs to the Turbine-Generator (T-G) basement. Exiting the stairway, and entering the T-G basement the sound was deafening. I grabbed the earplugs that were dangling around my neck and crammed them into my ears. Steam was pouring out of various pop-off valves. I ducked into the electric shop where across the room Andy Tubbs, one of the electric foreman was pulling large sheets of electric blueprints from the print cabinet and laying them across the work table that doubled as the lunch table.
Andy Tubbs – True Power Plant Electrician
When I asked Andy what happened, I learned that somehow when a crew was flushing out a fire hydrant the water somehow shot up and into the bus work in the Auxiliary Substation (that supplies backup power to the Power Plant) and it shorted out the 189,000 volt substation directly to ground. When that happened it tripped unit 1 and the auxiliary substation at the same time leaving it without power.
An example of part of an Auxiliary Substation
I will explain how a fire hydrant could possibly spray the bus work in a substation in a little while, but first let me tell you what this meant at the moment to not have any power for a Power Plant Boiler and Turbine Generator that has just tripped when it was at full load which was around 515 Megawatts of power at the time.
Normally when a unit trips, the boiler cools down as the large Force Draft (FD) Fans blow air through the boiler while the even larger Induced Draft (ID) fans suck the air from the boiler on the other end and blow the hot air up the smoke stack. This causes the steam in the boiler tubes to condense back into water. Steam valves open on the boiler that allow excessive steam to escape.
When the boiler is running there is a large orange fireball hovering in space in the middle of the boiler. The boiler water is being circulated through the boiler and the Boiler Feed Pump Turbines are pumping steam back and forth between the turbine generator and the boiler reheating the steam until every bit of heat from the boiler that can be safely harnessed is used.
When all this stop suddenly, then it is important that the large fans keep running to cool down the steam, since it is no longer losing energy in the generator as it was when it was busy supplying electricity to 1/2 million people in Oklahoma City. The power is fed to the fans from the Auxiliary substation located right outside the Main Switchgear where all the breakers reside that supply the power to the fans. Unfortunately, in this case, the Auxiliary substation was shutdown as well, leaving the boiler without any fans.
Without fans for cooling, and pumps to circulate the water, the walls of the boiler began heating up to dangerous temperatures. Steam was whistling out of pop off valves, but if the steam drum on the top of the boiler were to run dry, then the entire boiler structure could be compromised and begin melting down. — So, this was serious. Something had to be done right away. It wouldn’t be as bad as the China Syndrome since we were burning coal instead of nuclear power, but it would have caused a lot of damage nonetheless.
From the movie “The China Syndrome” where a similar emergency existed only in the movie, it was a Nuclear Plant
I have a side story about this picture, but I think I’ll save it for another post because I don’t want to digress from the main story at this point (Ok. Let me just say “Jack Maloy and Merl Wright” for those who can’t wait. See the post: “Power Plant Conspiracy Theory“).
With the prospect that the boiler might melt to the ground in a pile of rubble, it would seem that the main priority was to turn the Auxiliary Substation back on so the fans could be turned back on and prevent the boiler from collapsing. So, we walked out to the substation and looked at the switches that would have to be operated in order to first power up the main bus and then to close to supply power to the two big transformers and the six smaller transformers that supplied the Unit 1 Main Switchgear.
While inspecting the switches where the electricity had gone to ground we found that one of the main insulators was cracked.
A High Voltage Insulator like this
Since this insulator was cracked, we didn’t really want to operate the switch to test if another 189,000 volts would go straight to ground again, especially since one of us would be standing right underneath it cranking the switch. So, we went back to the shop to find an alternative.
By this time the Plant Manager, Ron Kilman arrived in the shop, and understanding the urgency to find a solution asked us what were the alternatives. He was relying on our expertise to make the decision.
Ron Kilman is the one on the left in the plaid shirt
The other solution would be to cut the power over from Unit 2 which was still humming away pushing electricity to Oklahoma City out of the 345,000 volt substation. The cut over would be very simple because the switchgear was designed with this in mind. We analyzed the power rating on the auxiliary transformers on Unit 2 and thought that we might be cutting it close to have them running both sets of fans at the same time, especially since the full load amps of a huge fan starting up was about 10 times the normal rate.
The transformer was rated to handle the load, but consider this. What if this caused Unit 2 to trip as well. With the Auxiliary substation offline, if Unit 2 tripped, we would be in twice the amount of trouble we were currently in. What a day it would have been if that had happened and two 250 foot boilers had come crashing to the ground in a pile of rubble. After reading the power ratings on the auxiliary transformers I was thinking, “Yeah, let’s do it! These transformers can handle it.” Andy was not so eager.
So, we were left with one alternative. That was to shut the switch in the Auxiliary substation that had the cracked insulator and take our chances that it wasn’t going to short to ground and blow up over our heads. I think I was eager to close the switch for Andy, but if I remember correctly, he didn’t want me to be the one to suffer the consequences and decided to close the switch himself. Needless to say. Andy closed the switch, and nothing blew up.
High Voltage Switch similar to the one we were closing
As soon as the power was restored to the switchgear, the fans were powered up and the temperature in the boiler was quickly reduced. The coffee pot in the Electric Shop began heating the coffee again. The power plant was saved from a major catastrophe. That was delayed for another day… of which I will talk about later (see the post “Destruction of a Power Plant God).”
So, how exactly does a fire hydrant shoot water up into the bus work of a substation like the picture of the switch directly above? The culprit fire hydrant wasn’t in the substation, it sat alongside it outside the fence a good 50 feet from the high voltage switch. No hose was attached to the fire hydrant. It was only being flushed out as part of a yearly activity to go around and make sure the fire hydrants are all operating correctly.
Here is the story about how the squirrel climbed into the transformer this time….
George Alley, Dale Mitchell and Mickey Postman were going around to the 30,000 fire hydrants on the plant ground (ok. maybe not that many, but we did have a lot of them), and they were opening up the valves and flushing them out. That means, they were letting them run for a while to clear them out from any contaminates that may have built up over the year of not being used.
Throughout their adventure they had opened a multitude of Hydrants situated out in the fields along the long belt conveyor from the coalyard and around the two one-million gallon #2 Diesel tanks.
Large Oil Tank about the size of the two that are at the Power Plant
The brave Power Plant Men, learned that when opening a fire hydrant wide open in the middle of field had unintended consequences. It tended to wash out the ground in front of the flow of the water shooting out of the hydrant. So the team of experts devised a plan to place a board in front of the hydrant when it would be in danger of tearing a hole in the terrain. The board would divert the water into the air where it would fan out and not cause damage to the surrounding area.
This was working fine, and when they arrived at the fire hydrant next to the substation, since the stream from the hydrant was pointing directly into the substation (hmm. a design flaw, I think), they decided to prop the board up against the fence to keep from washing away the gravel in the substation. Well. When a fire hydrant is opened that hasn’t been used for a year, the first flow of water to shoot out is dark brown.
You may think that this is because the water has somehow become dirty over the past year, but that isn’t quite the case. What has happened is that the pipe has been rusting little by little and the water has become saturated with the rust. So, the water shooting out of the hydrant was full of rust (hence the need to flush them out).
Well. Rust is made of metal. Metal is conductive, especially when it is mixed with water. When the water hit the board, it was deflected into the air and happened to direct itself directly into the high voltage switch in the substation. This caused a circuit to the ground which, once it created an arc pumped all the electricity directly into the ground.
Normally when something like this happens it doesn’t trip the Main Power Transformer to a Power Plant.
A Main Power Transformer
This time it did. I know there was a few heads scratching trying to figure it out. I think I figured out what happened a little while later. You see… here is the rest of the story….
Once the unit was back online and the emergency was over, someone finally noticed that the telephone system couldn’t call outside of the plant. Well. I was the main telephone person at the time, so the control room called me and asked me to look into the problem.
I checked the telephone computer and it was up and running just fine. Internal calls could be made. Only any call outside just concluded with a funny humming sound. After checking the circuit in the Logic Room next to the Rolm Telephone Computer I headed for…. guess where….. the Main Switchgear….
A ROLM Phone Computer
In the middle of the main switchgear in the back of the room right next to the Auxiliary Substation beyond the back wall, the outside telephone line came into the plant. The first thing it did was go through a special Telephone Surge Protector.
Telephone Grounding Panel
In this picture above, the silver circular buttons on the left side are really an old style surge protector. whenever there was a power surge, the carbon connection in the surge protector would quickly melt causing the circuit to go straight to ground. Thus protecting the rest of the telephone circuit. So, if some kid in their house decides to connect the 120 volts circuit to the telephone for fun to see what would happen, this circuit would protect the rest of the phone circuits. Keep in mind that this was during the early 1990 when “Surge Protection” still was basically all “mechanical”.
Anyway, when I arrived at this panel and I checked the surge protector to the main line going out of the plant, guess what I found…. Yep. Shorted to ground. Luckily there were some spares that were not wired to anything in the panel and I was able to swap them out for the ones that had been destroyed. — These were a one time use. Which meant, if they ever had to short to ground, they had to be replaced.
Ok. Fine. After a little while, we were able to call back out of the plant, though there was still some residual noise on the line. It was like this… when you called out of the plant, the person on the other end sounded like they were buried in a grave somewhere and they were trying to talk to someone living just like in an episode of the Twilight Episode where a phone line landed on a grave and the dead person tried to call his long lost love from the past.
Gladys Cooper in the Twilight Zone Episode “Night Call”
I didn’t give it much thought other than that I figured the 189,000 volt arc to ground must have shorted out the telephone line since the phone line ran directly under the auxiliary substation ground grid.
It wasn’t until the next morning when the Southwestern Bell repairman showed up at the plant. I knew him well, since he had been working on our phone lines since before the AT&T breakup in 1984. When I met him in the front of the electric shop, he said that he needed to check our telephone circuits. I told him that I knew that we had a problem because we had a high voltage short to ground yesterday and I found our surge protectors melted away.
He explained to me that not only was our circuit affected, but that every relay house from here to Ponca City was blown out. That’s when I realized that the problem was the reverse of the usual situation. What had happened was that the Ground Grid in the substation and the surrounding area (including the Unit 1 Main Power Transformer) had become hot. What do you do when the ground grid becomes charged?
The Ground Grid is what is supposed to protect you when a surge happens, but what happens when the ground grid itself is the problem? In this case, when the high voltage line about 60 feet from the telephone cable surge protector, arced to ground, it fed a tremendous amount of power back through the ground grid. when equipment detected the surge in voltage, they automatically defaulted their circuits to ground. That’s why the telephone circuit died. That’s what tripped the Main Power Transformer.
When the telephone circuit detected the high voltage surge, it shorted to ground (which was the problem), causing the high voltage to feed directly into the phone line and down the line to the next Southwestern Bell relay switch, which also defaulted to ground, trying to bleed off the surge as it went from relay switch to switch until enough of the power was able to be diverted to ground.
That day sure turned out to be a learning experience. I learned that when all the lights go out in the control room, that it is almost assured that the coffee pot in the electric shop is going to stop working. I also learned that in order to coax the plant manager to the electric shop, a major electrical tragedy is one good way. I learned that when shooting rusty water into the air don’t point it at a high voltage auxiliary substation switch. — I’m sure Mickey Postman learned that lesson too. I also learned that just like in Star Trek… whenever there is a dangerous job to do, the Captain is always the one that wants to do it. Does that make sense? Send a Peon like me in there…
I also learned something else about Power Plant Men…. You see…. People like Dale Mitchell, George Alley and Mickey Postman all are examples of incredibly wonderful Power Plant Men. When they were out there doing their duty and something tragic like this, all the Power Plant Men felt their pain. They knew that they all felt guilty for tripping the unit. It didn’t matter that a million dollars every so many minutes was walking out the door in revenue. The only thing that mattered was that these three men were safe.
Mickey Postman
Since I have left the Power Plant, I have found that the idea that the employee is the greatest asset that a company can possess is not a universal idea. You see, there was never the thought that any of these people should be fired for their mistake. On the contrary. The true Power Plant Men did whatever they could to let them know that they knew exactly how they felt. It could have happened to any of them.
Besides the friendship between Power Plant Men, one of the things I miss most about working at the Power Plant is that the employees are held in high esteem as a real asset to the company. Many could learn from their example.
Comments from the Original post
-
-
I like the mix of storytelling and information sharing you deliver here. Thanks again
-
-
-
Aug 8, 2011 I lost all reserve in and lost both units in 2 separate storms and A1 would not start for a few minutes and A2 was leaking antifreeze terribly not a scratch and back online in 34 hours both units and no one ever asked me one question even with a Safety Dept not one question asked of me the SS on duty. Not many men on the planet have ever experienced an uncontrolled total plant outage- you would of thought a learning opportunity would of took place. Feb 2011 worst winter temps in years 50 Units or more tripped in Texas let alone the trusty units of Redbud and McClain were fighting Sooner rolled right along with storm warnings for two weeks ahead – the ICS still went to Detroit for just tours of other facilities once again the fall guy Tarver McArthur stood alone. I had authored a Freeze Protection Plan for the plant and that seemed to save the day and explain to the regulatory bodies how we were online and everyone else in Texas was off and enjoying rolling blackouts in a terrible winter weather situation not to mention our powers that be were all stranded in Detroit and very few people could get to the plant without getting stuck trying to get there as well – but a few health heart issues later I am still here to tell about it all you would think folks would want to take advantage of someone that had went through the fire and Ice but thats ok I want them someday distant to get all the credit they deserve when the trumpets sound as that is truly what matters most. DT
-
-
Great story, and yes, you took exactly the proper lesson from it, and it is too bad that many of our bosses haven’t learned it.
And yup, I’ve buried a lot of electrocuted squirrels over the years.
Rivers and Rose in the Power Plant Palace
Originally posted January 25, 2013:
When is the appropriate time to call 911? Calling 911 in the Power Plant is when you call the Shift Supervisor to report something important. As Randy Dailey, our Safety Trainer extraordinaire, always taught us, first tap the person on the shoulder and say, “Are you all right?” Then you point your finger at someone and say, “Call 911!” That’s called “Activating the EMS” (Emergency Medical System). Besides medical emergencies, there are other reasons to call the Shift Supervisor.
I learned early on to ‘fess up when you have done something wrong.” People appreciate it when you tell them up front that you goofed. That way the problem can be dealt with directly. Dee Ball was that way. Any time he wrecked a truck, he didn’t hesitate to tell his boss. So, even as a summer help I had developed this philosophy. Never be afraid to expose your blunders. It works out better in the long run.
One example of someone not following this philosophy was Curtis Love. As I mentioned in the post Power Plant Spider Wars and Bugs in the Basement, Curtis didn’t want to tell anyone that he had been bitten by a brown recluse for the third time because he was afraid of losing his job.
The Oklahoma house spider — The Brown Recluse
His philosophy came back to bite him a year and a half later when he was on the labor crew when he was the designated truck driver. I had moved on to the electric shop by this time.
A red crew cab like this only the bed of the truck was longer
He was backing up the crew cab around a corner under the Fly Ash hoppers up at the coalyard when the side of the crew cab came into contact with one of those yellow poles designed to protect the structure from rogue vehicles. Unfortunately. This created a dent in the side of the truck.
An example of yellow poles protecting an area
Curtis, already on probation. worried that he would be fired if he told anyone about this mishap, failed to tell Larry Riley about this incident. Larry, on the other hand, was standing in front of the Coalyard Maintenance shop (the labor crew home), and saw the entire incident. At that moment, he turned to one of the labor crew hands and said, “I hope Curtis comes over here and tells me about that.” Unfortunately, Curtis decided to act as if nothing had happened. This resulted in his termination. As much as I cared about Curtis, I must admit that the Power Plant scene was probably not the best location for his vocation.
I had seen Dee Ball do the same thing over and over again, and he always reported his accidents immediately. He was never punished for an accident, though, for a number of years, he was banned from driving a truck. You can read more about this in the post: Experiencing Maggots, Mud and Motor Vehicles with Dee Ball.
One day during the summer of 1984 just after lunch, 1A PA fan tripped (PA stands for Primary Air). When this happened, number one unit had to lower it’s output from over 500 Megawatts down to around 200. The trip indicator on the 6900 volt breaker said that it had been grounded. Being grounded means that one of the three phases of the motor or cable had made a circuit with the ground (or something that was grounded). The trip circuits shut the fan down so fast that it prevents an explosion and saves the fan from being destroyed.
Diana Lucas (later Diana Brien), Andy Tubbs and I were given the task of finding the ground and seeing what we could do to fix it. We unwired the motor, which was no easy task, because the motor is about the size of a large van, and about 10 times heavier.
This is about 1/2 the size of the PA fan motor which is 1000 horsepower
So, we spent the rest of the day unwiring the motor (in the rain), and unwiring the cable to the motor from the breaker in the main switchgear and testing both the motor and the cable with various instruments looking for the grounded wire or coil that caused the motor to trip. We used a large “Megger” on the motor. It’s called a Megger because it measures Mega-Ohms. So, it’s technically called a Mega-Ohm meter. Ohms is a measurement of resistance in an electrical circuit. We usually use a small hand cranked megger, that is similar to an old hand crank telephone that generates a high voltage (good for shocking fish in a lake to make them rise to the surface). In the case of the hand cranked Megger, it would generate 1,000 volts.
A Hand cranked Megger made by the same company, only newer than the ones we had
The Megger this size would have been useless with this large motor. Instead we used one that was electric, and you ran the voltage up over 10,000 volts and watched the mega-ohms over a period of 1/2 hour or so.
For the cables, we hooked up a Hypot (or Hipot). This stands for High Potential. Potential in this case is another word for “Voltage”. It would charge up and then you pressed a button and it would send a high voltage pulse down the cable, and if there is a weak spot in the insulation,The Hypot will find it. So, we hooked a Hypot up to the cable and tried to find the grounded wire. No luck.
After spending 4 hours looking for the grounded cable or motor, we found nothing. We spent another hour and a half putting the motor and the breaker back in service. The Fan was put back into operation and we went home. As I was walking out to the car with Bill Rivers, he told me, “I knew they weren’t going to find anything wrong with that fan.” He had a big grin on his face.
At first I thought he was just making an educated guess as Rivers was apt to do on many occasions (daily). It was raining and I could see where water may have been sucked into the motor or something and had momentarily grounded the motor. Just because we didn’t find anything didn’t mean that the breaker didn’t trip for no reason.
When we were in the car and on our way to Stillwater, Oklahoma with Yvonne Taylor and Rich Litzer, Bill explained that he knew why the motor tripped. He had been walking through the main switchgear with Mike Rose, and Mike, for no apparent reason other than curiosity, had opened up the bottom door to the breaker for 1A PA fan. He looked at it for a moment and then slammed the door shut. When he did this, the breaker tripped.
So, the ground relay happened to be the one that tripped. It might as well been an over-current or a low voltage trip. It just happened to trip the ground trip. Bill said that he told Mike that he should call the Shift Supervisor and let him know so they could restart the motor. Mike on the other hand told Bill that he was already on probation and was afraid of losing his job if he reported that he had slammed the door on the breaker and tripped the fan.
If there was ever a reason to call 911, it was then. All he had to do was tell them, “I accidentally tripped the PA fan when I bumped the breaker cabinet.” They would have told him to reset the flag, and they would have started the fan right back up. No questions asked… I’m sure of it. And they wouldn’t have lost their generating capacity for the remainder of the afternoon and we wouldn’t have spent 4 hours unwiring, testing and rewiring the motor in the rain with a plastic umbrella over our head.
Bill wasn’t about to tell on Mike. If Mike didn’t want to report it, Bill wasn’t going to say anything, and I understand that. I probably would have kept it to myself at the time if I was in Bill’s shoes (I’m just glad I wasn’t because I probably wouldn’t have been able to sleep soundly for the next year). But 30 years later, I might write about it in a Blog. Even though I wouldn’t have looked to Mike to teach me much about being an electrician (he was more of an Air Condition man anyway), I still loved the guy.
Mike died almost two years ago on May 29, 2011. He was from England and had lived in Canada for a time. He used to work on trains. Trains, even though they are diesel, are really electric. The Diesel engine really runs a generator that generates electricity that runs the train. I know that Mike was a good man at heart. He loved his family with all his heart. Here is a picture of the Limey:
Mike Rose. A fair plant electrician, but a great family man!
Ok. So I know what you are thinking…. There must be a story about myself in here somewhere. Well, you would be right. First of all. I always ‘fessed up to my mistakes, as my current manager at Dell knows well (yes. I still mess up after all these years). I told my current manager the other day that CLM was my middle name. (CLM means “Career Limiting Move”). So here is my power plant “mess up” story (well one of them):
In January 1986, I returned from my Honeymoon with my new wife Kelly when I found that we had hired a new electrician. Gary Wehunt was replacing Jim Stephenson who had left the plant on February 15, 1985, which is a story all it’s own. We had just started an overhaul on Unit 1.
I remember the first Monday I spent with Gary. It was January 6, 1986 and we were working on cleaning out the exciter house on the end of the main power generator with Diana Brien (formerly Diana Lucas). We were discussing salaries and Gary was surprised to find out that I was making more than he was. Well… I had been an electrician for over 2 years and had been promoted regularly…. so I didn’t think there was anything strange about it, except that I still looked like I was only about 18 years old (even though I was 25) and Gary was about 34. I had already been promoted 4 times and my salary had gone from $7.15 to over $12 an hour.
Anyway, when that first Wednesday rolled around, Since Gary and I were assigned to Substation Inspection that week, (Some later time I may go into the details of what “Substation Inspection” entails), but for now, let’s just stick with my “911 call.” It is enough to say that we were in the main plant substation relay house on Wdnesday January 8, 1986 at 9:00 am. One of our jobs was to call other substations and perform a test called a “Transfer Trip and Carrier Test”. We had called Woodring Substation (Woodring is a town in Oklahoma and we had a 345 KV line going there), and I was talking to the man in the substation on the other end of the phone line.
At the same time I was showing Gary just how experienced I was at being an electrician. People had told me that you had to be a plant electrician for 5 years before you really became a “first class” electrician. Well. Here I was at 2 years, and I thought I was so good that I could do anything by now…. — Yeah… right. I told the guy on the other end of the line as I turned a switch…. Amber light… Back to Blue…. and I wrote down the value on the meter (paperwork… oh yes…. it’s that important. Like A-1 sauce).
Then I reached for the second switch. I said, “Carrier test”, then turned the switch. The lights in the relay house went out and we were in the dark. I told the guy on the other end of the line….. “Well. That’s not supposed to happen.” Then as I let go of the switch and it returned to it’s normal position, the lights turned back on. Okay……
I wrote the numbers down from the meter and said goodbye to the other faceless substation man on the other end of the line that I talked to over 100 times, but never met in person. He sounded like a nice guy. Then I headed for the gray phone. I heard the Shift Supervisor paging Leroy Godfrey (The Electrical Supervisor) on line 2 (we had 5 Gray phone lines. The Gray Phone was our PA system).
When I picked up the line I heard Leroy pick up the phone and the Shift Supervisor tell Leroy that we just lost station power in the main substation and it had switched over to Auxiliary power. I immediately jumped in and said, “Jim (for Jim Padgett, the Shift Supervisor), I did that. I was performing a Carrier test with Woodring and the moment I performed the carrier test the lights went out.” Leroy chimed in by saying, “That wouldn’t cause you to lose station power.”
Well, in my ‘inexperienced’ plant electrician way, I responded, “Well. All I know is that when I turned the switch to perform the carrier test, the lights went out, and when I let go of the switch, the lights came back on.” Leroy reiterated, “That wouldn’t cause you to lose station power.” I replied with, “I’m just saying….” and left it at that. I had done my job. They knew I was out here. They knew I had called 911 right away. I explained what I was doing…. they could take it from there.
I had hoped that I had showed Gary upfront that it doesn’t hurt to report your mistakes (even though I hadn’t made one as far as I could tell), but I was 100% sure I had done something to cause the relay house to lose power. Though, I couldn’t figure out why.
After lunch, Bill Bennett, our A foreman came down to the shop to tell me that they figured out how the substation lost station power. He said that a road grader had been grating the road down by the Otoe-Missouri reservation (which is actually called “Windmill road” I guess because there is a windmill down that road somewhere), and had hit an electric pole and knocked it over and had killed the power to the substation.
Substation Power Interrupting Road Grader
It turned out that the substation relay house was fed by a substation down that road where we have a radio tower. So, think about this. The exact time that I turned that switch in the substation, a road grater 2 1/2 miles away hits a telephone pole accidentally and knocks it to the ground and kills the power to the substation at the exact same time that I am performing a transfer-trip and Carrier test with Woodring Substation, and the time it takes to switch to auxiliary power is the exact time it took me to let go of the switch.
Don’t tell me that was by accident. I will never believe it. I think it was for the soul purpose of teaching me a useful lesson or two. First….. don’t be afraid to tell someone when you do something wrong. Second…. If you think you have control over the things that happen to you in your life… well, think again…… Third….. God watches you every moment, and if you let him, he will guide you to do the right thing when the time comes. Oh, and Fourth: “Doesn’t God have a great sense of humor?”
God bless you all.
COMMENTS FROM THE ORIGINAL POST:
-
Monty Hansen January 26, 2013
I had a similar thing happen to me, I was upgrading to shift foreman & system called to remove a tag in the switchyard & put the switch back to auto. The tag on the pistol grip was attached with a plastic zip tie & the previous operator had put it on real tight, as I was wrestling it off with my leatherman, the pliers slipped & I banged my elbow into the control panel, at that very instant there was a loud BANG as several 345 KV breakers opened simultaneously in the switchyard, I had the phone pinched between my shoulder & ear as I was wrestling with this switch & talking to the system control operator, he said a few bad words – gotta go – & hung up. The power plant lost all power & went in the black, I, of course was just sick in the pit of my stomach, after we got power restored, the plant back on etc. I called system back to see if they found the cause & fess up to causing the trip (I figured I must have caused a trip relay to close when I hit the panel) – anyway a crane at a plant down the road had got it’s boom tangled in the power line & went to ground – AT PRECISELY THE INSTANT MY ELBOW SLIPPED & HIT THE PANEL!!
-
Ron Kilman January 26, 2013
Some great illustrations of the truth in Proverbs 28:13 “He who conceals his transgressions will not prosper, But he who confesses and forsakes them will find compassion”.
-
justturnright January 28, 2013
CLM: I can relate.
My first boss 30 years ago once told me he was going to officially nickname me “I’m sorry” (and make me wear it for a name badge) if I said it one more time.
Hey, there’s worse things.
-
Roomy January 29, 2013
I had not thought about Mike Rose in years. He was a good guy to work with, now Rivers was a different story!!!
Sub checks, I used to love to do sub checks. I performed pilot wire & transfer trip checks for years. I hated it when they went to being done by automation.
Thanks for bringing back old memories.
Runaway Fire Hydrant Leaves Power Plant in the Dark
Originally posted May 17, 2014:
Don’t believe it when the Electric Company tells you that the reason your town lost electricity for an hour was because a squirrel climbed onto a transformer and shorted it out. The real reason just may be more bizarre than that and the company doesn’t want you to know all the different creative ways that power can be shut off. This is a tale of just one of those ways. So, get out your pencil and paper and take notes.
Power Plant Notepad
One spring day in 1993 while sitting at the Precipitator computer for Unit one at the Coal-fired Power Plant in North Central Oklahoma, while I was checking the controls to make sure all the cabinets were operating correctly, suddenly there was a distant boom, and the lights in the control room went out. The computer stayed on because it was connected to an electric panel called the VSP or Vital Services Panel, which in turn was supplied by the UPS system (Uninterruptible Power Supply). That was one of those moments where you may pause for a moment to make sure you aren’t still at home dreaming before you fly into a panic.
The Precipitator cabinets all indicated on the computer that they had just shutdown. I rose from the chair and walked around to the front of the Alarm Panel for Unit one, and found that the fluorescent lights were only out on Unit 1. The lights were still on for Unit 2. The Control Panel was lit up like a Christmas Tree with Green, Red, Blue and Yellow Lights. The Alarm Printer was spewing out paper at high speed. As the large sheets of paper were pouring out onto the floor, I watched as Pat Quiring and other brave Power Plant Control Room operators were scurrying back and forth turning switch handles, pushing buttons, and checking pressure gauges.
Just this site alone gave me confidence that everything was going to be all right. These Control Room operators were all well trained for emergencies just like this, and each person knew what their job was. No one was panicking. Everyone was concentrating on the task at hand.
Someone told me that we lost Unit 1, and the Auxiliary Power to Unit 1 at the same time. So, Unit 1 was dead in the water. This meant, no fans, no pumps, no lights, no vending machines, no cold water at the water fountain and most importantly, no hot coffee!!! I could hear steam valves on the T-G floor banging open and the loud sound of steam escaping.
I turned quickly to go to the electric shop to see what I could do there in case I was needed. I bolted out the door and down the six flights of stairs to the Turbine-Generator (T-G) basement. Exiting the stairway, and entering the T-G basement the sound was deafening. I grabbed the earplugs that were dangling around my neck and crammed them into my ears. Steam was pouring out of various pop-off valves. I ducked into the electric shop where across the room Andy Tubbs, one of the electric foreman was pulling large sheets of electric blueprints from the print cabinet and laying them across the work table that doubled as the lunch table.
Andy Tubbs – True Power Plant Electrician
When I asked Andy what happened, I learned that somehow when a crew was flushing out a fire hydrant the water somehow shot up and into the bus work in the Auxiliary Substation (that supplies backup power to the Power Plant) and it shorted out the 189,000 volt substation directly to ground. When that happened it tripped unit 1 and the auxiliary substation at the same time leaving it without power.
An example of part of an Auxiliary Substation
I will explain how a fire hydrant could possibly spray the bus work in a substation in a little while, but first let me tell you what this meant at the moment to not have any power for a Power Plant Boiler and Turbine Generator that has just tripped when it was at full load which was around 515 Megawatts of power at the time.
Normally when a unit trips, the boiler cools down as the large Force Draft (FD) Fans blow air through the boiler while the even larger Induced Draft (ID) fans suck the air from the boiler on the other end and blow the hot air up the smoke stack. This causes the steam in the boiler tubes to condense back into water. Steam valves open on the boiler that allow excessive steam to escape.
When the boiler is running there is a large orange fireball hovering in space in the middle of the boiler. The boiler water is being circulated through the boiler and the Boiler Feed Pump Turbines are pumping steam back and forth between the turbine generator and the boiler reheating the steam until every bit of heat from the boiler that can be safely harnessed is used.
When all this stop suddenly, then it is important that the large fans keep running to cool down the steam, since it is no longer losing energy in the generator as it was when it was busy supplying electricity to 1/2 million people in Oklahoma City. The power is fed to the fans from the Auxiliary substation located right outside the Main Switchgear where all the breakers reside that supply the power to the fans. Unfortunately, in this case, the Auxiliary substation was shutdown as well, leaving the boiler without any fans.
Without fans for cooling, and pumps to circulate the water, the walls of the boiler began heating up to dangerous temperatures. Steam was whistling out of pop off valves, but if the steam drum on the top of the boiler were to run dry, then the entire boiler structure could be compromised and begin melting down. — So, this was serious. Something had to be done right away. It wouldn’t be as bad as the China Syndrome since we were burning coal instead of nuclear power, but it would have caused a lot of damage nonetheless.
From the movie “The China Syndrome” where a similar emergency existed only in the movie, it was a Nuclear Plant
I have a side story about this picture, but I think I’ll save it for another post because I don’t want to digress from the main story at this point (Ok. Let me just say “Jack Maloy and Merl Wright” for those who can’t wait) See the post: “Power Plant Conspiracy Theory“.
With the prospect that the boiler might melt to the ground in a pile of rubble, it would seem that the main priority was to turn the Auxiliary Substation back on so the fans could be turned back on and prevent the boiler from collapsing. So, we walked out to the substation and looked at the switches that would have to be operated in order to first power up the main bus and then to close to supply power to the two big transformers and the six smaller transformers that supplied the Unit 1 Main Switchgear.
While inspecting the switches where the electricity had gone to ground we found that one of the main insulators was cracked.
A High Voltage Insulator like this
Since this insulator was cracked, we didn’t really want to operate the switch to test if another 189,000 volts would go straight to ground again, especially since one of us would be standing right underneath it cranking the switch. So, we went back to the shop to find an alternative.
By this time the Plant Manager, Ron Kilman arrived in the shop, and understanding the urgency to find a solution asked us what were the alternatives. He was relying on our expertise to make the decision.
Ron Kilman is the one on the left in the plaid shirt
The other solution would be to cut the power over from Unit 2 which was still humming away pushing electricity to Oklahoma City out of the 345,000 volt substation. The cut over would be very simple because the switchgear was designed with this in mind. We analyzed the power rating on the auxiliary transformers on Unit 2 and thought that we might be cutting it close to have them running both sets of fans at the same time, especially since the full load amps of a huge fan starting up was about 10 times the normal rate.
The transformer was rated to handle the load, but consider this. What if this caused Unit 2 to trip as well. With the Auxiliary substation offline, if Unit 2 tripped, we would be in twice the amount of trouble we were currently in. What a day it would have been if that had happened and two 250 foot boilers had come crashing to the ground in a pile of rubble. After reading the power ratings on the auxiliary transformers I was thinking, “Yeah, let’s do it! These transformers can handle it.” Andy was not so eager.
So, we were left with one alternative. That was to shut the switch in the Auxiliary substation that had the cracked insulator and take our chances that it wasn’t going to short to ground and blow up over our heads. I think I was eager to close the switch for Andy, but if I remember correctly, he didn’t want me to be the one to suffer the consequences and decided to close the switch himself. Needless to say. Andy closed the switch, and nothing blew up.
High Voltage Switch similar to the one we were closing
As soon as the power was restored to the switchgear, the fans were powered up and the temperature in the boiler was quickly reduced. The coffee pot in the Electric Shop began heating the coffee again. The power plant was saved from a major catastrophe. That was delayed for another day… of which I will talk about later (see the post “Destruction of a Power Plant God).”
So, how exactly does a fire hydrant shoot water up into the bus work of a substation like the picture of the switch directly above? The culprit fire hydrant wasn’t in the substation, it sat alongside it outside the fence a good 50 feet from the high voltage switch. No hose was attached to the fire hydrant. It was only being flushed out as part of a yearly activity to go around and make sure the fire hydrants are all operating correctly.
Here is the story about how the squirrel climbed into the transformer this time….
George Alley, Dale Mitchell and Mickey Postman were going around to the 30,000 fire hydrants on the plant ground (ok. maybe not that many, but we did have a lot of them), and they were opening up the valves and flushing them out. That means, they were letting them run for a while to clear them out from any contaminates that may have built up over the year of not being used.
Throughout their adventure they had opened a multitude of Hydrants situated out in the fields along the long belt conveyor from the coalyard and around the two one-million gallon #2 Diesel tanks.
Large Oil Tank about the size of the two that are at the Power Plant
The brave Power Plant Men, learned that when opening a fire hydrant wide open in the middle of field had unintended consequences. It tended to wash out the ground in front of the flow of the water shooting out of the hydrant. So the team of experts devised a plan to place a board in front of the hydrant when it would be in danger of tearing a hole in the terrain. The board would divert the water into the air where it would fan out and not cause damage to the surrounding area.
This was working fine, and when they arrived at the fire hydrant next to the substation, since the stream from the hydrant was pointing directly into the substation (hmm. a design flaw, I think), they decided to prop the board up against the fence to keep from washing away the gravel in the substation. Well. When a fire hydrant is opened that hasn’t been used for a year, the first flow of water to shoot out is dark brown.
You may think that this is because the water has somehow become dirty over the past year, but that isn’t quite the case. What has happened is that the pipe has been rusting little by little and the water has become saturated with the rust. So, the water shooting out of the hydrant was full of rust (hence the need to flush them out).
Well. Rust is made of metal. Metal is conductive, especially when it is mixed with water. When the water hit the board, it was deflected into the air and happened to direct itself directly into the high voltage switch in the substation. This caused a circuit to the ground which, once it created an arc pumped all the electricity directly into the ground.
Normally when something like this happens it doesn’t trip the Main Power Transformer to a Power Plant.
A Main Power Transformer
This time it did. I know there was a few heads scratching trying to figure it out. I think I figured out what happened a little while later. You see… here is the rest of the story….
Once the unit was back online and the emergency was over, someone finally noticed that the telephone system couldn’t call outside of the plant. Well. I was the main telephone person at the time, so the control room called me and asked me to look into the problem.
I checked the telephone computer and it was up and running just fine. Internal calls could be made. Only any call outside just concluded with a funny humming sound. After checking the circuit in the Logic Room next to the Rolm Telephone Computer I headed for…. guess where….. the Main Switchgear….
A ROLM Phone Computer
In the middle of the main switchgear in the back of the room right next to the Auxiliary Substation beyond the back wall, the outside telephone line came into the plant. The first thing it did was go through a special Telephone Surge Protector.
Telephone Grounding Panel
In this picture above, the silver circular buttons on the left side are really an old style surge protector. whenever there was a power surge, the carbon connection in the surge protector would quickly melt causing the circuit to go straight to ground. Thus protecting the rest of the telephone circuit. So, if some kid in their house decides to connect the 120 volts circuit to the telephone for fun to see what would happen, this circuit would protect the rest of the phone circuits. Keep in mind that this was during the early 1990 when “Surge Protection” still was basically all “mechanical”.
Anyway, when I arrived at this panel and I checked the surge protector to the main line going out of the plant, guess what I found…. Yep. Shorted to ground. Luckily there were some spares that were not wired to anything in the panel and I was able to swap them out for the ones that had been destroyed. — These were a one time use. Which meant, if they ever had to short to ground, they had to be replaced.
Ok. Fine. After a little while, we were able to call back out of the plant, though there was still some residual noise on the line. It was like this… when you called out of the plant, the person on the other end sounded like they were buried in a grave somewhere and they were trying to talk to someone living just like in an episode of the Twilight Episode where a phone line landed on a grave and the dead person tried to call his long lost love from the past.
Gladys Cooper in the Twilight Zone Episode “Night Call”
I didn’t give it much thought other than that I figured the 189,000 volt arc to ground must have shorted out the telephone line since the phone line ran directly under the auxiliary substation ground grid.
It wasn’t until the next morning when the Southwestern Bell repairman showed up at the plant. I knew him well, since he had been working on our phone lines since before the AT&T breakup in 1984. When I met him in the front of the electric shop, he said that he needed to check our telephone circuits. I told him that I knew that we had a problem because we had a high voltage short to ground yesterday and I found our surge protectors melted away.
He explained to me that not only was our circuit affected, but that every relay house from here to Ponca City was blown out. That’s when I realized that the problem was the reverse of the usual situation. What had happened was that the Ground Grid in the substation and the surrounding area (including the Unit 1 Main Power Transformer) had become hot. What do you do when the ground grid becomes charged?
The Ground Grid is what is supposed to protect you when a surge happens, but what happens when the ground grid itself is the problem? In this case, when the high voltage line about 60 feet from the telephone cable surge protector, arced to ground, it fed a tremendous amount of power back through the ground grid. when equipment detected the surge in voltage, they automatically defaulted their circuits to ground. That’s why the telephone circuit died. That’s what tripped the Main Power Transformer.
When the telephone circuit detected the high voltage surge, it shorted to ground (which was the problem), causing the high voltage to feed directly into the phone line and down the line to the next Southwestern Bell relay switch, which also defaulted to ground, trying to bleed off the surge as it went from relay switch to switch until enough of the power was able to be diverted to ground.
That day sure turned out to be a learning experience. I learned that when all the lights go out in the control room, that it is almost assured that the coffee pot in the electric shop is going to stop working. I also learned that in order to coax the plant manager to the electric shop, a major electrical tragedy is one good way. I learned that when shooting rusty water into the air don’t point it at a high voltage auxiliary substation switch. — I’m sure Mickey Postman learned that lesson too. I also learned that just like in Star Trek… whenever there is a dangerous job to do, the Captain is always the one that wants to do it. Does that make sense? Send a Peon like me in there…
I also learned something else about Power Plant Men…. You see…. People like Dale Mitchell, George Alley and Mickey Postman all are examples of incredibly wonderful Power Plant Men. When they were out there doing their duty and something tragic like this, all the Power Plant Men felt their pain. They knew that they all felt guilty for tripping the unit. It didn’t matter that a million dollars every so many minutes was walking out the door in revenue. The only thing that mattered was that these three men were safe.
Mickey Postman
Since I have left the Power Plant, I have found that the idea that the employee is the greatest asset that a company can possess is not a universal idea. You see, there was never the thought that any of these people should be fired for their mistake. On the contrary. The true Power Plant Men did whatever they could to let them know that they knew exactly how they felt. It could have happened to any of them.
Besides the friendship between Power Plant Men, one of the things I miss most about working at the Power Plant is that the employees are held in high esteem as a real asset to the company. Many could learn from their example.
Comments from the Original post
-
-
That was an exciting day! Another great story. Thanks for the memories.
-
-
-
I like the mix of storytelling and information sharing you deliver here. Thanks again
-
-
-
Aug 8, 2011 I lost all reserve in and lost both units in 2 separate storms and A1 would not start for a few minutes and A2 was leaking antifreeze terribly not a scratch and back online in 34 hours both units and no one ever asked me one question even with a Safety Dept not one question asked of me the SS on duty. Not many men on the planet have ever experienced an uncontrolled total plant outage- you would of thought a learning opportunity would of took place. Feb 2011 worst winter temps in years 50 Units or more tripped in Texas let alone the trusty units of Redbud and McClain were fighting Sooner rolled right along with storm warnings for two weeks ahead – the ICS still went to Detroit for just tours of other facilities once again the fall guy Tarver McArthur stood alone. I had authored a Freeze Protection Plan for the plant and that seemed to save the day and explain to the regulatory bodies how we were online and everyone else in Texas was off and enjoying rolling blackouts in a terrible winter weather situation not to mention our powers that be were all stranded in Detroit and very few people could get to the plant without getting stuck trying to get there as well – but a few health heart issues later I am still here to tell about it all you would think folks would want to take advantage of someone that had went through the fire and Ice but thats ok I want them someday distant to get all the credit they deserve when the trumpets sound as that is truly what matters most. DT
-
-
Great story, and yes, you took exactly the proper lesson from it, and it is too bad that many of our bosses haven’t learned it.
And yup, I’ve buried a lot of electrocuted squirrels over the years.
Rivers and Rose in the Power Plant Palace
Originally posted January 25, 2013:
When is the appropriate time to call 911? Calling 911 in the Power Plant is when you call the Shift Supervisor to report something important. As Randy Dailey, our Safety Trainer extraordinaire, always taught us, first tap the person on the shoulder and say, “Are you all right?” Then you point your finger at someone and say, “Call 911!” That’s called “Activating the EMS” (Emergency Medical System). Besides medical emergencies, there are other reasons to call the Shift Supervisor.
I learned early on to ‘fess up when you have done something wrong.” People appreciate it when you tell them up front that you goofed. That way the problem can be dealt with directly. Dee Ball was that way. Any time he wrecked a truck, he didn’t hesitate to tell his boss. So, even as a summer help I had developed this philosophy. Never be afraid to expose your blunders. It works out better in the long run.
One example of someone not following this philosoply was Curtis Love. As I mentioned in the post Power Plant Spider Wars and Bugs in the Basement, Curtis didn’t want to tell anyone that he had been bitten by a brown recluse for the third time because he was afraid of losing his job.
The Oklahoma house spider — The Brown Recluse
His philosophy came back to bite him a year and a half later when he was on the labor crew when he was the designated truck driver. I had moved on to the electric shop by this time.
A red crew cab like this only the bed of the truck was longer
He was backing up the crew cab around a corner under the Fly Ash hoppers up at the coalyard when the side of the crew cab came into contact with one of those yellow poles designed to protect the structure from rogue vehicles. Unfortunately. This created a dent in the side of the truck.
An example of yellow poles protecting an area
Curtis, already on probation. worried that he would be fired if he told anyone about this mishap, failed to tell Larry Riley about this incident. Larry, on the other hand, was standing in front of the Coalyard Maintenance shop (the labor crew home), and saw the entire incident. At that moment, he turned to one of the labor crew hands and said, “I hope Curtis comes over here and tells me about that.” Unfortunately, Curtis decided to act as if nothing had happened. This resulted in his termination. As much as I cared about Curtis, I must admit that the Power Plant scene was probably not the best location for his vocation.
I had seen Dee Ball do the same thing over and over again, and he always reported his accidents immediately. He was never punished for an accident, though, for a number of years, he was banned from driving a truck. You can read more about this in the post: Experiencing Maggots, Mud and Motor Vehicles with Dee Ball.
One day during the summer of 1984 just after lunch, 1A PA fan tripped (PA stands for Primary Air). When this happened, number one unit had to lower it’s output from over 500 Megawatts down to around 200. The trip indicator on the 6900 volt breaker said that it had been grounded. Being grounded means that one of the three phases of the motor or cable had made a circuit with the ground (or something that was grounded). The trip circuits shut the fan down so fast that it prevents an explosion and saves the fan from being destroyed.
Diana Lucas (later Diana Brien), Andy Tubbs and I were given the task of finding the ground and seeing what we could do to fix it. We unwired the motor, which was no easy task, because the motor is about the size of a large van, and about 10 times heavier.
This is about 1/2 the size of the PA fan motor which is 1000 horsepower
So, we spent the rest of the day unwiring the motor (in the rain), and unwiring the cable to the motor from the breaker in the main switchgear and testing both the motor and the cable with various instruments looking for the grounded wire or coil that caused the motor to trip. We used a large “Megger” on the motor. It’s called a Megger because it measures Mega-Ohms. So, it’s technically called a Mega-Ohm meter. Ohms is a measurement of resistance in an electrical circuit. We usually use a small hand cranked megger, that is similar to an old hand crank telephone that generates a high voltage (good for shocking fish in a lake to make them rise to the surface). In the case of the hand cranked Megger, it would generate 1,000 volts.
A Hand cranked Megger made by the same company, only newer than the ones we had
The Megger this size would have been useless with this large motor. Instead we used one that was electric, and you ran the voltage up over 10,000 volts and watched the mega-ohms over a period of 1/2 hour or so.
For the cables, we hooked up a Hypot (or Hipot). This stands for High Potential. Potential in this case is another word for “Voltage”. It would charge up and then you pressed a button and it would send a high voltage pulse down the cable, and if there is a weak spot in the insulation,The Hypot will find it. So, we hooked a Hypot up to the cable and tried to find the grounded wire. No luck.
After spending 4 hours looking for the grounded cable or motor, we found nothing. We spent another hour and a half putting the motor and the breaker back in service. The Fan was put back into operation and we went home. As I was walking out to the car with Bill Rivers, he told me, “I knew they weren’t going to find anything wrong with that fan.” He had a big grin on his face.
At first I thought he was just making an educated guess as Rivers was apt to do on many occasions (daily). It was raining and I could see where water may have been sucked into the motor or something and had momentarily grounded the motor. Just because we didn’t find anything didn’t mean that the breaker didn’t trip for no reason.
When we were in the car and on our way to Stillwater, Oklahoma with Yvonne Taylor and Rich Litzer, Bill explained that he knew why the motor tripped. He had been walking through the main switchgear with Mike Rose, and Mike, for no apparent reason other than curiosity, had opened up the bottom door to the breaker for 1A PA fan. He looked at it for a moment and then slammed the door shut. When he did this, the breaker tripped.
So, the ground relay happened to be the one that tripped. It might as well been an over-current or a low voltage trip. It just happened to trip the ground trip. Bill said that he told Mike that he should call the Shift Supervisor and let him know so they could restart the motor. Mike on the other hand told Bill that he was already on probation and was afraid of losing his job if he reported that he had slammed the door on the breaker and tripped the fan.
If there was ever a reason to call 911, it was then. All he had to do was tell them, “I accidentally tripped the PA fan when I bumped the breaker cabinet.” They would have told him to reset the flag, and they would have started the fan right back up. No questions asked… I’m sure of it. And they wouldn’t have lost their generating capacity for the remainder of the afternoon and we wouldn’t have spent 4 hours unwiring, testing and rewiring the motor in the rain with a plastic umbrella over our head.
Bill wasn’t about to tell on Mike. If Mike didn’t want to report it, Bill wasn’t going to say anything, and I understand that. I probably would have kept it to myself at the time if I was in Bill’s shoes (I’m just glad I wasn’t because I probably wouldn’t have been able to sleep soundly for the next year). But 30 years later, I might write about it in a Blog. Even though I wouldn’t have looked to Mike to teach me much about being an electrician (he was more of an Air Condition man anyway), I still loved the guy.
Mike died almost two years ago on May 29, 2011. He was from England and had lived in Canada for a time. He used to work on trains. Trains, even though they are diesel, are really electric. The Diesel engine really runs a generator that generates electricity that runs the train. I know that Mike was a good man at heart. He loved his family with all his heart. Here is a picture of the Limey:
Mike Rose. A fair plant electrician, but a great family man!
Ok. So I know what you are thinking…. There must be a story about myself in here somewhere. Well, you would be right. First of all. I always ‘fessed up to my mistakes, as my current manager at Dell knows well (yes. I still mess up after all these years). I told my current manager the other day that CLM was my middle name. (CLM means “Career Limiting Move”). So here is my power plant “mess up” story (well one of them):
In January 1986, I returned from my Honeymoon with my new wife Kelly when I found that we had hired a new electrician. Gary Wehunt was replacing Jim Stephenson who had left the plant on February 15, 1985, which is a story all it’s own. We had just started an overhaul on Unit 1.
I remember the first Monday I spent with Gary. It was January 6, 1986 and we were working on cleaning out the exciter house on the end of the main power generator with Diana Brien (formerly Diana Lucas). We were discussing salaries and Gary was surprised to find out that I was making more than he was. Well… I had been an electrician for over 2 years and had been promoted regularly…. so I didn’t think there was anything strange about it, except that I still looked like I was only about 18 years old (even though I was 25) and Gary was about 34. I had already been promoted 4 times and my salary had gone from $7.15 to over $12 an hour.
Anyway, when that first Friday rolled around, Gary and I were assigned to Substation Inspection. Some later time I may go into the details of what “Substation Inspection” entails, but for now, let’s just stick with my “911 call.” It is enough to say that we were in the main plant substation relay house on Friday January 10, 1986 at 9:00 am. One of our jobs was to call other substations and perform a test called a “Transfer Trip and Carrier Test”. We had called Woodring Substation (Woodring is a town in Oklahoma and we had a 345 KV line going there), and I was talking to the man in the substation on the other end of the phone line.
At the same time I was showing Gary just how experienced I was at being an electrician. People had told me that you had to be a plant electrician for 5 years before you really became a “first class” electrician. Well. Here I was at 2 years, and I thought I was so good that I could do anything by now…. — Yeah… right. I told the guy on the other end of the line as I turned a switch…. Amber light… Back to Blue…. and I wrote down the value on the meter (paperwork… oh yes…. it’s that important. Like A-1 sauce).
Then I reached for the second switch. I said, “Carrier test”, then turned the switch. The lights in the relay house went out and we were in the dark. I told the guy on the other end of the line….. “Well. That’s not supposed to happen.” Then as I let go of the switch and it returned to it’s normal position, the lights turned back on. Okay……
I wrote the numbers down from the meter and said goodbye to the other faceless substation man on the other end of the line that I talked to over 100 times, but never met in person. He sounded like a nice guy. Then I headed for the gray phone. I heard the Shift Supervisor paging Leroy Godfrey (The Electrical Supervisor) on line 2 (we had 5 Gray phone lines. The Gray Phone was our PA system).
When I picked up the line I heard Leroy pick up the phone and the Shift Supervisor tell Leroy that we just lost station power in the main substation and it had switched over to Auxiliary power. I immediately jumped in and said, “Jim (for Jim Padgett, the Shift Supervisor), I did that. I was performing a Carrier test with Woodring and the moment I performed the carrier test the lights went out.” Leroy chimed in by saying, “That wouldn’t cause you to lose station power.”
Well, in my ‘inexperienced’ plant electrician way, I responded, “Well. All I know is that when I turned the switch to perform the carrier test, the lights went out, and when I let go of the switch, the lights came back on.” Leroy reiterated, “That wouldn’t cause you to lose station power.” I replied with, “I’m just saying….” and left it at that. I had done my job. They knew I was out here. They knew I had called 911 right away. I explained what I was doing…. they could take it from there.
I had hoped that I had showed Gary upfront that it doesn’t hurt to report your mistakes (even though I hadn’t made one as far as I could tell), but I was 100% sure I had done something to cause the relay house to lose power. Though, I couldn’t figure out why.
After lunch, Bill Bennett, our A foreman came down to the shop to tell me that they figured out how the substation lost station power. He said that a road grader had been grating the road down by the Otoe-Missouri reservation (which is actually called “Windmill road” I guess because there is a windmill down that road somewhere), and had hit an electric pole and knocked it over and had killed the power to the substation.
Substation Power Interrupting Road Grader
It turned out that the substation relay house was fed by a substation down that road where we have a radio tower. So, think about this. The exact time that I turned that switch in the substation, a road grater 2 1/2 miles away hits a telephone pole accidentally and knocks it to the ground and kills the power to the substation at the exact same time that I am performing a transfer-trip and Carrier test with Woodring Substation, and the time it takes to switch to auxiliary power is the exact time it took me to let go of the switch.
Don’t tell me that was by accident. I will never believe it. I think it was for the soul purpose of teaching me a useful lesson or two. First….. don’t be afraid to tell someone when you do something wrong. Second…. If you think you have control over the things that happen to you in your life… well, think again…… Third….. God watches you every moment, and if you let him, he will guide you to do the right thing when the time comes.
God bless you all.
COMMENTS FROM THE ORIGINAL POST:
-
Monty Hansen January 26, 2013
I had a similar thing happen to me, I was upgrading to shift foreman & system called to remove a tag in the switchyard & put the switch back to auto. The tag on the pistol grip was attached with a plastic zip tie & the previous operator had put it on real tight, as I was wrestling it off with my leatherman, the pliers slipped & I banged my elbow into the control panel, at that very instant there was a loud BANG as several 345 KV breakers opened simultaniously in the swithyard, I had the phone pinched between my shoulder & ear as I was wrestling with this switch & talking to the system control operator, he said a few bad words – gotta go – & hung up. The power plant lost all power & went in the black, I, of course was just sick in the pit of my stomach, after we got power restored, the plant back on etc. I called system back to see if they found the cause & fess up to causing the trip (I figured I must have caused a trip relay to close when I hit the panel) – anyway a crane at a plant down the road had got it’s boom tangled in the power line & went to ground – AT PRECISELY THE INSTANT MY ELBOW SLIPPED & HIT THE PANEL!!
-
Ron Kilman January 26, 2013
Some great illustrations of the truth in Proverbs 28:13 “He who conceals his transgressions will not prosper, But he who confesses and forsakes them will find compassion”.
-
justturnright January 28, 2013
CLM: I can relate.
My first boss 30 years ago once told me he was going to officially nickname me “I’m sorry” (and make me wear it for a name badge) if I said it one more time.
Hey, there’s worse things.
-
Roomy January 29, 2013
I had not thought about Mike Rose in years. He was a good guy to work with, now Rivers was a different story!!!
Sub checks, I used to love to do sub checks. I performed pilot wire & transfer trip checks for years. I hated it when they went to being done by automation.
Thanks for bringing back old memories.
Runaway Fire Hydrant Leaves Power Plant in the Dark
Originally posted May 17, 2014:
Don’t believe it when the Electric Company tells you that the reason your town lost electricity for an hour was because a squirrel climbed onto a transformer and shorted it out. The real reason just may be more bizarre than that and the company doesn’t want you to know all the different creative ways that power can be shut off. This is a tale of just one of those ways. So, get out your pencil and paper and take notes.
Power Plant Notepad
One spring day in 1993 while sitting at the Precipitator computer for Unit one at the Coal-fired Power Plant in North Central Oklahoma, while I was checking the controls to make sure all the cabinets were operating correctly, suddenly there was a distant boom, and the lights in the control room went out. The computer stayed on because it was connected to an electric panel called the VSP or Vital Services Panel, which in turn was supplied by the UPS system (Uninterruptible Power Supply). That was one of those moments where you may pause for a moment to make sure you aren’t still at home dreaming before you fly into a panic.
The Precipitator cabinets all indicated on the computer that they had just shutdown. I rose from the chair and walked around to the front of the Alarm Panel for Unit one, and found that the fluorescent lights were only out on Unit 1. The lights were still on for Unit 2. The Control Panel was lit up like a Christmas Tree with Green, Red, Blue and Yellow Lights. The Alarm Printer was spewing out paper at high speed. As the large sheets of paper were pouring out onto the floor, I watched as Pat Quiring and other brave Power Plant Control Room operators were scurrying back and forth turning switch handles, pushing buttons, and checking pressure gauges.
Just this site alone gave me confidence that everything was going to be all right. These Control Room operators were all well trained for emergencies just like this, and each person knew what their job was. No one was panicking. Everyone was concentrating on the task at hand.
Someone told me that we lost Unit 1, and the Auxiliary Power to Unit 1 at the same time. So, Unit 1 was dead in the water. This meant, no fans, no pumps, no lights, no vending machines, no cold water at the water fountain and most importantly, no hot coffee!!! I could hear steam valves on the T-G floor banging open and the loud sound of steam escaping.
I turned quickly to go to the electric shop to see what I could do there in case I was needed. I bolted out the door and down the six flights of stairs to the Turbine-Generator (T-G) basement. Exiting the stairway, and entering the T-G basement the sound was deafening. I grabbed the earplugs that were dangling around my neck and crammed them into my ears. Steam was pouring out of various pop-off valves. I ducked into the electric shop where across the room Andy Tubbs, one of the electric foreman was pulling large sheets of electric blueprints from the print cabinet and laying them across the work table that doubled as the lunch table.
Andy Tubbs – True Power Plant Electrician
When I asked Andy what happened, I learned that somehow when a crew was flushing out a fire hydrant the water somehow shot up and into the bus work in the Auxiliary Substation (that supplies backup power to the Power Plant) and it shorted out the 189,000 volt substation directly to ground. When that happened it tripped unit 1 and the auxiliary substation at the same time leaving it without power.
An example of part of an Auxiliary Substation
I will explain how a fire hydrant could possibly spray the bus work in a substation in a little while, but first let me tell you what this meant at the moment to not have any power for a Power Plant Boiler and Turbine Generator that has just tripped when it was at full load which was around 515 Megawatts of power at the time.
Normally when a unit trips, the boiler cools down as the large Force Draft (FD) Fans blow air through the boiler while the even larger Induced Draft (ID) fans suck the air from the boiler on the other end and blow the hot air up the smoke stack. This causes the steam in the boiler tubes to condense back into water. Steam valves open on the boiler that allow excessive steam to escape.
When the boiler is running there is a large orange fireball hovering in space in the middle of the boiler. The boiler water is being circulated through the boiler and the Boiler Feed Pump Turbines are pumping steam back and forth between the turbine generator and the boiler reheating the steam until every bit of heat from the boiler that can be safely harnessed is used.
When all this stop suddenly, then it is important that the large fans keep running to cool down the steam, since it is no longer losing energy in the generator as it was when it was busy supplying electricity to 1/2 million people in Oklahoma City. The power is fed to the fans from the Auxiliary substation located right outside the Main Switchgear where all the breakers reside that supply the power to the fans. Unfortunately, in this case, the Auxiliary substation was shutdown as well, leaving the boiler without any fans.
Without fans for cooling, and pumps to circulate the water, the walls of the boiler began heating up to dangerous temperatures. Steam was whistling out of pop off valves, but if the steam drum on the top of the boiler were to run dry, then the entire boiler structure could be compromised and begin melting down. — So, this was serious. Something had to be done right away. It wouldn’t be as bad as the China Syndrome since we were burning coal instead of nuclear power, but it would have caused a lot of damage nonetheless.
From the movie “The China Syndrome” where a similar emergency existed only in the movie, it was a Nuclear Plant
I have a side story about this picture, but I think I’ll save it for another post because I don’t want to digress from the main story at this point (Ok. Let me just say “Jack Maloy and Merl Wright” for those who can’t wait) See the post: “Power Plant Conspiracy Theory“.
With the prospect that the boiler might melt to the ground in a pile of rubble, it would seem that the main priority was to turn the Auxiliary Substation back on so the fans could be turned back on and prevent the boiler from collapsing. So, we walked out to the substation and looked at the switches that would have to be operated in order to first power up the main bus and then to close to supply power to the two big transformers and the six smaller transformers that supplied the Unit 1 Main Switchgear.
While inspecting the switches where the electricity had gone to ground we found that one of the main insulators was cracked.
A High Voltage Insulator like this
Since this insulator was cracked, we didn’t really want to operate the switch to test if another 189,000 volts would go straight to ground again, especially since one of us would be standing right underneath it cranking the switch. So, we went back to the shop to find an alternative.
By this time the Plant Manager, Ron Kilman arrived in the shop, and understanding the urgency to find a solution asked us what were the alternatives. He was relying on our expertise to make the decision.
Ron Kilman is the one on the left in the plaid shirt
The other solution would be to cut the power over from Unit 2 which was still humming away pushing electricity to Oklahoma City out of the 345,000 volt substation. The cut over would be very simple because the switchgear was designed with this in mind. We analyzed the power rating on the auxiliary transformers on Unit 2 and thought that we might be cutting it close to have them running both sets of fans at the same time, especially since the full load amps of a huge fan starting up was about 10 times the normal rate.
The transformer was rated to handle the load, but consider this. What if this caused Unit 2 to trip as well. With the Auxiliary substation offline, if Unit 2 tripped, we would be in twice the amount of trouble we were currently in. What a day it would have been if that had happened and two 250 foot boilers had come crashing to the ground in a pile of rubble. After reading the power ratings on the auxiliary transformers I was thinking, “Yeah, let’s do it! These transformers can handle it.” Andy was not so eager.
So, we were left with one alternative. That was to shut the switch in the Auxiliary substation that had the cracked insulator and take our chances that it wasn’t going to short to ground and blow up over our heads. I think I was eager to close the switch for Andy, but if I remember correctly, he didn’t want me to be the one to suffer the consequences and decided to close the switch himself. Needless to say. Andy closed the switch, and nothing blew up.
High Voltage Switch similar to the one we were closing
As soon as the power was restored to the switchgear, the fans were powered up and the temperature in the boiler was quickly reduced. The coffee pot in the Electric Shop began heating the coffee again. The power plant was saved from a major catastrophe. That was delayed for another day… of which I will talk about later (see the post “Destruction of a Power Plant God).”
So, how exactly does a fire hydrant shoot water up into the bus work of a substation like the picture of the switch directly above? The culprit fire hydrant wasn’t in the substation, it sat alongside it outside the fence a good 50 feet from the high voltage switch. No hose was attached to the fire hydrant. It was only being flushed out as part of a yearly activity to go around and make sure the fire hydrants are all operating correctly.
Here is the story about how the squirrel climbed into the transformer this time….
George Alley, Dale Mitchell and Mickey Postman were going around to the 30,000 fire hydrants on the plant ground (ok. maybe not that many, but we did have a lot of them), and they were opening up the valves and flushing them out. That means, they were letting them run for a while to clear them out from any contaminates that may have built up over the year of not being used.
Throughout their adventure they had opened a multitude of Hydrants situated out in the fields along the long belt conveyor from the coalyard and around the two one-million gallon #2 Diesel tanks.
Large Oil Tank about the size of the two that are at the Power Plant
The brave Power Plant Men, learned that when opening a fire hydrant wide open in the middle of field had unintended consequences. It tended to wash out the ground in front of the flow of the water shooting out of the hydrant. So the team of experts devised a plan to place a board in front of the hydrant when it would be in danger of tearing a hole in the terrain. The board would divert the water into the air where it would fan out and not cause damage to the surrounding area.
This was working fine, and when they arrived at the fire hydrant next to the substation, since the stream from the hydrant was pointing directly into the substation (hmm. a design flaw, I think), they decided to prop the board up against the fence to keep from washing away the gravel in the substation. Well. When a fire hydrant is opened that hasn’t been used for a year, the first flow of water to shoot out is dark brown.
You may think that this is because the water has somehow become dirty over the past year, but that isn’t quite the case. What has happened is that the pipe has been rusting little by little and the water has become saturated with the rust. So, the water shooting out of the hydrant was full of rust (hence the need to flush them out).
Well. Rust is made of metal. Metal is conductive, especially when it is mixed with water. When the water hit the board, it was deflected into the air and happened to direct itself directly into the high voltage switch in the substation. This caused a circuit to the ground which, once it created an arc pumped all the electricity directly into the ground.
Normally when something like this happens it doesn’t trip the Main Power Transformer to a Power Plant.
A Main Power Transformer
This time it did. I know there was a few heads scratching trying to figure it out. I think I figured out what happened a little while later. You see… here is the rest of the story….
Once the unit was back online and the emergency was over, someone finally noticed that the telephone system couldn’t call outside of the plant. Well. I was the main telephone person at the time, so the control room called me and asked me to look into the problem.
I checked the telephone computer and it was up and running just fine. Internal calls could be made. Only any call outside just concluded with a funny humming sound. After checking the circuit in the Logic Room next to the Rolm Telephone Computer I headed for…. guess where….. the Main Switchgear….
A ROLM Phone Computer
In the middle of the main switchgear in the back of the room right next to the Auxiliary Substation beyond the back wall, the outside telephone line came into the plant. The first thing it did was go through a special Telephone Surge Protector.
Telephone Grounding Panel
In this picture above, the silver circular buttons on the left side are really an old style surge protector. whenever there was a power surge, the carbon connection in the surge protector would quickly melt causing the circuit to go straight to ground. Thus protecting the rest of the telephone circuit. So, if some kid in their house decides to connect the 120 volts circuit to the telephone for fun to see what would happen, this circuit would protect the rest of the phone circuits. Keep in mind that this was during the early 1990 when “Surge Protection” still was basically all “mechanical”.
Anyway, when I arrived at this panel and I checked the surge protector to the main line going out of the plant, guess what I found…. Yep. Shorted to ground. Luckily there were some spares that were not wired to anything in the panel and I was able to swap them out for the ones that had been destroyed. — These were a one time use. Which meant, if they ever had to short to ground, they had to be replaced.
Ok. Fine. After a little while, we were able to call back out of the plant, though there was still some residual noise on the line. It was like this… when you called out of the plant, the person on the other end sounded like they were buried in a grave somewhere and they were trying to talk to someone living just like in an episode of the Twilight Episode where a phone line landed on a grave and the dead person tried to call his long lost love from the past.
Gladys Cooper in the Twilight Zone Episode “Night Call”
I didn’t give it much thought other than that I figured the 189,000 volt arc to ground must have shorted out the telephone line since the phone line ran directly under the auxiliary substation ground grid.
It wasn’t until the next morning when the Southwestern Bell repairman showed up at the plant. I knew him well, since he had been working on our phone lines since before the AT&T breakup in 1984. When I met him in the front of the electric shop, he said that he needed to check our telephone circuits. I told him that I knew that we had a problem because we had a high voltage short to ground yesterday and I found our surge protectors melted away.
He explained to me that not only was our circuit affected, but that every relay house from here to Ponca City was blown out. That’s when I realized that the problem was the reverse of the usual situation. What had happened was that the Ground Grid in the substation and the surrounding area (including the Unit 1 Main Power Transformer) had become hot. What do you do when the ground grid becomes charged?
The Ground Grid is what is supposed to protect you when a surge happens, but what happens when the ground grid itself is the problem? In this case, when the high voltage line about 60 feet from the telephone cable surge protector, arced to ground, it fed a tremendous amount of power back through the ground grid. when equipment detected the surge in voltage, they automatically defaulted their circuits to ground. That’s why the telephone circuit died. That’s what tripped the Main Power Transformer.
When the telephone circuit detected the high voltage surge, it shorted to ground (which was the problem), causing the high voltage to feed directly into the phone line and down the line to the next Southwestern Bell relay switch, which also defaulted to ground, trying to bleed off the surge as it went from relay switch to switch until enough of the power was able to be diverted to ground.
That day sure turned out to be a learning experience. I learned that when all the lights go out in the control room, that it is almost assured that the coffee pot in the electric shop is going to stop working. I also learned that in order to coax the plant manager to the electric shop, a major electrical tragedy is one good way. I learned that when shooting rusty water into the air don’t point it at a high voltage auxiliary substation switch. — I’m sure Mickey Postman learned that lesson too. I also learned that just like in Star Trek… whenever there is a dangerous job to do, the Captain is always the one that wants to do it. Does that make sense? Send a Peon like me in there…
I also learned something else about Power Plant Men…. You see…. People like Dale Mitchell, George Alley and Mickey Postman all are examples of incredibly wonderful Power Plant Men. When they were out there doing their duty and something tragic like this, all the Power Plant Men felt their pain. They knew that they all felt guilty for tripping the unit. It didn’t matter that a million dollars every so many minutes was walking out the door in revenue. The only thing that mattered was that these three men were safe.
Mickey Postman
Since I have left the Power Plant, I have found that the idea that the employee is the greatest asset that a company can possess is not a universal idea. You see, there was never the thought that any of these people should be fired for their mistake. On the contrary. The true Power Plant Men did whatever they could to let them know that they knew exactly how they felt. It could have happened to any of them.
Besides the friendship between Power Plant Men, one of the things I miss most about working at the Power Plant is that the employees are held in high esteem as a real asset to the company. Many could learn from their example.
Comments from the Original post
-
-
That was an exciting day! Another great story. Thanks for the memories.
-
-
-
I like the mix of storytelling and information sharing you deliver here. Thanks again
-
-
-
Aug 8, 2011 I lost all reserve in and lost both units in 2 separate storms and A1 would not start for a few minutes and A2 was leaking antifreeze terribly not a scratch and back online in 34 hours both units and no one ever asked me one question even with a Safety Dept not one question asked of me the SS on duty. Not many men on the planet have ever experienced an uncontrolled total plant outage- you would of thought a learning opportunity would of took place. Feb 2011 worst winter temps in years 50 Units or more tripped in Texas let alone the trusty units of Redbud and McClain were fighting Sooner rolled right along with storm warnings for two weeks ahead – the ICS still went to Detroit for just tours of other facilities once again the fall guy Tarver McArthur stood alone. I had authored a Freeze Protection Plan for the plant and that seemed to save the day and explain to the regulatory bodies how we were online and everyone else in Texas was off and enjoying rolling blackouts in a terrible winter weather situation not to mention our powers that be were all stranded in Detroit and very few people could get to the plant without getting stuck trying to get there as well – but a few health heart issues later I am still here to tell about it all you would think folks would want to take advantage of someone that had went through the fire and Ice but thats ok I want them someday distant to get all the credit they deserve when the trumpets sound as that is truly what matters most. DT
-
-
Great story, and yes, you took exactly the proper lesson from it, and it is too bad that many of our bosses haven’t learned it.
And yup, I’ve buried a lot of electrocuted squirrels over the years.
Runaway Fire Hydrant Leaves Power Plant in the Dark
Originally posted May 17, 2014:
Don’t believe it when the Electric Company tells you that the reason your town lost electricity for an hour was because a squirrel climbed onto a transformer and shorted it out. The real reason just may be more bizarre than that and the company doesn’t want you to know all the different creative ways that power can be shut off. This is a tale of just one of those ways. So, get out your pencil and paper and take notes.
Power Plant Notepad
One spring day in 1993 while sitting at the Precipitator computer for Unit one at the Coal-fired Power Plant in North Central Oklahoma, while I was checking the controls to make sure all the cabinets were operating correctly, suddenly there was a distant boom, and the lights in the control room went out. The computer stayed on because it was connected to an electric panel called the VSP or Vital Services Panel, which in turn was supplied by the UPS system (Uninterruptible Power Supply). That was one of those moments where you may pause for a moment to make sure you aren’t still at home dreaming before you fly into a panic.
The Precipitator cabinets all indicated on the computer that they had just shutdown. I rose from the chair and walked around to the front of the Alarm Panel for Unit one, and found that the fluorescent lights were only out on Unit 1. The lights were still on for Unit 2. The Control Panel was lit up like a Christmas Tree with Green, Red, Blue and Yellow Lights. The Alarm Printer was spewing out paper at high speed. As the large sheets of paper were pouring out onto the floor, I watched as Pat Quiring and other brave Power Plant Control Room operators were scurrying back and forth turning switch handles, pushing buttons, and checking pressure gauges.
Just this site alone gave me confidence that everything was going to be all right. These Control Room operators were all well trained for emergencies just like this, and each person knew what their job was. No one was panicking. Everyone was concentrating on the task at hand.
Someone told me that we lost Unit 1, and the Auxiliary Power to Unit 1 at the same time. So, Unit 1 was dead in the water. This meant, no fans, no pumps, no lights, no vending machines, no cold water at the water fountain and most importantly, no hot coffee!!! I could hear steam valves on the T-G floor banging open and the loud sound of steam escaping.
I turned quickly to go to the electric shop to see what I could do there in case I was needed. I bolted out the door and down the six flights of stairs to the Turbine-Generator (T-G) basement. Exiting the stairway, and entering the T-G basement the sound was deafening. I grabbed the earplugs that were dangling around my neck and crammed them into my ears. Steam was pouring out of various pop-off valves. I ducked into the electric shop where across the room Andy Tubbs, one of the electric foreman was pulling large sheets of electric blueprints from the print cabinet and laying them across the work table that doubled as the lunch table.
Andy Tubbs – True Power Plant Electrician
When I asked Andy what happened, I learned that somehow when a crew was flushing out a fire hydrant the water somehow shot up and into the bus work in the Auxiliary Substation (that supplies backup power to the Power Plant) and it shorted out the 189,000 volt substation directly to ground. When that happened it tripped unit 1 and the auxiliary substation at the same time leaving it without power.
An example of part of an Auxiliary Substation
I will explain how a fire hydrant could possibly spray the bus work in a substation in a little while, but first let me tell you what this meant at the moment to not have any power for a Power Plant Boiler and Turbine Generator that has just tripped when it was at full load which was around 515 Megawatts of power at the time.
Normally when a unit trips, the boiler cools down as the large Force Draft (FD) Fans blow air through the boiler while the even larger Induced Draft (ID) fans suck the air from the boiler on the other end and blow the hot air up the smoke stack. This causes the steam in the boiler tubes to condense back into water. Steam valves open on the boiler that allow excessive steam to escape.
When the boiler is running there is a large orange fireball hovering in space in the middle of the boiler. The boiler water is being circulated through the boiler and the Boiler Feed Pump Turbines are pumping steam back and forth between the turbine generator and the boiler reheating the steam until every bit of heat from the boiler that can be safely harnessed is used.
When all this stop suddenly, then it is important that the large fans keep running to cool down the steam, since it is no longer losing energy in the generator as it was when it was busy supplying electricity to 1/2 million people in Oklahoma City. The power is fed to the fans from the Auxiliary substation located right outside the Main Switchgear where all the breakers reside that supply the power to the fans. Unfortunately, in this case, the Auxiliary substation was shutdown as well, leaving the boiler without any fans.
Without fans for cooling, and pumps to circulate the water, the walls of the boiler began heating up to dangerous temperatures. Steam was whistling out of pop off valves, but if the steam drum on the top of the boiler were to run dry, then the entire boiler structure could be compromised and begin melting down. — So, this was serious. Something had to be done right away. It wouldn’t be as bad as the China Syndrome since we were burning coal instead of nuclear power, but it would have caused a lot of damage nonetheless.
From the movie “The China Syndrome” where a similar emergency existed only in the movie, it was a Nuclear Plant
I have a side story about this picture, but I think I’ll save it for another post because I don’t want to digress from the main story at this point (Ok. Let me just say “Jack Maloy and Merl Wright” for those who can’t wait) See the post: “Power Plant Conspiracy Theory“.
With the prospect that the boiler might melt to the ground in a pile of rubble, it would seem that the main priority was to turn the Auxiliary Substation back on so the fans could be turned back on and prevent the boiler from collapsing. So, we walked out to the substation and looked at the switches that would have to be operated in order to first power up the main bus and then to close to supply power to the two big transformers and the six smaller transformers that supplied the Unit 1 Main Switchgear.
While inspecting the switches where the electricity had gone to ground we found that one of the main insulators was cracked.
A High Voltage Insulator like this
Since this insulator was cracked, we didn’t really want to operate the switch to test if another 189,000 volts would go straight to ground again, especially since one of us would be standing right underneath it cranking the switch. So, we went back to the shop to find an alternative.
By this time the Plant Manager, Ron Kilman arrived in the shop, and understanding the urgency to find a solution asked us what were the alternatives. He was relying on our expertise to make the decision.
Ron Kilman is the one on the left in the plaid shirt
The other solution would be to cut the power over from Unit 2 which was still humming away pushing electricity to Oklahoma City out of the 345,000 volt substation. The cut over would be very simple because the switchgear was designed with this in mind. We analyzed the power rating on the auxiliary transformers on Unit 2 and thought that we might be cutting it close to have them running both sets of fans at the same time, especially since the full load amps of a huge fan starting up was about 10 times the normal rate.
The transformer was rated to handle the load, but consider this. What if this caused Unit 2 to trip as well. With the Auxiliary substation offline, if Unit 2 tripped, we would be in twice the amount of trouble we were currently in. What a day it would have been if that had happened and two 250 foot boilers had come crashing to the ground in a pile of rubble. After reading the power ratings on the auxiliary transformers I was thinking, “Yeah, let’s do it! These transformers can handle it.” Andy was not so eager.
So, we were left with one alternative. That was to shut the switch in the Auxiliary substation that had the cracked insulator and take our chances that it wasn’t going to short to ground and blow up over our heads. I think I was eager to close the switch for Andy, but if I remember correctly, he didn’t want me to be the one to suffer the consequences and decided to close the switch himself. Needless to say. Andy closed the switch, and nothing blew up.
High Voltage Switch similar to the one we were closing
As soon as the power was restored to the switchgear, the fans were powered up and the temperature in the boiler was quickly reduced. The coffee pot in the Electric Shop began heating the coffee again. The power plant was saved from a major catastrophe. That was delayed for another day… of which I will talk about later (see the post “Destruction of a Power Plant God).”
So, how exactly does a fire hydrant shoot water up into the bus work of a substation like the picture of the switch directly above? The culprit fire hydrant wasn’t in the substation, it sat alongside it outside the fence a good 50 feet from the high voltage switch. No hose was attached to the fire hydrant. It was only being flushed out as part of a yearly activity to go around and make sure the fire hydrants are all operating correctly.
Here is the story about how the squirrel climbed into the transformer this time….
George Alley, Dale Mitchell and Mickey Postman were going around to the 30,000 fire hydrants on the plant ground (ok. maybe not that many, but we did have a lot of them), and they were opening up the valves and flushing them out. That means, they were letting them run for a while to clear them out from any contaminates that may have built up over the year of not being used.
Throughout their adventure they had opened a multitude of Hydrants situated out in the fields along the long belt conveyor from the coalyard and around the two one-million gallon #2 Diesel tanks.
Large Oil Tank about the size of the two that are at the Power Plant
The brave Power Plant Men, learned that when opening a fire hydrant wide open in the middle of field had unintended consequences. It tended to wash out the ground in front of the flow of the water shooting out of the hydrant. So the team of experts devised a plan to place a board in front of the hydrant when it would be in danger of tearing a hole in the terrain. The board would divert the water into the air where it would fan out and not cause damage to the surrounding area.
This was working fine, and when they arrived at the fire hydrant next to the substation, since the stream from the hydrant was pointing directly into the substation (hmm. a design flaw, I think), they decided to prop the board up against the fence to keep from washing away the gravel in the substation. Well. When a fire hydrant is opened that hasn’t been used for a year, the first flow of water to shoot out is dark brown.
You may think that this is because the water has somehow become dirty over the past year, but that isn’t quite the case. What has happened is that the pipe has been rusting little by little and the water has become saturated with the rust. So, the water shooting out of the hydrant was full of rust (hence the need to flush them out).
Well. Rust is made of metal. Metal is conductive, especially when it is mixed with water. When the water hit the board, it was deflected into the air and happened to direct itself directly into the high voltage switch in the substation. This caused a circuit to the ground which, once it created an arc pumped all the electricity directly into the ground.
Normally when something like this happens it doesn’t trip the Main Power Transformer to a Power Plant.
A Main Power Transformer
This time it did. I know there was a few heads scratching trying to figure it out. I think I figured out what happened a little while later. You see… here is the rest of the story….
Once the unit was back online and the emergency was over, someone finally noticed that the telephone system couldn’t call outside of the plant. Well. I was the main telephone person at the time, so the control room called me and asked me to look into the problem.
I checked the telephone computer and it was up and running just fine. Internal calls could be made. Only any call outside just concluded with a funny humming sound. After checking the circuit in the Logic Room next to the Rolm Telephone Computer I headed for…. guess where….. the Main Switchgear….
A ROLM Phone Computer
In the middle of the main switchgear in the back of the room right next to the Auxiliary Substation beyond the back wall, the outside telephone line came into the plant. The first thing it did was go through a special Telephone Surge Protector.
Telephone Grounding Panel
In this picture above, the silver circular buttons on the left side are really an old style surge protector. whenever there was a power surge, the carbon connection in the surge protector would quickly melt causing the circuit to go straight to ground. Thus protecting the rest of the telephone circuit. So, if some kid in their house decides to connect the 120 volts circuit to the telephone for fun to see what would happen, this circuit would protect the rest of the phone circuits. Keep in mind that this was during the early 1990 when “Surge Protection” still was basically was all “mechanical”.
Anyway, when I arrived at this panel and I checked the surge protector to the main line going out of the plant, guess what I found…. Yep. Shorted to ground. Luckily there were some spares that were not wired to anything in the panel and I was able to swap them out for the ones that had been destroyed. — These were a one time use. Which meant, if they ever had to short to ground, they had to be replaced.
Ok. Fine. After a little while, we were able to call back out of the plant, though there was still some residual noise on the line. It was like this… when you called out of the plant, the person on the other end sounded like they were buried in a grave somewhere and they were trying to talk to someone living just like in an episode of the Twilight Episode where a phone line landed on a grave and the dead person tried to call his long lost love from the past.
Gladys Cooper in the Twilight Zone Episode “Night Call”
I didn’t give it much thought other than that I figured the 189,000 volt arc to ground must have shorted out the telephone line since the phone line ran directly under the auxiliary substation ground grid.
It wasn’t until the next morning when the Southwestern Bell repairman showed up at the plant. I knew him well, since he had been working on our phone lines since before the AT&T breakup in 1984. When I met him in the front of the electric shop, he said that he needed to check our telephone circuits. I told him that I knew that we had a problem because we had a high voltage short to ground yesterday and I found our surge protectors melted away.
He explained to me that not only was our circuit affected, but that every relay house from here to Ponca City was blown out. That’s when I realized that the problem was the reverse of the usual situation. What had happened was that the Ground Grid in the substation and the surrounding area (including the Unit 1 Main Power Transformer) had become hot. What do you do when the ground grid becomes charged?
The Ground Grid is what is supposed to protect you when a surge happens, but what happens when the ground grid itself is the problem? In this case, when the high voltage line about 60 feet from the telephone cable surge protector, arced to ground, it fed a tremendous amount of power back through the ground grid. when equipment detected the surge in voltage, they automatically defaulted their circuits to ground. That’s why the telephone circuit died. That’s what tripped the Main Power Transformer.
When the telephone circuit detected the high voltage surge, it shorted to ground (which was the problem), causing the high voltage to feed directly into the phone line and down the line to the next Southwestern Bell relay switch, which also defaulted to ground, trying to bleed off the surge as it went from relay switch to switch until enough of the power was able to be diverted to ground.
That day sure turned out to be a learning experience. I learned that when all the lights go out in the control room, that it is almost assured that the coffee pot in the electric shop is going to stop working. I also learned that in order to coax the plant manager to the electric shop, a major electrical tragedy is one good way. I learned that when shooting rusty water into the air don’t point it at a high voltage auxiliary substation switch. — I’m sure Mickey Postman learned that lesson too. I also learned that just like in Star Trek… whenever there is a dangerous job to do, the Captain is always the one that wants to do it. Does that make sense? Send a Peon like me in there…
I also learned something else about Power Plant Men…. You see…. People like Dale Mitchell, George Alley and Mickey Postman all are examples of incredibly wonderful Power Plant Men. When they were out there doing their duty and something tragic like this, all the Power Plant Men felt their pain. They knew that they all felt guilty for tripping the unit. It didn’t matter that a million dollars every so many minutes was walking out the door in revenue. The only thing that mattered was that these three men were safe.
Mickey Postman
Since I have left the Power Plant, I have found that the idea that the employee is the greatest asset that a company can possess is not a universal idea. You see, there was never the thought that any of these people should be fired for their mistake. On the contrary. The true Power Plant Men did whatever they could to let them know that they knew exactly how they felt. It could have happened to any of them.
Besides the friendship between Power Plant Men, one of the things I miss most about working at the Power Plant is that the employees are held in high esteem as a real asset to the company. Many could learn from their example.
Comments from the Original post
-
That was an exciting day! Another great story. Thanks for the memories.
-
I like the mix of storytelling and information sharing you deliver here. Thanks again
-
Aug 8, 2011 I lost all reserve in and lost both units in 2 separate storms and A1 would not start for a few minutes and A2 was leaking antifreeze terribly not a scratch and back online in 34 hours both units and no one ever asked me one question even with a Safety Dept not one question asked of me the SS on duty. Not many men on the planet have ever experienced an uncontrolled total plant outage- you would of thought a learning opportunity would of took place. Feb 2011 worst winter temps in years 50 Units or more tripped in Texas let alone the trusty units of Redbud and McClain were fighting Sooner rolled right along with storm warnings for two weeks ahead – the ICS still went to Detroit for just tours of other facilities once again the fall guy Tarver McArthur stood alone. I had authored a Freeze Protection Plan for the plant and that seemed to save the day and explain to the regulatory bodies how we were online and everyone else in Texas was off and enjoying rolling blackouts in a terrible winter weather situation not to mention our powers that be were all stranded in Detroit and very few people could get to the plant without getting stuck trying to get there as well – but a few health heart issues later I am still here to tell about it all you would think folks would want to take advantage of someone that had went through the fire and Ice but thats ok I want them someday distant to get all the credit they deserve when the trumpets sound as that is truly what matters most. DT
-
Great story, and yes, you took exactly the proper lesson from it, and it is too bad that many of our bosses haven’t learned it.
And yup, I’ve buried a lot of electrocuted squirrels over the years.
Rivers and Rose in the Power Plant Palace
Originally posted January 25, 2013:
When is the appropriate time to call 911? Calling 911 in the Power Plant is when you call the Shift Supervisor to report something important. As Randy Dailey, our Safety Trainer extraordinaire, always taught us, first tap the person on the shoulder and say, “Are you all right?” Then you point you finger at someone and say, “Call 911!” That’s called “Activating the EMS” (Emergency Medical System). Besides medical emergencies, there are other reasons to call the Shift Supervisor.
I learned early on to ‘fess up when you have done something wrong.” People appreciate it when you tell them up front that you goofed. That way the problem can be dealt with directly. Dee Ball was that way. Any time he wrecked a truck, he didn’t hesitate to tell his boss. So, even as a summer help I had developed this philosophy. Never be afraid to expose your blunders. It works out better in the long run.
One example of someone not following this philosoply was Curtis Love. As I mentioned in the post Power Plant Spider Wars and Bugs in the Basement, Curtis didn’t want to tell anyone that he had been bitten by a brown recluse for the third time because he was afraid of losing his job.
The Oklahoma house spider — The Brown Recluse
His philosophy came back to bite him a year and a half later when he was on the labor crew when he was the designated truck driver. I had moved on to the electric shop by this time.
A red crew cab like this only the bed of the truck was longer
He was backing up the crew cab around a corner under the Fly Ash hoppers up at the coalyard when the side of the crew cab came into contact with one of those yellow poles designed to protect the structure from rogue vehicles. Unfortunately. This created a dent in the side of the truck.
An example of yellow poles protecting an area
Curtis, already on probation. worried that he would be fired if he told anyone about this mishap, failed to tell Larry Riley about this incident. Larry, on the other hand, was standing in front of the Coalyard Maintenance shop (the labor crew home), and saw the entire incident. At that moment, he turned to one of the labor crew hands and said, “I hope Curtis comes over here and tells me about that.” Unfortunately, Curtis decided to act as if nothing had happened. This resulted in his termination. As much as I cared about Curtis, I must admit that the Power Plant scene was probably not the best location for his vocation.
I had seen Dee Ball do the same thing over and over again, and he always reported his accidents immediately. He was never punished for an accident, though, for a number of years, he was banned from driving a truck. You can read more about this in the post: Experiencing Maggots, Mud and Motor Vehicles with Dee Ball.
One day during the summer of 1984 just after lunch, 1A PA fan tripped (PA stands for Primary Air). When this happened, number one unit had to lower it’s output from over 500 Megawatts down to around 200. The trip indicator on the 6900 volt breaker said that it had been grounded. Being grounded means that one of the three phases of the motor or cable had made a circuit with the ground (or something that was grounded). The trip circuits shut the fan down so fast that it prevents an explosion and saves the fan from being destroyed.
Diana Lucas (later Diana Brien), Andy Tubbs and I were given the task of finding the ground and seeing what we could do to fix it. We unwired the motor, which was no easy task, because the motor is about the size of a large van, and about 10 times heavier.
This is about 1/2 the size of the PA fan motor which is 1000 horsepower
So, we spent the rest of the day unwiring the motor (in the rain), and unwiring the cable to the motor from the breaker in the main switchgear and testing both the motor and the cable with various instruments looking for the grounded wire or coil that caused the motor to trip. We used a large “Megger” on the motor. It’s called a Megger because it measures Mega-Ohms. So, it’s technically called a Mega-Ohm meter. Ohms is a measurement of resistance in an electrical circuit. We usually use a small hand cranked megger, that is similar to an old hand crank telephone that generates a high voltage (good for shocking fish in a lake to make them rise to the surface). In the case of the hand cranked Megger, it would generate 1,000 volts.
A Hand cranked Megger made by the same company, only newer than the ones we had
The Megger this size would have been useless with this large motor. Instead we used one that was electric, and you ran the voltage up over 10,000 volts and watched the mega-ohms over a period of 1/2 hour or so.
For the cables, we hooked up a Hypot (or Hipot). This stands for High Potential. Potential in this case is another word for “Voltage”. It would charge up and then you pressed a button and it would send a high voltage pulse down the cable, and if there is a weak spot in the insulation,The Hypot will find it. So, we hooked a Hypot up to the cable and tried to find the grounded wire. No luck.
After spending 4 hours looking for the grounded cable or motor, we found nothing. We spent another hour and a half putting the motor and the breaker back in service. The Fan was put back into operation and we went home. As I was walking out to the car with Bill Rivers, he told me, “I knew they weren’t going to find anything wrong with that fan.” He had a big grin on his face.
At first I thought he was just making an educated guess as Rivers was apt to do on many occasions (daily). It was raining and I could see where water may have been sucked into the motor or something and had momentarily grounded the motor. Just because we didn’t find anything didn’t mean that the breaker didn’t trip for no reason.
When we were in the car and on our way to Stillwater, Oklahoma with Yvonne Taylor and Rich Litzer, Bill explained that he knew why the motor tripped. He had been walking through the main switchgear with Mike Rose, and Mike, for no apparent reason other than curiosity, had opened up the bottom door to the breaker for 1A PA fan. He looked at it for a moment and then slammed the door shut. When he did this, the breaker tripped.
So, the ground relay happened to be the one that tripped. It might as well been an over-current or a low voltage trip. It just happened to trip the ground trip. Bill said that he told Mike that he should call the Shift Supervisor and let him know so they could restart the motor. Mike on the other hand told Bill that he was already on probation and was afraid of losing his job if he reported that he had slammed the door on the breaker and tripped the fan.
If there was ever a reason to call 911, it was then. All he had to do was tell them, “I accidentally tripped the PA fan when I bumped the breaker cabinet.” They would have told him to reset the flag, and they would have started the fan right back up. No questions asked… I’m sure of it. And they wouldn’t have lost their generating capacity for the remainder of the afternoon and we wouldn’t have spent 4 hours unwiring, testing and rewiring the motor in the rain with a plastic umbrella over our head.
Bill wasn’t about to tell on Mike. If Mike didn’t want to report it, Bill wasn’t going to say anything, and I understand that. I probably would have kept it to myself at the time if I was in Bill’s shoes (I’m just glad I wasn’t because I probably wouldn’t have been able to sleep soundly for the next year). But 30 years later, I might write about it in a Blog. Even though I wouldn’t have looked to Mike to teach me much about being an electrician (he was more of an Air Condition man anyway), I still loved the guy.
Mike died almost two years ago on May 29, 2011. He was from England and had lived in Canada for a time. He used to work on trains. Trains, even though they are diesel, are really electric. The Diesel engine really runs a generator that generates electricity that runs the train. I know that Mike was a good man at heart. He loved his family with all his heart. Here is a picture of the Limey:
Mike Rose. A fair plant electrician, but a great family man!
Ok. So I know what you are thinking…. There must be a story about myself in here somewhere. Well, you would be right. First of all. I always ‘fessed up to my mistakes, as my current manager at Dell knows well (yes. I still mess up after all these years). I told my current manager the other day that CLM was my middle name. (CLM means “Career Limiting Move”). So here is my power plant “mess up” story (well one of them):
In January 1986, I returned from my Honeymoon with my new wife Kelly when I found that we had hired a new electrician. Gsry Wehunt was replacing Jim Stephenson who had left the plant on February 15, 1985, which is a story all it’s own. We had just started an overhaul on Unit 1.
I remember the first Monday I spent with Gary. It was January 6, 1986 and we were working on cleaning out the exciter house on the end of the main power generator with Diana Brien (formerly Diana Lucas). We were discussing salaries and Gary was surprised to find out that I was making more than he was. Well… I had been an electrician for over 2 years and had been promoted regularly…. so I didn’t think there was anything strange about it, except that I still looked like I was only about 18 years old (even though I was 25) and Gary was about 34. I had already been promoted 4 times and my salary had gone from $7.15 to over $12 an hour.
Anyway, when that first Friday rolled around, Gary and I were assigned to Substation Inspection. Some later time I may go into the details of what “Substation Inspection” entails, but for now, let’s just stick with my “911 call.” It is enough to say that we were in the main plant substation relay house on Friday January 10, 1986 at 9:00 am. One of our jobs was to call other substations and perform a test called a “Transfer Trip and Carrier Test”. We had called Woodring Substation (Woodring is a town in Oklahoma and we had a 345 KV line going there), and I was talking to the man in the substation on the other end of the phone line.
At the same time I was showing Gary just how experienced I was at being an electrician. People had told me that you had to be a plant electrician for 5 years before you really became a “first class” electrician. Well. Here I was at 2 years, and I thought I was so good that I could do anything by now…. — Yeah… right. I told the guy on the other end of the line as I turned a switch…. Amber light… Back to Blue…. and I wrote down the value on the meter (paperwork… oh yes…. it’s that important. Like A-1 sauce).
Then I reached for the second switch. I said, “Carrier test”, then turned the switch. The lights in the relay house went out and we were in the dark. I told the guy on the other end of the line….. “Well. That’s not supposed to happen.” Then as I let go of the switch and it returned to it’s normal position, the lights turned back on. Okay……
I wrote the numbers down from the meter and said goodbye to the other faceless substation man on the other end of the line that I talked to over 100 times, but never met in person. He sounded like a nice guy. Then I headed for the gray phone. I heard the Shift Supervisor paging Leroy Godfrey (The Electrical Supervisor) on line 2 (we had 5 Gray phone lines. The Gray Phone was our PA system).
When I picked up the line I heard Leroy pick up the phone and the Shift Supervisor tell Leroy that we just lost station power in the main substation and it had switched over to Auxiliary power. I immediately jumped in and said, “Jim (for Jim Padgett, the Shift Supervisor), I did that. I was performing a Carrier test with Woodring and the moment I performed the carrier test the lights went out.” Leroy chimed in by saying, “That wouldn’t cause you to lose station power.”
Well, in my ‘inexperienced’ plant electrician way, I responded, “Well. All I know is that when I turned the switch to perform the carrier test, the lights went out, and when I let go of the switch, the lights came back on.” Leroy reiterated, “That wouldn’t cause you to lose station power.” I replied with, “I’m just saying….” and left it at that. I had done my job. They knew I was out here. They knew I had called 911 right away. I explained what I was doing…. they could take it from there.
I had hoped that I had showed Gary upfront that it doesn’t hurt to report your mistakes (even though I hadn’t made one as far as I could tell), but I was 100% sure I had done something to cause the relay house to lose power. Though, I couldn’t figure out why.
After lunch, Bill Bennett, our A foreman came down to the shop to tell me that they figured out how the substation lost station power. He said that a road grader had been grating the road down by the Otoe-Missouri reservation (which is actually called “Windmill road” I guess because there is a windmill down that road somewhere), and had hit an electric pole and knocked it over and had killed the power to the substation.
Substation Power Interrupting Road Grader
It turned out that the substation relay house was fed by a substation down that road where we have a radio tower. So, think about this. The exact time that I turned that switch in the substation, a road grater 2 1/2 miles away hits a telephone pole accidentally and knocks it to the ground and kills the power to the substation at the exact same time that I am performing a transfer-trip and Carrier test with Woodring Substation, and the time it takes to switch to auxiliary power is the exact time it took me to let go of the switch.
Don’t tell me that was by accident. I will never believe it. I think it was for the soul purpose of teaching me a useful lesson or two. First….. don’t be afraid to tell someone when you do something wrong. Second…. If you think you have control over the things that happen to you in your life… well, think again…… Third….. God watches you every moment, and if you let him, he will guide you to do the right thing when the time comes.
God bless you all.
COMMENTS FROM THE ORIGINAL POST:
-
Monty Hansen January 26, 2013
I had a similar thing happen to me, I was upgrading to shift foreman & system called to remove a tag in the switchyard & put the switch back to auto. The tag on the pistol grip was attached with a plastic zip tie & the previous operator had put it on real tight, as I was wrestling it off with my leatherman, the pliers slipped & I banged my elbow into the control panel, at that very instant there was a loud BANG as several 345 KV breakers opened simultaniously in the swithyard, I had the phone pinched between my shoulder & ear as I was wrestling with this switch & talking to the system control operator, he said a few bad words – gotta go – & hung up. The power plant lost all power & went in the black, I, of course was just sick in the pit of my stomach, after we got power restored, the plant back on etc. I called system back to see if they found the cause & fess up to causing the trip (I figured I must have caused a trip relay to close when I hit the panel) – anyway a crane at a plant down the road had got it’s boom tangled in the power line & went to ground – AT PRECISELY THE INSTANT MY ELBOW SLIPPED & HIT THE PANEL!!
-
Ron Kilman January 26, 2013
Some great illustrations of the truth in Proverbs 28:13 “He who conceals his transgressions will not prosper, But he who confesses and forsakes them will find compassion”.
-
justturnright January 28, 2013
CLM: I can relate.
My first boss 30 years ago once told me he was going to officially nickname me “I’m sorry” (and make me wear it for a name badge) if I said it one more time.
Hey, there’s worse things.
-
Roomy January 29, 2013
I had not thought about Mike Rose in years. He was a good guy to work with, now Rivers was a different story!!!
Sub checks, I used to love to do sub checks. I performed pilot wire & transfer trip checks for years. I hated it when they went to being done by automation.
Thanks for brining back old memorys.
Runaway Fire Hydrant Leaves Power Plant in the Dark
Don’t believe it when the Electric Company tells you that the reason your town lost electricity for an hour was because a squirrel climbed onto a transformer and shorted it out. The real reason just may be more bizarre than that and the company doesn’t want you to know all the different creative ways that power can be shut off. This is a tale of just one of those ways. So, get out your pencil and paper and take notes.
Power Plant Notepad
One spring day 1993 while sitting at the Precipitator computer for Unit one at the Coal-fired Power Plant in North Central Oklahoma, while I was checking the controls to make sure all the cabinets were operating correctly, suddenly there was a distant boom, and the lights in the control room went out. The computer stayed on because it was connected to an electric panel called the VSP or Vital Services Panel, which in turn was supplied by the UPS system (Uninterruptible Power Supply). That was one of those moments where you may pause for a moment to make sure you aren’t still at home dreaming before you fly into a panic.
The Precipitator cabinets all indicated on the computer that they had just shutdown. I rose from the chair and walked around to the front of the Alarm Panel for Unit one, and found that the lights were only out on Unit 1. The lights were still on for Unit 2. The Control Panel was lit up like a Christmas Tree with Green, Red, Blue and Yellow Lights. The Alarm Printer was spewing out printer at high speed. As the large sheets of paper were pouring out onto the floor, I watched as Pat Quiring and other brave Power Plant Control Room operators were scurrying back and forth turning switch handles, pushing buttons, and checking pressure gauges.
Just this site alone gave me confidence that everything was going to be all right. These Control Room operators were all well trained for emergencies just like this, and each person knew what their job was. No one was panicking. Everyone was concentrating on the task at hand.
Someone told me that we lost Unit 1, and the Auxiliary Power to Unit 1 at the same time. So, Unit 1 was dead in the water. This meant, no fans, no pumps, no lights, no vending machines, no cold water at the water fountain and most importantly, no hot coffee!!! I could hear steam valves on the T-G floor banging open and the loud sound of steam escaping.
I turned quickly to go to the electric shop to see what I could do there in case I was needed. I bolted out the door and down the six flights of stairs to the Turbine-Generator (T-G) basement. Exiting the stairway, and entering the T-G basement the sound was deafening. I grabbed the earplugs that were dangling around my neck and crammed them into my ears. Steam was pouring out of various pop-off valves. I ducked into the electric shop where across the room Andy Tubbs, one of the electric foreman was pulling large sheets of electric blueprints from the print cabinet and laying them across the work table that doubled as the lunch table.
Andy Tubbs – True Power Plant Electrician
When I asked Andy what happened, I learned that somehow when a crew was flushing out a fire hydrant the water somehow shot up and into the bus work in the Auxiliary Substation (that supplies backup power to the Power Plant) and it shorted out the 189,000 volt substation directly to ground. When that happened it tripped unit 1 and the auxiliary substation at the same time leaving it without power.
An example of part of an Auxiliary Substation
I will explain how a fire hydrant could possibly spray the bus work in a substation in a little while, but first let me tell you what this meant at the moment to not have any power for a Power Plant Boiler and Turbine Generator that has just tripped when it was at full load which was around 515 Megawatts of power at the time.
Normally when a unit trips, the boiler cools down as the large Force Draft (FD) Fans blow air through the boiler while the even larger Induced Draft (ID) fans suck the air from the boiler on the other end and blow the hot air up the smoke stack. This causes the steam in the boiler tubes to condense back into water. Steam valves open on the boiler that allow excessive steam to escape.
When the boiler is running there is a large orange fireball hovering in space in the middle of the boiler. The boiler water is being circulated through the boiler and the Boiler Feed Pump Turbines are pumping steam back and forth between the turbine generator and the boiler reheating the steam until every bit of heat from the boiler that can be safely harnessed is used.
When all this stop suddenly, then it is important that the large fans keep running to cool down the steam, since it is no longer loosing energy in the generator as it was when it was busy supplying electricity to 1/2 million people in Oklahoma City. The power is fed to the fans from the Auxiliary substation located right outside the Main Switchgear where all the breakers reside that supply the power to the fans. Unfortunately, in this case, the Auxiliary substation was shutdown as well, leaving the boiler without any fans.
Without fans for cooling, and pumps to circulate the water, the walls of the boiler began heating up to dangerous temperatures. Steam was whistling out of pop off valves, but if the steam drum on the top of the boiler were to run dry, then the entire boiler structure could be compromised and begin melting down. — So, this was serious. Something had to be done right away. It wouldn’t be as bad as the China Syndrome since we were burning coal, but it would have caused a lot of damage nonetheless.
From the movie “The China Syndrome” where a similar emergency existed only in the movie, it was a Nuclear Plant
I have a side story about this picture, but I think I’ll save it for another post because I don’t want to digress from the main story at this point (Ok. Let me just say “Jack Maloy and Merl Wright” for those who can’t wait).
With the prospect that the boiler might melt to the ground in a pile of rubble, it would seem that the main priority was to turn the Auxiliary Substation back on so the fans could be turned back on and prevent the boiler from collapsing. So, we walked out to the substation and looked at the switches that would have to be operated in order to first power up the main bus and then to close to supply power to the two big transformers and the six smaller transformers that supplied the Unit 1 Main Switchgear.
While inspecting the switches where the electricity had gone to ground we found that one of the main insulators was cracked.
A High Voltage Insulator like this
Since this insulator was cracked, we didn’t really want to operate the switch to test if another 189,000 volts would go straight to ground again, especially since one of us would be standing right underneath it cranking the switch. So, we went back to the shop to find an alternative.
By this time the Plant Manager, Ron Kilman arrived in the shop, and understanding the urgency to find a solution asked us what were the alternatives. He was relying on our expertise to make the decision.
Ron Kilman is the one on the left in the plaid shirt
The next solution would be to cut the power over from Unit 2 which was still humming away pushing electricity to Oklahoma City out of the 345,000 volt substation. The cut over would be very simple because the switchgear was designed with this in mind. We analyzed the power rating on the auxiliary transformers on Unit 2 and thought that we might be cutting it close to have them running both sets of fans at the same time, especially since the full load amps of a huge fan starting up was about 10 times the normal rate.
The transformer was rated to handle the load, but consider this. What if this caused Unit 2 to trip as well. With the Auxiliary substation offline, if Unit 2 tripped, we would be in twice the amount of trouble we were currently in. What a day it would have been if that had happened and two 250 foot boilers had come crashing to the ground in a pile of rubble. After reading the power ratings on the auxiliary transformers I was thinking, “Yeah, let’s do it! These transformers can handle it.” Andy was not so eager.
So, we were left with one alternative. That was to shut the switch in the Auxiliary substation that had the cracked insulator and take our chances that it wasn’t going to short to ground and blow up over our heads. I think I was eager to close the switch for Andy, but if I remember correctly, he didn’t want me to be the one to suffer the consequences and decided to close the switch himself. Needless to say. Andy closed the switch, and nothing blew up.
High Voltage Switch similar to the one we were closing
As soon as the power was restored to the switchgear, the fans were powered up and the temperature in the boiler was quickly reduced. The coffee pot in the Electric Shop began heating the coffee again. The power plant was saved from a major catastrophe. That was delayed for another day… of which I will talk about later.
So, how exactly does a fire hydrant shoot water up into the bus work of a substation like the picture of the switch directly above? The culprit fire hydrant wasn’t in the substation, it sat alongside it outside the fence a good 50 feet from the high voltage switch. No hose was attached to the fire hydrant. It was only being flushed out as part of a yearly activity to go around and make sure the fire hydrants are all operating correctly.
Here is the story about how the squirrel climbed into the transformer this time….
George Alley, Dale Mitchell and Mickey Postman were going around to the 30,000 fire hydrants on the plant ground (ok. maybe not that many, but we did have a lot of them), and they were opening up the valves and flushing them out. That means, they were letting them run for a while to clear them out from any contaminates that may have built up over the year of not being used.
Throughout their adventure they had opened a multitude of Hydrants situated out in the fields along the long belt conveyor from the coalyard and around the two one-million gallon #2 Diesel tanks.
Large Oil Tank about the size of the two that are at the Power Plant
The brave Power Plant Men, learned that when opening a fire hydrant wide open in the middle of field had unintended consequences. It tended to wash out the ground in front of the flow of the water shooting out of the hydrant. So the team of experts devised a plan to place a board in front of the hydrant when it would be in danger of tearing a hole in the terrain. The board would divert the water into the air where it would fan out and not cause damage to the surrounding area.
This was working fine, and when they arrived at the fire hydrant next to the substation, since the stream from the hydrant was pointing directly into the substation (hmm. a design flaw, I think), they decided to prop the board up against the fence to keep from washing away the gravel in the substation. Well. When a fire hydrant is opened that hasn’t been used for a year, the first flow of water to shoot out is dark brown.
You may think that this is because the water has somehow become dirty over the past year, but that isn’t quite the case. What has happened is that the pipe has been rusting little by little and the water has become saturated with the rust. So, the water shooting out of the hydrant was full of rust (hence the need to flush them out).
Well. Rust is made of metal. Metal is conductive, especially when it is mixed with water. When the water hit the board, it was deflected into the air and happened to direct itself directly into the high voltage switch in the substation. This caused a circuit to the the ground which, once it created an arc pumped all the electricity directly into the ground.
Normally when something like this happens it doesn’t trip the Main Power Transformer to a Power Plant.
A Main Power Transformer
This time it did. I know there was a few heads scratching trying to figure it out. I think I figured out what happened a little while later. You see… here is the rest of the story….
Once the unit was back online and the emergency was over, someone finally noticed that the telephone system couldn’t call outside of the plant. Well. I was the main telephone person at the time, so the control room called me and asked me to look into the problem.
I checked the telephone computer and it was up and running just fine. Internal calls could be made. Only any call outside just concluded with a funny humming sound. After checking the circuit in the Logic Room next to the Rolm Telephone Computer I headed for…. guess where….. the Main Switchgear….
A ROLM Phone Computer
In the middle of the main switchgear in the back of the room right next to the Auxiliary Substation beyond the back wall, the outside telephone line came into the plant. The first thing it did was go through a special Telephone Surge Protector.
Telephone Grounding Panel
In this picture above, the silver circular buttons on the left side are really an old style surge protector. whenever there was a power surge, the carbon connection in the surge protector would quickly melt causing the circuit to go straight to ground. Thus protecting the rest of the telephone circuit. So, if some kid in their house decides to connect the 120 volts circuit to the telephone for fun to see what would happen, this circuit would protect the rest of the phone circuits. Keep in mind that this was during the early 1990 when “Surge Protection” still had a “mechanical” element to it.
Anyway, when I arrived at this panel and I checked the surge protector to the main line going out of the plant, guess what I found…. Yep. Shorted to ground. Luckily there were some spares that were not wired to anything in the panel and I was able to swap them out for the ones that had been destroyed. — These were a one time use. Which meant, if they ever had to short to ground, they had to be replaced.
Ok. Fine. After a little while, we were able to call back out of the plant, though there was still some residual noise on the line. It was like this… when you called out of the plant, the person on the other end sounded like they were buried in a grave somewhere and they were trying to talk to someone living just like in an episode of the Twilight Episode where a phone line landed on a grave and the dead person tried to call his long lost love from the past.
Gladys Cooper in the Twilight Zone Episode “Night Call”
I didn’t give it much thought other than that I figured the 189,000 volt arc to ground must have shorted out the telephone line since the phone line ran directly under the auxiliary substation ground grid.
It wasn’t until the next morning when the Southwestern Bell repairman showed up at the plant. I knew him well, since he had been working on our phone lines since before the AT&T breakup in 1984. When I met him in the front of the electric shop, he said that he needed to check our telephone circuits. I told him that I knew that we had a problem because we had a high voltage short to ground yesterday and I found our surge protectors melted away.
He explained to me that not only was our circuit affected, but that every relay house from here to Ponca City was blown out. That’s when I realized that the problem was the reverse of the usual situation. What had happened was that the Ground Grid in the substation and the surrounding area (including the Unit 1 Main Power Transformer) had become hot. What do you do when the ground grid becomes charged?
The Ground Grid is what is supposed to protect you when a surge happens, but what happens when the ground grid itself is the problem? In this case, when the high voltage line about 60 feet from the telephone cable surge protector, arced to ground, it fed a tremendous amount of power back through the ground grid. when equipment detected the surge in voltage, they automatically defaulted their circuits to ground. That’s what the telephone circuit die. That’s what tripped the Main Power Transformer.
When the telephone circuit detected the high voltage surge, it shorted to ground (which was the problem), causing the high voltage to feed directly into the phone line and down the line to the next Southwestern Bell relay switch, which also defaulted to ground, trying to bleed off the surge as it went from relay switch to switch until enough of the power was able to be diverted to ground.
That day sure turned out to be a learning experience. I learned that when all the lights go out in the control room, that it is almost assured that the coffee pot in the electric shop is going to stop working. I also learned that in order to coax the plant manager to the electric shop, a major electrical tragedy is one good way. I learned that when shooting rusty water into the air don’t point it at a high voltage auxiliary substation switch. — I’m sure Mickey Postman learned that lesson too. I also learned that just like in Star Trek… whenever there is a dangerous job to do, the Captain is always the one that wants to do it. Does that make sense? Send a Peon like me in there…
I also learned something else about Power Plant Men…. You see…. People like Dale Mitchell, George Alley and Mickey Postman all are examples of incredibly wonderful Power Plant Men. When they were out there doing their duty and something tragic like this, all the Power Plant Men felt their pain. They knew that they all felt guilty for tripping the unit. It didn’t matter that a million dollars every so many minutes was walking out the door in revenue. The only thing that mattered was that these three men were safe.
Mickey Postman
Since I have left the Power Plant, I have found that the idea that the employee is the greatest asset that a company can possess is not a universal idea. You see, there was never the thought that any of these people should be fired for their mistake. On the contrary. The true Power Plant Men did whatever they could to let them know that they knew exactly how they felt. It could have happened to any of them.
Besides the friendship between Power Plant Men, one of the things I miss most about working at the Power Plant is that the employees are held in high esteem as a real asset to the company. Many could learn from their example.
Rivers and Rose in the Power Plant Palace — Repost
Originally posted January 25, 2013:
When is the appropriate time to call 911? Calling 911 in the Power Plant is when you call the Shift Supervisor to report something important. As Randy Dailey, our Safety Trainer extraordinaire, always taught us, first tap the person on the shoulder and say, “Are you all right?” Then you point you finger at someone and say, “Call 911!” That’s called “Activating the EMS” (Emergency Medical System). Besides medical emergencies, there are other reasons to call the Shift Supervisor.
I learned early on to ‘fess up when you have done something wrong.” People appreciate it when you tell them up front that you goofed. That way the problem can be dealt with directly. Dee Ball was that way. Any time he wrecked a truck, he didn’t hesitate to tell his boss. So, even as a summer help I had developed this philosophy. Never be afraid to expose your blunders. It works out better in the long run.
One example of someone not following this philosoply was Curtis Love. As I mentioned in the post Power Plant Spider Wars and Bugs in the Basement, Curtis didn’t want to tell anyone that he had been bitten by a brown recluse for the third time because he was afraid of losing his job.
The Oklahoma house spider — The Brown Recluse
His philosophy came back to bite him a year and a half later when he was on the labor crew when he was the designated truck driver. I had moved on to the electric shop by this time.
A red crew cab like this only the bed of the truck was longer
He was backing up the crew cab around a corner under the Fly Ash hoppers up at the coalyard when the side of the crew cab came into contact with one of those yellow poles designed to protect the structure from rogue vehicles. Unfortunately. This created a dent in the side of the truck.
An example of yellow poles protecting an area
Curtis, already on probation. worried that he would be fired if he told anyone about this mishap, failed to tell Larry Riley about this incident. Larry, on the other hand, was standing in front of the Coalyard Maintenance shop (the labor crew home), and saw the entire incident. At that moment, he turned to one of the labor crew hands and said, “I hope Curtis comes over here and tells me about that.” Unfortunately, Curtis decided to act as if nothing had happened. This resulted in his termination. As much as I cared about Curtis, I must admit that the Power Plant scene was probably not the best location for his vocation.
I had seen Dee Ball do the same thing over and over again, and he always reported his accidents immediately. He was never punished for an accident, though, for a number of years, he was banned from driving a truck. You can read more about this in the post: Experiencing Maggots, Mud and Motor Vehicles with Dee Ball.
One day during the summer of 1984 just after lunch, 1A PA fan tripped (PA stands for Primary Air). When this happened, number one unit had to lower it’s output from over 500 Megawatts down to around 200. The trip indicator on the 6900 volt breaker said that it had been grounded. Being grounded means that one of the three phases of the motor or cable had made a circuit with the ground (or something that was grounded). The trip circuits shut the fan down so fast that it prevents an explosion and saves the fan from being destroyed.
Diana Lucas (later Diana Brien), Andy Tubbs and I were given the task of finding the ground and seeing what we could do to fix it. We unwired the motor, which was no easy task, because the motor is about the size of a large van, and about 10 times heavier.
This is about 1/2 the size of the PA fan motor which is 1000 horsepower
So, we spent the rest of the day unwiring the motor (in the rain), and unwiring the cable to the motor from the breaker in the main switchgear and testing both the motor and the cable with various instruments looking for the grounded wire or coil that caused the motor to trip. We used a large “Megger” on the motor. It’s called a Megger because it measures Mega-Ohms. So, it’s technically called a Mega-Ohm meter. Ohms is a measurement of resistance in an electrical circuit. We usually use a small hand cranked megger, that is similar to an old hand crank telephone that generates a high voltage (good for shocking fish in a lake to make them rise to the surface). In the case of the hand cranked Megger, it would generate 1,000 volts.
A Hand cranked Megger made by the same company, only newer than the ones we had
The Megger this size would have been useless with this large motor. Instead we used one that was electric, and you ran the voltage up over 10,000 volts and watched the mega-ohms over a period of 1/2 hour or so.
For the cables, we hooked up a Hypot (or Hipot). This stands for High Potential. Potential in this case is another word for “Voltage”. It would charge up and then you pressed a button and it would send a high voltage pulse down the cable, and if there is a weak spot in the insulation,The Hypot will find it. So, we hooked a Hypot up to the cable and tried to find the grounded wire. No luck.
After spending 4 hours looking for the grounded cable or motor, we found nothing. We spent another hour and a half putting the motor and the breaker back in service. The Fan was put back into operation and we went home. As I was walking out to the car with Bill Rivers, he told me, “I knew they weren’t going to find anything wrong with that fan.” He had a big grin on his face.
At first I thought he was just making an educated guess as Rivers was apt to do on many occasions (daily). It was raining and I could see where water may have been sucked into the motor or something and had momentarily grounded the motor. Just because we didn’t find anything didn’t mean that the breaker didn’t trip for no reason.
When we were in the car and on our way to Stillwater, Oklahoma with Yvonne Taylor and Rich Litzer, Bill explained that he knew why the motor tripped. He had been walking through the main switchgear with Mike Rose, and Mike, for no apparent reason other than curiosity, had opened up the bottom door to the breaker for 1A PA fan. He looked at it for a moment and then slammed the door shut. When he did this, the breaker tripped.
So, the ground relay happened to be the one that tripped. It might as well been an over-current or a low voltage trip. It just happened to trip the ground trip. Bill said that he told Mike that he should call the Shift Supervisor and let him know so they could restart the motor. Mike on the other hand told Bill that he was already on probation and was afraid of losing his job if he reported that he had slammed the door on the breaker and tripped the fan.
If there was ever a reason to call 911, it was then. All he had to do was tell them, “I accidentally tripped the PA fan when I bumped the breaker cabinet.” They would have told him to reset the flag, and they would have started the fan right back up. No questions asked… I’m sure of it. And they wouldn’t have lost their generating capacity for the remainder of the afternoon and we wouldn’t have spent 4 hours unwiring, testing and rewiring the motor in the rain with a plastic umbrella over our head.
Bill wasn’t about to tell on Mike. If Mike didn’t want to report it, Bill wasn’t going to say anything, and I understand that. I probably would have kept it to myself at the time if I was in Bill’s shoes (I’m just glad I wasn’t because I probably wouldn’t have been able to sleep soundly for the next year). But 30 years later, I might write about it in a Blog. Even though I wouldn’t have looked to Mike to teach me much about being an electrician (he was more of an Air Condition man anyway), I still loved the guy.
Mike died almost two years ago on May 29, 2011. He was from England and had lived in Canada for a time. He used to work on trains. Trains, even though they are diesel, are really electric. The Diesel engine really runs a generator that generates electricity that runs the train. I know that Mike was a good man at heart. He loved his family with all his heart. Here is a picture of the Limey:
Mike Rose. A fair plant electrician, but a great family man!
Ok. So I know what you are thinking…. There must be a story about myself in here somewhere. Well, you would be right. First of all. I always ‘fessed up to my mistakes, as my current manager at Dell knows well (yes. I still mess up after all these years). I told my current manager the other day that CLM was my middle name. (CLM means “Career Limiting Move”). So here is my power plant “mess up” story (well one of them):
In January 1986, I returned from my Honeymoon with my new wife Kelly when I found that we had hired a new electrician. Gsry Wehunt was replacing Jim Stephenson who had left the plant on February 15, 1985, which is a story all it’s own. We had just started an overhaul on Unit 1.
I remember the first Monday I spent with Gary. It was January 6, 1986 and we were working on cleaning out the exciter house on the end of the main power generator with Diana Brien (formerly Diana Lucas). We were discussing salaries and Gary was surprised to find out that I was making more than he was. Well… I had been an electrician for over 2 years and had been promoted regularly…. so I didn’t think there was anything strange about it, except that I still looked like I was only about 18 years old (even though I was 25) and Gary was about 34. I had already been promoted 4 times and my salary had gone from $7.15 to over $12 an hour.
Anyway, when that first Friday rolled around, Gary and I were assigned to Substation Inspection. Some later time I may go into the details of what “Substation Inspection” entails, but for now, let’s just stick with my “911 call.” It is enough to say that we were in the main plant substation relay house on Friday January 10, 1986 at 9:00 am. One of our jobs was to call other substations and perform a test called a “Transfer Trip and Carrier Test”. We had called Woodring Substation (Woodring is a town in Oklahoma and we had a 345 KV line going there), and I was talking to the man in the substation on the other end of the phone line.
At the same time I was showing Gary just how experienced I was at being an electrician. People had told me that you had to be a plant electrician for 5 years before you really became a “first class” electrician. Well. Here I was at 2 years, and I thought I was so good that I could do anything by now…. — Yeah… right. I told the guy on the other end of the line as I turned a switch…. Amber light… Back to Blue…. and I wrote down the value on the meter (paperwork… oh yes…. it’s that important. Like A-1 sauce).
Then I reached for the second switch. I said, “Carrier test”, then turned the switch. The lights in the relay house went out and we were in the dark. I told the guy on the other end of the line….. “Well. That’s not supposed to happen.” Then as I let go of the switch and it returned to it’s normal position, the lights turned back on. Okay……
I wrote the numbers down from the meter and said goodbye to the other faceless substation man on the other end of the line that I talked to over 100 times, but never met in person. He sounded like a nice guy. Then I headed for the gray phone. I heard the Shift Supervisor paging Leroy Godfrey (The Electrical Supervisor) on line 2 (we had 5 Gray phone lines. The Gray Phone was our PA system).
When I picked up the line I heard Leroy pick up the phone and the Shift Supervisor tell Leroy that we just lost station power in the main substation and it had switched over to Auxiliary power. I immediately jumped in and said, “Jim (for Jim Padgett, the Shift Supervisor), I did that. I was performing a Carrier test with Woodring and the moment I performed the carrier test the lights went out.” Leroy chimed in by saying, “That wouldn’t cause you to lose station power.”
Well, in my ‘inexperienced’ plant electrician way, I responded, “Well. All I know is that when I turned the switch to perform the carrier test, the lights went out, and when I let go of the switch, the lights came back on.” Leroy reiterated, “That wouldn’t cause you to lose station power.” I replied with, “I’m just saying….” and left it at that. I had done my job. They knew I was out here. They knew I had called 911 right away. I explained what I was doing…. they could take it from there.
I had hoped that I had showed Gary upfront that it doesn’t hurt to report your mistakes (even though I hadn’t made one as far as I could tell), but I was 100% sure I had done something to cause the relay house to lose power. Though, I couldn’t figure out why.
After lunch, Bill Bennett, our A foreman came down to the shop to tell me that they figured out how the substation lost station power. He said that a road grader had been grating the road down by the Otoe-Missouri reservation (which is actually called “Windmill road” I guess because there is a windmill down that road somewhere), and had hit an electric pole and knocked it over and had killed the power to the substation.
Substation Power Interrupting Road Grader
It turned out that the substation relay house was fed by a substation down that road where we have a radio tower. So, think about this. The exact time that I turned that switch in the substation, a road grater 2 1/2 miles away hits a telephone pole accidentally and knocks it to the ground and kills the power to the substation at the exact same time that I am performing a transfer-trip and Carrier test with Woodring Substation, and the time it takes to switch to auxiliary power is the exact time it took me to let go of the switch.
Don’t tell me that was by accident. I will never believe it. I think it was for the soul purpose of teaching me a useful lesson or two. First….. don’t be afraid to tell someone when you do something wrong. Second…. If you think you have control over the things that happen to you in your life… well, think again…… Third….. God watches you every moment, and if you let him, he will guide you to do the right thing when the time comes.
God bless you all.
COMMENTS FROM THE ORIGINAL POST:
-
Monty Hansen January 26, 2013
I had a similar thing happen to me, I was upgrading to shift foreman & system called to remove a tag in the switchyard & put the switch back to auto. The tag on the pistol grip was attached with a plastic zip tie & the previous operator had put it on real tight, as I was wrestling it off with my leatherman, the pliers slipped & I banged my elbow into the control panel, at that very instant there was a loud BANG as several 345 KV breakers opened simultaniously in the swithyard, I had the phone pinched between my shoulder & ear as I was wrestling with this switch & talking to the system control operator, he said a few bad words – gotta go – & hung up. The power plant lost all power & went in the black, I, of course was just sick in the pit of my stomach, after we got power restored, the plant back on etc. I called system back to see if they found the cause & fess up to causing the trip (I figured I must have caused a trip relay to close when I hit the panel) – anyway a crane at a plant down the road had got it’s boom tangled in the power line & went to ground – AT PRECISELY THE INSTANT MY ELBOW SLIPPED & HIT THE PANEL!!
-
Ron Kilman January 26, 2013
Some great illustrations of the truth in Proverbs 28:13 “He who conceals his transgressions will not prosper, But he who confesses and forsakes them will find compassion”.
-
justturnright January 28, 2013
CLM: I can relate.
My first boss 30 years ago once told me he was going to officially nickname me “I’m sorry” (and make me wear it for a name badge) if I said it one more time.
Hey, there’s worse things.
-
Roomy January 29, 2013
I had not thought about Mike Rose in years. He was a good guy to work with, now Rivers was a different story!!!
Sub checks, I used to love to do sub checks. I performed pilot wire & transfer trip checks for years. I hated it when they went to being done by automation.
Thanks for brining back old memorys.
Rivers and the Rose in the Power Plant Palace
When is the appropriate time to call 911? Calling 911 in the Power Plant is when you call the Shift Supervisor to report something important. As Randy Dailey, our Safety Trainer extraordinaire, always taught us, first tap the person on the shoulder and say, “Are you all right?” Then you point you finger at someone and say, “Call 911!” That’s called “Activating the EMS” (Emergency Medical System). Besides medical emergencies, there are other reasons to call the Shift Supervisor.
I learned early on to ‘fess up when you have done something wrong.” People appreciate it when you tell them up front that you goofed. That way the problem can be dealt with directly. Dee Ball was that way. Any time he wrecked a truck, he didn’t hesitate to tell his boss. So, even as a summer help I had developed this philosophy. Never be afraid to expose your blunders. It works out better in the long run.
One example of someone not following this philosoply was Curtis Love. As I mentioned in the post Power Plant Spider Wars and Bugs in the Basement, Curtis didn’t want to tell anyone that he had been bitten by a brown recluse for the third time because he was afraid of losing his job.
The Oklahoma house spider — The Brown Recluse
His philosophy came back to bite him a year and a half later when he was on the labor crew when he was the designated truck driver. I had moved on to the electric shop by this time.
A red crew cab like this only the bed of the truck was longer
He was backing up the crew cab around a corner under the Fly Ash hoppers up at the coalyard when the side of the crew cab came into contact with one of those yellow poles designed to protect the structure from rogue vehicles. Unfortunately. This created a dent in the side of the truck.
An example of yellow poles protecting an area
Curtis, already on probation. worried that he would be fired if he told anyone about this mishap, failed to tell Larry Riley about this incident. Larry, on the other hand, was standing in front of the Coalyard Maintenance shop (the labor crew home), and saw the entire incident. At that moment, he turned to one of the labor crew hands and said, “I hope Curtis comes over here and tells me about that.” Unfortunately, Curtis decided to act as if nothing had happened. This resulted in his termination. As much as I cared about Curtis, I must admit that the Power Plant scene was probably not the best location for his vocation.
I had seen Dee Ball do the same thing over and over again, and he always reported his accidents immediately. He was never punished for an accident, though, for a number of years, he was banned from driving a truck. You can read more about this in the post: Experiencing Maggots, Mud and Motor Vehicles with Dee Ball.
One day during the summer of 1984 just after lunch, 1A PA fan tripped (PA stands for Primary Air). When this happened, number one unit had to lower it’s output from over 500 Megawatts down to around 200. The trip indicator on the 6900 volt breaker said that it had been grounded. Being grounded means that one of the three phases of the motor or cable had made a circuit with the ground (or something that was grounded). The trip circuits shut the fan down so fast that it prevents an explosion and saves the fan from being destroyed.
Diana Lucas (later Diana Brien), Andy Tubbs and I were given the task of finding the ground and seeing what we could do to fix it. We unwired the motor, which was no easy task, because the motor is about the size of a large van, and about 10 times heavier.
This is about 1/2 the size of the PA fan motor which is 1000 horsepower
So, we spent the rest of the day unwiring the motor (in the rain), and unwiring the cable to the motor from the breaker in the main switchgear and testing both the motor and the cable with various instruments looking for the grounded wire or coil that caused the motor to trip. We used a large “Megger” on the motor. It’s called a Megger because it measures Mega-Ohms. So, it’s technically called a Mega-Ohm meter. Ohms is a measurement of resistance in an electrical circuit. We usually use a small hand cranked megger, that is similar to an old hand crank telephone that generates a high voltage (good for shocking fish in a lake to make them rise to the surface). In the case of the hand cranked Megger, it would generate 1,000 volts.
A Hand cranked Megger made by the same company, only newer than the ones we had
The Megger this size would have been useless with this large motor. Instead we used one that was electric, and you ran the voltage up over 10,000 volts and watched the mega-ohms over a period of 1/2 hour or so.
For the cables, we hooked up a Hypot (or Hipot). This stands for High Potential. Potential in this case is another word for “Voltage”. It would charge up and then you pressed a button and it would send a high voltage pulse down the cable, and if there is a weak spot in the insulation,The Hypot will find it. So, we hooked a Hypot up to the cable and tried to find the grounded wire. No luck.
After spending 4 hours looking for the grounded cable or motor, we found nothing. We spent another hour and a half putting the motor and the breaker back in service. The Fan was put back into operation and we went home. As I was walking out to the car with Bill Rivers, he told me, “I knew they weren’t going to find anything wrong with that fan.” He had a big grin on his face.
At first I thought he was just making an educated guess as Rivers was apt to do on many occasions (daily). It was raining and I could see where water may have been sucked into the motor or something and had momentarily grounded the motor. Just because we didn’t find anything didn’t mean that the breaker didn’t trip for no reason.
When we were in the car and on our way to Stillwater, Oklahoma with Yvonne Taylor and Rich Litzer, Bill explained that he knew why the motor tripped. He had been walking through the main switchgear with Mike Rose, and Mike, for no apparent reason other than curiosity, had opened up the bottom door to the breaker for 1A PA fan. He looked at it for a moment and then slammed the door shut. When he did this, the breaker tripped.
So, the ground relay happened to be the one that tripped. It might as well been an over-current or a low voltage trip. It just happened to trip the ground trip. Bill said that he told Mike that he should call the Shift Supervisor and let him know so they could restart the motor. Mike on the other hand told Bill that he was already on probation and was afraid of losing his job if he reported that he had slammed the door on the breaker and tripped the fan.
If there was ever a reason to call 911, it was then. All he had to do was tell them, “I accidentally tripped the PA fan when I bumped the breaker cabinet.” They would have told him to reset the flag, and they would have started the fan right back up. No questions asked… I’m sure of it. And they wouldn’t have lost their generating capacity for the remainder of the afternoon and we wouldn’t have spent 4 hours unwiring, testing and rewiring the motor in the rain with a plastic umbrella over our head.
Bill wasn’t about to tell on Mike. If Mike didn’t want to report it, Bill wasn’t going to say anything, and I understand that. I probably would have kept it to myself at the time if I was in Bill’s shoes (I’m just glad I wasn’t because I probably wouldn’t have been able to sleep soundly for the next year). But 30 years later, I might write about it in a Blog. Even though I wouldn’t have looked to Mike to teach me much about being an electrician (he was more of an Air Condition man anyway), I still loved the guy.
Mike died almost two years ago on May 29, 2011. He was from England and had lived in Canada for a time. He used to work on trains. Trains, even though they are diesel, are really electric. The Diesel engine really runs a generator that generates electricity that runs the train. I know that Mike was a good man at heart. He loved his family with all his heart. Here is a picture of the Limey:
Mike Rose. A fair plant electrician, but a great family man!
Ok. So I know what you are thinking…. There must be a story about myself in here somewhere. Well, you would be right. First of all. I always ‘fessed up to my mistakes, as my current manager at Dell knows well (yes. I still mess up after all these years). I told my current manager the other day that CLM was my middle name. (CLM means “Career Limiting Move”). So here is my power plant “mess up” story (well one of them):
In January 1986, I returned from my Honeymoon with my new wife Kelly when I found that we had hired a new electrician. Gsry Wehunt was replacing Jim Stephenson who had left the plant on February 15, 1985, which is a story all it’s own. We had just started an overhaul on Unit 1.
I remember the first Monday I spent with Gary. It was January 6, 1986 and we were working on cleaning out the exciter house on the end of the main power generator with Diana Brien (formerly Diana Lucas). We were discussing salaries and Gary was surprised to find out that I was making more than he was. Well… I had been an electrician for over 2 years and had been promoted regularly…. so I didn’t think there was anything strange about it, except that I still looked like I was only about 18 years old (even though I was 25) and Gary was about 34. I had already been promoted 4 times and my salary had gone from $7.15 to over $12 an hour.
Anyway, when that first Friday rolled around, Gary and I were assigned to Substation Inspection. Some later time I may go into the details of what “Substation Inspection” entails, but for now, let’s just stick with my “911 call.” It is enough to say that we were in the main plant substation relay house on Friday January 10, 1986 at 9:00 am. One of our jobs was to call other substations and perform a test called a “Transfer Trip and Carrier Test”. We had called Woodring Substation (Woodring is a town in Oklahoma and we had a 345 KV line going there), and I was talking to the man in the substation on the other end of the phone line.
At the same time I was showing Gary just how experienced I was at being an electrician. People had told me that you had to be a plant electrician for 5 years before you really became a “first class” electrician. Well. Here I was at 2 years, and I thought I was so good that I could do anything by now…. — Yeah… right. I told the guy on the other end of the line as I turned a switch…. Amber light… Back to Blue…. and I wrote down the value on the meter (paperwork… oh yes…. it’s that important. Like A-1 sauce).
Then I reached for the second switch. I said, “Carrier test”, then turned the switch. The lights in the relay house went out and we were in the dark. I told the guy on the other end of the line….. “Well. That’s not supposed to happen.” Then as I let go of the switch and it returned to it’s normal position, the lights turned back on. Okay……
I wrote the numbers down from the meter and said goodbye to the other faceless substation man on the other end of the line that I talked to over 100 times, but never met in person. He sounded like a nice guy. Then I headed for the gray phone. I heard the Shift Supervisor paging Leroy Godfrey (The Electrical Supervisor) on line 2 (we had 5 Gray phone lines. The Gray Phone was our PA system).
When I picked up the line I heard Leroy pick up the phone and the Shift Supervisor tell Leroy that we just lost station power in the main substation and it had switched over to Auxiliary power. I immediately jumped in and said, “Jim (for Jim Padgett, the Shift Supervisor), I did that. I was performing a Carrier test with Woodring and the moment I performed the carrier test the lights went out.” Leroy chimed in by saying, “That wouldn’t cause you to lose station power.”
Well, in my ‘inexperienced’ plant electrician way, I responded, “Well. All I know is that when I turned the switch to perform the carrier test, the lights went out, and when I let go of the switch, the lights came back on.” Leroy reiterated, “That wouldn’t cause you to lose station power.” I replied with, “I’m just saying….” and left it at that. I had done my job. They knew I was out here. They knew I had called 911 right away. I explained what I was doing…. they could take it from there.
I had hoped that I had showed Gary upfront that it doesn’t hurt to report your mistakes (even though I hadn’t made one as far as I could tell), but I was 100% sure I had done something to cause the relay house to lose power. Though, I couldn’t figure out why.
After lunch, Bill Bennett, our A foreman came down to the shop to tell me that they figured out how the substation lost station power. He said that a road grader had been grating the road down by the Otoe-Missouri reservation (which is actually called “Windmill road” I guess because there is a windmill down that road somewhere), and had hit an electric pole and knocked it over and had killed the power to the substation.
Substation Power Interrupting Road Grader
It turned out that the substation relay house was fed by a substation down that road where we have a radio tower. So, think about this. The exact time that I turned that switch in the substation, a road grater 2 1/2 miles away hits a telephone pole accidentally and knocks it to the ground and kills the power to the substation at the exact same time that I am performing a transfer-trip and Carrier test with Woodring Substation, and the time it takes to switch to auxiliary power is the exact time it took me to let go of the switch.
Don’t tell me that was by accident. I will never believe it. I think it was for the soul purpose of teaching me a useful lesson or two. First….. don’t be afraid to tell someone when you do something wrong. Second…. If you think you have control over the things that happen to you in your life… well, think again…… Third….. God watches you every moment, and if you let him, he will guide you to do the right thing when the time comes.
God bless you all.
Power Plant Men 
That was an exciting day! Another great story. Thanks for the memories.