Runaway Fire Hydrant Leaves Power Plant in the Dark
Favorites Post #13 (posted in no particular order)
Originally posted May 17, 2014:
Don’t believe it when the Electric Company tells you that the reason your town lost electricity for an hour was because a squirrel climbed onto a transformer and shorted it out. The real reason just may be more bizarre than that and the company doesn’t want you to know all the different creative ways that power can be shut off. This is a tale of just one of those ways. So, get out your pencil and paper and take notes.
One spring day in 1993 while sitting at the Precipitator computer for Unit one at the Coal-fired Power Plant in North Central Oklahoma, while I was checking the controls to make sure all the cabinets were operating correctly, suddenly there was a distant boom, and the lights in the control room went out. The computer stayed on because it was connected to an electric panel called the VSP or Vital Services Panel, which in turn was supplied by the UPS system (Uninterruptible Power Supply). That was one of those moments where you may pause for a moment to make sure you aren’t still at home dreaming before you fly into a panic.
The Precipitator cabinets all indicated on the computer that they had just shutdown. I rose from the chair and walked around to the front of the Alarm Panel for Unit one, and found that the fluorescent lights were only out on Unit 1. The lights were still on for Unit 2. The Control Panel was lit up like a Christmas Tree with Green, Red, Blue and Yellow Lights. The Alarm Printer was spewing out paper at high speed. As the large sheets of paper were pouring out onto the floor, I watched as Pat Quiring and other brave Power Plant Control Room operators were scurrying back and forth turning switch handles, pushing buttons, and checking pressure gauges.
Just this site alone gave me confidence that everything was going to be all right. These Control Room operators were all well trained for emergencies just like this, and each person knew what their job was. No one was panicking. Everyone was concentrating on the task at hand.
Someone told me that we lost Unit 1, and the Auxiliary Power to Unit 1 at the same time. So, Unit 1 was dead in the water. This meant, no fans, no pumps, no lights, no vending machines, no cold water at the water fountain and most importantly, no hot coffee!!! I could hear steam valves on the T-G floor banging open and the loud sound of steam escaping.
I turned quickly to go to the electric shop to see what I could do there in case I was needed. I bolted out the door and down the six flights of stairs to the Turbine-Generator (T-G) basement. Exiting the stairway, and entering the T-G basement the sound was deafening. I grabbed the earplugs that were dangling around my neck and crammed them into my ears. Steam was pouring out of various pop-off valves. I ducked into the electric shop where across the room Andy Tubbs, one of the electric foreman was pulling large sheets of electric blueprints from the print cabinet and laying them across the work table that doubled as the lunch table.
When I asked Andy what happened, I learned that somehow when a crew was flushing out a fire hydrant the water somehow shot up and into the bus work in the Auxiliary Substation (that supplies backup power to the Power Plant) and it shorted out the 189,000 volt substation directly to ground. When that happened it tripped unit 1 and the auxiliary substation at the same time leaving it without power.
I will explain how a fire hydrant could possibly spray the bus work in a substation in a little while, but first let me tell you what this meant at the moment to not have any power for a Power Plant Boiler and Turbine Generator that has just tripped when it was at full load which was around 515 Megawatts of power at the time.
Normally when a unit trips, the boiler cools down as the large Force Draft (FD) Fans blow air through the boiler while the even larger Induced Draft (ID) fans suck the air from the boiler on the other end and blow the hot air up the smoke stack. This causes the steam in the boiler tubes to condense back into water. Steam valves open on the boiler that allow excessive steam to escape.
When the boiler is running there is a large orange fireball hovering in space in the middle of the boiler. The boiler water is being circulated through the boiler and the Boiler Feed Pump Turbines are pumping steam back and forth between the turbine generator and the boiler reheating the steam until every bit of heat from the boiler that can be safely harnessed is used.
When all this stop suddenly, then it is important that the large fans keep running to cool down the steam, since it is no longer losing energy in the generator as it was when it was busy supplying electricity to 1/2 million people in Oklahoma City. The power is fed to the fans from the Auxiliary substation located right outside the Main Switchgear where all the breakers reside that supply the power to the fans. Unfortunately, in this case, the Auxiliary substation was shutdown as well, leaving the boiler without any fans.
Without fans for cooling, and pumps to circulate the water, the walls of the boiler began heating up to dangerous temperatures. Steam was whistling out of pop off valves, but if the steam drum on the top of the boiler were to run dry, then the entire boiler structure could be compromised and begin melting down. — So, this was serious. Something had to be done right away. It wouldn’t be as bad as the China Syndrome since we were burning coal instead of nuclear power, but it would have caused a lot of damage nonetheless.

From the movie “The China Syndrome” where a similar emergency existed only in the movie, it was a Nuclear Plant
I have a side story about this picture, but I think I’ll save it for another post because I don’t want to digress from the main story at this point (Ok. Let me just say “Jack Maloy and Merl Wright” for those who can’t wait. See the post: “Power Plant Conspiracy Theory“).
With the prospect that the boiler might melt to the ground in a pile of rubble, it would seem that the main priority was to turn the Auxiliary Substation back on so the fans could be turned back on and prevent the boiler from collapsing. So, we walked out to the substation and looked at the switches that would have to be operated in order to first power up the main bus and then to close to supply power to the two big transformers and the six smaller transformers that supplied the Unit 1 Main Switchgear.
While inspecting the switches where the electricity had gone to ground we found that one of the main insulators was cracked.
Since this insulator was cracked, we didn’t really want to operate the switch to test if another 189,000 volts would go straight to ground again, especially since one of us would be standing right underneath it cranking the switch. So, we went back to the shop to find an alternative.
By this time the Plant Manager, Ron Kilman arrived in the shop, and understanding the urgency to find a solution asked us what were the alternatives. He was relying on our expertise to make the decision.
The other solution would be to cut the power over from Unit 2 which was still humming away pushing electricity to Oklahoma City out of the 345,000 volt substation. The cut over would be very simple because the switchgear was designed with this in mind. We analyzed the power rating on the auxiliary transformers on Unit 2 and thought that we might be cutting it close to have them running both sets of fans at the same time, especially since the full load amps of a huge fan starting up was about 10 times the normal rate.
The transformer was rated to handle the load, but consider this. What if this caused Unit 2 to trip as well. With the Auxiliary substation offline, if Unit 2 tripped, we would be in twice the amount of trouble we were currently in. What a day it would have been if that had happened and two 250 foot boilers had come crashing to the ground in a pile of rubble. After reading the power ratings on the auxiliary transformers I was thinking, “Yeah, let’s do it! These transformers can handle it.” Andy was not so eager.
So, we were left with one alternative. That was to shut the switch in the Auxiliary substation that had the cracked insulator and take our chances that it wasn’t going to short to ground and blow up over our heads. I think I was eager to close the switch for Andy, but if I remember correctly, he didn’t want me to be the one to suffer the consequences and decided to close the switch himself. Needless to say. Andy closed the switch, and nothing blew up.
As soon as the power was restored to the switchgear, the fans were powered up and the temperature in the boiler was quickly reduced. The coffee pot in the Electric Shop began heating the coffee again. The power plant was saved from a major catastrophe. That was delayed for another day… of which I will talk about later (see the post “Destruction of a Power Plant God).”
So, how exactly does a fire hydrant shoot water up into the bus work of a substation like the picture of the switch directly above? The culprit fire hydrant wasn’t in the substation, it sat alongside it outside the fence a good 50 feet from the high voltage switch. No hose was attached to the fire hydrant. It was only being flushed out as part of a yearly activity to go around and make sure the fire hydrants are all operating correctly.
Here is the story about how the squirrel climbed into the transformer this time….
George Alley, Dale Mitchell and Mickey Postman were going around to the 30,000 fire hydrants on the plant ground (ok. maybe not that many, but we did have a lot of them), and they were opening up the valves and flushing them out. That means, they were letting them run for a while to clear them out from any contaminates that may have built up over the year of not being used.
Throughout their adventure they had opened a multitude of Hydrants situated out in the fields along the long belt conveyor from the coalyard and around the two one-million gallon #2 Diesel tanks.
The brave Power Plant Men, learned that when opening a fire hydrant wide open in the middle of field had unintended consequences. It tended to wash out the ground in front of the flow of the water shooting out of the hydrant. So the team of experts devised a plan to place a board in front of the hydrant when it would be in danger of tearing a hole in the terrain. The board would divert the water into the air where it would fan out and not cause damage to the surrounding area.
This was working fine, and when they arrived at the fire hydrant next to the substation, since the stream from the hydrant was pointing directly into the substation (hmm. a design flaw, I think), they decided to prop the board up against the fence to keep from washing away the gravel in the substation. Well. When a fire hydrant is opened that hasn’t been used for a year, the first flow of water to shoot out is dark brown.
You may think that this is because the water has somehow become dirty over the past year, but that isn’t quite the case. What has happened is that the pipe has been rusting little by little and the water has become saturated with the rust. So, the water shooting out of the hydrant was full of rust (hence the need to flush them out).
Well. Rust is made of metal. Metal is conductive, especially when it is mixed with water. When the water hit the board, it was deflected into the air and happened to direct itself directly into the high voltage switch in the substation. This caused a circuit to the ground which, once it created an arc pumped all the electricity directly into the ground.
Normally when something like this happens it doesn’t trip the Main Power Transformer to a Power Plant.
This time it did. I know there was a few heads scratching trying to figure it out. I think I figured out what happened a little while later. You see… here is the rest of the story….
Once the unit was back online and the emergency was over, someone finally noticed that the telephone system couldn’t call outside of the plant. Well. I was the main telephone person at the time, so the control room called me and asked me to look into the problem.
I checked the telephone computer and it was up and running just fine. Internal calls could be made. Only any call outside just concluded with a funny humming sound. After checking the circuit in the Logic Room next to the Rolm Telephone Computer I headed for…. guess where….. the Main Switchgear….
In the middle of the main switchgear in the back of the room right next to the Auxiliary Substation beyond the back wall, the outside telephone line came into the plant. The first thing it did was go through a special Telephone Surge Protector.
In this picture above, the silver circular buttons on the left side are really an old style surge protector. whenever there was a power surge, the carbon connection in the surge protector would quickly melt causing the circuit to go straight to ground. Thus protecting the rest of the telephone circuit. So, if some kid in their house decides to connect the 120 volts circuit to the telephone for fun to see what would happen, this circuit would protect the rest of the phone circuits. Keep in mind that this was during the early 1990 when “Surge Protection” still was basically all “mechanical”.
Anyway, when I arrived at this panel and I checked the surge protector to the main line going out of the plant, guess what I found…. Yep. Shorted to ground. Luckily there were some spares that were not wired to anything in the panel and I was able to swap them out for the ones that had been destroyed. — These were a one time use. Which meant, if they ever had to short to ground, they had to be replaced.
Ok. Fine. After a little while, we were able to call back out of the plant, though there was still some residual noise on the line. It was like this… when you called out of the plant, the person on the other end sounded like they were buried in a grave somewhere and they were trying to talk to someone living just like in an episode of the Twilight Episode where a phone line landed on a grave and the dead person tried to call his long lost love from the past.
I didn’t give it much thought other than that I figured the 189,000 volt arc to ground must have shorted out the telephone line since the phone line ran directly under the auxiliary substation ground grid.
It wasn’t until the next morning when the Southwestern Bell repairman showed up at the plant. I knew him well, since he had been working on our phone lines since before the AT&T breakup in 1984. When I met him in the front of the electric shop, he said that he needed to check our telephone circuits. I told him that I knew that we had a problem because we had a high voltage short to ground yesterday and I found our surge protectors melted away.
He explained to me that not only was our circuit affected, but that every relay house from here to Ponca City was blown out. That’s when I realized that the problem was the reverse of the usual situation. What had happened was that the Ground Grid in the substation and the surrounding area (including the Unit 1 Main Power Transformer) had become hot. What do you do when the ground grid becomes charged?
The Ground Grid is what is supposed to protect you when a surge happens, but what happens when the ground grid itself is the problem? In this case, when the high voltage line about 60 feet from the telephone cable surge protector, arced to ground, it fed a tremendous amount of power back through the ground grid. when equipment detected the surge in voltage, they automatically defaulted their circuits to ground. That’s why the telephone circuit died. That’s what tripped the Main Power Transformer.
When the telephone circuit detected the high voltage surge, it shorted to ground (which was the problem), causing the high voltage to feed directly into the phone line and down the line to the next Southwestern Bell relay switch, which also defaulted to ground, trying to bleed off the surge as it went from relay switch to switch until enough of the power was able to be diverted to ground.
That day sure turned out to be a learning experience. I learned that when all the lights go out in the control room, that it is almost assured that the coffee pot in the electric shop is going to stop working. I also learned that in order to coax the plant manager to the electric shop, a major electrical tragedy is one good way. I learned that when shooting rusty water into the air don’t point it at a high voltage auxiliary substation switch. — I’m sure Mickey Postman learned that lesson too. I also learned that just like in Star Trek… whenever there is a dangerous job to do, the Captain is always the one that wants to do it. Does that make sense? Send a Peon like me in there…
I also learned something else about Power Plant Men…. You see…. People like Dale Mitchell, George Alley and Mickey Postman all are examples of incredibly wonderful Power Plant Men. When they were out there doing their duty and something tragic like this, all the Power Plant Men felt their pain. They knew that they all felt guilty for tripping the unit. It didn’t matter that a million dollars every so many minutes was walking out the door in revenue. The only thing that mattered was that these three men were safe.
Since I have left the Power Plant, I have found that the idea that the employee is the greatest asset that a company can possess is not a universal idea. You see, there was never the thought that any of these people should be fired for their mistake. On the contrary. The true Power Plant Men did whatever they could to let them know that they knew exactly how they felt. It could have happened to any of them.
Besides the friendship between Power Plant Men, one of the things I miss most about working at the Power Plant is that the employees are held in high esteem as a real asset to the company. Many could learn from their example.
Comments from the Original post
-
-
I like the mix of storytelling and information sharing you deliver here. Thanks again
-
-
-
Aug 8, 2011 I lost all reserve in and lost both units in 2 separate storms and A1 would not start for a few minutes and A2 was leaking antifreeze terribly not a scratch and back online in 34 hours both units and no one ever asked me one question even with a Safety Dept not one question asked of me the SS on duty. Not many men on the planet have ever experienced an uncontrolled total plant outage- you would of thought a learning opportunity would of took place. Feb 2011 worst winter temps in years 50 Units or more tripped in Texas let alone the trusty units of Redbud and McClain were fighting Sooner rolled right along with storm warnings for two weeks ahead – the ICS still went to Detroit for just tours of other facilities once again the fall guy Tarver McArthur stood alone. I had authored a Freeze Protection Plan for the plant and that seemed to save the day and explain to the regulatory bodies how we were online and everyone else in Texas was off and enjoying rolling blackouts in a terrible winter weather situation not to mention our powers that be were all stranded in Detroit and very few people could get to the plant without getting stuck trying to get there as well – but a few health heart issues later I am still here to tell about it all you would think folks would want to take advantage of someone that had went through the fire and Ice but thats ok I want them someday distant to get all the credit they deserve when the trumpets sound as that is truly what matters most. DT
-
-
Great story, and yes, you took exactly the proper lesson from it, and it is too bad that many of our bosses haven’t learned it.
And yup, I’ve buried a lot of electrocuted squirrels over the years.
Runaway Fire Hydrant Leaves Power Plant in the Dark
Originally posted May 17, 2014:
Don’t believe it when the Electric Company tells you that the reason your town lost electricity for an hour was because a squirrel climbed onto a transformer and shorted it out. The real reason just may be more bizarre than that and the company doesn’t want you to know all the different creative ways that power can be shut off. This is a tale of just one of those ways. So, get out your pencil and paper and take notes.
One spring day in 1993 while sitting at the Precipitator computer for Unit one at the Coal-fired Power Plant in North Central Oklahoma, while I was checking the controls to make sure all the cabinets were operating correctly, suddenly there was a distant boom, and the lights in the control room went out. The computer stayed on because it was connected to an electric panel called the VSP or Vital Services Panel, which in turn was supplied by the UPS system (Uninterruptible Power Supply). That was one of those moments where you may pause for a moment to make sure you aren’t still at home dreaming before you fly into a panic.
The Precipitator cabinets all indicated on the computer that they had just shutdown. I rose from the chair and walked around to the front of the Alarm Panel for Unit one, and found that the fluorescent lights were only out on Unit 1. The lights were still on for Unit 2. The Control Panel was lit up like a Christmas Tree with Green, Red, Blue and Yellow Lights. The Alarm Printer was spewing out paper at high speed. As the large sheets of paper were pouring out onto the floor, I watched as Pat Quiring and other brave Power Plant Control Room operators were scurrying back and forth turning switch handles, pushing buttons, and checking pressure gauges.
Just this site alone gave me confidence that everything was going to be all right. These Control Room operators were all well trained for emergencies just like this, and each person knew what their job was. No one was panicking. Everyone was concentrating on the task at hand.
Someone told me that we lost Unit 1, and the Auxiliary Power to Unit 1 at the same time. So, Unit 1 was dead in the water. This meant, no fans, no pumps, no lights, no vending machines, no cold water at the water fountain and most importantly, no hot coffee!!! I could hear steam valves on the T-G floor banging open and the loud sound of steam escaping.
I turned quickly to go to the electric shop to see what I could do there in case I was needed. I bolted out the door and down the six flights of stairs to the Turbine-Generator (T-G) basement. Exiting the stairway, and entering the T-G basement the sound was deafening. I grabbed the earplugs that were dangling around my neck and crammed them into my ears. Steam was pouring out of various pop-off valves. I ducked into the electric shop where across the room Andy Tubbs, one of the electric foreman was pulling large sheets of electric blueprints from the print cabinet and laying them across the work table that doubled as the lunch table.
When I asked Andy what happened, I learned that somehow when a crew was flushing out a fire hydrant the water somehow shot up and into the bus work in the Auxiliary Substation (that supplies backup power to the Power Plant) and it shorted out the 189,000 volt substation directly to ground. When that happened it tripped unit 1 and the auxiliary substation at the same time leaving it without power.
I will explain how a fire hydrant could possibly spray the bus work in a substation in a little while, but first let me tell you what this meant at the moment to not have any power for a Power Plant Boiler and Turbine Generator that has just tripped when it was at full load which was around 515 Megawatts of power at the time.
Normally when a unit trips, the boiler cools down as the large Force Draft (FD) Fans blow air through the boiler while the even larger Induced Draft (ID) fans suck the air from the boiler on the other end and blow the hot air up the smoke stack. This causes the steam in the boiler tubes to condense back into water. Steam valves open on the boiler that allow excessive steam to escape.
When the boiler is running there is a large orange fireball hovering in space in the middle of the boiler. The boiler water is being circulated through the boiler and the Boiler Feed Pump Turbines are pumping steam back and forth between the turbine generator and the boiler reheating the steam until every bit of heat from the boiler that can be safely harnessed is used.
When all this stop suddenly, then it is important that the large fans keep running to cool down the steam, since it is no longer losing energy in the generator as it was when it was busy supplying electricity to 1/2 million people in Oklahoma City. The power is fed to the fans from the Auxiliary substation located right outside the Main Switchgear where all the breakers reside that supply the power to the fans. Unfortunately, in this case, the Auxiliary substation was shutdown as well, leaving the boiler without any fans.
Without fans for cooling, and pumps to circulate the water, the walls of the boiler began heating up to dangerous temperatures. Steam was whistling out of pop off valves, but if the steam drum on the top of the boiler were to run dry, then the entire boiler structure could be compromised and begin melting down. — So, this was serious. Something had to be done right away. It wouldn’t be as bad as the China Syndrome since we were burning coal instead of nuclear power, but it would have caused a lot of damage nonetheless.

From the movie “The China Syndrome” where a similar emergency existed only in the movie, it was a Nuclear Plant
I have a side story about this picture, but I think I’ll save it for another post because I don’t want to digress from the main story at this point (Ok. Let me just say “Jack Maloy and Merl Wright” for those who can’t wait) See the post: “Power Plant Conspiracy Theory“.
With the prospect that the boiler might melt to the ground in a pile of rubble, it would seem that the main priority was to turn the Auxiliary Substation back on so the fans could be turned back on and prevent the boiler from collapsing. So, we walked out to the substation and looked at the switches that would have to be operated in order to first power up the main bus and then to close to supply power to the two big transformers and the six smaller transformers that supplied the Unit 1 Main Switchgear.
While inspecting the switches where the electricity had gone to ground we found that one of the main insulators was cracked.
Since this insulator was cracked, we didn’t really want to operate the switch to test if another 189,000 volts would go straight to ground again, especially since one of us would be standing right underneath it cranking the switch. So, we went back to the shop to find an alternative.
By this time the Plant Manager, Ron Kilman arrived in the shop, and understanding the urgency to find a solution asked us what were the alternatives. He was relying on our expertise to make the decision.
The other solution would be to cut the power over from Unit 2 which was still humming away pushing electricity to Oklahoma City out of the 345,000 volt substation. The cut over would be very simple because the switchgear was designed with this in mind. We analyzed the power rating on the auxiliary transformers on Unit 2 and thought that we might be cutting it close to have them running both sets of fans at the same time, especially since the full load amps of a huge fan starting up was about 10 times the normal rate.
The transformer was rated to handle the load, but consider this. What if this caused Unit 2 to trip as well. With the Auxiliary substation offline, if Unit 2 tripped, we would be in twice the amount of trouble we were currently in. What a day it would have been if that had happened and two 250 foot boilers had come crashing to the ground in a pile of rubble. After reading the power ratings on the auxiliary transformers I was thinking, “Yeah, let’s do it! These transformers can handle it.” Andy was not so eager.
So, we were left with one alternative. That was to shut the switch in the Auxiliary substation that had the cracked insulator and take our chances that it wasn’t going to short to ground and blow up over our heads. I think I was eager to close the switch for Andy, but if I remember correctly, he didn’t want me to be the one to suffer the consequences and decided to close the switch himself. Needless to say. Andy closed the switch, and nothing blew up.
As soon as the power was restored to the switchgear, the fans were powered up and the temperature in the boiler was quickly reduced. The coffee pot in the Electric Shop began heating the coffee again. The power plant was saved from a major catastrophe. That was delayed for another day… of which I will talk about later (see the post “Destruction of a Power Plant God).”
So, how exactly does a fire hydrant shoot water up into the bus work of a substation like the picture of the switch directly above? The culprit fire hydrant wasn’t in the substation, it sat alongside it outside the fence a good 50 feet from the high voltage switch. No hose was attached to the fire hydrant. It was only being flushed out as part of a yearly activity to go around and make sure the fire hydrants are all operating correctly.
Here is the story about how the squirrel climbed into the transformer this time….
George Alley, Dale Mitchell and Mickey Postman were going around to the 30,000 fire hydrants on the plant ground (ok. maybe not that many, but we did have a lot of them), and they were opening up the valves and flushing them out. That means, they were letting them run for a while to clear them out from any contaminates that may have built up over the year of not being used.
Throughout their adventure they had opened a multitude of Hydrants situated out in the fields along the long belt conveyor from the coalyard and around the two one-million gallon #2 Diesel tanks.
The brave Power Plant Men, learned that when opening a fire hydrant wide open in the middle of field had unintended consequences. It tended to wash out the ground in front of the flow of the water shooting out of the hydrant. So the team of experts devised a plan to place a board in front of the hydrant when it would be in danger of tearing a hole in the terrain. The board would divert the water into the air where it would fan out and not cause damage to the surrounding area.
This was working fine, and when they arrived at the fire hydrant next to the substation, since the stream from the hydrant was pointing directly into the substation (hmm. a design flaw, I think), they decided to prop the board up against the fence to keep from washing away the gravel in the substation. Well. When a fire hydrant is opened that hasn’t been used for a year, the first flow of water to shoot out is dark brown.
You may think that this is because the water has somehow become dirty over the past year, but that isn’t quite the case. What has happened is that the pipe has been rusting little by little and the water has become saturated with the rust. So, the water shooting out of the hydrant was full of rust (hence the need to flush them out).
Well. Rust is made of metal. Metal is conductive, especially when it is mixed with water. When the water hit the board, it was deflected into the air and happened to direct itself directly into the high voltage switch in the substation. This caused a circuit to the ground which, once it created an arc pumped all the electricity directly into the ground.
Normally when something like this happens it doesn’t trip the Main Power Transformer to a Power Plant.
This time it did. I know there was a few heads scratching trying to figure it out. I think I figured out what happened a little while later. You see… here is the rest of the story….
Once the unit was back online and the emergency was over, someone finally noticed that the telephone system couldn’t call outside of the plant. Well. I was the main telephone person at the time, so the control room called me and asked me to look into the problem.
I checked the telephone computer and it was up and running just fine. Internal calls could be made. Only any call outside just concluded with a funny humming sound. After checking the circuit in the Logic Room next to the Rolm Telephone Computer I headed for…. guess where….. the Main Switchgear….
In the middle of the main switchgear in the back of the room right next to the Auxiliary Substation beyond the back wall, the outside telephone line came into the plant. The first thing it did was go through a special Telephone Surge Protector.
In this picture above, the silver circular buttons on the left side are really an old style surge protector. whenever there was a power surge, the carbon connection in the surge protector would quickly melt causing the circuit to go straight to ground. Thus protecting the rest of the telephone circuit. So, if some kid in their house decides to connect the 120 volts circuit to the telephone for fun to see what would happen, this circuit would protect the rest of the phone circuits. Keep in mind that this was during the early 1990 when “Surge Protection” still was basically all “mechanical”.
Anyway, when I arrived at this panel and I checked the surge protector to the main line going out of the plant, guess what I found…. Yep. Shorted to ground. Luckily there were some spares that were not wired to anything in the panel and I was able to swap them out for the ones that had been destroyed. — These were a one time use. Which meant, if they ever had to short to ground, they had to be replaced.
Ok. Fine. After a little while, we were able to call back out of the plant, though there was still some residual noise on the line. It was like this… when you called out of the plant, the person on the other end sounded like they were buried in a grave somewhere and they were trying to talk to someone living just like in an episode of the Twilight Episode where a phone line landed on a grave and the dead person tried to call his long lost love from the past.
I didn’t give it much thought other than that I figured the 189,000 volt arc to ground must have shorted out the telephone line since the phone line ran directly under the auxiliary substation ground grid.
It wasn’t until the next morning when the Southwestern Bell repairman showed up at the plant. I knew him well, since he had been working on our phone lines since before the AT&T breakup in 1984. When I met him in the front of the electric shop, he said that he needed to check our telephone circuits. I told him that I knew that we had a problem because we had a high voltage short to ground yesterday and I found our surge protectors melted away.
He explained to me that not only was our circuit affected, but that every relay house from here to Ponca City was blown out. That’s when I realized that the problem was the reverse of the usual situation. What had happened was that the Ground Grid in the substation and the surrounding area (including the Unit 1 Main Power Transformer) had become hot. What do you do when the ground grid becomes charged?
The Ground Grid is what is supposed to protect you when a surge happens, but what happens when the ground grid itself is the problem? In this case, when the high voltage line about 60 feet from the telephone cable surge protector, arced to ground, it fed a tremendous amount of power back through the ground grid. when equipment detected the surge in voltage, they automatically defaulted their circuits to ground. That’s why the telephone circuit died. That’s what tripped the Main Power Transformer.
When the telephone circuit detected the high voltage surge, it shorted to ground (which was the problem), causing the high voltage to feed directly into the phone line and down the line to the next Southwestern Bell relay switch, which also defaulted to ground, trying to bleed off the surge as it went from relay switch to switch until enough of the power was able to be diverted to ground.
That day sure turned out to be a learning experience. I learned that when all the lights go out in the control room, that it is almost assured that the coffee pot in the electric shop is going to stop working. I also learned that in order to coax the plant manager to the electric shop, a major electrical tragedy is one good way. I learned that when shooting rusty water into the air don’t point it at a high voltage auxiliary substation switch. — I’m sure Mickey Postman learned that lesson too. I also learned that just like in Star Trek… whenever there is a dangerous job to do, the Captain is always the one that wants to do it. Does that make sense? Send a Peon like me in there…
I also learned something else about Power Plant Men…. You see…. People like Dale Mitchell, George Alley and Mickey Postman all are examples of incredibly wonderful Power Plant Men. When they were out there doing their duty and something tragic like this, all the Power Plant Men felt their pain. They knew that they all felt guilty for tripping the unit. It didn’t matter that a million dollars every so many minutes was walking out the door in revenue. The only thing that mattered was that these three men were safe.
Since I have left the Power Plant, I have found that the idea that the employee is the greatest asset that a company can possess is not a universal idea. You see, there was never the thought that any of these people should be fired for their mistake. On the contrary. The true Power Plant Men did whatever they could to let them know that they knew exactly how they felt. It could have happened to any of them.
Besides the friendship between Power Plant Men, one of the things I miss most about working at the Power Plant is that the employees are held in high esteem as a real asset to the company. Many could learn from their example.
Comments from the Original post
-
-
That was an exciting day! Another great story. Thanks for the memories.
-
-
-
I like the mix of storytelling and information sharing you deliver here. Thanks again
-
-
-
Aug 8, 2011 I lost all reserve in and lost both units in 2 separate storms and A1 would not start for a few minutes and A2 was leaking antifreeze terribly not a scratch and back online in 34 hours both units and no one ever asked me one question even with a Safety Dept not one question asked of me the SS on duty. Not many men on the planet have ever experienced an uncontrolled total plant outage- you would of thought a learning opportunity would of took place. Feb 2011 worst winter temps in years 50 Units or more tripped in Texas let alone the trusty units of Redbud and McClain were fighting Sooner rolled right along with storm warnings for two weeks ahead – the ICS still went to Detroit for just tours of other facilities once again the fall guy Tarver McArthur stood alone. I had authored a Freeze Protection Plan for the plant and that seemed to save the day and explain to the regulatory bodies how we were online and everyone else in Texas was off and enjoying rolling blackouts in a terrible winter weather situation not to mention our powers that be were all stranded in Detroit and very few people could get to the plant without getting stuck trying to get there as well – but a few health heart issues later I am still here to tell about it all you would think folks would want to take advantage of someone that had went through the fire and Ice but thats ok I want them someday distant to get all the credit they deserve when the trumpets sound as that is truly what matters most. DT
-
-
Great story, and yes, you took exactly the proper lesson from it, and it is too bad that many of our bosses haven’t learned it.
And yup, I’ve buried a lot of electrocuted squirrels over the years.
Turning the Tables on a Power Plant Telephone Interloper
Originally posted June 13, 2014:
When discussing Telephones at the Coal-Fired Power Plant in North Central Oklahoma, I have to remember that some of my readers have a completely different perspective of telephones than me. My children grew up probably never seeing a real rotary dial phone except in movies or old TV shows. It might be a little hard for them to imagine a telephone being a possible murder weapon. Telephones have come a long way since I grew up in the 1960’s and 70’s.
When you turned the dial on a Rotary phone you put your finger in the hole on the number you want to dial and then you swing it around until your finger bumps up against the metal bracket. When you pull your finger out of the hole, the phone sends a rapid succession of pulses to the telephone company telling them what number you just dialed. It was very… well…. tedious and manual…. and not even electronic. It was electric signals and switches. “Mechanical” is the word I think I’m trying to say.
Even the way you received a dial tone was by sending something called a “Ring-to-ground” signal to the telephone company. That would happen when you would lift the receiver off the hook. There are only two wires used to communicate in an old phone and only one of those had voltage on it. when you ground that wire (called the “Ring”) momentarily, the phone company would then send a dial tone to your phone.
You could actually do this on a dead phone line at times when the phone company had shut off your service. On an old pay phone, when the proper coin was inserted in the phone, the coin itself was used to ground the ring wire, thus telling the telephone company to send the dial tone, allowing you to use the phone. In 1983 there was a movie called “Wargames”.
I had learned about how these telephones worked from Bill Rivers just before going to watch this movie. During the movie Matthew Broderick’s character needed to make a phone call at a pay phone but didn’t have a coin. By taking the mouthpiece off of the transmitter, and using a metal pop top he found on the ground, he was able to ground the “ring” wire to the pay phone, and he received a dial tone. There was a good ol’ boy sitting behind me in the movie theater that said, “You can’t do that!” — Being the newly educated smart (-alec) guy I was, I turned around and said, “Yeah. You really can.”
Anyway. This isn’t a story so much about how old phones work. I just wanted to bring the younger readers back-to-date on phones since now they don’t really call them telephones anymore. It is more like, “Smart Phone” and “Cell Phone”, “Mobile Phone” or just “Phone”. The phone in the house isn’t even referred to as a telephone. We now call them “Home Phone” to distinguish them from the actual phones that we use.
Anyway, when I joined the electric shop in 1983, I learned about the phone system. We didn’t use the older Rotary Dial phones at the plant. We were one step up. We had “Touch Tone” Phones.
As I have mentioned in previous posts, we had our own telephone computer at the Power Plant. It was called a ROLM phone system. See the post “A Slap In the Face at a Gas-fired Power Plant“.
To give you an idea of the technology used by this phone system, you connected to it using a “teletype” terminal that you connected to a telephone by clipping the receiver in a cradle. Then you dialed the phone computer. When you connected, it was at 300 Baud. Think of 300 bytes per second, only using audio…. like a fax machine. — It was like connecting using a modem. 300 baud meant that when it typed out the results on the paper that scrolled out the top, you could watch it as it slowly printed out each line. The maximum speed of the terminal was 300 baud.

This is the TI Silent 700 Terminal. We used this exact model of Teletype terminal at the Power Plant
In this picture you can see the cradle in the back where the phone receiver would fit in those two rubber cups.
After many years of going to the lab to connect to the telephone computer to make changes and to monitor the telephone traffic, in 1992 I decided to bring my 8088 computer to work and set it on the desk in the electric shop. We didn’t have our own computer yet. At that time the only people that had computers were office workers and the Shift Supervisor. We had started a computer club and having a computer in the shop was a big help. I had just replaced this computer at home with a 486.

This is a Leading Edge computer. My father had this one. An earlier version than the 8088 that I was using.
I had a modem on my computer, so I tried connecting to the telephone computer, and it worked! So, sometimes during lunch when Charles Foster and I were sitting there talking about movies we had seen while eating vegetables from his garden, I would connect to the ROLM computer and just watch the call log. I could see whenever someone was dialing in and out of the plant.
We had a special call in number into the plant that allowed you to make “trunk” calls. This is another term you don’t hear much anymore. You see….. for the younger readers (again)…. long distant calls used to cost a lot of money. You would be charged by how many minutes you were on the call. During the day, it could be as high as $3.00 a minute to call across the country. Amazing huh? Because today, most of you with cell phones and even your land lines (which are rarely real land lines anymore) long distant phone calls are now free with your phone plan.
Yeah, if you wanted to call someone in the next town over, you would have to pay a fee for every minute you were on the call…. That was when AT&T had a monopoly on the phone lines in the United States. Sure, you only payed $7.00 each month for your phone, but you could only call people in your immediate area or you would be charged extra.
A Trunk line gave you access to a much wider area. The Electric company had a trunk line that gave them access to most of Oklahoma. You could dial into a local number that would connect you to the company phone system. Then after entering the correct password number, you could dial access numbers that would take you to another office location in the electric company. Once on that phone system, you could dial to get an outside line, and then dial a local number in that area.
Our plant had three access numbers that allowed you to dial out locally to Stillwater, Ponca City and Pawnee. This was useful when a foreman needed to call people out to work. They could dial into the plant, then back out to one of these other towns and then dial the local phone number of the crew member they were trying to reach without incurring a personal charge on their phone line.
So, here I was in 1992 during lunch watching the phone traffic in and out of the plant (not exactly NSA style, but sort of), when I saw something unexpected. A long string of numbers showed up. Someone had dialed in on the Stillwater trunk, then dialed out to the Corporate Headquarters trunk, then out to Oklahoma City and from there they placed a long distance call to a phone number in the same area code. The prefix on the phone number was familiar to me. It was a Ponca City phone number. I had lived in Ponca City for three years when I had just been married, from 1986 to 1989. I knew a Ponca City phone number when I saw one.
I thought this was odd, because it wouldn’t be normal for someone to dial from Stillwater through our plant to Oklahoma City only to call a Ponca City phone number when they could have dialed the local Ponca City access code. Then they wouldn’t have had to make a long distance call which bypassed our trunk call system causing the electric company to be billed for the long distance telephone call.
At the time I was a CompuServe user. This was when the World Wide Web was in it’s infancy. I was still using a DOS computer. When I connected to the Internet, it was either by using my dad’s Internet account from Oklahoma State University where I would use Telnet to access a bunch of mainframe computers all over the country, or I would use the DOS-based version of CompuServe. CompuServe was the king of Internet access before America Online came around and seemingly overnight made CompuServe obsolete.
In 1992, CompuServe had a service where you could look up phone numbers and find out whose number it was. Imagine that! Yeah. That was one of the neatest features on CompuServe! That and getting stock quotes. — Like I said…. There was no “www.whitepages.com” online. The only catch to using the reverse phone number feature, was that it was like making a long distance call. It cost money. You were charged by the minute for using the CompuServe reverse telephone number service, with the least amount being a dollar.
So, I bit the bullet and accessed the Phone Number lookup section of CompuServe. I quickly typed in the number. When the name and address of the user popped up, I quickly hit “Print Screen”, and then exited the service. My fee came to $1.00, but at least I knew what number had been dialed in Ponca City.
Charles, Scott Hubbard and I were a little excited by the time Terry Blevins walked into the electric shop office after lunch was over, I told him what I had seen.
When I told Terry the name of the person that had received the long distance call, he recognized the name right away. When I gave him the address, he was sure he knew who it was. The phone number belonged to the Music Director at the Ponca City High School. His son was attending college in Stillwater, Oklahoma.
Well, that sort of cinched it. We had a pretty good idea who had made the call. It was a college student calling home, who had been given the phone number most likely by a fellow student who knew the code to call home in Oklahoma City. So, the only local access code this guy knew was how to dial through our plant to Oklahoma City and back out where he was free (but it was not free for us) to make a long distance call home.
Armed with this knowledge, I headed up to the front office. I went straight to the Plant Manager, Ron Kilman’s office. I told Ron what I had found. I explained in detail how the person had dialed from Stillwater into our plant and then to Oklahoma City and out and then placed a long distance call to Ponca City leaving us with the phone bill. Since it was the middle of the day, the cost of a long distance call was not cheap.
I told Ron that I had used CompuServe to lookup the phone number and that Terry had said that it belonged to the Music Director at the Ponca City High School and that he had a son in college in Stillwater. I was all ready to pounce on this guy. This was a fraudulent use of the telephone service and there were some pretty strict laws then about stealing long distance from someone else.
Ron, being the more level-headed of the two of us thought about it for a minute and said, “What would be the best way to stop this from happening?” — Oh. Well. I was so intent on catching the culprit, I hadn’t thought about that angle…. “Well….” I said, “We could change the pass code used to log into our phone system. We would just have to tell our supervisors what the new number is.”
Ron asked me what it would take to do that. I told him I could do it in two minutes. We quickly settled on a new 4 digit pass code and I left his office and returned to the electric shop and made the change essentially turning the tables on the Telephone Interloper. I suppose the college student in Stillwater was lucky that our plant manager at the time was the type to forgive and forget.
Three years later the entire electric company phone system was replaced by a new AT&T computer which was managed by AT&T. As you can tell… Technology just keeps moving forward making seemingly really neat new inventions quickly obsolete.
Comments from the original post:
-
-
I still stand in awe at all the talent we had at the plant- never has one place had so many guys of remarkable skill and overall just good people and kind hearted.
-
-
I can’t agree with you more! We had the cream of the crop for sure.
-
-
-
-
Your memory still amazes me. I don’t remember that at all.
I’ll bet most young people today don’t know why we say to “dial” a phone number!-
It was just a moment in your busy day. It was the highlight of my week.
-
Runaway Fire Hydrant Leaves Power Plant in the Dark
Originally posted May 17, 2014:
Don’t believe it when the Electric Company tells you that the reason your town lost electricity for an hour was because a squirrel climbed onto a transformer and shorted it out. The real reason just may be more bizarre than that and the company doesn’t want you to know all the different creative ways that power can be shut off. This is a tale of just one of those ways. So, get out your pencil and paper and take notes.
One spring day in 1993 while sitting at the Precipitator computer for Unit one at the Coal-fired Power Plant in North Central Oklahoma, while I was checking the controls to make sure all the cabinets were operating correctly, suddenly there was a distant boom, and the lights in the control room went out. The computer stayed on because it was connected to an electric panel called the VSP or Vital Services Panel, which in turn was supplied by the UPS system (Uninterruptible Power Supply). That was one of those moments where you may pause for a moment to make sure you aren’t still at home dreaming before you fly into a panic.
The Precipitator cabinets all indicated on the computer that they had just shutdown. I rose from the chair and walked around to the front of the Alarm Panel for Unit one, and found that the fluorescent lights were only out on Unit 1. The lights were still on for Unit 2. The Control Panel was lit up like a Christmas Tree with Green, Red, Blue and Yellow Lights. The Alarm Printer was spewing out paper at high speed. As the large sheets of paper were pouring out onto the floor, I watched as Pat Quiring and other brave Power Plant Control Room operators were scurrying back and forth turning switch handles, pushing buttons, and checking pressure gauges.
Just this site alone gave me confidence that everything was going to be all right. These Control Room operators were all well trained for emergencies just like this, and each person knew what their job was. No one was panicking. Everyone was concentrating on the task at hand.
Someone told me that we lost Unit 1, and the Auxiliary Power to Unit 1 at the same time. So, Unit 1 was dead in the water. This meant, no fans, no pumps, no lights, no vending machines, no cold water at the water fountain and most importantly, no hot coffee!!! I could hear steam valves on the T-G floor banging open and the loud sound of steam escaping.
I turned quickly to go to the electric shop to see what I could do there in case I was needed. I bolted out the door and down the six flights of stairs to the Turbine-Generator (T-G) basement. Exiting the stairway, and entering the T-G basement the sound was deafening. I grabbed the earplugs that were dangling around my neck and crammed them into my ears. Steam was pouring out of various pop-off valves. I ducked into the electric shop where across the room Andy Tubbs, one of the electric foreman was pulling large sheets of electric blueprints from the print cabinet and laying them across the work table that doubled as the lunch table.
When I asked Andy what happened, I learned that somehow when a crew was flushing out a fire hydrant the water somehow shot up and into the bus work in the Auxiliary Substation (that supplies backup power to the Power Plant) and it shorted out the 189,000 volt substation directly to ground. When that happened it tripped unit 1 and the auxiliary substation at the same time leaving it without power.
I will explain how a fire hydrant could possibly spray the bus work in a substation in a little while, but first let me tell you what this meant at the moment to not have any power for a Power Plant Boiler and Turbine Generator that has just tripped when it was at full load which was around 515 Megawatts of power at the time.
Normally when a unit trips, the boiler cools down as the large Force Draft (FD) Fans blow air through the boiler while the even larger Induced Draft (ID) fans suck the air from the boiler on the other end and blow the hot air up the smoke stack. This causes the steam in the boiler tubes to condense back into water. Steam valves open on the boiler that allow excessive steam to escape.
When the boiler is running there is a large orange fireball hovering in space in the middle of the boiler. The boiler water is being circulated through the boiler and the Boiler Feed Pump Turbines are pumping steam back and forth between the turbine generator and the boiler reheating the steam until every bit of heat from the boiler that can be safely harnessed is used.
When all this stop suddenly, then it is important that the large fans keep running to cool down the steam, since it is no longer losing energy in the generator as it was when it was busy supplying electricity to 1/2 million people in Oklahoma City. The power is fed to the fans from the Auxiliary substation located right outside the Main Switchgear where all the breakers reside that supply the power to the fans. Unfortunately, in this case, the Auxiliary substation was shutdown as well, leaving the boiler without any fans.
Without fans for cooling, and pumps to circulate the water, the walls of the boiler began heating up to dangerous temperatures. Steam was whistling out of pop off valves, but if the steam drum on the top of the boiler were to run dry, then the entire boiler structure could be compromised and begin melting down. — So, this was serious. Something had to be done right away. It wouldn’t be as bad as the China Syndrome since we were burning coal instead of nuclear power, but it would have caused a lot of damage nonetheless.

From the movie “The China Syndrome” where a similar emergency existed only in the movie, it was a Nuclear Plant
I have a side story about this picture, but I think I’ll save it for another post because I don’t want to digress from the main story at this point (Ok. Let me just say “Jack Maloy and Merl Wright” for those who can’t wait) See the post: “Power Plant Conspiracy Theory“.
With the prospect that the boiler might melt to the ground in a pile of rubble, it would seem that the main priority was to turn the Auxiliary Substation back on so the fans could be turned back on and prevent the boiler from collapsing. So, we walked out to the substation and looked at the switches that would have to be operated in order to first power up the main bus and then to close to supply power to the two big transformers and the six smaller transformers that supplied the Unit 1 Main Switchgear.
While inspecting the switches where the electricity had gone to ground we found that one of the main insulators was cracked.
Since this insulator was cracked, we didn’t really want to operate the switch to test if another 189,000 volts would go straight to ground again, especially since one of us would be standing right underneath it cranking the switch. So, we went back to the shop to find an alternative.
By this time the Plant Manager, Ron Kilman arrived in the shop, and understanding the urgency to find a solution asked us what were the alternatives. He was relying on our expertise to make the decision.
The other solution would be to cut the power over from Unit 2 which was still humming away pushing electricity to Oklahoma City out of the 345,000 volt substation. The cut over would be very simple because the switchgear was designed with this in mind. We analyzed the power rating on the auxiliary transformers on Unit 2 and thought that we might be cutting it close to have them running both sets of fans at the same time, especially since the full load amps of a huge fan starting up was about 10 times the normal rate.
The transformer was rated to handle the load, but consider this. What if this caused Unit 2 to trip as well. With the Auxiliary substation offline, if Unit 2 tripped, we would be in twice the amount of trouble we were currently in. What a day it would have been if that had happened and two 250 foot boilers had come crashing to the ground in a pile of rubble. After reading the power ratings on the auxiliary transformers I was thinking, “Yeah, let’s do it! These transformers can handle it.” Andy was not so eager.
So, we were left with one alternative. That was to shut the switch in the Auxiliary substation that had the cracked insulator and take our chances that it wasn’t going to short to ground and blow up over our heads. I think I was eager to close the switch for Andy, but if I remember correctly, he didn’t want me to be the one to suffer the consequences and decided to close the switch himself. Needless to say. Andy closed the switch, and nothing blew up.
As soon as the power was restored to the switchgear, the fans were powered up and the temperature in the boiler was quickly reduced. The coffee pot in the Electric Shop began heating the coffee again. The power plant was saved from a major catastrophe. That was delayed for another day… of which I will talk about later (see the post “Destruction of a Power Plant God).”
So, how exactly does a fire hydrant shoot water up into the bus work of a substation like the picture of the switch directly above? The culprit fire hydrant wasn’t in the substation, it sat alongside it outside the fence a good 50 feet from the high voltage switch. No hose was attached to the fire hydrant. It was only being flushed out as part of a yearly activity to go around and make sure the fire hydrants are all operating correctly.
Here is the story about how the squirrel climbed into the transformer this time….
George Alley, Dale Mitchell and Mickey Postman were going around to the 30,000 fire hydrants on the plant ground (ok. maybe not that many, but we did have a lot of them), and they were opening up the valves and flushing them out. That means, they were letting them run for a while to clear them out from any contaminates that may have built up over the year of not being used.
Throughout their adventure they had opened a multitude of Hydrants situated out in the fields along the long belt conveyor from the coalyard and around the two one-million gallon #2 Diesel tanks.
The brave Power Plant Men, learned that when opening a fire hydrant wide open in the middle of field had unintended consequences. It tended to wash out the ground in front of the flow of the water shooting out of the hydrant. So the team of experts devised a plan to place a board in front of the hydrant when it would be in danger of tearing a hole in the terrain. The board would divert the water into the air where it would fan out and not cause damage to the surrounding area.
This was working fine, and when they arrived at the fire hydrant next to the substation, since the stream from the hydrant was pointing directly into the substation (hmm. a design flaw, I think), they decided to prop the board up against the fence to keep from washing away the gravel in the substation. Well. When a fire hydrant is opened that hasn’t been used for a year, the first flow of water to shoot out is dark brown.
You may think that this is because the water has somehow become dirty over the past year, but that isn’t quite the case. What has happened is that the pipe has been rusting little by little and the water has become saturated with the rust. So, the water shooting out of the hydrant was full of rust (hence the need to flush them out).
Well. Rust is made of metal. Metal is conductive, especially when it is mixed with water. When the water hit the board, it was deflected into the air and happened to direct itself directly into the high voltage switch in the substation. This caused a circuit to the ground which, once it created an arc pumped all the electricity directly into the ground.
Normally when something like this happens it doesn’t trip the Main Power Transformer to a Power Plant.
This time it did. I know there was a few heads scratching trying to figure it out. I think I figured out what happened a little while later. You see… here is the rest of the story….
Once the unit was back online and the emergency was over, someone finally noticed that the telephone system couldn’t call outside of the plant. Well. I was the main telephone person at the time, so the control room called me and asked me to look into the problem.
I checked the telephone computer and it was up and running just fine. Internal calls could be made. Only any call outside just concluded with a funny humming sound. After checking the circuit in the Logic Room next to the Rolm Telephone Computer I headed for…. guess where….. the Main Switchgear….
In the middle of the main switchgear in the back of the room right next to the Auxiliary Substation beyond the back wall, the outside telephone line came into the plant. The first thing it did was go through a special Telephone Surge Protector.
In this picture above, the silver circular buttons on the left side are really an old style surge protector. whenever there was a power surge, the carbon connection in the surge protector would quickly melt causing the circuit to go straight to ground. Thus protecting the rest of the telephone circuit. So, if some kid in their house decides to connect the 120 volts circuit to the telephone for fun to see what would happen, this circuit would protect the rest of the phone circuits. Keep in mind that this was during the early 1990 when “Surge Protection” still was basically all “mechanical”.
Anyway, when I arrived at this panel and I checked the surge protector to the main line going out of the plant, guess what I found…. Yep. Shorted to ground. Luckily there were some spares that were not wired to anything in the panel and I was able to swap them out for the ones that had been destroyed. — These were a one time use. Which meant, if they ever had to short to ground, they had to be replaced.
Ok. Fine. After a little while, we were able to call back out of the plant, though there was still some residual noise on the line. It was like this… when you called out of the plant, the person on the other end sounded like they were buried in a grave somewhere and they were trying to talk to someone living just like in an episode of the Twilight Episode where a phone line landed on a grave and the dead person tried to call his long lost love from the past.
I didn’t give it much thought other than that I figured the 189,000 volt arc to ground must have shorted out the telephone line since the phone line ran directly under the auxiliary substation ground grid.
It wasn’t until the next morning when the Southwestern Bell repairman showed up at the plant. I knew him well, since he had been working on our phone lines since before the AT&T breakup in 1984. When I met him in the front of the electric shop, he said that he needed to check our telephone circuits. I told him that I knew that we had a problem because we had a high voltage short to ground yesterday and I found our surge protectors melted away.
He explained to me that not only was our circuit affected, but that every relay house from here to Ponca City was blown out. That’s when I realized that the problem was the reverse of the usual situation. What had happened was that the Ground Grid in the substation and the surrounding area (including the Unit 1 Main Power Transformer) had become hot. What do you do when the ground grid becomes charged?
The Ground Grid is what is supposed to protect you when a surge happens, but what happens when the ground grid itself is the problem? In this case, when the high voltage line about 60 feet from the telephone cable surge protector, arced to ground, it fed a tremendous amount of power back through the ground grid. when equipment detected the surge in voltage, they automatically defaulted their circuits to ground. That’s why the telephone circuit died. That’s what tripped the Main Power Transformer.
When the telephone circuit detected the high voltage surge, it shorted to ground (which was the problem), causing the high voltage to feed directly into the phone line and down the line to the next Southwestern Bell relay switch, which also defaulted to ground, trying to bleed off the surge as it went from relay switch to switch until enough of the power was able to be diverted to ground.
That day sure turned out to be a learning experience. I learned that when all the lights go out in the control room, that it is almost assured that the coffee pot in the electric shop is going to stop working. I also learned that in order to coax the plant manager to the electric shop, a major electrical tragedy is one good way. I learned that when shooting rusty water into the air don’t point it at a high voltage auxiliary substation switch. — I’m sure Mickey Postman learned that lesson too. I also learned that just like in Star Trek… whenever there is a dangerous job to do, the Captain is always the one that wants to do it. Does that make sense? Send a Peon like me in there…
I also learned something else about Power Plant Men…. You see…. People like Dale Mitchell, George Alley and Mickey Postman all are examples of incredibly wonderful Power Plant Men. When they were out there doing their duty and something tragic like this, all the Power Plant Men felt their pain. They knew that they all felt guilty for tripping the unit. It didn’t matter that a million dollars every so many minutes was walking out the door in revenue. The only thing that mattered was that these three men were safe.
Since I have left the Power Plant, I have found that the idea that the employee is the greatest asset that a company can possess is not a universal idea. You see, there was never the thought that any of these people should be fired for their mistake. On the contrary. The true Power Plant Men did whatever they could to let them know that they knew exactly how they felt. It could have happened to any of them.
Besides the friendship between Power Plant Men, one of the things I miss most about working at the Power Plant is that the employees are held in high esteem as a real asset to the company. Many could learn from their example.
Comments from the Original post
-
-
That was an exciting day! Another great story. Thanks for the memories.
-
-
-
I like the mix of storytelling and information sharing you deliver here. Thanks again
-
-
-
Aug 8, 2011 I lost all reserve in and lost both units in 2 separate storms and A1 would not start for a few minutes and A2 was leaking antifreeze terribly not a scratch and back online in 34 hours both units and no one ever asked me one question even with a Safety Dept not one question asked of me the SS on duty. Not many men on the planet have ever experienced an uncontrolled total plant outage- you would of thought a learning opportunity would of took place. Feb 2011 worst winter temps in years 50 Units or more tripped in Texas let alone the trusty units of Redbud and McClain were fighting Sooner rolled right along with storm warnings for two weeks ahead – the ICS still went to Detroit for just tours of other facilities once again the fall guy Tarver McArthur stood alone. I had authored a Freeze Protection Plan for the plant and that seemed to save the day and explain to the regulatory bodies how we were online and everyone else in Texas was off and enjoying rolling blackouts in a terrible winter weather situation not to mention our powers that be were all stranded in Detroit and very few people could get to the plant without getting stuck trying to get there as well – but a few health heart issues later I am still here to tell about it all you would think folks would want to take advantage of someone that had went through the fire and Ice but thats ok I want them someday distant to get all the credit they deserve when the trumpets sound as that is truly what matters most. DT
-
-
Great story, and yes, you took exactly the proper lesson from it, and it is too bad that many of our bosses haven’t learned it.
And yup, I’ve buried a lot of electrocuted squirrels over the years.
Turning the Tables on a Power Plant Telephone Interloper
Originally posted June 13, 2014:
When discussing Telephones at the Coal-Fired Power Plant in North Central Oklahoma, I have to remember that some of my readers have a completely different perspective of telephones than me. My children grew up probably never seeing a real rotary dial phone except in movies or old TV shows. It might be a little hard for them to imagine a telephone being a possible murder weapon. Telephones have come a long way since I grew up in the 1960’s and 70’s.
When you turned the dial on a Rotary phone you put your finger in the hole on the number you want to dial and then you swing it around until your finger bumps up against the metal bracket. When you pull your finger out of the hole, the phone sends a rapid succession of pulses to the telephone company telling them what number you just dialed. It was very… well…. tedious and manual…. and not even electronic. It was electric signals and switches. “Mechanical” is the word I think I’m trying to say.
Even the way you received a dial tone was by sending something called a “Ring-to-ground” signal to the telephone company. That would happen when you would lift the receiver off the hook. There are only two wires used to communicate in an old phone and only one of those had voltage on it. when you ground that wire (called the “Ring”) momentarily, the phone company would then send a dial tone to your phone.
You could actually do this on a dead phone line at times when the phone company had shut off your service. On an old pay phone, when the proper coin was inserted in the phone, the coin itself was used to ground the ring wire, thus telling the telephone company to send the dial tone, allowing you to use the phone. In 1983 there was a movie called “Wargames”.
I had learned about how these telephones worked from Bill Rivers just before going to watch this movie. During the movie Matthew Broderick’s character needed to make a phone call at a pay phone but didn’t have a coin. By taking the mouthpiece off of the transmitter, and using a metal pop top he found on the ground, he was able to ground the “ring” wire to the pay phone, and he received a dial tone. There was a good ol’ boy sitting behind me in the movie theater that said, “You can’t do that!” — Being the newly educated smart (-alec) guy I was, I turned around and said, “Yeah. You really can.”
Anyway. This isn’t a story so much about how old phones work. I just wanted to bring the younger readers back-to-date on phones since now they don’t really call them telephones anymore. It is more like, “Smart Phone” and “Cell Phone”, “Mobile Phone” or just “Phone”. The phone in the house isn’t even referred to as a telephone. We now call them “Home Phone” to distinguish them from the actual phones that we use.
Anyway, when I joined the electric shop in 1983, I learned about the phone system. We didn’t use the older Rotary Dial phones at the plant. We were one step up. We had “Touch Tone” Phones.
As I have mentioned in previous posts, we had our own telephone computer at the Power Plant. It was called a ROLM phone system. See the post “A Slap In the Face at a Gas-fired Power Plant“.
To give you an idea of the technology used by this phone system, you connected to it using a “teletype” terminal that you connected to a telephone by clipping the receiver in a cradle. Then you dialed the phone computer. When you connected, it was at 300 Baud. Think of 300 bytes per second, only using audio…. like a fax machine. — It was like connecting using a modem. 300 baud meant that when it typed out the results on the paper that scrolled out the top, you could watch it as it slowly printed out each line. The maximum speed of the terminal was 300 baud.

This is the TI Silent 700 Terminal. We used this exact model of Teletype terminal at the Power Plant
In this picture you can see the cradle in the back where the phone receiver would fit in those two rubber cups.
After many years of going to the lab to connect to the telephone computer to make changes and to monitor the telephone traffic, in 1992 I decided to bring my 8088 computer to work and set it on the desk in the electric shop. We didn’t have our own computer yet. At that time the only people that had computers were office workers and the Shift Supervisor. We had started a computer club and having a computer in the shop was a big help. I had just replaced this computer at home with a 486.

This is a Leading Edge computer. My father had this one. An earlier version than the 8088 that I was using.
I had a modem on my computer, so I tried connecting to the telephone computer, and it worked! So, sometimes during lunch when Charles Foster and I were sitting there talking about movies we had seen while eating vegetables from his garden, I would connect to the ROLM computer and just watch the call log. I could see whenever someone was dialing in and out of the plant.
We had a special call in number into the plant that allowed you to make “trunk” calls. This is another term you don’t hear much anymore. You see….. for the younger readers (again)…. long distant calls used to cost a lot of money. You would be charged by how many minutes you were on the call. During the day, it could be as high as $3.00 a minute to call across the country. Amazing huh? Because today, most of you with cell phones and even your land lines (which are rarely real land lines anymore) long distant phone calls are now free with your phone plan.
Yeah, if you wanted to call someone in the next town over, you would have to pay a fee for every minute you were on the call…. That was when AT&T had a monopoly on the phone lines in the United States. Sure, you only payed $7.00 each month for your phone, but you could only call people in your immediate area or you would be charged extra.
A Trunk line gave you access to a much wider area. The Electric company had a trunk line that gave them access to most of Oklahoma. You could dial into a local number that would connect you to the company phone system. Then after entering the correct password number, you could dial access numbers that would take you to another office location in the electric company. Once on that phone system, you could dial to get an outside line, and then dial a local number in that area.
Our plant had three access numbers that allowed you to dial out locally to Stillwater, Ponca City and Pawnee. This was useful when a foreman needed to call people out to work. They could dial into the plant, then back out to one of these other towns and then dial the local phone number of the crew member they were trying to reach without incurring a personal charge on their phone line.
So, here I was in 1992 during lunch watching the phone traffic in and out of the plant (not exactly NSA style, but sort of), when I saw something unexpected. A long string of numbers showed up. Someone had dialed in on the Stillwater trunk, then dialed out to the Corporate Headquarters trunk, then out to Oklahoma City and from there they placed a long distance call to a phone number in the same area code. The prefix on the phone number was familiar to me. It was a Ponca City phone number. I had lived in Ponca City for three years when I had been married, from 1986 to 1989. I knew a Ponca City phone number when I saw one.
I thought this was odd, because it wouldn’t be normal for someone to dial from Stillwater through out plant to Oklahoma City only to call a Ponca City phone number when they could have dialed the local Ponca City access code. Then they wouldn’t have had to make a long distance call which bypassed our trunk call system causing the electric company to be billed for the long distance telephone call.
At the time I was a CompuServe user. This was when the World Wide Web was in it’s infancy. I was still using a DOS computer. When I connected to the Internet, it was either by using my dad’s Internet account from Oklahoma State University where I would use Telnet to access a bunch of mainframe computers all over the country, or I would use the DOS-based version of CompuServe. CompuServe was the king of Internet access before America Online came around and seemingly overnight made CompuServe obsolete.
In 1992, CompuServe had a service where you could look up phone numbers and find out whose number it was. Imagine that! Yeah. That was one of the neatest features on CompuServe! That and getting stock quotes. — Like I said…. There was no “www.whitepages.com” online. The only catch to using the reverse phone number feature, was that it was like making a long distance call. It cost money. You were charged by the minute for using the CompuServe reverse telephone number service, with the least amount being a dollar.
So, I bit the bullet and accessed the Phone Number lookup section of CompuServe. I quickly typed in the number. When the name and address of the user popped up, I quickly hit “Print Screen”, and then exited the service. My fee came to $1.00, but at least I knew what number had been dialed in Ponca City.
Charles, Scott Hubbard and I were a little excited by the time Terry Blevins walked into the electric shop office after lunch was over, I told him what I had seen.
When I told Terry the name of the person that had received the long distance call, he recognized the name right away. When I gave him the address, he was sure he knew who it was. The phone number belonged to the Music Director at the Ponca City High School. His son was attending college in Stillwater, Oklahoma.
Well, that sort of cinched it. We had a pretty good idea who had made the call. It was a college student calling home, who had been given the phone number most likely by a fellow student who knew the code to call home in Oklahoma City. So, the only local access code this guy knew was how to dial through our plant to Oklahoma City and back out where he was free (but it was not free for us) to make a long distance call home.
Armed with this knowledge, I headed up to the front office. I went straight to the Plant Manager, Ron Kilman’s office. I told Ron what I had found. I explained in detail how the person had dialed from Stillwater into our plant and then to Oklahoma City and out and then placed a long distance call to Ponca City leaving us with the phone bill. Since it was the middle of the day, the cost of a long distance call was not cheap.
I told Ron that I had used CompuServe to lookup the phone number and that Terry had said that it belonged to the Music Director at the Ponca City High School and that he had a son in college in Stillwater. I was all ready to pounce on this guy. This was a fraudulent use of the telephone service and there were some pretty strict laws then about stealing long distance from someone else.
Ron, being the more level-headed of the two of us thought about it for a minute and said, “What would be the best way to stop this from happening?” — Oh. Well. I was so intent on catching the culprit, I hadn’t thought about that angle…. “Well….” I said, “We could change the pass code used to log into our phone system. We would just have to tell our supervisors what the new number is.”
Ron asked me what it would take to do that. I told him I could do it in two minutes. We quickly settled on a new 4 digit pass code and I left his office and returned to the electric shop and made the change essentially turning the tables on the Telephone Interloper. I suppose the college student in Stillwater was lucky that our plant manager at the time was the type to forgive and forget.
Three years later the entire electric company phone system was replaced by a new AT&T computer which was managed by AT&T. As you can tell… Technology just keeps moving forward making seemingly really neat new inventions quickly obsolete.
Comments from the original post:
-
-
I still stand in awe at all the talent we had at the plant- never has one place had so many guys of remarkable skill and overall just good people and kind hearted.
-
-
I can’t agree with you more! We had the cream of the crop for sure.
-
-
-
-
Your memory still amazes me. I don’t remember that at all.
I’ll bet most young people today don’t know why we say to “dial” a phone number!-
It was just a moment in your busy day. It was the highlight of my week.
-
That was an exciting day! Another great story. Thanks for the memories.