Black Ops Raid Power Plant — Power Plant Men Ignore Attackers
Favorites Post #75
Originally posted January 17, 2015
I don’t know if they called them “Black Ops” in 1994, but when the control room operator David Evans answered the phone that day in October, I don’t think he ever expected to have the person on the other end of the line tell him that a military special forces unit was going to stage a mock raid on the coal-fired power plant in North Central Oklahoma some time that night. I’m sure Jack Maloy, the shift supervisor, was equally surprised when David told him about the phone call. I heard later that Jack was pretty upset to find out that a military force was going to be attacking our plant in the middle of the night without his permission!
David Evans waits for Jim Padgett to get a cup of coffee — Thanks Jim Cave for the photo
The first we heard about the call was when Jasper Christensen called a meeting of the entire maintenance department on the spur of the moment in the main break room. He told us about the phone call. He said we didn’t have any more information than that. Though the maintenance department shouldn’t be working that night, Jasper said that just in case we were called out for something, we should know that a group of commandos were going to be performing some sort of mock raid on our plant. If we encountered any soldiers sneaking around the plant in the middle of the night in full military gear, not to be alarmed. Just go on doing what you’re doing and don’t bother them.
Pay no attention to the man behind the mask. Photo from Call of Duty
Now that it is 21 years later (well, almost) the truth can finally come out…. Isn’t that how it goes? When we are sworn to secrecy, isn’t it 21 years before we can finally speak out? (That’s what Shadow Warriors always told me). I don’t remember us taking an oath or anything, but that’s the way it is with Power Plant Men. They just assume that if the military is staging a mock raid on our plant, it is a matter of national security. It seemed as if our plant sort of matched the layout of a power plant somewhere in Central America where the real raid was going to take place.
The main difference between our Power Plant and the one in Honduras, or wherever it was, is that our plant had recently gone through a downsizing. So, our operators at night now had to perform the duties that had before been done by the labor crew. They had to do coal cleanup throughout the conveyor system.
Coal conveyor carrying coal to the coal silos from the coalyard
This meant that if one of our auxiliary operators happened to run across someone dressed in the outfit above, they would have naturally handed him either a water hose or a shovel and pointed to the nearest conveyor and said something like, “I’ll start on this end, and you can start over there.” After all. He would already be wearing his respirator.
That day on the way home, Scott Hubbard and I discussed the significance of such a raid on our Power Plant. A year and a half earlier, Janet Reno had really messed up the raid on the Branch Davidian compound in Texas when it burned down and burned everyone to death including women and children. So, it would be good to go into a situation like this more prepared.
I had often thought about the steps that could covertly be taken to single-handed destroy the power plant without using any kind of explosives. Those who understood how all the systems worked together could do it if they really wanted to. Of course, that was just how I might occupy my mind when I was doing a repetitive job, like sweeping out the main switchgear. What better place for those thoughts to drift into your mind.
A picture of a clean switchgear. Picture 6 rows of switchgear like this
Actually, now that I think about it, instead of sending in the Special Forces, just send in a few Plant Operators, Electricians and Instrument and Controls guys and they could totally destroy the plant in a matter of hours if that was their intent. The same thing could be said about putting a few incompetent people in upper management even if it isn’t their intent, only it takes longer than a couple of hours to destroy the plant in that case.
The next morning when we arrived at the plant, our foreman Alan Kramer told us the stories about the raid that happened the night before. This is what I can remember about it (if any Power Plant Men want to correct me, or add some more stories, please do in the comments below).
Alan Kramer
First he said that it appeared as if the commandos had landed in some kind of stealth helicopter out on the north side of the intake because later when the power plant men had investigated the site they could see where two wheels on the helicopter had left an impression in the mud. Dan Landes had been keeping a lookout from the top of the Unit 1 boiler, and he thought for a moment that he saw the flash of a red light…. which… thinking about it now, could have been one of those laser sites taking aim at him and mock assassinating him by shooting him in the eye from about 1/2 mile. You know how good American Snipers can be (my plug for the new movie). Good thing he was wearing his auto-tinting safety glasses.
That’s Dan Landes in the second row with the Gray Shirt – Otherwise known as Deputy Dan
We also heard that one of the operators, Maybe Charles Peavler (Charles is standing next to Dan wearing the pink shirt and carrying something in his lower lip) had stepped out of the office elevator on the ground floor only to come face-to-face with a soldier. When the soldier was seen by the operator, he just turned around and walked out of the door… he evidently was considered a casualty if he was seen by anyone. Either that, or he had to go do coal cleanup the rest of the night.
I think it was Jeff Meyers (front row, left in the picture above) who told us later that the Special Ops forces had left a present for the operators on the Turbine-Generator Room floor. Tracked across the clean shiny red T-G floor were muddy boot prints leading from the Unit 1 boiler entrance to the door to the control room. The tracks ended at the control room door.
The Red T-G floor is always kept clean. The control room entrance is under the grating where this picture was taken. – Thanks Jim Cave for the picture
The tracks were extra muddy as if someone had intentionally wanted us to see that someone had walked right up to the control room door. The tracks did not lead away from the door. They just ended right there.
So, we did have proof that the commandos had actually visited our plant that night, only because one of the operators had come face-to-face with one in the main lobby. If that hadn’t happened, then they would have come and gone and we would have been none-the-wiser… other than wondering about the strange muddy footprints and the impression left in the mud by the stealth helicopter.
Stealth Helicopter
I suppose it was easy for the Power Plant operators to ignore the commandos since for the most part, they never saw them coming or going. The Power Plant Men were happy to play their part in the mock raid. Of all that has been asked of these Power Plant Men over the years, this was one of the more “unique” events. How many Power Plant Men across the country can say that they took part in a Special Ops Commando Raid on their Power Plant?
All I can say is that the commandos sure picked a great bunch of Power Plant Men and Women to attack. We were all honored (even those of us who were at home in bed asleep at the time) to be able to help out the military any way we could.
Psychological Profile of a Power Plant Control Room Operator
Favorites Post #38
Originally posted on July 6, 2012:
I suppose that many parents while raising their children would hear them say, “Dad, can you read that story to us again about the pirates that go to the island to find the treasure but Jim Hawkins fights them single-handed?” Or their children might say to their mother, “Will you tell us the story again about how you met daddy?” In my household, my children would say, “Dad, tell us another story about how you played a joke on Gene Day when you were working at the Coal-Fired Power Plant in Oklahoma? Those are always the best!”
As I pointed out in my third post this year called “Power Plant Humor And Joking With Gene Day” the first time I met Gene Day, I could tell that he was the type of person that would take a joke well, or so I thought…. One of the favorite stories my daughter would like for me to tell her as she was growing up was the one where I had created a Psychological Profile of Gene Day, who at that time was an Auxiliary Operator.
It began one day when I was leaving the electric shop through the Turbine Generator (T-G) building ground floor. A very noisy location as large steam pipes wound around under the Turbines where the steam caused a rumbling or whining sound. It was normal when walking through this area to reach up to the ear plugs that were draped over your shoulders and put them in your ears because the decibels were dangerously high if you were exposed earplug-less too long.
Earplugs on a cord that can be draped over your shoulders when not in use.
I stepped from the landing leading from the electric shop and started toward number 1 boiler when I spied Gene Day making his way around the first floor of the T-G building inspecting equipment and marking his documents indicating that they were operating correctly. As I saw him turn toward my direction I quickly dodged behind the nearest metal pillar (I-Beam). I peaked my head out from behind the pillar and took out the notepad that was in my back pocket, and the pen from my vest pocket pocket protector.
A pocket protector is a must for electricians and computer nerds who need a place to keep their small tools.
Gene saw me and gave me a suspicious look as I began feverishly writing in my notepad while looking from my notepad back to Gene and then back to what I was writing. After I had done this for about 10 seconds or so, I put my pad away, my pen back in the pocket protector and strolled away toward unit 1 to continue my work.
A notepad like this
It happened that this particular week was Gene’s week to monitor the equipment in the T-G building, so throughout the week he would be making his way somewhere around the T-G building with his clipboard in hand. Each time I encountered him I would do the same thing. I would visibly hide behind a beam and write notes in my notepad. I saw Gene rather frequently during the week because the majority of the time, I left the electric shop by going through the T-G room to one of the boilers and then to the precipitators, where I spent most of my time working at this time in my career (but that is another story for a later time).
Each time, Gene would watch me suspiciously knowing that I was just messing with him, but not exactly sure what I was “up to”. At the time, I wasn’t sure either. So I just wrote down what I saw Gene doing, that way, if he ran over and grabbed my notepad from me, it wouldn’t say anything other than what I saw.
It had happened at the plant a few years earlier when I was a janitor that the company had hired an efficiency expert to monitor the employees at the plant. He would walk around the plant with a stopwatch observing the employees. When he saw them he would write notes on his clipboard. It became very unnerving because you would walk around the corner and there he would be standing writing something down about you.
I went to the Assistant Plant Manager Bill Moler and told him that this creepy guy keeps showing up in the main switchgear by the janitors closet. And every time he sees me, he writes something down. I told Bill that it really bothered me. He explained that he is an efficiency expert and he has a certain path that he takes throughout the day and takes a snapshot of what the workers are doing at that moment and writes it down. By doing that he calculates how efficient we are. It seemed pretty silly to me, because most mechanics when they saw him coming put their tools down and did nothing while he walked by until he was out of sight again.
When I was on the labor crew a few weeks later, and I was blowing coal dust off of all the I-Beams above the bowl mills with a high pressure air hose, I looked down, and through my fogged up goggles I could see this guy standing directly under me. I was about 50 feet above him crawling across an I-Beam with the air hose blowing black dust everywhere. He had crossed my barrier tape to go into the bowl mill area to see how efficient I was being.
This is the type of barrier tape I was using. It is made of woven plastic fibers.
I was so mad I turned off my air hose, climbed down the wall Spiderman-like (no. not head first) and went straight into the A-Foreman’s office and told Marlin McDaniel that the Efficiency Expert had crossed my barrier tape and was standing directly under me as I was blowing down the beams with air.
It turned out that the Efficiency Expert had gone upstairs to complain that some guy in the bowl mill had dumped a bunch of coal dust on him while he was monitoring him from below. Evidently, he wasn’t expert enough to know you weren’t supposed to cross someone’s barrier tape without permission, as was indicated on the caution tags that were tied to the barrier tape. From that point on, the efficiency expert (now in lowercase) had to be accompanied by someone from the plant to make sure he wasn’t breaking any safety rules and putting himself at harm.
To make a long side story short, we turned out to be so efficient, people came from all over the world to study us. Somebody downtown hired the efficiency expert full time, but later he was laid off during the first downsizing.
He reminds me of a person on an episode of Star Trek The Next Generation where this guy Lieutenant Commander Remmick comes aboard the Enterprise and he walks around inspecting everything and asking everyone questions that make them uncomfortable and at the end asks Picard if he could come work on the Enterprise. He looked so much like him, that I thought maybe our efficiency expert went to Hollywood to become an actor.
Lieutenant Commander Remmick
Back to Gene Day. I suppose the thought of the efficiency expert may have been going through my mind as I was taking notes about Gene Day at work. Like I said, at the time, I didn’t know what I was going to do with it.
Oh. Let me include a picture of Gene Day:
Gene Day is the one standing on the right with the Orange shirt.
It finally came to me on Thursday morning. This was the last day that Gene was going to be on the day shift, so I figured I might as well do something about it. So when I entered the shop that morning, I sat down at the desk of my foreman Andy Tubbs and began to write. The title at the top of the page was: “The Psychological Profile of Gene Day”.
Using the notes I had taken during the week, I wrote things like the following: “Gene Day walks around the Turbine Generator building with a clipboard in his hand trying desperately to look like he’s doing something important. He constantly hopes that someone is watching him because he dislikes doing so much work to act busy for no reason.
At times Gene Day gets paranoid and believes that he sees people spying on him from behind every corner (especially I-Beams). Sometimes Gene Day stands in the middle of the T-G floor staring up into space as if he forgot what he was supposed to be doing.” I wrote more similar observations, but I don’t remember everything… but I do remember the last two sentences. I will save that for a couple of paragraphs from now.
I took my Psychological Profile of Gene Day and went up to the Control Room. I noticed out of the corner of my eye that Gene had just walked over toward the break room behind the Auxiliary Control Panel so I walked over by the Shift Supervisor’s office and I leaned against the top of the large Blue Monitor and placed the Psychological Profile on the top of the monitor in front of me.
Less than a minute later Gene came walking around behind me and seeing the paper on the top of the monitor came up behind me and looked over my shoulder and began to read….. He wasn’t trying to hide the fact that he was reading the paper, and I obviously knew he was there, and the title in large bold letters did say, “The Psychological Profile of Gene Day”. So, he read on.
I heard a few chuckles as he read through my interpretation of what he had done during the week. Then he came to the “Second from the Last Sentence”, as I could hear him reading quietly in my ear… The sentence read, “Gene Day sneaks up behind people and reads their private material over their shoulder!” — Bingo! I had him!
When he read that he grabbed me by the throat and started to Throttle me! Shaking me back and forth. This would have been a humdinger of a joke at that point, but I had one more sentence up my sleeve… ur… I mean on the paper….
As I was wavering (is that a word?) back and forth between life and death I managed to eke out something like: “Wait! There’s More!”…. Gene Day let up on me a little and looked down at the page and read the last sentence…… It read….. “Gene Day tries to strangle people who are only trying to help him by creating his Psychological Profile.”
That was all it took. Another perfect joke played on Gene Day, and I was able to live to tell about it. When Gene read that he was stunned into dismay. Giggling as hard as he could he retreated shaking his head in defeat.
Now, I know that Gene reads these posts, and he may remember this story a little differently, and I’ll give him that because Gene is older than dirt and his memory isn’t that good. But I was the one that was being strangled, and I still have a vivid image of those few moments. Not only do I, but so do my children, who will one day tell their children, who will say, “Mommy, will you tell us that story again about how Grandpa was strangled by Gene Day that time in the Control Room at the Coal-Fired Power Plant in Oklahoma?”
Pain in the Neck Muskogee Power Plant Relay Testing
Favorites Post #23
Originally post 2/21/2015
Don’t let the title fool you. I love testing Power Plant Protective Relays. There is a sense of satisfaction when you have successfully cleaned, calibrated and tested a relay that is going to protect the equipment you have to work on every day. With that said, I was hit with such an unbelievable situation when testing Muskogee Relays in 1995 that I was left with a serious pain in the neck.
On August 14, 2003 the electric power in the Northeast United States and Canada went out. The Blackout lasted long enough to be a major annoyance for those in the that region of the United States.
Map of the power blackout in 2003
When I heard about how the blackout had moved across the region, I immediately knew what had happened. I was quickly reminded of the following story. I told my wife Kelly, “I know exactly why such a large area lost power! They hadn’t done proper preventative maintenance on the Protective Relays in the substations! Just like….” Well…. I’ll tell you that part now:
I have mentioned in a couple of earlier posts that something always seemed a little “off” at the Muskogee Power Plant. I had decided early on that while working there I would stick to drinking sodas instead of water. See the post: “Something is in the Water at the Muskogee Power Plant“. Even with that knowledge, I was still shocked at what I found while testing relays at the plant.
This story really begins one Sunday at Muskogee when one of the Auxiliary Operators was making his rounds inspecting equipment. He was driving his truck around the south edge of the Unit 6 parking lot on the service road. He glanced over at a pump next to the road, and at first, he thought he was just seeing things. After stopping the truck and backing up for a second glance, he was sure he wasn’t dreaming. It’s just that what he was seeing seemed so strange, he wasn’t sure what was happening.
The operator could see what appeared to be silver paint chips popping off of the large pump motor in all directions. After closer examination, he figured out that the motor was burning up. It was still running, but it had become so hot that the paint was literally burning off of the motor.
Like this Horizontal pump only much bigger and painted silver
A motor like this would get hot if the bearings shell out. Before the motor is destroyed, the protective relays on the breaker in the 4,000 Volt switchgear shuts the motor off. In this case, the relay hadn’t tripped the motor, so, it had become extremely hot and could have eventually exploded if left running. The operator shut the motor down and wrote a work order for the electricians.
Doyle Fullen was the foreman in the electric shop that received the work order. When he looked into what had happened, he realized that the protective relay had not been inspected for a couple of years for this motor.
I couldn’t find a picture of Doyle. In his youth he reminds me of a very smart Daryl in Walking Dead:
Norman Reedus from Walking Dead
In fact, since before the downsizing in 1994, none of the Protective Relays at the plant had been inspected. The person that had been inspecting the relays for many years had moved to another job or retired in 1994. This was just a warning shot across the bow that could have had major consequences.
No one at Muskogee had been trained to test Protective Relays since the downsizing, so they reached out to our plant in North Central Oklahoma for help. That was when I was told that I was going to be going to Muskogee during the next overhaul (outage). I had been formally trained to inspect, clean, calibrate and test Protective Relays with two of my Power Plant Heroes, Ben Davis and Sonny Kendrick years earlier. See the post: “Relay Tests and Radio Quizzes with Ben Davis“.
My Protective Relay Maintenance course book
Without going into too much detail about the actual tests we performed as I don’t want to make this a long rambling post (like… well…. like most of my posts…..I can already tell this is going to be a long one), I will just say that I took our antiquated relay tester down to Muskogee to inspect their relays and teach another electrician Charles Lay, how to perform those tests in the future. Muskogee had a similar Relay Test Set. These were very outdated, but they did everything we needed, and it helped you understand exactly what was going on when you don’t have a newfangled Relay Test Set.
AVO Multi-Amp SR-76 Relay Test Set
You need to periodically test both mechanical and electronic protective relays. In the electronic relays the components change their properties slightly over time, changing the time it takes to trip a breaker under a given circumstance (we’re talking about milliseconds). In the mechanical relays (which I have always found to be more reliable), they sit inside a black box all the time, heating up and cooling as the equipment is used. Over time, the varnish on the copper coils evaporates and settles on all the components. This becomes sticky so that the relay won’t operate at the point where it should.
A panel of Protective Relays
In the picture above, the black boxes on the top, middle and right are mechanical relays. This means that something actually has to turn or pick up in order to trip the equipment. The electronic relays may have a couple of small relays, but for the most part, they are made up of transistors, resistors, capacitors and diodes.
So, with all that said, let me start the real story…. gee…. It’s about time…
So, here I am sitting in the electric shop lab just off of the Unit 6 T-G floor. We set up all the equipment and had taken a couple of OverCurrent relays out of some high voltage breakers in the switchgear. I told Charles that before you actually start testing the relays, you need to have the test documents from the previous test and we also needed the instruction manuals for each of the relays because the manuals will have the diagrams that you use to determine the exact time that the relays should trip for each of the tests. So, we went up to the print room to find the old tests and manuals. Since they weren’t well organized, we just grabbed the entire folder where all the relays tests were kept since Unit 6 had been in operation.
When we began testing the relays at first I thought that the relay test set wasn’t working correctly. Here I was trying to impress my new friend, Charles Lay, a 63 year old highly religious fundamental Christian that I knew what I was doing, and I couldn’t even make a relay trip. I was trying to find the “As Found” tripping level. That is, before you clean up the relay. Just like you found it. Only, it wouldn’t trip.
It turned out that the relay was stuck from the varnish as I explained above. It appeared as if the relay hadn’t been tested or even operated for years. The paperwork showed that it had been tested three years earlier. Protective Relays should be tested at least every two years, but I wouldn’t have thought that the relay would be in such a bad condition in just three years. It had been sitting in a sealed container to keep out dust. But it was what it was.
I told Charles that in order to find the “As Found” point where the relay would trip, we would need to crank up the test set as high as needed to find when it actually did trip. It turned out that the relay which should have instantaneously tripped somewhere around 150 amps wouldn’t have tripped until the motor was pulling over 4,000 amps. I could tell right away why the Auxiliary Operator found that motor burning up without tripping. The protective relays were stuck.
As it turned out… almost all of the 125 or so relays were in the same condition. We cleaned them all up and made them operational.
There is an overcurrent relay for the main bus on each section of a main switchgear.
A picture of a clean switchgear. Picture 6 rows of switchgear like this
When I tested the “As Found” instantaneous trip for the main bus relay, I found that it was so high that the Unit 6 Main Turbine Generator would have melted down before the protective relay would have tripped the power to that one section of switchgear. The entire electric bus would have been nothing but molten metal by that time.
As I tested each of these relays, I kept shaking my head in disbelief. But that wasn’t the worst of it. The mystery as to why these relays were all glued shut by varnish was finally solved, and that reason was even more unbelievable.
Here is what I found….. The first thing you do when you are going to test a relay is that you fill out a form that includes all the relay information, such as, what it is for, what are the settings on the relay, and what are the levels of tests that you are going to perform on it. You also include a range of milliseconds that are acceptable for the relay for each of the tests. Normally, you just copy what was used in the previous test, because you need to include the time it took for the Previous “As Left” test on your form. That is why we needed the forms from the previous test.
So, I had copied the information from the previous test form and began testing the relay (one of the first overcurrent relays we tested)… Again… I was a 34 year old teacher trying to impress my 63 year old student. So, I was showing him how you mechanically adjust the relay in order for it to trip within the acceptable range. No matter how hard I tried, I couldn’t adjust the relay so that it would even be close to the desired range for the longer time trip times…. like the 2 second to 25 second range. It wasn’t even close to the range that was on the form from the last test.
The form from the last test showed that the relay was in the right range for all the levels of test. When I tested it, like I said, it wasn’t even close. So, I went to the diagram in the instruction manual for this type of relay. The diagram looks similar to this one used for thermal overloads:
Thermal Overload Tripping curves
See all those red lines? Well, when you setup a relay, you have a dial where you set the range depending on the needs for the type of motor you are trying to trip. Each red line represents each setting on the dial. Most of the relays were set on the same number, so we would be using the same red line on the diagram to figure out at different currents how long it should take for a relay to trip….
Here is the clincher….The time range that was written on the previous form wasn’t for the correct relay setting. The person that tested the relay had accidentally looked at the wrong red line. — That in itself is understandable, since it could be easy to get on the wrong line… The only thing is that as soon as you test the relay, you would know that something is wrong, because the relay wouldn’t trip in that range, just like I had found.
I double and triple checked everything to make sure we were looking at the same thing. The previous form indicated the same settings on the relay as now, yet, the time ranges were for a different line! — Ok. I know. I have bored you to tears with all this stuff about time curves and overcurrent trips… so I will just tell you what this means…
This meant that when the person completed the forms the last time, they didn’t test the relays at all. They just filled out the paperwork. They put in random values that were in the acceptable range and sat around in the air conditioned lab during the entire overhaul smoking his pipe. — Actually, I don’t remember if he smoked a pipe or not. He was the Electrical Specialist for the plant. I remembered seeing him sitting in the lab with a relay hooked up to the test set throughout the entire overhaul when I had been there during previous overhauls, but I realized finally that he never tested the relays. He didn’t even go so far as try to operate them.
I went back through the records to when the plant was first “checked out”. Doyle Fullen had done the check out on the relays and the test after that. Doyle had written the correct values from the manual on his forms. I could see where he had actually performed the tests on the relays and was getting the same values I was finding when I tested the relays, so I was certain that I wasn’t overlooking anything.
As I tested each of the relays, I kept shaking my head in disbelief. It was so unbelievable. How could someone do such a thing? Someone could have been killed because a protective relay wasn’t working correctly. This was serious stuff.
One day while Charles and I were working away on the relays, Jack Coffman, the Superintendent of all the Power Plants came walking through the lab. He asked us how we were doing. I swiveled around in my chair to face him and I said, “Pretty good, except for this pain in my neck” as I rubbed the back of my neck.
Jack stopped and asked me what happened. I told him that I had been shaking my head in disbelief for the last two weeks, and it gave me a pain in the neck. Of course, I knew this would get his attention, so he asked, “Why?” I went through all the details of what I had found.
I showed him how since the time that Doyle Fullen last tested the relays more than 10 years earlier, these relays hadn’t been tested at all. I showed him how the main bus relays were so bad that it would take over 100,000 amps to have tripped the 7100 KV switchgear bus or 710 Megawatts! More power than the entire generator could generate. The main generator was only rated at about 550 Megawatts at the most.
Jack stood there looking off into space for a few seconds, and then walked out the door…. I thought I saw him shaking his head as he left. Maybe he was just looking both ways for safety reasons, but to me, it looked like a shake of disbelief. I wonder if I had given him the same pain in the neck.
That is really the end of the relay story, but I do want to say a few words about Charles Lay. He was a hard working electrician that was nearing retirement. People would come around to hear us discussing religion. I am Catholic, and he went to a Fundamental Christian Church. We would debate the differences between our beliefs and just Christian beliefs in general. We respected each other during our time together, even though he was sure I am going to hell when I die.
People would come in just to hear our discussion for a while as we were cleaning and calibrating the relays. One day Charles asked me if I could help him figure out how much he was going to receive from his retirement from the electric company. He had only been working there for three years. Retirement at that time was determined by your years of service. So, three years didn’t give him too much.
When I calculated his amount, he was upset. He said, “Am I going to have to work until I die?” I said, “Well, there’s always your 401k and Social Security.” He replied that he can’t live on Social Security. I said, “Well, there’s your 401k.” He asked, “What’s that?” (oh. not a good sign).
I explained that it was a retirement plan where you are able to put money in taxed deferred until you take it out when you retire. He said, “Oh. I never put anything in something like that.” My heart just sank as I looked in his eyes. He had suddenly realized that he wasn’t going to receive a retirement like those around him who had spent 35 years working in the Power Plant.
When I left the plant after teaching Charles Lay how to test the relays, that was the last time I ever saw him. I don’t know what became of Charles. I figure he would be 83 years old today. I wonder if he finally retired when he reached the 80 points for your age and years of service. He would have never reached enough years of service to receive a decent amount of retirement from the Electric Company since he didn’t start working there until he was 60 years old. That is, unless he’s still working there now.
As I said earlier in this post, Charles Lay was a very good worker. He always struck me as the “Hardworking type”. I often think about the time we spent together, especially when I hear about a power blackout somewhere. — A word of caution to Power Companies…. keep your protective relays in proper working condition. Don’t slack off on the Preventative Maintenance. — I guess that’s true for all of us… isn’t it? Don’t slack off on Preventative Maintenance in all aspects of your life.
Added note: On 7/6/2019, 3 weeks after re-posting this story, look what happened: Con Edison says cause of NYC blackout was substation’s faulty relay protection system
Comments from previous posts:
Great story! And I don’t mind the lengthy technical stuff. I like it.
We bought our current house in 2004. It was built in 1983. About 4 years ago we lost power to the GFI outlet in our “guest” bathroom and to an outlet in the “Florida room”. I replaced the GFI, but that didn’t fix anything. The power to the GFI was off. The affected bathroom was used very little, so it wasn’t a big deal. In the “Florida room” we just ran an extension cord from a different outlet and went on. I just assumed that a mouse or squirrel in the attack had messed with the wiring. Maybe I had pulled some wiring lose when I was crawling in the attic. Last December, I finally “bit the bullet” and called an electrician. What he found was a GFI in another bathroom had been wired backwards. It was wired in series with the GFI in the other bathroom. The mis-wired GFI had tripped but it didn’t cut the power to it’s own outlet. It only cut power to the other bathroom GFI and the “Florida room” outlet. It cost me $80 to re-learn what I had learned long ago – “Never assume previous work was does correctly – even if done by a “professional””.
KB,
I suppose the old timers knew more about this! not sure as in they had a saying at relief time between shifts No sparks out of any motors, no paint burning off any bearings, the truck is in the parking spot and filled with gas, all in all had a smooth shift! LOL! What would fascinate me was the mill fires in the early days – you go down there and here is this whole thing cherry red glowing! unbelievable! I have been over to said plant a few times and their switch gears were pretty dirty typically ours were aways very clean and the breakers always worked very well, very reliable and dependable.
-
Yeah. That was the first thing I noticed about Muskogee. Our plant took more pride in keeping things clean. To be fair, the Muskogee plant was close to the coal pile and the plant was down wind. So coal dust would blow right into the plant covering everything.
Charles came to our Plant and Calibrated Relays on Unit #1 in Sept. of 2011. Brought his wife with him and called it a working vacation. Still a very nice and intelligent man.
This is bringing back the memories. Worked in a coal fired plant for many years as a T.A. (Technicians Assistant) doing the protection maintenance and even the commissioning of a new power station for a couple of years in Australia.
And I agree, it is a place that gets in your blood. And as you are attesting here, some of the stories are unbelievable.
And I must put in my bit about those AVO multi Amp test sets. The day that we received one they were affectionately known as the ‘T.A.’s’. Apparently they could be relied on to not make a mistake (unlike the occasional human 🙂 ), and give correct settings. Us human T.A.’s were told, ‘if they could have made coffee they would have sacked the lot of us!’ 😀
Great story Plant Electrician, a good throwback to yesteryear. I wonder with all the computer gear they now have what has become of the technicians. It’s all probably digitised and in constant readouts to a computer test panel somewhere. One technician probably does all testing from a clean, air-conditioned room somewhere 🙂
Psychological Profile of a Power Plant Control Room Operator
Originally posted on July 6, 2012:
I suppose that many parents while raising their children would hear them say, “Dad, can you read that story to us again about the pirates that go to the island to find the treasure but Jim Hawkins fights them single-handed?” Or their children might say to their mother, “Will you tell us the story again about how you met daddy?” In my household, my children would say, “Dad, tell us another story about how you played a joke on Gene Day when you were working at the Coal-Fired Power Plant in Oklahoma? Those are always the best!”
As I pointed out in my third post this year called “Power Plant Humor And Joking With Gene Day” the first time I met Gene Day, I could tell that he was the type of person that would take a joke well, or so I thought…. One of the favorite stories my daughter would like for me to tell her as she was growing up was the one where I had created a Psychological Profile of Gene Day, who at that time was an Auxiliary Operator.
It began one day when I was leaving the electric shop through the Turbine Generator (T-G) building ground floor. A very noisy location as large steam pipes wound around under the Turbines where the steam caused a rumbling or whining sound. It was normal when walking through this area to reach up to the ear plugs that were draped over your shoulders and put them in your ears because the decibels were dangerously high if you were exposed earplugless too long.
Earplugs on a cord that can be draped over your shoulders when not in use.
I stepped from the landing leading from the electric shop and started toward number 1 boiler when I spied Gene Day making his way around the first floor of the T-G building inspecting equipment and marking his documents indicating that they were operating correctly. As I saw him turn toward my direction I quickly dodged behind the nearest metal pillar (I-Beam). I peaked my head out from behind the pillar and took out the notepad that was in my back pocket, and the pen from my vest pocket pocket protector.
A pocket protector is a must for electricians and computer nerds who need a place to keep their small tools.
Gene saw me and gave me a suspicious look as I began feverishly writing in my notepad while looking from my notepad back to Gene and then back to what I was writing. After I had done this for about 10 seconds or so, I put my pad away, my pen back in the pocket protector and strolled away toward unit 1 to continue my work.
A notepad like this
It happened that this particular week was Gene’s week to monitor the equipment in the T-G building, so throughout the week he would be making his way somewhere around the T-G building with his clipboard in hand. Each time I encountered him I would do the same thing. I would visibly hide behind a beam and write notes in my notepad. I saw Gene rather frequently during the week because the majority of the time, I left the electric shop by going through the T-G room to one of the boilers and then to the precipitators, where I spent most of my time working at this time in my career (but that is another story for a later time).
Each time, Gene would watch me suspiciously knowing that I was just messing with him, but not exactly sure what I was “up to”. At the time, I wasn’t sure either. So I just wrote down what I saw Gene doing, that way, if he ran over and grabbed my notepad from me, it wouldn’t say anything other than what I saw.
It had happened at the plant a few years earlier when I was a janitor that the company had hired an efficiency expert to monitor the employees at the plant. He would walk around the plant with a stopwatch observing the employees. When he saw them he would write notes on his clipboard. It became very unnerving because you would walk around the corner and there he would be standing writing something down about you.
I went to the Assistant Plant Manager Bill Moler and told him that this creepy guy keeps showing up in the main switchgear by the janitors closet. And every time he sees me, he writes something down. I told Bill that it really bothered me. He explained that he is an efficiency expert and he has a certain path that he takes throughout the day and takes a snapshot of what the workers are doing at that moment and writes it down. By doing that he calculates how efficient we are. It seemed pretty silly to me, because most mechanics when they saw him coming put their tools down and did nothing while he walked by until he was out of sight again.
When I was on the labor crew a few weeks later, and I was blowing coal dust off of all the I-Beams above the bowl mills with a high pressure air hose, I looked down, and through my fogged up goggles I could see this guy standing directly under me. I was about 50 feet above him crawling across an I-Beam with the air hose blowing black dust everywhere. He had crossed my barrier tape to go into the bowl mill area to see how efficient I was being.
This is the type of barrier tape I was using. It is made of woven plastic fibers.
I was so mad I turned off my air hose, climbed down the wall Spiderman-like (no. not head first) and went straight into the A-Foreman’s office and told Marlin McDaniel that the Efficiency Expert had crossed my barrier tape and was standing directly under me as I was blowing down the beams with air.
It turned out that the Efficiency Expert had gone upstairs to complain that some guy in the bowl mill had dumped a bunch of coal dust on him while he was monitoring him from below. Evidently, he wasn’t expert enough to know you weren’t supposed to cross someone’s barrier tape without permission, as was indicated on the caution tags that were tied to the barrier tape. From that point on, the efficiency expert (now in lowercase) had to be accompanied by someone from the plant to make sure he wasn’t breaking any safety rules and putting himself at harm.
To make a long side story short, we turned out to be so efficient, people came from all over the world to study us. Somebody downtown hired the efficiency expert full time, but later he was laid off during the first downsizing.
He reminds me of a person on an episode of Star Trek The Next Generation where this guy Lieutenant Commander Remmick comes aboard the Enterprise and he walks around inspecting everything and asking everyone questions that make them uncomfortable and at the end asks Picard if he could come work on the Enterprise. He looked so much like him, that I thought maybe our efficiency expert went to Hollywood to become an actor.
Lieutenant Commander Remmick
Back to Gene Day. I suppose the thought of the efficiency expert may have been going through my mind as I was taking notes about Gene Day at work. Like I said, at the time, I didn’t know what I was going to do with it.
It finally came to me on Thursday morning. This was the last day that Gene was going to be on the day shift, so I figured I might as well do something about it. So when I entered the shop that morning, I sat down at the desk of my foreman Andy Tubbs and began to write. The title at the top of the page was: “The Psychological Profile of Gene Day”.
Using the notes I had taken during the week, I wrote things like the following: “Gene Day walks around the Turbine Generator building with a clipboard in his hand trying desperately to look like he’s doing something important. He constantly hopes that someone is watching him because he dislikes doing so much work to act busy for no reason.
At times Gene Day gets paranoid and believes that he sees people spying on him from behind every corner (especially I-Beams). Sometimes Gene Day stands in the middle of the T-G floor staring up into space as if he forgot what he was supposed to be doing.” I’m sure I wrote more, but I don’t remember everything… but I do remember the second from the last sentence. I will save that for a couple of paragraphs from now.
I took my Psychological Profile of Gene Day and went up to the Control Room. I noticed out of the corner of my eye that Gene had just walked over toward the break room behind the Auxiliary Control Panel so I walked over by the Shift Supervisor’s office and I leaned against the top of the large Blue Monitor and placed the Psychological Profile on the top of the monitor in front of me.
Less than a minute later Gene came walking around behind me and seeing the paper on the top of the monitor came up behind me and looked over my shoulder and began to read….. He wasn’t trying to hide the fact that he was reading the paper, and I obviously knew he was there, and the title in large bold letters did say, “The Psychological Profile of Gene Day”. So, he read on.
I heard a few chuckles as he read through my interpretation of what he had done during the week. Then he came to the “Second from the Last Sentence”, as I could hear him reading quietly in my ear… The sentence read, “Gene Day sneaks up behind people and reads their private material over their shoulder!” — Bingo! I had him!
When he read that he grabbed me by the throat and started to Throttle me! Shaking me back and forth. This would have been a humdinger of a joke at that point, but I had one more sentence up my sleeve… ur… I mean on the paper….
As I was wavering (is that a word?) back and forth between life and death I managed to eke out something like: “Wait! There’s More!”…. Gene Day let up on me a little and looked down at the page and read the last sentence…… It read….. “Gene Day tries to strangle people who are only trying to help him by creating his Psychological Profile.”
That was all it took. Another perfect joke played on Gene Day, and I was able to live to tell about it. When Gene read that he was stunned into dismay. Giggling as hard as he could he retreated shaking his head in defeat.
Now, I know that Gene reads these posts, and he may remember this story a little differently, and I’ll give him that because Gene is older than dirt and his memory isn’t that good. But I was the one that was being strangled, and I still have a vivid image of those few moments. Not only do I, but so do my children, who will one day tell their children, who will say, “Mommy, will you tell us that story again about how Grandpa was strangled by Gene Day that time in the Control Room at the Coal-Fired Power Plant in Oklahoma?”
Black Ops Raid Power Plant — Power Plant Men Ignore Attackers
I don’t know if they called them “Black Ops” in 1994, but when the control room operator David Evans answered the phone that day in October, I don’t think he ever expected to have the person on the other end of the line tell him that a military special forces unit was going to stage a mock raid on the coal-fired power plant in North Central Oklahoma some time that night. I’m sure Jack Maloy, the shift supervisor, was equally surprised when David told him about the phone call. I heard later that Jack was pretty upset to find out that a military force was going to be attacking our plant in the middle of the night without his permission!
David Evans waits for Jim Padgett to get a cup of coffee — Thanks Jim Cave for the photo
The first we heard about the call was when Jasper Christensen called a meeting of the entire maintenance department on the spur of the moment in the main break room. He told us about the phone call. He said we didn’t have any more information than that. Though the maintenance department shouldn’t be working that night, Jasper said that just in case we were called out for something, we should know that a group of commandos were going to be performing some sort of mock raid on our plant. If we encountered any soldiers sneaking around the plant in the middle of the night in full military gear, not to be alarmed. Just go on doing what you’re doing and don’t bother them.
Pay no attention to the man behind the mask. Photo from Call of Duty
Now that it is 21 years later (well, almost) the truth can finally come out…. Isn’t that how it goes? When we are sworn to secrecy, isn’t it 21 years before we can finally speak out? (That’s what Shadow Warriors always told me). I don’t remember us taking an oath or anything, but that’s the way it is with Power Plant Men. They just assume that if the military is staging a mock raid on our plant, it is a matter of national security. It seemed as if our plant sort of matched the layout of a power plant somewhere in Central America where the real raid was going to take place.
The main difference between our Power Plant and the one in Honduras, or wherever it was, is that our plant had recently gone through a downsizing. So, our operators at night now had to perform the duties that had before been done by the labor crew. They had to do coal cleanup throughout the conveyor system.
Coal conveyor carrying coal to the coal silos from the coalyard
This meant that if one of our auxiliary operators happened to run across someone dressed in the outfit above, they would have naturally handed him either a water hose or a shovel and pointed to the nearest conveyor and said something like, “I’ll start on this end, and you can start over there.” After all. He would already be wearing his respirator.
That day on the way home, Scott Hubbard and I discussed the significance of such a raid on our Power Plant. A year and a half earlier, Janet Reno had really messed up the raid on the Branch Davidian compound in Texas when it burned down and burned everyone to death including women and children. So, it would be good to go into a situation like this more prepared.
I had often thought about the steps that could covertly be taken to single-handed destroy the power plant without using any kind of explosives. Those who understood how all the systems worked together could do it if they really wanted to. Of course, that was just how I might occupy my mind when I was doing a repetitive job, like sweeping out the main switchgear. What better place for those thoughts to drift into your mind.
A picture of a clean switchgear. Picture 6 rows of switchgear like this
Actually, now that I think about it, instead of sending in the Special Forces, just send in a few Plant Operators, Electricians and Instrument and Controls guys and they could totally destroy the plant in a matter of hours if that was their intent. The same thing could be said about putting a few incompetent people in upper management even if it isn’t their intent, only it takes longer than a couple of hours to destroy the plant in that case.
The next morning when we arrived at the plant, our foreman Alan Kramer told us the stories about the raid that happened the night before. This is what I can remember about it (if any Power Plant Men want to correct me, or add some more stories, please do in the comments below).
Alan Kramer
First he said that it appeared as if the commandos had landed in some kind of stealth helicopter out on the north side of the intake because later when the power plant men had investigated the site they could see where two wheels on the helicopter had left an impression in the mud. Dan Landes had been keeping a lookout from the top of the Unit 1 boiler, and he thought for a moment that he saw the flash of a red light…. which… thinking about it now, could have been one of those laser sites taking aim at him and mock assassinating him by shooting him in the eye from about 1/2 mile. You know how good American Snipers can be (my plug for the new movie). Good thing he was wearing his auto-tinting safety glasses.
That’s Dan Landes in the second row with the Gray Shirt – Otherwise known as Deputy Dan
We also heard that one of the operators, Maybe Charles Peavler (Charles is standing next to Dan wearing the pink shirt and carrying something in his lower lip) had stepped out of the office elevator on the ground floor only to come face-to-face with a soldier. When the soldier was seen by the operator, he just turned around and walked out of the door… he evidently was considered a casualty if he was seen by anyone. Either that, or he had to go do coal cleanup the rest of the night.
I think it was Jeff Meyers (front row, left in the picture above) who told us later that the Special Ops forces had left a present for the operators on the Turbine-Generator Room floor. Tracked across the clean shiny red T-G floor were muddy boot prints leading from the Unit 1 boiler entrance to the door to the control room. The tracks ended at the control room door.
The Red T-G floor is always kept clean. The control room entrance is under the grating where this picture was taken. – Thanks Jim Cave for the picture
The tracks were extra muddy as if someone had intentionally wanted us to see that someone had walked right up to the control room door. The tracks did not lead away from the door. They just ended right there.
So, we did have proof that the commandos had actually visited our plant that night, only because one of the operators had come face-to-face with one in the main lobby. If that hadn’t happened, then they would have come and gone and we would have been none-the-wiser… other than wondering about the strange muddy footprints and the impression left in the mud by the stealth helicopter.
Stealth Helicopter
I suppose it was easy for the Power Plant operators to ignore the commandos since for the most part, they never saw them coming or going. The Power Plant Men were happy to play their part in the mock raid. Of all that has been asked of these Power Plant Men over the years, this was one of the more “unique” events. How many Power Plant Men across the country can say that they took part in a Special Ops Commando Raid on their Power Plant?
All I can say is that the commandos sure picked a great bunch of Power Plant Men and Women to attack. We were all honored (even those of us who were at home in bed asleep at the time) to be able to help out the military any way we could.
Pain in the Neck Muskogee Power Plant Relay Testing
Don’t let the title fool you. I love testing Power Plant Protective Relays. There is a sense of satisfaction when you have successfully cleaned, calibrated and tested a relay that is going to protect the equipment you have to work on every day. With that said, I was hit with such an unbelievable situation when testing Muskogee Relays in 1995 that I was left with a serious pain in the neck.
On August 14, 2003 the electric power in the Northeast United States and Canada went out. The Blackout lasted long enough to be a major annoyance for those in the that region of the United States.
Map of the power blackout in 2003
When I heard about how the blackout had moved across the region, I immediately knew what had happened. I was quickly reminded of the following story. I told my wife Kelly, “I know exactly why such a large area lost power! They hadn’t done proper preventative maintenance on the Protective Relays in the substations! Just like….” Well…. I’ll tell you that part now:
I have mentioned in a couple of earlier posts that something always seemed a little “off” at the Muskogee Power Plant. I had decided early on that while working there I would stick to drinking sodas instead of water. See the post: “Something’s In the Water at the Muskogee Power Plant“. Even with that knowledge, I was still shocked at what I found while testing relays at the plant.
This story really begins one Sunday at Muskogee when one of the Auxiliary Operators was making his rounds inspecting equipment. He was driving his truck around the south edge of the Unit 6 parking lot on the service road. He glanced over at a pump next to the road, and at first, he thought he was just seeing things. After stopping the truck and backing up for a second glance, he was sure he wasn’t dreaming. It’s just that what he was seeing seemed so strange, he wasn’t sure what was happening.
The operator could see what appeared to be silver paint chips popping off of the large pump motor in all directions. After closer examination, he figured out that the motor was burning up. It was still running, but it had become so hot that the paint was literally burning off of the motor.
Like this Horizontal pump only much bigger and painted silver
A motor like this would get hot if the bearings shell out. Before the motor is destroyed, the protective relays on the breaker in the 4,000 Volt switchgear shuts the motor off. In this case, the relay hadn’t tripped the motor, so, it had become extremely hot and could have eventually exploded if left running. The operator shut the motor down and wrote a work order for the electricians.
Doyle Fullen was the foreman in the electric shop that received the work order. When he looked into what had happened, he realized that the protective relay had not been inspected for a couple of years for this motor.
I couldn’t find a picture of Doyle. In his youth he reminds me of a very smart Daryl in Walking Dead:
Norman Reedus from Walking Dead
In fact, since before the downsizing in 1994, none of the Protective Relays at the plant had been inspected. The person that had been inspecting the relays for many years had moved to another job or retired in 1994. This was just a warning shot across the bow that could have had major consequences.
No one at Muskogee had been trained to test Protective Relays since the downsizing, so they reached out to our plant in North Central Oklahoma for help. That was when I was told that I was going to be going to Muskogee during the next overhaul (outage). I had been formally trained to inspect, clean, calibrate and test Protective Relays with two of my Power Plant Heroes, Ben Davis and Sonny Kendrick years earlier. See the post: “Relay Tests and Radio Quizzes with Ben Davis“.
My Protective Relay Maintenance course book
Without going into too much detail about the actual tests we performed as I don’t want to make this a long rambling post (like… well…. like most of my posts…..I can already tell this is going to be a long one), I will just say that I took our antiquated relay tester down to Muskogee to inspect their relays and teach another electrician Charles Lay, how to perform those tests in the future. Muskogee had a similar Relay Test Set. These were really outdated, but they did everything we needed, and it helped you understand exactly what was going on when you don’t have a newfangled Relay Test Set.
AVO Multi-Amp SR-76 Relay Test Set
You need to periodically test both mechanical and electronic protective relays. In the electronic relays the components change their properties slightly over time, changing the time it takes to trip a breaker under a given circumstance (we’re talking about milliseconds). In the mechanical relays (which I have always found to be more reliable), they sit inside a black box all the time, heating up and cooling as the equipment is used. Over time, the varnish on the copper coils evaporates and settles on all the components. This becomes sticky so that the relay won’t operate at the point where it should.
A panel of Protective Relays
In the picture above, the black boxes on the top, middle and right are mechanical relays. This means that something actually has to turn or pick up in order to trip the equipment. The electronic relays may have a couple of small relays, but for the most part, they are made up of transistors, resistors, capacitors and diodes.
So, with all that said, let me start the real story…. gee…. It’s about time…
So, here I am sitting in the electric shop lab just off of the Unit 6 T-G floor. We set up all the equipment and had taken a couple of OverCurrent relays out of some high voltage breakers in the switchgear. I told Charles that before you actually start testing the relays, you need to have the test documents from the previous test and we also needed the instruction manuals for each of the relays because the manuals will have the diagrams that you use to determine the exact time that the relays should trip for each of the tests. So, we went up to the print room to find the old tests and manuals. Since they weren’t well organized, we just grabbed the entire folder where all the relays tests were kept since Unit 6 had been in operation.
When we began testing the relays at first I thought that the relay test set wasn’t working correctly. Here I was trying to impress my new friend, Charles Lay, a 63 year old highly religious fundamental Christian that I knew what I was doing, and I couldn’t even make a relay trip. I was trying to find the “As Found” tripping level. That is, before you clean up the relay. Just like you found it. Only, it wouldn’t trip.
It turned out that the relay was stuck from the varnish as I explained above. It appeared as if the relay hadn’t been tested or even operated for years. The paperwork showed that it had been tested three years earlier. Protective Relays should be tested at least every two years, but I wouldn’t have thought that the relay would be in such a bad condition in just three years. It had been sitting in a sealed container to keep out dust. But it was what it was.
I told Charles that in order to find the “As Found” point where the relay would trip, we would need to crank up the test set as high as needed to find when it actually did trip. It turned out that the relay which should have instantaneously tripped somewhere around 150 amps wouldn’t have tripped until the motor was pulling over 4,000 amps. I could tell right away why the Auxiliary Operator found that motor burning up without tripping. The protective relays were stuck.
As it turned out… almost all of the 125 or so relays were in the same condition. We cleaned them all up and made them operational.
There is an overcurrent relay for the main bus on each section of a main switchgear.
A picture of a clean switchgear. Picture 6 rows of switchgear like this
When I tested the “As Found” instantaneous trip for the main bus relay, I found that it was so high that the Unit 6 Main Turbine Generator would have melted down before the protective relay would have tripped the power to that one section of switchgear. The entire electric bus would have been nothing but molten metal by that time.
As I tested each of these relays, I kept shaking my head in disbelief. But that wasn’t the worst of it. The mystery as to why these relays were all glued shut by varnish was finally solved, and that reason was even more unbelievable.
Here is what I found….. The first thing you do when you are going to test a relay is that you fill out a form that includes all the relay information, such as, what it is for, what are the settings on the relay, and what are the levels of tests that you are going to perform on it. You also include a range of milliseconds that are acceptable for the relay for each of the tests. Normally, you just copy what was used in the previous test, because you need to include the time it took for the Previous “As Left” test on your form. That is why we needed the forms from the previous test.
So, I had copied the information from the previous test form and began testing the relay (one of the first overcurrent relays we tested)… Again… I was a 34 year old teacher trying to impress my 63 year old student. So, I was showing him how you mechanically adjust the relay in order for it to trip within the acceptable range. No matter how hard I tried, I couldn’t adjust the relay so that it would even be close to the desired range for the longer time trip times…. like the 2 second to 25 second range. It wasn’t even close to the range that was on the form from the last test.
The form from the last test showed that the relay was in the right range for all the levels of test. When I tested it, like I said, it wasn’t even close. So, I went to the diagram in the instruction manual for this type of relay. The diagram looks similar to this one used for thermal overloads:
Thermal Overload Tripping curves
See all those red lines? Well, when you setup a relay, you have a dial where you set the range depending on the needs for the type of motor you are trying to trip. Each red line represents each setting on the dial. Most of the relays were set on the same number, so we would be using the same red line on the diagram to figure out at different currents how long it should take for a relay to trip….
Here is the clincher….The time range that was written on the previous form wasn’t for the correct relay setting. The person that tested the relay had accidentally looked at the wrong red line. — That in itself is understandable, since it could be easy to get on the wrong line… The only thing is that as soon as you test the relay, you would know that something is wrong, because the relay wouldn’t trip in that range, just like I had found.
I double and triple checked everything to make sure we were looking at the same thing. The previous form indicated the same settings on the relay as now, yet, the time ranges were for a different line! — Ok. I know. I have bored you to tears with all this stuff about time curves and overcurrent trips… so I will just tell you what this means…
This meant that when the person completed the forms the last time, they didn’t test the relays at all. They just filled out the paperwork. They put in random values that were in the acceptable range and sat around in the air conditioned lab during the entire overhaul smoking his pipe. — Actually, I don’t remember if he smoked a pipe or not. He was the Electrical Specialist for the plant. I remembered seeing him sitting in the lab with a relay hooked up to the test set throughout the entire overhaul when I had been there during previous overhauls, but I realized finally that he never tested the relays. He didn’t even go so far as try to operate them.
I went back through the records to when the plant was first “checked out”. Doyle Fullen had done the check out on the relays and the test after that. Doyle had written the correct values from the manual on his forms. I could see where he had actually performed the tests on the relays and was getting the same values I was finding when I tested the relays, so I was certain that I wasn’t overlooking anything.
As I tested each of the relays, I kept shaking my head in disbelief. It was so unbelievable. How could someone do such a thing? Someone could have been killed because a protective relay wasn’t working correctly. This was serious stuff.
One day while Charles and I were working away on the relays, Jack Coffman, the Superintendent of all the Power Plants came walking through the lab. He asked us how we were doing. I swiveled around in my chair to face him and I said, “Pretty good, except for this pain in my neck” as I rubbed the back of my neck.
Jack stopped and asked me what happened. I told him that I had been shaking my head in disbelief for the last two weeks, and it gave me a pain in the neck. Of course, I knew this would get his attention, so he asked, “Why?” I went through all the details of what I had found.
I showed him how since the time that Doyle Fullen last tested the relays more than 10 years earlier, these relays hadn’t been tested at all. I showed him how the main bus relays were so bad that it would take over 100,000 amps to have tripped the 7100 KV switchgear bus or 710 Megawatts! More power than the entire generator could generate. It was only rated at about 550 Megawatts at the most.
Jack stood there looking off into space for a few seconds, and then walked out the door…. I thought I saw him shaking his head as he left. Maybe he was just looking both ways for safety reasons, but to me, it looked like a shake of disbelief. I wonder if I had given him the same pain in the neck.
That is really the end of the relay story, but I do want to say a few words about Charles Lay. He was a hard working electrician that was nearing retirement. People would come around to hear us discussing religion. I am Catholic, and he went to a Fundamental Christian Church. We would debate the differences between our beliefs and just Christian beliefs in general. We respected each other during our time together, even though he was sure I am going to hell when I die.
People would come in just to hear our discussion for a while as we were cleaning and calibrating the relays. One day Charles asked me if I could help him figure out how much he was going to receive from his retirement from the electric company. He had only been working there for three years. Retirement at that time was determined by your years of service. So, three years didn’t give him too much.
When I calculated his amount, he was upset. He said, “Am I going to have to work until I die?” I said, “Well, there’s always your 401k and Social Security.” He replied that he can’t live on Social Security. I said, “Well, there’s your 401k.” He asked, “What’s that?” (oh. not a good sign).
I explained that it was a retirement plan where you are able to put money in taxed deferred until you take it out when you retire. He said, “Oh. I never put anything in something like that.” My heart just sank as I looked in his eyes. He had suddenly realized that he wasn’t going to receive a retirement like those around him who had spent 35 years working in the Power Plant.
When I left the plant after teaching Charles Lay how to test the relays, that was the last time I ever saw him. I don’t know what became of Charles. I figure he would be 83 years old today. I wonder if he finally retired when he reached the 80 points for your age and years of service. He would have never reached enough years of service to receive a decent amount of retirement from the Electric Company since he didn’t start working there until he was 60 years old. That is, unless he’s still working there now.
As I said earlier in this post, Charles Lay was a very good worker. He always struck me as the “Hardworking type”. I often think about the time we spent together, especially when I hear about a power blackout somewhere. — A word of caution to Power Companies…. keep your protective relays in proper working condition. Don’t slack off on the Preventative Maintenance. — I guess that’s true for all of us… isn’t it? Don’t slack off on Preventative Maintenance in all aspects of your life.
Added note: On 7/6/2019, 3 weeks after re-posting this story, look what happened: Con Edison says cause of NYC blackout was substation’s faulty relay protection system
Psychological Profile of a Power Plant Control Room Operator
Originally posted on July 6, 2012:
I suppose that many parents while raising their children would hear them say, “Dad, can you read that story to us again about the pirates that go to the island to find the treasure but Jim Hawkins fights them single-handed?” Or their children might say to their mother, “Will you tell us the story again about how you met daddy?” In my household, my children would say, “Dad, tell us another story about how you played a joke on Gene Day when you were working at the Coal-Fired Power Plant in Oklahoma? Those are always the best!”
As I pointed out in my third post this year called “Power Plant Humor And Joking With Gene Day” the first time I met Gene Day, I could tell that he was the type of person that would take a joke well, or so I thought…. One of the favorite stories my daughter would like for me to tell her as she was growing up was the one where I had created a Psychological Profile of Gene Day, who at that time was an Auxiliary Operator.
It began one day when I was leaving the electric shop through the Turbine Generator (T-G) building ground floor. A very noisy location as large steam pipes wound around under the Turbines where the steam caused a rumbling or whining sound. It was normal when walking through this area to reach up to the ear plugs that were draped over your shoulders and put them in your ears because the decibels were dangerously high if you were exposed earplugless too long.
Earplugs on a cord that can be draped over your shoulders when not in use.
I stepped from the landing leading from the electric shop and started toward number 1 boiler when I spied Gene Day making his way around the first floor of the T-G building inspecting equipment and marking his documents indicating that they were operating correctly. As I saw him turn toward my direction I quickly dodged behind the nearest metal pillar (I-Beam). I peaked my head out from behind the pillar and took out the notepad that was in my back pocket, and the pen from my vest pocket pocket protector.
A pocket protector is a must for electricians and computer nerds who need a place to keep their small tools.
Gene saw me and gave me a suspicious look as I began feverishly writing in my notepad while looking from my notepad back to Gene and then back to what I was writing. After I had done this for about 10 seconds or so, I put my pad away, my pen back in the pocket protector and strolled away toward unit 1 to continue my work.
A notepad like this
It happened that this particular week was Gene’s week to monitor the equipment in the T-G building, so throughout the week he would be making his way somewhere around the T-G building with his clipboard in hand. Each time I encountered him I would do the same thing. I would visibly hide behind a beam and write notes in my notepad. I saw Gene rather frequently during the week because the majority of the time, I left the electric shop by going through the T-G room to one of the boilers and then to the precipitators, where I spent most of my time working at this time in my career (but that is another story for a later time).
Each time, Gene would watch me suspiciously knowing that I was just messing with him, but not exactly sure what I was “up to”. At the time, I wasn’t sure either. So I just wrote down what I saw Gene doing, that way, if he ran over and grabbed my notepad from me, it wouldn’t say anything other than what I saw.
It had happened at the plant a few years earlier when I was a janitor that the company had hired an efficiency expert to monitor the employees at the plant. He would walk around the plant with a stopwatch observing the employees. When he saw them he would write notes on his clipboard. It became very unnerving because you would walk around the corner and there he would be standing writing something down about you.
I went to the Assistant Plant Manager Bill Moler and told him that this creepy guy keeps showing up in the main switchgear by the janitors closet. And every time he sees me, he writes something down. I told Bill that it really bothered me. He explained that he is an efficiency expert and he has a certain path that he takes throughout the day and takes a snapshot of what the workers are doing at that moment and writes it down. By doing that he calculates how efficient we are. It seemed pretty silly to me, because most mechanics when they saw him coming put their tools down and did nothing while he walked by until he was out of sight again.
When I was on the labor crew a few weeks later, and I was blowing coal dust off of all the I-Beams above the bowl mills with a high pressure air hose, I looked down, and through my fogged up goggles I could see this guy standing directly under me. I was about 50 feet above him crawling across an I-Beam with the air hose blowing black dust everywhere. He had crossed my barrier tape to go into the bowl mill area to see how efficient I was being.
This is the type of barrier tape I was using. It is made of woven plastic fibers.
I was so mad I turned off my air hose, climbed down the wall Spiderman-like (no. not head first) and went straight into the A-Foreman’s office and told Marlin McDaniel that the Efficiency Expert had crossed my barrier tape and was standing directly under me as I was blowing down the beams with air.
It turned out that the Efficiency Expert had gone upstairs to complain that some guy in the bowl mill had dumped a bunch of coal dust on him while he was monitoring him from below. Evidently, he wasn’t expert enough to know you weren’t supposed to cross someone’s barrier tape without permission, as was indicated on the caution tags that were tied to the barrier tape. From that point on, the efficiency expert (now in lowercase) had to be accompanied by someone from the plant to make sure he wasn’t breaking any safety rules and putting himself at harm.
To make a long side story short, we turned out to be so efficient, people came from all over the world to study us. Somebody downtown hired the efficiency expert full time, but later he was laid off during the first downsizing.
He reminds me of a person on an episode of Star Trek The Next Generation where this guy Lieutenant Commander Remmick comes aboard the Enterprise and he walks around inspecting everything and asking everyone questions that make them uncomfortable and at the end asks Picard if he could come work on the Enterprise. He looked so much like him, that I thought maybe our efficiency expert went to Hollywood to become an actor.
Lieutenant Commander Remmick
Back to Gene Day. I suppose the thought of the efficiency expert may have been going through my mind as I was taking notes about Gene Day at work. Like I said, at the time, I didn’t know what I was going to do with it.
It finally came to me on Thursday morning. This was the last day that Gene was going to be on the day shift, so I figured I might as well do something about it. So when I entered the shop that morning, I sat down at the desk of my foreman Andy Tubbs and began to write. The title at the top of the page was: “The Psychological Profile of Gene Day”.
Using the notes I had taken during the week, I wrote things like the following: “Gene Day walks around the Turbine Generator building with a clipboard in his hand trying desperately to look like he’s doing something important. He constantly hopes that someone is watching him because he dislikes doing so much work to act busy for no reason.
At times Gene Day gets paranoid and believes that he sees people spying on him from behind every corner (especially I-Beams). Sometimes Gene Day stands in the middle of the T-G floor staring up into space as if he forgot what he was supposed to be doing.” I’m sure I wrote more, but I don’t remember everything… but I do remember the second from the last sentence. I will save that for a couple of paragraphs from now.
I took my Psychological Profile of Gene Day and went up to the Control Room. I noticed out of the corner of my eye that Gene had just walked over toward the break room behind the Auxiliary Control Panel so I walked over by the Shift Supervisor’s office and I leaned against the top of the large Blue Monitor and placed the Psychological Profile on the top of the monitor in front of me.
Less than a minute later Gene came walking around behind me and seeing the paper on the top of the monitor came up behind me and looked over my shoulder and began to read….. He wasn’t trying to hide the fact that he was reading the paper, and I obviously knew he was there, and the title in large bold letters did say, “The Psychological Profile of Gene Day”. So, he read on.
I heard a few chuckles as he read through my interpretation of what he had done during the week. Then he came to the “Second from the Last Sentence”, as I could hear him reading quietly in my ear… The sentence read, “Gene Day sneaks up behind people and reads their private material over their shoulder!” — Bingo! I had him!
When he read that he grabbed me by the throat and started to Throttle me! Shaking me back and forth. This would have been a humdinger of a joke at that point, but I had one more sentence up my sleeve… ur… I mean on the paper….
As I was waivering (is that a word?) back and forth between life and death I managed to eke out something like: “Wait! There’s More!”…. Gene Day let up on me a little and looked down at the page and read the last sentence…… It read….. “Gene Day tries to strangle people who are only trying to help him by creating his Psychological Profile.”
That was all it took. Another perfect joke played on Gene Day, and I was able to live to tell about it. When Gene read that he was stunned into dismay. Giggling as hard as he could he retreated shaking his head in defeat.
Now, I know that Gene reads these posts, and he may remember this story a little differently, and I’ll give him that because Gene is older than dirt and his memory isn’t that good. But I was the one that was being strangled, and I still have a vivid image of those few moments. Not only do I, but so do my children, who will one day tell their children, who will say, “Mommy, will you tell us that story again about how Grandpa was strangled by Gene Day that time in the Control Room at the Coal-Fired Power Plant in Oklahoma?”
Black Ops Raid Power Plant — Power Plant Men Ignore Attackers
I don’t know if they called them “Black Ops” in 1994, but when the control room operator David Evans answered the phone that day in October, I don’t think he ever expected to have the person on the other end of the line tell him that a military special forces unit was going to stage a mock raid on the coal-fired power plant in North Central Oklahoma some time that night. I’m sure Jack Maloy, the shift supervisor, was equally surprised when David told him about the phone call. I heard later that Jack was pretty upset to find out that a military force was going to be attacking our plant in the middle of the night without his permission!
David Evans waits for Jim Padgett to get a cup of coffee — Thanks Jim Cave for the photo
The first we heard about the call was when Jasper Christensen called a meeting of the entire maintenance department on the spur of the moment in the main break room. He told us about the phone call. He said we didn’t have any more information than that. Though the maintenance department shouldn’t be working that night, Jasper said that just in case we were called out for something, we should know that a group of commandos were going to be performing some sort of mock raid on our plant. If we encountered any soldiers sneaking around the plant in the middle of the night in full military gear, not to be alarmed. Just go on doing what you’re doing and don’t bother them.
Pay no attention to the man behind the mask. Photo from Call of Duty
Now that it is 21 years later (well, almost) the truth can finally come out…. Isn’t that how it goes? When we are sworn to secrecy, isn’t it 21 years before we can finally speak out? I don’t remember us taking an oath or anything, but that’s the way it is with Power Plant Men. They just assume that if the military is staging a mock raid on our plant, it is a matter of national security. It seemed as if our plant sort of matched the layout of a power plant somewhere in Central America where the real raid was going to take place.
The main difference between our Power Plant and the one in Honduras, or wherever it was, is that our plant had recently gone through a downsizing. So, our operators at night now had to perform the duties that had before been done by the labor crew. They had to do coal cleanup throughout the conveyor system.
Coal conveyor carrying coal to the coal silos from the coalyard
This meant that if one of our auxiliary operators happened to run across someone dressed in the outfit above, they would have naturally handed him either a water hose or a shovel and pointed to the nearest conveyor and said something like, “I’ll start on this end, and you can start over there.” After all. He would already be wearing his respirator.
That day on the way home that day, Scott Hubbard and I discussed the significance of such a raid on our Power Plant. A year and a half earlier, Janet Reno had really messed up the raid on the Branch Davidian compound in Texas when it burned down and burned everyone to death including women and children. So, it would be good to go into a situation like this more prepared.
I had often thought about the steps that could covertly be taken to single-handed destroy the power plant without using any kind of explosives. Those who understood how all the systems worked together could do it if they really wanted to. Of course, that was just how I might occupy my mind when I was doing a repetitive job, like sweeping out the main switchgear. What better place for those thoughts to drift into your mind.
A picture of a clean switchgear. Picture 6 rows of switchgear like this
Actually, now that I think about it, instead of sending in the Special Forces, just send in a few Plant Operators, Electricians and Instrument and Controls guys and they could totally destroy the plant in a matter of hours if that was their intent. The same thing could be said about putting a few incompetent people in upper management even if it isn’t their intent, only it takes longer than a couple of hours to destroy the plant in that case.
The next morning when we arrived at the plant, our foreman Alan Kramer told us the stories about the raid that happened the night before. This is what I can remember about it (if any Power Plant Men want to correct me, or add some more stories, please do in the comments below).
Alan Kramer
First he said that it appeared as if the commandos had landed in some kind of stealth helicopter out on the north side of the intake because later when the power plant men had investigated the site they could see where two wheels on the helicopter had left an impression in the mud. Dan Landes had been keeping a lookout from the top of the Unit 1 boiler, and he thought for a moment that he saw the flash of a red light…. which… thinking about it now, could have been one of those laser sites taking aim at him and mock assassinating him by shooting him in the eye from about 1/2 mile. You know how good American Snipers can be (my plug for the new movie). Good thing he was wearing his auto-tinting safety glasses.
That’s Dan Landes in the second row with the Gray Shirt – Otherwise known as Deputy Dan
We also heard that one of the operators, Maybe Charles Peavler (Charles is standing next to Dan wearing the pink shirt and carrying something in his lower lip) had stepped out of the office elevator on the ground floor only to come face-to-face with a soldier. When the soldier was seen by the operator, he just turned around and walked out of the door… he evidently was considered a casualty if he was seen by anyone. Either that, or he had to go do coal cleanup the rest of the night.
I think it was Jeff Meyers (front row, left in the picture above) who told us later that the Special Ops forces had left a present for the operators on the Turbine-Generator Room floor. Tracked across the clean shiny red T-G floor were muddy boot prints leading from the Unit 1 boiler entrance to the door to the control room. The tracks ended at the control room door.
The Red T-G floor is always kept clean. The control room entrance is under the grating where this picture was taken. – Thanks Jim Cave for the picture
The tracks were extra muddy as if someone had intentionally wanted us to see that someone had walked right up to the control room door. The tracks did not lead away from the door. They just ended right there.
So, we did have proof that the commandos had actually visited our plant that night, only because one of the operators had come face-to-face with one in the main lobby. If that hadn’t happened, then they would have come and gone and we would have been none-the-wiser… other than wondering about the strange muddy footprints and the impression left in the mud by the stealth helicopter.
Stealth Helicopter
I suppose it was easy for the Power Plant operators to ignore the commandos since for the most part, they never saw them coming or going. The Power Plant Men were happy to play their part in the mock raid. Of all that has been asked of these Power Plant Men over the years, this was one of the more “unique” events. How many Power Plant Men across the country can say that they took part in a Special Ops Commando Raid on their Power Plant?
All I can say is that the commandos sure picked a great bunch of Power Plant Men and Women to attack. We were all honored (even those of us who were at home in bed asleep at the time) to be able to help out the military any way we could.
Pain in the Neck Muskogee Power Plant Relay Testing
Don’t let the title fool you. I love testing Power Plant Protective Relays. There is a sense of satisfaction when you have successfully cleaned, calibrated and tested a relay that is going to protect the equipment you have to work on every day. With that said, I was hit with such an unbelievable situation when testing Muskogee Relays in 1995 that I was left with a serious pain in the neck.
On August 14, 2003 the electric power in the Northeast United States and Canada went out. The Blackout lasted long enough to be a major annoyance for those in the that region of the United States.
Map of the power blackout in 2003
When I heard about how the blackout had moved across the region, I immediately knew what had happened. I was quickly reminded of the following story. I told my wife Kelly, “I know exactly why such a large area lost power! They hadn’t done proper preventative maintenance on the Protective Relays in the substations! Just like….” Well…. I’ll tell you that part now:
I have mentioned in a couple of earlier posts that something always seemed a little “off” at the Muskogee Power Plant. I had decided early on that while working there I would stick to drinking sodas instead of water. See the post: “Something’s In the Water at the Muskogee Power Plant“. Even with that knowledge, I was still shocked at what I found while testing relays at the plant.
This story really begins one Sunday at Muskogee when one of the Auxiliary Operators was making his rounds inspecting equipment. He was driving his truck around the south edge of the Unit 6 parking lot on the service road. He glanced over at a pump next to the road, and at first, he thought he was just seeing things. After stopping the truck and backing up for a second glance, he was sure he wasn’t dreaming. It’s just that what he was seeing seemed so strange, he wasn’t sure what was happening.
The operator could see what appeared to be silver paint chips popping off of the large pump motor in all directions. After closer examination, he figured out that the motor was burning up. It was still running, but it had become so hot that the paint was literally burning off of the motor.
Like this Horizontal pump only much bigger and painted silver
A motor like this would get hot if the bearings shell out. Before the motor is destroyed, the protective relays on the breaker in the 4,000 Volt switchgear shuts the motor off. In this case, the relay hadn’t tripped the motor, so, it had become extremely hot and could have eventually exploded if left running. The operator shut the motor down and wrote a work order for the electricians.
Doyle Fullen was the foreman in the electric shop that received the work order. When he looked into what had happened, he realized that the protective relay had not been inspected for a couple of years for this motor.
I couldn’t find a picture of Doyle. In his youth he reminds me of a very smart Daryl in Walking Dead:
Norman Reedus from Walking Dead
In fact, since before the downsizing in 1994, none of the Protective Relays at the plant had been inspected. The person that had been inspecting the relays for many years had moved to another job or retired in 1994. This was just a warning shot across the bow that could have had major consequences.
No one at Muskogee had been trained to test Protective Relays since the downsizing, so they reached out to our plant in North Central Oklahoma for help. That was when I was told that I was going to be going to Muskogee during the next overhaul (outage). I had been formally trained to inspect, clean, calibrate and test Protective Relays with two of my Power Plant Heroes, Ben Davis and Sonny Kendrick years earlier. See the post: “Relay Tests and Radio Quizzes with Ben Davis“.
My Protective Relay Maintenance course book
Without going into too much detail about the actual tests we performed as I don’t want to make this a long rambling post (like… well…. like most of my posts…..I can already tell this is going to be a long one), I will just say that I took our antiquated relay tester down to Muskogee to inspect their relays and teach another electrician Charles Lay, how to perform those tests in the future. Muskogee had a similar Relay Test Set. These were really outdated, but they did everything we needed, and it helped you understand exactly what was going on when you don’t have a newfangled Relay Test Set.
AVO Multi-Amp SR-76 Relay Test Set
You need to periodically test both mechanical and electronic protective relays. In the electronic relays the components change their properties slightly over time, changing the time it takes to trip a breaker under a given circumstance (we’re talking about milliseconds). In the mechanical relays (which I have always found to be more reliable), they sit inside a black box all the time, heating up and cooling as the equipment is used. Over time, the varnish on the copper coils evaporates and settles on all the components. This becomes sticky so that the relay won’t operate at the point where it should.
A panel of Protective Relays
In the picture above, the black boxes on the top, middle and right are mechanical relays. This means that something actually has to turn or pick up in order to trip the equipment. The electronic relays may have a couple of small relays, but for the most part, they are made up of transistors, resistors, capacitors and diodes.
So, with all that said, let me start the real story…. gee…. It’s about time…
So, here I am sitting in the electric shop lab just off of the Unit 6 T-G floor. We set up all the equipment and had taken a couple of OverCurrent relays out of some high voltage breakers in the switchgear. I told Charles that before you actually start testing the relays, you need to have the test documents from the previous test and we also needed the instruction manuals for each of the relays because the manuals will have the diagrams that you use to determine the exact time that the relays should trip for each of the tests. So, we went up to the print room to find the old tests and manuals. Since they weren’t well organized, we just grabbed the entire folder where all the relays tests were kept since Unit 6 had been in operation.
When we began testing the relays at first I thought that the relay test set wasn’t working correctly. Here I was trying to impress my new friend, Charles Lay, a 63 year old highly religious fundamental Christian that I knew what I was doing, and I couldn’t even make a relay trip. I was trying to find the “As Found” tripping level. That is, before you clean up the relay. Just like you found it. Only, it wouldn’t trip.
It turned out that the relay was stuck from the varnish as I explained above. It appeared as if the relay hadn’t been tested or even operated for years. The paperwork showed that it had been tested three years earlier. Protective Relays should be tested at least every two years, but I wouldn’t have thought that the relay would be in such a bad condition in just three years. It had been sitting in a sealed container to keep out dust. But it was what it was.
I told Charles that in order to find the “As Found” point where the relay would trip, we would need to crank up the test set as high as needed to find when it actually did trip. It turned out that the relay which should have instantaneously tripped somewhere around 150 amps wouldn’t have tripped until the motor was pulling over 4,000 amps. I could tell right away why the Auxiliary Operator found that motor burning up without tripping. The protective relays were stuck.
As it turned out… almost all of the 125 or so relays were in the same condition. We cleaned them all up and made them operational.
There is an overcurrent relay for the main bus on each section of a main switchgear.
A picture of a clean switchgear. Picture 6 rows of switchgear like this
When I tested the “As Found” instantaneous trip for the main bus relay, I found that it was so high that the Unit 6 Main Turbine Generator would have melted down before the protective relay would have tripped the power to that one section of switchgear. The entire electric bus would have been nothing but molten metal by that time.
As I tested each of these relays, I kept shaking my head in disbelief. But that wasn’t the worst of it. The mystery as to why these relays were all glued shut by varnish was finally solved, and that reason was even more unbelievable.
Here is what I found….. The first thing you do when you are going to test a relay is that you fill out a form that includes all the relay information, such as, what it is for, what are the settings on the relay, and what are the levels of tests that you are going to perform on it. You also include a range of milliseconds that are acceptable for the relay for each of the tests. Normally, you just copy what was used in the previous test, because you need to include the time it took for the Previous “As Left” test on your form. That is why we needed the forms from the previous test.
So, I had copied the information from the previous test form and began testing the relay (one of the first overcurrent relays we tested)… Again… I was a 34 year old teacher trying to impress my 63 year old student. So, I was showing him how you mechanically adjust the relay in order for it to trip within the acceptable range. No matter how hard I tried, I couldn’t adjust the relay so that it would even be close to the desired range for the longer time trip times…. like the 2 second to 25 second range. It wasn’t even close to the range that was on the form from the last test.
The form from the last test showed that the relay was in the right range for all the levels of test. When I tested it, like I said, it wasn’t even close. So, I went to the diagram in the instruction manual for this type of relay. The diagram looks similar to this one used for thermal overloads:
Thermal Overload Tripping curves
See all those red lines? Well, when you setup a relay, you have a dial where you set the range depending on the needs for the type of motor you are trying to trip. Each red line represents each setting on the dial. Most of the relays were set on the same number, so we would be using the same red line on the diagram to figure out at different currents how long it should take for a relay to trip….
Here is the clincher….The time range that was written on the previous form wasn’t for the correct relay setting. The person that tested the relay had accidentally looked at the wrong red line. — That in itself is understandable, since it could be easy to get on the wrong line… The only thing is that as soon as you test the relay, you would know that something is wrong, because the relay wouldn’t trip in that range, just like I had found.
I double and triple checked everything to make sure we were looking at the same thing. The previous form indicated the same settings on the relay as now, yet, the time ranges were for a different line! — Ok. I know. I have bored you to tears with all this stuff about time curves and overcurrent trips… so I will just tell you what this means…
This meant that when the person completed the forms the last time, they didn’t test the relays at all. They just filled out the paperwork. They put in random values that were in the acceptable range and sat around in the air conditioned lab during the entire overhaul smoking his pipe. — Actually, I don’t remember if he smoked a pipe or not. He was the Electrical Specialist for the plant. I remembered seeing him sitting in the lab with a relay hooked up to the test set throughout the entire overhaul, but I realized finally that he never tested the relays. He didn’t even go so far as try to operate them.
I went back through the records to when the plant was first “checked out”. Doyle Fullen had done the check out on the relays and the test after that. Doyle had written the correct values from the manual on his forms. I could see where he had actually performed the tests on the relays and was getting the same values I was finding when I tested the relays, so I was certain that I wasn’t overlooking anything.
As I tested each of the relays, I kept shaking my head in disbelief. It was so unbelievable. How could someone do such a thing? Someone could have been killed because a protective relay wasn’t working correctly. This was serious stuff.
One day while Charles and I were working away on the relays, Jack Coffman, the Superintendent of all the Power Plants came walking through the lab. He asked us how we were doing. I swiveled around in my chair to face him and I said, “Pretty good, except for this pain in my neck” as I rubbed the back of my neck.
Jack stopped and asked me what happened. I told him that I had been shaking my head in disbelief for the last two weeks, and it gave me a pain in the neck. Of course, I knew this would get his attention, so he asked, “Why?” I went through all the details of what I had found.
I showed him how since the time that Doyle Fullen last tested the relays more than 10 years earlier, these relays hadn’t been tested at all. I showed him how the main bus relays were so bad that it would take over 100,000 amps to have tripped the 7100 KV switchgear bus or 710 Megawatts! More power than the entire generator could generate. It was only rated at about 550 Megawatts at the most.
Jack stood there looking off into space for a few seconds, and then walked out the door…. I thought I saw him shaking his head as he left. Maybe he was just looking both ways for safety reasons, but to me, it looked like a shake of disbelief. I wonder if I had given him the same pain in the neck.
That is really the end of the relay story, but I do want to say a few words about Charles Lay. He was a hard working electrician that was nearing retirement. People would come around to hear us discussing religion. I am Catholic, and he went to a Fundamental Christian Church. We would debate the differences between our beliefs and just Christian beliefs in general. We respected each other during our time together, even though he was sure I am going to hell when I die.
People would come in just to hear our discussion for a while as we were cleaning and calibrating the relays. One day Charles asked me if I could help him figure out how much he was going to receive from his retirement from the electric company. He had only been working there for three years. Retirement at that time was determined by your years of service. So, three years didn’t give him too much.
When I calculated his amount, he was upset. He said, “Am I going to have to work until I die?” I said, “Well, there’s always your 401k and Social Security.” He replied that he can’t live on Social Security. I said, “Well, there’s your 401k.” He asked, “What’s that?” (oh. not a good sign).
I explained that it was a retirement plan where you are able to put money in taxed deferred until you take it out when you retire. He said, “Oh. I never put anything in something like that.” My heart just sank as I looked in his eyes. He had suddenly realized that he wasn’t going to receive a retirement like those around him who had spent 35 years working in the Power Plant.
When I left the plant after teaching Charles Lay how to test the relays, that was the last time I ever saw him. I don’t know what became of Charles. I figure he would be 83 years old today. I wonder if he finally retired when he reached the 80 points for your age and years of service. He would have never reached enough years of service to receive a decent amount of retirement from the Electric Company since he didn’t start working there until he was 60 years old. That is, unless he’s still working there now.
As I said earlier in this post, Charles Lay was a very good worker. He always struck me as the “Hardworking type”. I often think about the time we spent together, especially when I hear about a power blackout somewhere. — A word of caution to Power Companies…. keep your protective relays in proper working condition. Don’t slack off on the Preventative Maintenance. — I guess that’s true for all of us… isn’t it? Don’t slack off on Preventative Maintenance in all aspects of your life.
Pain in the Neck Muskogee Power Plant Relay Testing
RapDon’t let the title fool you. I love testing Power Plant Protective Relays. There is a sense of satisfaction when you have successfully cleaned, calibrated and tested a relay that is going to protect the equipment you have to work on every day. With that said, I was hit with such an unbelievable situation when testing Muskogee Relays in 1995 that I was left with a serious pain in the neck.
On August 14, 2003 the electric power in the Northeast United States and Canada went out. The Blackout lasted long enough to be a major annoyance for those in the that region of the United States.
Map of the power blackout in 2003
When I heard about how the blackout had moved across the region, I immediately knew what had happened. I was quickly reminded of the following story. I told my wife Kelly, “I know exactly why such a large area lost power! They hadn’t done proper preventative maintenance on the Protective Relays in the substations! Just like….” Well…. I’ll tell you that part now:
I have mentioned in a couple of earlier posts that something always seemed a little “off” at the Muskogee Power Plant. I had decided early on that while working there I would stick to drinking sodas instead of water. See the post: “Something’s In the Water at the Muskogee Power Plant“. Even with that knowledge, I was still shocked at what I found while testing relays at the plant.
This story really begins when one Sunday at Muskogee when one of the Auxiliary Operators was making his rounds inspecting equipment. He was driving his truck around the south edge of the Unit 6 parking lot on the service road. He glanced over at a pump next to the road, and at first, he thought he was just seeing things. After stopping the truck and backing up for a second glance, he was sure he wasn’t dreaming. It’s just that what he was seeing seemed so strange, he wasn’t sure what was happening.
The operator could see what appeared to be silver paint chips popping off of the large pump motor in all directions. After closer examination, he figured out that the motor was burning up. It was still running, but it had become so hot that the paint was literally burning off of the motor.
Like this Horizontal pump only much bigger and painted silver
A motor like this would get hot if the bearings shell out. Before the motor is destroyed, the protective relays on the breaker in the 4,000 Volt switchgear shuts the motor off. In this case, the relay hadn’t tripped the motor, so, it had become extremely hot and could have eventually exploded if left running. The operator shut the motor down and wrote a work order for the electricians.
Doyle Fullen was the foreman in the electric shop that received the work order. When he looked into what had happened, he realized that the protective relay had no been inspected for a couple of years for this motor.
I couldn’t find a picture of Doyle. In his youth he reminds me of a very smart Daryl in Walking Dead:
Norman Reedus from Walking Dead
In fact, since before the downsizing in 1994, none of the Protective Relays at the plant had been inspected. The person that had been inspecting the relays for many years had moved to another job or retired in 1994. This was just a warning shot across the bow that could have had major consequences.
No one at Muskogee had been trained to test Protective Relays since the downsizing, so they reached out to our plant in North Central Oklahoma for help. That was when I was told that I was going to be going to Muskogee during the next overhaul (outage). I had been formally trained to inspect, clean, calibrate and test Protective Relays with two of my Power Plant Heroes, Ben Davis and Sonny Kendrick years earlier. See the post: “Relay Tests and Radio Quizzes with Ben Davis“.
My Protective Relay Maintenance course book
Without going into too much detail about the actual tests we performed as I don’t want to make this a long rambling post (like… well…. like most of my posts…..I can already tell this is going to be a long one), I will just say that I took our antiquated relay tester down to Muskogee to inspect their relays and teach another electrician Charles Lay, how to perform those tests in the future. Muskogee had a similar Relay Test Set. These were really outdated, but they did everything we needed, and it helped you understand exactly what was going on when you don’t have a newfangled Relay Test Set.
AVO Multi-Amp SR-76 Relay Test Set
You need to periodically test both mechanical and electronic protective relays. In the electronic relays the components change their properties slightly over time, changing the time it takes to trip a breaker under a given circumstance (we’re talking about milliseconds). In the mechanical relays (which I have always found to be more reliable), they sit inside a black box all the time, heating up and cooling as the equipment is used. Over time, the varnish on the copper coils evaporates and settles on all the components. This becomes sticky so that the relay won’t operate at the point where it should.
A panel of Protective Relays
In the picture above, the black boxes on the top, middle and right are mechanical relays. This means that something actually has to turn or pick up in order to trip the equipment. The electronic relays may have a couple of small relays, but for the most part, they are made up of transistors, resistors, capacitors and diodes.
So, with all that said, let me start the real story…. gee…. It’s about time…
So, here I am sitting in the electric shop lab just off of the Unit 6 T-G floor. We set up all the equipment and had taken a couple of OverCurrent relays out of some high voltage breakers in the switchgear. I told Charles that before you actually start testing the relays, you need to have the test documents from the previous test and we also needed the instruction manuals for each of the relays because the manuals will have the diagrams that you use to determine the exact time that the relays should trip for each of the tests. So, we went up to the print room to find the old tests and manuals. Since they weren’t well organized, we just grabbed the entire folder where all the relays tests were kept since Unit 6 had been in operation.
When we began testing the relays at first I thought that the relay test set wasn’t working correctly. Here I was trying to impress my new friend, Charles Lay, a 63 year old highly religious fundamental Christian that I knew what I was doing, and I couldn’t even make a relay trip. I was trying to find the “As Found” tripping level. That is, before you clean up the relay. Just like you found it. Only, it wouldn’t trip.
It turned out that the relay was stuck from the varnish as I explained above. It appeared as if the relay hadn’t been tested or even operated for years. The paperwork showed that it had been tested three years earlier. Protective Relays should be tested at least every two years, but I wouldn’t have thought that the relay would be in such a bad condition in just three years. It had been sitting in a sealed container to keep out dust. But it was what it was.
I told Charles that in order to find the “As Found” point where the relay would trip, we would need to crank up the test set as high as needed to find when it actually did trip. It turned out that the relay which should have instantaneously tripped somewhere around 150 amps wouldn’t have tripped until the motor was pulling over 4,000 amps. I could tell right away why the Auxiliary Operator found that motor burning up without tripping. The protective relays were stuck.
As it turned out… almost all of the 125 or so relays were in the same condition. We cleaned them all up and made them operational.
There is an overcurrent relay for the main bus on each section of a main switchgear.
A picture of a clean switchgear. Picture 6 rows of switchgear like this
When I tested the “As Found” instantaneous trip for the main bus relay, I found that it was so high that the Unit 6 Main Turbine Generator would have melted down before the protective relay would have tripped the power to that one section of switchgear. The entire electric bus would have been nothing but molten metal by that time.
As I tested each of these relays, I kept shaking my head in disbelief. But that wasn’t the worst of it. The mystery as to why these relays were all glued shut by varnish was finally solved, and that reason was even more unbelievable.
Here is what I found….. The first thing you do when you are going to test a relay is that you fill out a form that includes all the relay information, such as, what it is for, what are the settings on the relay, and what are the levels of tests that you are going to perform on it. You also include a range of milliseconds that are acceptable for the relay for each of the tests. Normally, you just copy what was used in the previous test, because you need to include the time it took for the Previous “As Left” test on your form. That is why we needed the forms from the previous test.
So, I had copied the information from the previous test form and began testing the relay (one of the first overcurrent relays we tested)… Again… I was a 34 year old teacher trying to impress my 63 year old student. So, I was showing him how you mechanically adjust the relay in order for it to trip within the acceptable range. No matter how hard I tried, I couldn’t adjust the relay so that it would even be close to the desired range for the longer time trip times…. like the 2 second to 25 second range. It wasn’t even close to the range that was on the form from the last test.
The form from the last test showed that the relay was in the right range for all the levels of test. When I tested it, like I said, it wasn’t even close. So, I went to the diagram in the instruction manual for this type of relay. The diagram looks similar to this one used for thermal overloads:
Thermal Overload Tripping curves
See all those red lines? Well, when you setup a relay, you have a dial where you set the range depending on the needs for the type of motor you are trying to trip. Each red line represents each setting on the dial. Most of the relays were set on the same number, so we would be using the same red line on the diagram to figure out at different currents how long it should take for a relay to trip….
Here is the clincher….The time range that was written on the previous form wasn’t for the correct relay setting. The person that tested the relay had accidentally looked at the wrong red line. — That in itself is understandable, since it could be easy to get on the wrong line… The only thing is that as soon as you test the relay, you would know that something is wrong, because the relay wouldn’t trip in that range, just like I had found.
I double and triple checked everything to make sure we were looking at the same thing. The previous form indicated the same settings on the relay as now, yet, the time ranges were for a different line! — Ok. I know. I have bored you to tears with all this stuff about time curves and overcurrent trips… so I will just tell you what this means…
This meant that when the person completed the forms the last time, they didn’t test the relays at all. They just filled out the paperwork. They put in random values that were in the acceptable range and sat around in the air conditioned lab during the entire overhaul smoking his pipe. — Actually, I don’t remember if he smoked a pipe or not. He was the Electrical Specialist for the plant. I remembered seeing him sitting in the lab with a relay hooked up to the test set throughout the entire overhaul, but I realized finally that he never tested the relays. He didn’t even go so far as try to operate them.
I went back through the records to when the plant was first “checked out”. Doyle Fullen had done the check out on the relays and the test after that. Doyle had written the correct values from the manual on his forms. I could see where he had actually performed the tests on the relays and was getting the same values I was finding when I tested the relays, so I was certain that I wasn’t overlooking anything.
As I tested each of the relays, I kept shaking my head in disbelief. It was so unbelievable. How could someone do such a thing? Someone could have been killed because a protective relay wasn’t working correctly. This was serious stuff.
One day while Charles and I were working away on the relays, Jack Coffman, the Superintendent of all the Power Plants came walking through the lab. He asked us how we were doing. I swiveled around in my chair to face him and I said, “Pretty good, except for this pain in my neck” as I rubbed the back of my neck.
Jack stopped and asked me what happened. I told him that I had been shaking my head in disbelief for the last two weeks, and it gave me a pain in the neck. Of course, I knew this would get his attention, so he asked, “Why?” I went through all the details of what I had found.
I showed him how since the time that Doyle Fullen last tested the relays more than 10 years earlier, these relays hadn’t been tested at all. I showed him how the main bus relays were so bad that it would take over 100,000 amps to have tripped the 7100 KV switchgear bus.
Jack stood there looking off into space for a few seconds, and then walked out the door…. I thought I saw him shaking his head as he left. Maybe he was just looking both ways for safety reasons, but to me, it looked like a shake of disbelief. I wonder if I had given him the same pain in the neck.
That is really the end of the relay story, but I do want to say a few words about Charles Lay. He was a hard working electrician that was nearing retirement. People would come around to hear us discussing religion. I am Catholic, and he went to a Fundamental Christian Church. We would debate the differences between our beliefs and just Christian beliefs in general. We respected each other during our time together, even though he was sure I am going to hell when I die.
People would come in just to hear our discussion for a while as we were cleaning and calibrating the relays. One day Charles asked me if I could help him figure out how much he was going to receive from his retirement from the electric company. He had only been working there for three years. Retirement at that time was determined by your years of service. So, three years didn’t give him too much.
When I calculated his amount, he was upset. He said, “Am I going to have to work until I die?” I said, “Well, there’s always your 401k and Social Security.” He replied that he can’t live on Social Security. I said, “Well, there’s your 401k.” He asked, “What’s that?” (oh. not a good sign).
I explained that it was a retirement plan where you are able to put money in taxed deferred until you take it out when you retire. He said, “Oh. I never put anything in something like that.” My heart just sank as I looked in his eyes. He had suddenly realized that he wasn’t going to receive a retirement like those around him who had spent 35 years working in the Power Plant.
When I left the plant after teaching Charles Lay how to test the relays, that was the last time I ever saw him. I don’t know what became of Charles. I figure he would be 83 years old today. I wonder if he finally retired when he reached the 80 points for your age and years of service. He would have never reached enough years of service to receive a decent amount of retirement from the Electric Company since he didn’t start working there until he was 60 years old. That is, unless he’s still working there now.
As I said earlier in this post, Charles Lay was a very good worker. He always struck me as the “Hardworking type”. I often think about the time we spent together, especially when I hear about a power blackout somewhere. — A word of caution to Power Companies…. keep your protective relays in proper working condition. Don’t slack off on the Preventative Maintenance. — I guess that’s true for all of us… isn’t it? Don’t slack off on Preventative Maintenance in all aspects of your life.
Power Plant Men 