Don’t let the title fool you. I love testing Power Plant Protective Relays. There is a sense of satisfaction when you have successfully cleaned, calibrated and tested a relay that is going to protect the equipment you have to work on every day. With that said, I was hit with such an unbelievable situation when testing Muskogee Relays in 1995 that I was left with a serious pain in the neck.
On August 14, 2003 the electric power in the Northeast United States and Canada went out. The Blackout lasted long enough to be a major annoyance for those in the that region of the United States.
When I heard about how the blackout had moved across the region, I immediately knew what had happened. I was quickly reminded of the following story. I told my wife Kelly, “I know exactly why such a large area lost power! They hadn’t done proper preventative maintenance on the Protective Relays in the substations! Just like….” Well…. I’ll tell you that part now:
I have mentioned in a couple of earlier posts that something always seemed a little “off” at the Muskogee Power Plant. I had decided early on that while working there I would stick to drinking sodas instead of water. See the post: “Something’s In the Water at the Muskogee Power Plant“. Even with that knowledge, I was still shocked at what I found while testing relays at the plant.
This story really begins one Sunday at Muskogee when one of the Auxiliary Operators was making his rounds inspecting equipment. He was driving his truck around the south edge of the Unit 6 parking lot on the service road. He glanced over at a pump next to the road, and at first, he thought he was just seeing things. After stopping the truck and backing up for a second glance, he was sure he wasn’t dreaming. It’s just that what he was seeing seemed so strange, he wasn’t sure what was happening.
The operator could see what appeared to be silver paint chips popping off of the large pump motor in all directions. After closer examination, he figured out that the motor was burning up. It was still running, but it had become so hot that the paint was literally burning off of the motor.
A motor like this would get hot if the bearings shell out. Before the motor is destroyed, the protective relays on the breaker in the 4,000 Volt switchgear shuts the motor off. In this case, the relay hadn’t tripped the motor, so, it had become extremely hot and could have eventually exploded if left running. The operator shut the motor down and wrote a work order for the electricians.
Doyle Fullen was the foreman in the electric shop that received the work order. When he looked into what had happened, he realized that the protective relay had not been inspected for a couple of years for this motor.
I couldn’t find a picture of Doyle. In his youth he reminds me of a very smart Daryl in Walking Dead:
In fact, since before the downsizing in 1994, none of the Protective Relays at the plant had been inspected. The person that had been inspecting the relays for many years had moved to another job or retired in 1994. This was just a warning shot across the bow that could have had major consequences.
No one at Muskogee had been trained to test Protective Relays since the downsizing, so they reached out to our plant in North Central Oklahoma for help. That was when I was told that I was going to be going to Muskogee during the next overhaul (outage). I had been formally trained to inspect, clean, calibrate and test Protective Relays with two of my Power Plant Heroes, Ben Davis and Sonny Kendrick years earlier. See the post: “Relay Tests and Radio Quizzes with Ben Davis“.
Without going into too much detail about the actual tests we performed as I don’t want to make this a long rambling post (like… well…. like most of my posts…..I can already tell this is going to be a long one), I will just say that I took our antiquated relay tester down to Muskogee to inspect their relays and teach another electrician Charles Lay, how to perform those tests in the future. Muskogee had a similar Relay Test Set. These were really outdated, but they did everything we needed, and it helped you understand exactly what was going on when you don’t have a newfangled Relay Test Set.
You need to periodically test both mechanical and electronic protective relays. In the electronic relays the components change their properties slightly over time, changing the time it takes to trip a breaker under a given circumstance (we’re talking about milliseconds). In the mechanical relays (which I have always found to be more reliable), they sit inside a black box all the time, heating up and cooling as the equipment is used. Over time, the varnish on the copper coils evaporates and settles on all the components. This becomes sticky so that the relay won’t operate at the point where it should.
In the picture above, the black boxes on the top, middle and right are mechanical relays. This means that something actually has to turn or pick up in order to trip the equipment. The electronic relays may have a couple of small relays, but for the most part, they are made up of transistors, resistors, capacitors and diodes.
So, with all that said, let me start the real story…. gee…. It’s about time…
So, here I am sitting in the electric shop lab just off of the Unit 6 T-G floor. We set up all the equipment and had taken a couple of OverCurrent relays out of some high voltage breakers in the switchgear. I told Charles that before you actually start testing the relays, you need to have the test documents from the previous test and we also needed the instruction manuals for each of the relays because the manuals will have the diagrams that you use to determine the exact time that the relays should trip for each of the tests. So, we went up to the print room to find the old tests and manuals. Since they weren’t well organized, we just grabbed the entire folder where all the relays tests were kept since Unit 6 had been in operation.
When we began testing the relays at first I thought that the relay test set wasn’t working correctly. Here I was trying to impress my new friend, Charles Lay, a 63 year old highly religious fundamental Christian that I knew what I was doing, and I couldn’t even make a relay trip. I was trying to find the “As Found” tripping level. That is, before you clean up the relay. Just like you found it. Only, it wouldn’t trip.
It turned out that the relay was stuck from the varnish as I explained above. It appeared as if the relay hadn’t been tested or even operated for years. The paperwork showed that it had been tested three years earlier. Protective Relays should be tested at least every two years, but I wouldn’t have thought that the relay would be in such a bad condition in just three years. It had been sitting in a sealed container to keep out dust. But it was what it was.
I told Charles that in order to find the “As Found” point where the relay would trip, we would need to crank up the test set as high as needed to find when it actually did trip. It turned out that the relay which should have instantaneously tripped somewhere around 150 amps wouldn’t have tripped until the motor was pulling over 4,000 amps. I could tell right away why the Auxiliary Operator found that motor burning up without tripping. The protective relays were stuck.
As it turned out… almost all of the 125 or so relays were in the same condition. We cleaned them all up and made them operational.
There is an overcurrent relay for the main bus on each section of a main switchgear.
When I tested the “As Found” instantaneous trip for the main bus relay, I found that it was so high that the Unit 6 Main Turbine Generator would have melted down before the protective relay would have tripped the power to that one section of switchgear. The entire electric bus would have been nothing but molten metal by that time.
As I tested each of these relays, I kept shaking my head in disbelief. But that wasn’t the worst of it. The mystery as to why these relays were all glued shut by varnish was finally solved, and that reason was even more unbelievable.
Here is what I found….. The first thing you do when you are going to test a relay is that you fill out a form that includes all the relay information, such as, what it is for, what are the settings on the relay, and what are the levels of tests that you are going to perform on it. You also include a range of milliseconds that are acceptable for the relay for each of the tests. Normally, you just copy what was used in the previous test, because you need to include the time it took for the Previous “As Left” test on your form. That is why we needed the forms from the previous test.
So, I had copied the information from the previous test form and began testing the relay (one of the first overcurrent relays we tested)… Again… I was a 34 year old teacher trying to impress my 63 year old student. So, I was showing him how you mechanically adjust the relay in order for it to trip within the acceptable range. No matter how hard I tried, I couldn’t adjust the relay so that it would even be close to the desired range for the longer time trip times…. like the 2 second to 25 second range. It wasn’t even close to the range that was on the form from the last test.
The form from the last test showed that the relay was in the right range for all the levels of test. When I tested it, like I said, it wasn’t even close. So, I went to the diagram in the instruction manual for this type of relay. The diagram looks similar to this one used for thermal overloads:
See all those red lines? Well, when you setup a relay, you have a dial where you set the range depending on the needs for the type of motor you are trying to trip. Each red line represents each setting on the dial. Most of the relays were set on the same number, so we would be using the same red line on the diagram to figure out at different currents how long it should take for a relay to trip….
Here is the clincher….The time range that was written on the previous form wasn’t for the correct relay setting. The person that tested the relay had accidentally looked at the wrong red line. — That in itself is understandable, since it could be easy to get on the wrong line… The only thing is that as soon as you test the relay, you would know that something is wrong, because the relay wouldn’t trip in that range, just like I had found.
I double and triple checked everything to make sure we were looking at the same thing. The previous form indicated the same settings on the relay as now, yet, the time ranges were for a different line! — Ok. I know. I have bored you to tears with all this stuff about time curves and overcurrent trips… so I will just tell you what this means…
This meant that when the person completed the forms the last time, they didn’t test the relays at all. They just filled out the paperwork. They put in random values that were in the acceptable range and sat around in the air conditioned lab during the entire overhaul smoking his pipe. — Actually, I don’t remember if he smoked a pipe or not. He was the Electrical Specialist for the plant. I remembered seeing him sitting in the lab with a relay hooked up to the test set throughout the entire overhaul when I had been there during previous overhauls, but I realized finally that he never tested the relays. He didn’t even go so far as try to operate them.
I went back through the records to when the plant was first “checked out”. Doyle Fullen had done the check out on the relays and the test after that. Doyle had written the correct values from the manual on his forms. I could see where he had actually performed the tests on the relays and was getting the same values I was finding when I tested the relays, so I was certain that I wasn’t overlooking anything.
As I tested each of the relays, I kept shaking my head in disbelief. It was so unbelievable. How could someone do such a thing? Someone could have been killed because a protective relay wasn’t working correctly. This was serious stuff.
One day while Charles and I were working away on the relays, Jack Coffman, the Superintendent of all the Power Plants came walking through the lab. He asked us how we were doing. I swiveled around in my chair to face him and I said, “Pretty good, except for this pain in my neck” as I rubbed the back of my neck.
Jack stopped and asked me what happened. I told him that I had been shaking my head in disbelief for the last two weeks, and it gave me a pain in the neck. Of course, I knew this would get his attention, so he asked, “Why?” I went through all the details of what I had found.
I showed him how since the time that Doyle Fullen last tested the relays more than 10 years earlier, these relays hadn’t been tested at all. I showed him how the main bus relays were so bad that it would take over 100,000 amps to have tripped the 7100 KV switchgear bus or 710 Megawatts! More power than the entire generator could generate. It was only rated at about 550 Megawatts at the most.
Jack stood there looking off into space for a few seconds, and then walked out the door…. I thought I saw him shaking his head as he left. Maybe he was just looking both ways for safety reasons, but to me, it looked like a shake of disbelief. I wonder if I had given him the same pain in the neck.
That is really the end of the relay story, but I do want to say a few words about Charles Lay. He was a hard working electrician that was nearing retirement. People would come around to hear us discussing religion. I am Catholic, and he went to a Fundamental Christian Church. We would debate the differences between our beliefs and just Christian beliefs in general. We respected each other during our time together, even though he was sure I am going to hell when I die.
People would come in just to hear our discussion for a while as we were cleaning and calibrating the relays. One day Charles asked me if I could help him figure out how much he was going to receive from his retirement from the electric company. He had only been working there for three years. Retirement at that time was determined by your years of service. So, three years didn’t give him too much.
When I calculated his amount, he was upset. He said, “Am I going to have to work until I die?” I said, “Well, there’s always your 401k and Social Security.” He replied that he can’t live on Social Security. I said, “Well, there’s your 401k.” He asked, “What’s that?” (oh. not a good sign).
I explained that it was a retirement plan where you are able to put money in taxed deferred until you take it out when you retire. He said, “Oh. I never put anything in something like that.” My heart just sank as I looked in his eyes. He had suddenly realized that he wasn’t going to receive a retirement like those around him who had spent 35 years working in the Power Plant.
When I left the plant after teaching Charles Lay how to test the relays, that was the last time I ever saw him. I don’t know what became of Charles. I figure he would be 83 years old today. I wonder if he finally retired when he reached the 80 points for your age and years of service. He would have never reached enough years of service to receive a decent amount of retirement from the Electric Company since he didn’t start working there until he was 60 years old. That is, unless he’s still working there now.
As I said earlier in this post, Charles Lay was a very good worker. He always struck me as the “Hardworking type”. I often think about the time we spent together, especially when I hear about a power blackout somewhere. — A word of caution to Power Companies…. keep your protective relays in proper working condition. Don’t slack off on the Preventative Maintenance. — I guess that’s true for all of us… isn’t it? Don’t slack off on Preventative Maintenance in all aspects of your life.
Added note: On 7/6/2019, 3 weeks after re-posting this story, look what happened: Con Edison says cause of NYC blackout was substation’s faulty relay protection system