Pain in the Neck Muskogee Power Plant Relay Testing

RapDon’t let the title fool you.  I love testing Power Plant Protective Relays.  There is a sense of satisfaction when you have successfully cleaned, calibrated and tested a relay that is going to protect the equipment you have to work on every day.  With that said, I was hit with such an unbelievable situation when testing Muskogee Relays in 1995 that I was left with a serious pain in the neck.

On August 14, 2003 the electric power in the Northeast United States and Canada went out.  The Blackout lasted long enough to be a major annoyance for those in the that region of the United States.

 

Map of the power blackout in 2003

Map of the power blackout in 2003

When I heard about how the blackout had moved across the region, I immediately knew what had happened.  I was quickly reminded of the following story.  I told my wife Kelly, “I know exactly why such a large area lost power!  They hadn’t done proper preventative maintenance on the Protective Relays in the substations!  Just like….”  Well…. I’ll tell you that part now:

I have mentioned in a couple of earlier posts that something always seemed a little “off” at the Muskogee Power Plant.  I had decided early on that while working there I would stick to drinking sodas instead of water.  See the post:  “Something’s In the Water at the Muskogee Power Plant“.  Even with that knowledge, I was still shocked at what I found while testing relays at the plant.

This story really begins when one Sunday at Muskogee when one of the Auxiliary Operators was making his rounds inspecting equipment.  He was driving his truck around the south edge of the Unit 6 parking lot on the service road.  He glanced over at a pump next to the road, and at first, he thought he was just seeing things.  After stopping the truck and backing up for a second glance, he was sure he wasn’t dreaming.  It’s just that what he was seeing seemed so strange, he wasn’t sure what was happening.

The operator could see what appeared to be silver paint chips popping off of the large pump motor in all directions.  After closer examination, he figured out that the motor was burning up.  It was still running, but it had become so hot that the paint was literally burning off of the motor.

Horizontal pump

Like this Horizontal pump only much bigger and painted silver

A motor like this would get hot if the bearings shell out.  Before the motor is destroyed, the protective relays on the breaker in the 4,000 Volt switchgear shuts the motor off.  In this case, the relay hadn’t tripped the motor, so, it had become extremely hot and could have eventually exploded if left running.  The operator shut the motor down and wrote a work order for the electricians.

Doyle Fullen was the foreman in the electric shop that received the work order.  When he looked into what had happened, he realized that the protective relay had no been inspected for a couple of years for this motor.

I couldn’t find a picture of Doyle.  In his youth he reminds me of a very smart Daryl in Walking Dead:

Norman Reedus from Walking Dead

Norman Reedus from Walking Dead

In fact, since before the downsizing in 1994, none of the Protective Relays at the plant had been inspected.  The person that had been inspecting the relays for many years had moved to another job or retired in 1994.  This was just a warning shot across the bow that could have had major consequences.

No one at Muskogee had been trained to test Protective Relays since the downsizing, so they reached out to our plant in North Central Oklahoma for help.  That was when I was told that I was going to be going to Muskogee during the next overhaul (outage).  I had been formally trained to inspect, clean, calibrate and test Protective Relays with two of my Power Plant Heroes, Ben Davis and Sonny Kendrick years earlier.  See the post:  “Relay Tests and Radio Quizzes with Ben Davis“.

My Protective Relay Maintenance course book

My Protective Relay Maintenance course book

Without going into too much detail about the actual tests we performed as I don’t want to make this a long rambling post (like… well…. like most of my posts…..I can already tell this is going to be a long one), I will just say that I took our antiquated relay tester down to Muskogee to inspect their relays and teach another electrician Charles Lay, how to perform those tests in the future.  Muskogee had a similar Relay Test Set.  These were really outdated, but they did everything we needed, and it helped you understand exactly what was going on when you don’t have a newfangled Relay Test Set.

AVO Multi-Amp SR-76 Relay Test Set

AVO Multi-Amp SR-76 Relay Test Set

You need to periodically test both mechanical and electronic protective relays.  In the electronic relays the components change their properties slightly over time, changing the time it takes to trip a breaker under a given circumstance (we’re talking about milliseconds).  In the mechanical relays (which I have always found to be more reliable), they sit inside a black box all the time, heating up and cooling as the equipment is used.  Over time, the varnish on the copper coils evaporates and settles on all the components.  This becomes sticky so that the relay won’t operate at the point where it should.

A panel of Protective Relays

A panel of Protective Relays

In the picture above, the black boxes on the top, middle and right are mechanical relays.  This means that something actually has to turn or pick up in order to trip the equipment.  The electronic relays may have a couple of small relays, but for the most part, they are made up of transistors, resistors, capacitors and diodes.

So, with all that said, let me start the real story…. gee…. It’s about time…

So, here I am sitting in the electric shop lab just off of the Unit 6 T-G floor.  We set up all the equipment and had taken a couple of OverCurrent relays out of some high voltage breakers in the switchgear.  I told Charles that before you actually start testing the relays, you need to have the test documents from the previous test and we also needed the instruction manuals for each of the relays because the manuals will have the diagrams that you use to determine the exact time that the relays should trip for each of the tests.  So, we went up to the print room to find the old tests and manuals.  Since they weren’t well organized, we just grabbed the entire folder where all the relays tests were kept since Unit 6 had been in operation.

When we began testing the relays at first I thought that the relay test set wasn’t working correctly.  Here I was trying to impress my new friend, Charles Lay, a 63 year old highly religious fundamental Christian that I knew what I was doing, and I couldn’t even make a relay trip.  I was trying to find the “As Found” tripping level.  That is, before you clean up the relay.  Just like you found it.  Only, it wouldn’t trip.

It turned out that the relay was stuck from the varnish as I explained above.  It appeared as if the relay hadn’t been tested or even operated for years.  The paperwork showed that it had been tested three years earlier.  Protective Relays should be tested at least every two years, but I wouldn’t have thought that the relay would be in such a bad condition in just three years.  It had been sitting in a sealed container to keep out dust.  But it was what it was.

I told Charles that in order to find the “As Found” point where the relay would trip, we would need to crank up the test set as high as needed to find when it actually did trip.  It turned out that the relay which should have instantaneously tripped somewhere around 150 amps wouldn’t have tripped until the motor was pulling over 4,000 amps.  I could tell right away why the Auxiliary Operator found that motor burning up without tripping.  The protective relays were stuck.

As it turned out… almost all of the 125 or so relays were in the same condition.  We cleaned them all up and made them operational.

There is an overcurrent relay for the main bus on each section of a main switchgear.

A picture of a clean switchgear. Picture 6 rows of switchgear like this

A picture of a clean switchgear. Picture 6 rows of switchgear like this

When I tested the “As Found” instantaneous trip for the main bus relay, I found that it was so high that the Unit 6 Main Turbine Generator would have melted down before the protective relay would have tripped the power to that one section of switchgear.  The entire electric bus would have been nothing but molten metal by that time.

As I tested each of these relays, I kept shaking my head in disbelief.  But that wasn’t the worst of it.  The mystery as to why these relays were all glued shut by varnish was finally solved, and that reason was even more unbelievable.

Here is what I found…..  The first thing you do when you are going to test a relay is that you fill out a form that includes all the relay information, such as, what it is for, what are the settings on the relay, and what are the levels of tests that you are going to perform on it.  You also include a range of milliseconds that are acceptable for the relay for each of the tests.  Normally, you just copy what was used in the previous test, because you need to include the time it took for the Previous “As Left” test on your form.  That is why we needed the forms from the previous test.

So, I had copied the information from the previous test form and began testing the relay (one of the first overcurrent relays we tested)…  Again… I was a 34 year old teacher trying to impress my 63 year old student.  So, I was showing him how you mechanically adjust the relay in order for it to trip within the acceptable range.  No matter how hard I tried, I couldn’t adjust the relay so that it would even be close to the desired range for the longer time trip times…. like the 2 second to 25 second range.  It wasn’t even close to the range that was on the form from the last test.

The form from the last test showed that the relay was in the right range for all the levels of test.  When I tested it, like I said, it wasn’t even close.  So, I went to the diagram in the instruction manual for this type of relay.  The diagram looks similar to this one used for thermal overloads:

Thermal Overload Tripping curves

Thermal Overload Tripping curves

See all those red lines?  Well, when you setup a relay, you have a dial where you set the range depending on the needs for the type of motor you are trying to trip. Each red line represents each setting on the dial.  Most of the relays were set on the same number, so we would be using the same red line on the diagram to figure out at different currents how long it should take for a relay to trip….

Here is the clincher….The time range that was written on the previous form wasn’t for the correct relay setting.  The person that tested the relay had accidentally looked at the wrong red line.  — That in itself is understandable, since it could be easy to get on the wrong line… The only thing is that as soon as you test the relay, you would know that something is wrong, because the relay wouldn’t trip in that range, just like I had found.

I double and triple checked everything to make sure we were looking at the same thing.  The previous form indicated the same settings on the relay as now, yet, the time ranges were for a different line! — Ok.  I know.  I have bored you to tears with all this stuff about time curves and overcurrent trips… so I will just tell you what this means…

This meant that when the person completed the forms the last time, they didn’t test the relays at all.  They just filled out the paperwork.  They put in random values that were in the acceptable range and sat around in the air conditioned lab during the entire overhaul smoking his pipe. — Actually, I don’t remember if he smoked a pipe or not.  He was the Electrical Specialist for the plant.  I remembered seeing him sitting in the lab with a relay hooked up to the test set throughout the entire overhaul, but I realized finally that he never tested the relays.  He didn’t even go so far as try to operate them.

I went back through the records to when the plant was first “checked out”.  Doyle Fullen had done the check out on the relays and the test after that.  Doyle had written the correct values from the manual on his forms.  I could see where he had actually performed the tests on the relays and was getting the same values I was finding when I tested the relays, so I was certain that I wasn’t overlooking anything.

As I tested each of the relays, I kept shaking my head in disbelief.  It was so unbelievable.  How could someone do such a thing?  Someone could have been killed because a protective relay wasn’t working correctly.  This was serious stuff.

One day while Charles and I were working away on the relays, Jack Coffman, the Superintendent of all the Power Plants came walking through the lab.  He asked us how we were doing.  I swiveled around in my chair to face him and I said, “Pretty good, except for this pain in my neck” as I rubbed the back of my neck.

Jack stopped and asked me what happened.  I told him that I had been shaking my head in disbelief for the last two weeks, and it gave me a pain in the neck.  Of course, I knew this would get his attention, so he asked, “Why?”  I went through all the details of what I had found.

I showed him how since the time that Doyle Fullen last tested the relays more than 10 years earlier, these relays hadn’t been tested at all.  I showed him how the main bus relays were so bad that it would take over 100,000 amps to have tripped the 7100 KV switchgear bus.

Jack stood there looking off into space for a few seconds, and then walked out the door…. I thought I saw him shaking his head as he left.  Maybe he was just looking both ways for safety reasons, but to me, it looked like a shake of disbelief.  I wonder if I had given him the same pain in the neck.

That is really the end of the relay story, but I do want to say a few words about Charles Lay.  He was a hard working electrician that was nearing retirement.  People would come around to hear us discussing religion.  I am Catholic, and he went to a Fundamental Christian Church.  We would debate the differences between our beliefs and just Christian beliefs in general.  We respected each other during our time together, even though he was sure I am going to hell when I die.

People would come in just to hear our discussion for a while as we were cleaning and calibrating the relays.  One day Charles asked me if I could help him figure out how much he was going to receive from his retirement from the electric company.  He had only been working there for three years.  Retirement at that time was determined by your years of service.  So, three years didn’t give him too much.

When I calculated his amount, he was upset.  He said, “Am I going to have to work until I die?”  I said, “Well, there’s always your 401k and Social Security.”  He replied that he can’t live on Social Security.  I said, “Well, there’s your 401k.”  He asked, “What’s that?” (oh.  not a good sign).

I explained that it was a retirement plan where you are able to put money in taxed deferred until you take it out when you retire.  He said, “Oh.  I never put anything in something like that.”  My heart just sank as I looked in his eyes.  He had suddenly realized that he wasn’t going to receive a retirement like those around him who had spent 35 years working in the Power Plant.

When I left the plant after teaching Charles Lay how to test the relays, that was the last time I ever saw him.  I don’t know what became of Charles.  I figure he would be 83 years old today.  I wonder if he finally retired when he reached the 80 points for your age and years of service.  He would have never reached enough years of service to receive a decent amount of retirement from the Electric Company since he didn’t start working there until he was 60 years old.  That is, unless he’s still working there now.

As I said earlier in this post, Charles Lay was a very good worker.  He always struck me as the “Hardworking type”.  I often think about the time we spent together, especially when I hear about a power blackout somewhere.  — A word of caution to Power Companies…. keep your protective relays in proper working condition.  Don’t slack off on the Preventative Maintenance.  — I guess that’s true for all of us… isn’t it?  Don’t slack off on Preventative Maintenance in all aspects of your life.

Advertisements

13 responses

  1. Reblogged this on A Conservative Christian Man.

    Like

  2. Thank you. Solves the mystery or conspiracies of 2003 and with a somewhat hopeful story. I hope Charles is doing well.

    Liked by 1 person

  3. Great story! And I don’t mind the lengthy technical stuff. I like it.

    We bought our current house in 2004. It was built in 1983. About 4 years ago we lost power to the GFI outlet in our “guest” bathroom and to an outlet in the “Florida room”. I replaced the GFI, but that didn’t fix anything. The power to the GFI was off. The affected bathroom was used very little, so it wasn’t a big deal. In the “Florida room” we just ran an extension cord from a different outlet and went on. I just assumed that a mouse or squirrel in the attack had messed with the wiring. Maybe I had pulled some wiring lose when I was crawling in the attic. Last December, I finally “bit the bullet” and called an electrician. What he found was a GFI in another bathroom had been wired backwards. It was wired in series with the GFI in the other bathroom. The mis-wired GFI had tripped but it didn’t cut the power to it’s own outlet. It only cut power to the other bathroom GFI and the “Florida room” outlet. It cost me $80 to re-learn what I had learned long ago – “Never assume previous work was does correctly – even if done by a “professional””.

    Liked by 1 person

    1. Great story Ron! Thanks for sharing that. That could be serious if the other outlet bypassed the GIF all together.

      Like

  4. KB,
    I suppose the old timers knew more about this! not sure as in they had a saying at relief time between shifts No sparks out of any motors, no paint burning off any bearings, the truck is in the parking spot and filled with gas, all in all had a smooth shift! LOL! What would fascinate me was the mill fires in the early days – you go down there and here is this whole thing cherry red glowing! unbelievable! I have been over to said plant a few times and their switch gears were pretty dirty typically ours were aways very clean and the breakers always worked very well, very reliable and dependable.

    Liked by 1 person

    1. Yeah. That was the first thing I noticed about Muskogee. Our plant took more pride in keeping things clean. To be fair, their plant was close to the coal pile and the plant was down wind. So coal dust would blow right into the plant covering everything.

      Like

  5. Reblogged this on Ace Friends News and commented:
    Really nice write-up. Please note our New World News Post and Site you can visit and post your local or world news snippets here: http://wp.me/5Mvky

    Like

  6. Charles came to our Plant and Calibrated Relays on Unit #1 in Sept. of 2011. Brought his wife with him and called it a working vacation. Still a very nice and intelligent man.

    Liked by 1 person

    1. Thanks Andy for the update.

      Like

  7. Still entertaining reading even for the non-technical types like me.

    Liked by 1 person

  8. This is bringing back the memories. Worked in a coal fired plant for many years as a T.A. (Technicians Assistant) doing the protection maintenance and even the commissioning of a new power station for a couple of years in Australia.
    And I agree, it is a place that gets in your blood. And as you are attesting here, some of the stories are unbelievable.
    And I must put in my bit about those AVO multi Amp test sets. The day that we received one they were affectionately known as the ‘T.A.’s’. Apparently they could be relied on to not make a mistake (unlike the occasional human 🙂 ), and give correct settings. Us human T.A.’s were told, ‘if they could have made coffee they would have sacked the lot of us!’ 😀
    Great story Plant Electrician, a good throwback to yesteryear. I wonder with all the computer gear they now have what has become of the technicians. It’s all probably digitised and in constant readouts to a computer test panel somewhere. One technician probably does all testing from a clean, air-conditioned room somewhere 🙂

    Liked by 1 person

    1. Thanks Mark for your comment. I’m always happy to connect up with fellow former Power Planters.

      Like

  9. Excellent informative post, thank you.

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: