Tag Archives: protective relays

Relay Tests and Radio Quizzes with Ben Davis

Originally posted January 24, 2014:

Reorganizations naturally shuffle things around.  People are generally resistant to change and don’t like to find that their routine has been changed without having their input on how to make things better.  When the coal-fired power plant in North Central Oklahoma went through a downsizing and reorganization in the latter part of 1987, my job changed slightly.  Personally, I was grateful for the changes.

Before the reorganization, I had inherited both the precipitators (the large boxes at a power plant that take the ash out of the exhaust from the boiler).  This meant that every overhaul, I knew what I was doing.  I was working on and in the precipitator.  This was generally a dirty and thankless job.

The plant has a similar electrostatic precipitator, only it is twice as long

The plant has a similar electrostatic precipitator, only it is twice as long

After the reorganization, however, Terry Blevins was assigned to work on the Unit 2 precipitator, while I worked on Unit 1.  I will go into this in more detail later, but for this post, I’ll just point out that this meant that when Unit 2 was on an overhaul (that means the unit is taken offline for one to three months in order to fix and repair things that can only be done while it is offline) I wasn’t automatically assigned to the precipitator.  So, I could work on other things.

Before the reorganization, Sonny Kendrick had the title “Electric Specialist”.  After the reorganization we no longer had a specialist.  I’m not sure exactly why.  I know that at Muskogee, they still had a specialist in the electric shop.  — I will talk about him next year (the specialist at Muskogee).  Anyway, I know that Sonny, at the time, was not too happy about his change in job title.  I don’t blame him.  I would be too.

One of the things that the Electric Specialist did during overhauls was test tripping relays.  Now that we no longer had a specialist, that was left up to whomever…. The first electricians, besides Sonny, that were assigned to relay testing was Ben Davis and myself.  I had started doing it on my own and after about a week, Ben Davis was assigned to help me out.

Ben Davis

Ben Davis

We were on a major overhaul on Unit 2 and it had been decided that we were not only going to test the regular super-high voltage breaker relays, we were also going to test all the 480 volt switchgear relays for Unit 2, as well as the intake and coalyard switchgears.  I don’t remember if we made it to the river pump switchgear, but I wouldn’t be surprised.  Once we started, there was no stopping us.

When I first was told to test the relays, Bill Bennett (our A foreman) told me to have Sonny tell me how to do them.  So, I walked into the lab and told Sonny that Bill had told me to ask him to help me learn how to test the protective relays on the switchgear.  Sonny, not looking too happy, grabbed a small stack of manuals, walked out into the main switchgear with me, and said, “Here is the relay test set.  Here are the manuals that tell you how to hook up the test set and test them.”  He turned and walked away…. I was sort of hoping for a more intimate lesson…

I knew the reason Sonny was so upset.  Later I learned why he would be as upset as he was to not be able to test the protective relays.  It was because when you test, clean and adjust protective relays you have an immediate rush of satisfaction that you have just done something very important.  Let me just say quickly (because in another post I will expound upon this), a protective relay is what keeps motors from blowing up.  It is what prevents blackouts from happening across the nation.  Without properly calibrated protective relays, a power company is just asking for a disaster (or… well….. their insurance company is, because they are the ones that usually end up paying for the damage — which I will also talk about in a later post).

I thought the relay test set that Sonny showed me was the neatest thing I had seen so far in the electric shop.  There were two boxes that hooked together with an umbilical cord.  They had dials, switches, connectors, meters and a digital readout down to the millisecond.  That is, you can read the time to trip a relay down to the one thousandth of a second.

AVO Multi-Amp SR-76 Relay Test Set

AVO Multi-Amp SR-76 Relay Test Set

I only wish that I had a bigger picture of this relay test set so that you could admire it as much as I did.  Even today it gives me goosebumps!  Ok.  I can imagine those relay technicians that read this blog are looking at this and thinking…. “What kind of piece of junk is this?”  Hey (as Mark Fielder used to say), this was my “baby” (only he was referring to the precipitator).

So, back to the story at hand…

Even though I was having a heck of a fun time trying to figure out how to perform these relay tests by reading these manuals about the different kinds of relays, I was glad when Ben Davis was assigned to work with me.  I don’t know if he had worked on relays before, but he seemed to know just what to do to hook up the test set and make things easier.

A panel of Protective Relays

A panel of Protective Relays

The best suggestion that Ben had right off the bat was that we should be listening to the radio while we were working.  This might have been a preventative measure after the first couple of days to prevent the same situation from occurring that happened to Ed Shiever when he and I were trapped inside a confined space for a couple of weeks (See the post:  “Ed Shiever Trapped in a Confined Space with a disciple of Ramblin’ Ann“).  Either way, it was a great idea.

You wouldn’t think that inside a switchgear 20 miles from the nearest town with a radio station, that we would have any reception on a little transistor radio, but we were able to manage.  It seemed that we had to be a little creative at times with the antenna in certain locations, but, like I said.  We managed.

My perception of Ben Davis up to this point was that he was a “Good-ol’ boy”.  That is, a country music type Oklahoman that had grown up in Shidler, Oklahoma where the major attraction in the town was the High School.  To my surprise, I quickly found out that he was a connoisseur of Rock and Roll.

It wasn’t until I was in college before I realized that the easy listening station I had been listening to on our family radio at home while I was growing up was playing rock and roll songs using an orchestra with violins and clarinets instead of electric guitars.  I learned from my dorm mates all about groups like Simon and Garfunkel, The Beatles (yeah… can you believe it?  I mean.  I knew “Hey Jude”, “Let it Be” and a few others, but most of the Beatles I thought were instrumentals normally played on violins with a man waving a wand) and many others.  When I found out about “Rock and Roll”, I had to go out and buy dozens of 8-track tapes, as fast as I could find them.

A stack of 8 Track Tapes

A stack of 8 Track Tapes

So, here was Ben Davis.  Even better than the “Good Ol’ Boy” that I already thought he was.  And he loved classical rock and roll.  I can only say that the next month and a half while we tested relays all over the plant, were one of the best times I have ever spent in my life!  He knew all the 60’s and 70’s rock and roll bands.

As each song would come on the radio, we would guess (well, I was guessing most of the time…. most of the time Ben already knew), what the name of the song was and the name of the band.  So, not only were we doing one of the most satisfying jobs at a power plant, but I was also have a lot of fun with Ben listening to the radio!  Who would have thought it?  No wonder Sonny was upset he wasn’t testing relays this overhaul.

I could go on about all the different bands and their backgrounds that I learned from Ben during that overhaul, but (unlike me), you probably already know all that stuff.  It never ceases to amaze me how many holes I have in my education until one is staring at me in the face.

This reminds me of a side story, and I apologize if I have told this before…. I don’t think I have….

After the Reorganization, and after I moved to Stillwater from Ponca City, Scott Hubbard (and Toby O’Brien) and I began carpooling.  One morning as we were listening to NPR, Scott Hubbard mentioned something about a “cur”.  I asked him, “What’s a cur?”  Well, he had the exact same reaction when 11 years earlier I had asked my friends in college at Oklahoma University, Tim Flowers and Kirby Davis, “What’s an orgasm?”  —  See how little holes in your education can make a big impact?

Just so you don’t get caught in the same predicament…  A “Cur” is a mongrel dog.  Scott Hubbard couldn’t believe that someone that read the dictionary for fun wouldn’t know what a “cur” was.  What the heck?  I didn’t grow up in Oklahoma!  — end of side story… which really isn’t a side story, since it was about a Power Plant Man — Scott Hubbard.  He probably knew what a “cur” was before he could walk.  — I know I haven’t told that story before!  I would have remembered that.

I’m not going to go on about all the fun that I had with Ben Davis testing protective relays.  I enjoy my memories, but I’m sure you don’t want to hear all about how much I looked up to this Power Plant Hero.  The only thing I will add is that the time I spent with Ben during that overhaul has been etched into my memory as one of the most enjoyable times of my life.  So, I’ll go onto the next step in our Protective Relay story….

A few years later, in 1993, Sonny Kendrick and Ben Davis and I were sent to “Advanced Protective Relay Maintenance” training in Dallas, Texas.  I remember this time so well, I remember the address where we were went.  It was at 4271 Bronze Way, Dallas, Texas.  It was hosted by the same company that made that wonderful test set I pictured above.  The AVO Multi-Amp Corporation.

I brought my wife Kelly and my three year old daughter Elizabeth with me.  They stayed at the hotel during the day and played in the swimming pool, while I went to class.

The classes lasted four days, Monday through Thursday.  That was where I learned that even though I thought our relay test set was the coolest piece of equipment in the electric shop, it turned out to be archaic by “Protective Relay Maintenance” standards.  Not that it didn’t do the job….   So, in order to train us properly, they let us use our own old test set during the training so that we could see how to properly test really advanced relays such as Distant Relays, Syncro-verifier relays, Negative Sequence Relays,directional distance relays and Pilot Wire relays.  — These are relays that are found in a large substation that trips high voltage lines that run long distances across the country.  — I can tell you’re jealous.  — Well.. I imagine it anyway.  Knowing what I know now.

This is the book we used in class

So, why drag you all the way to Dallas for this story?  There’s a reason.

time for a second side story:

You see. Tim Flowers, whom I mentioned above, knew not too long after he met me that I have the knack of running into people that I know (or should have known in this case), would love this story.  You see, I met Tim and Kirby at Oklahoma University and they drove with me to Columbia Missouri in 1979 (along with my brother Greg) when I went to register for classes at Missouri University when I decided to go back to school in my home town.

When we arrived in the town, we were hungry after driving for 8 hours straight from Stillwater, Oklahoma to Columbia, Missouri… so we stopped at Godfather’s Pizza.  As we walked in, there was a girl and a guy standing at the counter ordering a pizza.  The pretty girl (Pamela Ramsey) with long red hair turned and saw me.  She immediately came toward me saying “Kevin Breazile!!!!  You owe Me!!!  Slightly shocked and pleased, I said, “What for?” She reminded me that I never gave her the pictures that were taken during the Senior Prom.  You see.  I had taken her to the Senior Prom.

Later I explained that this happens to me a lot.  I meet people that I know in the oddest places (even though this wasn’t so odd, since I had grown up in Columbia). It was just that this was the first person we had seen since we entered town.  From that point on, Tim (who later worked as a summer help at the power plant) expected that everywhere we went we would run into someone I knew….

End of the second side story.  I’m sorry that this is making the post a little longer than usual.  I know you have to get back to work….

So, back to the relay training course in 1993 that Ben Davis, Sonny Kendrick and I were taking in Dallas…. On Wednesday night during the training there was a dinner held in a small banquet room in the hotel.  Well… of course I had to take my wife and my daughter.   So here we were sitting around this table at dinner with the rest of the class of about 10 other non-Sooner Plant employees….

I decided to talk to the guy next to me.  He said something back and my wife Kelly asked him, “Where in New Jersey are you from?”  She had picked up on a New Jersey accent.  He said, Well..  I work in the east for a company called Ebasco, but I’m really from the Midwest.  (oh.  That was my territory).  So I asked a follow-up question.  “Where in the Midwest are you from?”  He said, “From Missouri.”  — Oh.  I thought.   This is interesting. So was I.

I asked a follow-up question.  “Where in Missouri are you from?”  He answered…. “Columbia, Missouri.”  (What?   Where I had grown up?)….  So, I asked a second follow-up Question…. “What High School did you go to?”  With a curious look the man answered….. “Rockbridge High School…”   (Man!!!  the same one as me!!!)…. The third follow-up question….. “What year did you graduate?”  Now, looking really suspicious… he said, “1978”.   Trying to contain my excitement… I replied….. “Oh… so, you graduated from Rockbridge High School the same year I did….”

What are the odds?  There were 254 students in our graduating class.  This guy who currently lived somewhere in the east is sitting next to me at a dinner of about 10 people attending Advanced Protective Relay Training in Dallas, Texas where neither of us are from, and we both graduated from the same school back in Columbia, Missouri 15 years earlier!  His name is Randy Loesing.  He was working for a company called Ebasco at the time.  He said, “I thought I recognized you!  I just wasn’t sure.”  I didn’t recognize him at all until I went back home and looked in my yearbook.

It turned out that he kept in touch with two of my oldest friends from the second grade, Mark Schlemper and Brent Stewart.   So we talked about them.  What an incredible coincidence.   Like I may have mentioned before.   It happens to me all the time.  It turns out that an old friend of mine from the 3rd grade in Columbia, Missouri that I used to go to his house when we were stamp collectors and had a stamp collecting club, lives 5 miles south of me today in Round Rock Texas (He’s in Pflugerville).

Russell Somers lives in the  same direction and just about the same number of miles as when we were kids.  Not only that, but he worked at Dell while I was working at Dell (though I didn’t know it at the time).  He has an older daughter and a younger son, just like me only younger.  The same is true for another 3rd grade friend that I  graduated from Rockbridge Highschool and the University of Missouri with, Caryn Lile (now Caryn Iber) who lives in Wisconsin.  She has a daughter and a son the same age as my kids.  She was living in Tulsa when I was living in Stillwater, Oklahoma.  — Like I said… happens to me all the time.

Tim Flowers realized this odd phenomenon  in college.  I had told him earlier that my father told me that if I was every stranded somewhere that I could look up the local Veterinarian and tell him that I was the son of Dr. James Edward Breazile, and they would help me.  So, when we were hiking in the mountains in Colorado and we met a man walking along a trail in the middle of nowhere above Estes Park near the Great Divide, when I told him who I was, he gave us a curious look…. then divulged his most intimate secrets of his life and where he had stashed his most values possessions, Tim told me later.  “I really thought he was going to know who you were when he gave us that funny look.”  I replied.  “I think he did..”

I again apologize for the length of this post.  It is rare that I ramble on this long.  I can thank Ramblin’ Ann for the ability to Ramble so well.  I can thank Ben Davis for recognizing a rambling situation and replacing it with a rock and roll learning opportunity.  As I said earlier.   One of the most enjoyable times I have spent in my entire life is the time I spent with Ben Davis testing Protective Relays!  Bless you Ben and I pray for you, your wife, your son and your daughter on the way to work each morning.

Today when I hear any of the hundreds of rock and roll songs come on the radio that we listened to that month and a half, I can see us testing the relays, looking off into space saying, “Rolling Stones?”  “No.   Steve Miller Band?”  Really?  I thought Browneyed Girl was sung by the Rolling Stone!  It turned out that the version that we listened to was from the creator of the song, Van Morrison. Who would have thought that he would sound so much like Mick Jagger.  I can see Ben saying… I see what you mean…  it kind of sounds like Mick Jagger.

As an add on to this story…

I now work at General Motors in Austin Texas.  My best friend in High School was a guy named Jesse Cheng (I have mentioned him in other posts, especially in reference to the phrase “Jesse!  Come get your Chili!).  He was two years older than me, and throughout the years we would lose track of each other and then reconnect.  He went to Yale to become an Engineer, then to the University of Missouri to become a Medical Doctor, then to Harvard to earn a Masters in Public Health and Epidemiology.

It turns out that we both now work at General Motors where he works in Arlington Texas as a Medical Director and I work in IT in Austin.  We can IM (Instant Message) each other whenever we want, and we talk now at least once every week.

Advertisements

Pain in the Neck Muskogee Power Plant Relay Testing

Don’t let the title fool you.  I love testing Power Plant Protective Relays.  There is a sense of satisfaction when you have successfully cleaned, calibrated and tested a relay that is going to protect the equipment you have to work on every day.  With that said, I was hit with such an unbelievable situation when testing Muskogee Relays in 1995 that I was left with a serious pain in the neck.

On August 14, 2003 the electric power in the Northeast United States and Canada went out.  The Blackout lasted long enough to be a major annoyance for those in the that region of the United States.

Map of the power blackout in 2003

Map of the power blackout in 2003

When I heard about how the blackout had moved across the region, I immediately knew what had happened.  I was quickly reminded of the following story.  I told my wife Kelly, “I know exactly why such a large area lost power!  They hadn’t done proper preventative maintenance on the Protective Relays in the substations!  Just like….”  Well…. I’ll tell you that part now:

I have mentioned in a couple of earlier posts that something always seemed a little “off” at the Muskogee Power Plant.  I had decided early on that while working there I would stick to drinking sodas instead of water.  See the post:  “Something’s In the Water at the Muskogee Power Plant“.  Even with that knowledge, I was still shocked at what I found while testing relays at the plant.

This story really begins one Sunday at Muskogee when one of the Auxiliary Operators was making his rounds inspecting equipment.  He was driving his truck around the south edge of the Unit 6 parking lot on the service road.  He glanced over at a pump next to the road, and at first, he thought he was just seeing things.  After stopping the truck and backing up for a second glance, he was sure he wasn’t dreaming.  It’s just that what he was seeing seemed so strange, he wasn’t sure what was happening.

The operator could see what appeared to be silver paint chips popping off of the large pump motor in all directions.  After closer examination, he figured out that the motor was burning up.  It was still running, but it had become so hot that the paint was literally burning off of the motor.

Horizontal pump

Like this Horizontal pump only much bigger and painted silver

A motor like this would get hot if the bearings shell out.  Before the motor is destroyed, the protective relays on the breaker in the 4,000 Volt switchgear shuts the motor off.  In this case, the relay hadn’t tripped the motor, so, it had become extremely hot and could have eventually exploded if left running.  The operator shut the motor down and wrote a work order for the electricians.

Doyle Fullen was the foreman in the electric shop that received the work order.  When he looked into what had happened, he realized that the protective relay had not been inspected for a couple of years for this motor.

I couldn’t find a picture of Doyle.  In his youth he reminds me of a very smart Daryl in Walking Dead:

Norman Reedus from Walking Dead

Norman Reedus from Walking Dead

In fact, since before the downsizing in 1994, none of the Protective Relays at the plant had been inspected.  The person that had been inspecting the relays for many years had moved to another job or retired in 1994.  This was just a warning shot across the bow that could have had major consequences.

No one at Muskogee had been trained to test Protective Relays since the downsizing, so they reached out to our plant in North Central Oklahoma for help.  That was when I was told that I was going to be going to Muskogee during the next overhaul (outage).  I had been formally trained to inspect, clean, calibrate and test Protective Relays with two of my Power Plant Heroes, Ben Davis and Sonny Kendrick years earlier.  See the post:  “Relay Tests and Radio Quizzes with Ben Davis“.

My Protective Relay Maintenance course book

My Protective Relay Maintenance course book

Without going into too much detail about the actual tests we performed as I don’t want to make this a long rambling post (like… well…. like most of my posts…..I can already tell this is going to be a long one), I will just say that I took our antiquated relay tester down to Muskogee to inspect their relays and teach another electrician Charles Lay, how to perform those tests in the future.  Muskogee had a similar Relay Test Set.  These were really outdated, but they did everything we needed, and it helped you understand exactly what was going on when you don’t have a newfangled Relay Test Set.

AVO Multi-Amp SR-76 Relay Test Set

AVO Multi-Amp SR-76 Relay Test Set

You need to periodically test both mechanical and electronic protective relays.  In the electronic relays the components change their properties slightly over time, changing the time it takes to trip a breaker under a given circumstance (we’re talking about milliseconds).  In the mechanical relays (which I have always found to be more reliable), they sit inside a black box all the time, heating up and cooling as the equipment is used.  Over time, the varnish on the copper coils evaporates and settles on all the components.  This becomes sticky so that the relay won’t operate at the point where it should.

A panel of Protective Relays

A panel of Protective Relays

In the picture above, the black boxes on the top, middle and right are mechanical relays.  This means that something actually has to turn or pick up in order to trip the equipment.  The electronic relays may have a couple of small relays, but for the most part, they are made up of transistors, resistors, capacitors and diodes.

So, with all that said, let me start the real story…. gee…. It’s about time…

So, here I am sitting in the electric shop lab just off of the Unit 6 T-G floor.  We set up all the equipment and had taken a couple of OverCurrent relays out of some high voltage breakers in the switchgear.  I told Charles that before you actually start testing the relays, you need to have the test documents from the previous test and we also needed the instruction manuals for each of the relays because the manuals will have the diagrams that you use to determine the exact time that the relays should trip for each of the tests.  So, we went up to the print room to find the old tests and manuals.  Since they weren’t well organized, we just grabbed the entire folder where all the relays tests were kept since Unit 6 had been in operation.

When we began testing the relays at first I thought that the relay test set wasn’t working correctly.  Here I was trying to impress my new friend, Charles Lay, a 63 year old highly religious fundamental Christian that I knew what I was doing, and I couldn’t even make a relay trip.  I was trying to find the “As Found” tripping level.  That is, before you clean up the relay.  Just like you found it.  Only, it wouldn’t trip.

It turned out that the relay was stuck from the varnish as I explained above.  It appeared as if the relay hadn’t been tested or even operated for years.  The paperwork showed that it had been tested three years earlier.  Protective Relays should be tested at least every two years, but I wouldn’t have thought that the relay would be in such a bad condition in just three years.  It had been sitting in a sealed container to keep out dust.  But it was what it was.

I told Charles that in order to find the “As Found” point where the relay would trip, we would need to crank up the test set as high as needed to find when it actually did trip.  It turned out that the relay which should have instantaneously tripped somewhere around 150 amps wouldn’t have tripped until the motor was pulling over 4,000 amps.  I could tell right away why the Auxiliary Operator found that motor burning up without tripping.  The protective relays were stuck.

As it turned out… almost all of the 125 or so relays were in the same condition.  We cleaned them all up and made them operational.

There is an overcurrent relay for the main bus on each section of a main switchgear.

A picture of a clean switchgear. Picture 6 rows of switchgear like this

A picture of a clean switchgear. Picture 6 rows of switchgear like this

When I tested the “As Found” instantaneous trip for the main bus relay, I found that it was so high that the Unit 6 Main Turbine Generator would have melted down before the protective relay would have tripped the power to that one section of switchgear.  The entire electric bus would have been nothing but molten metal by that time.

As I tested each of these relays, I kept shaking my head in disbelief.  But that wasn’t the worst of it.  The mystery as to why these relays were all glued shut by varnish was finally solved, and that reason was even more unbelievable.

Here is what I found…..  The first thing you do when you are going to test a relay is that you fill out a form that includes all the relay information, such as, what it is for, what are the settings on the relay, and what are the levels of tests that you are going to perform on it.  You also include a range of milliseconds that are acceptable for the relay for each of the tests.  Normally, you just copy what was used in the previous test, because you need to include the time it took for the Previous “As Left” test on your form.  That is why we needed the forms from the previous test.

So, I had copied the information from the previous test form and began testing the relay (one of the first overcurrent relays we tested)…  Again… I was a 34 year old teacher trying to impress my 63 year old student.  So, I was showing him how you mechanically adjust the relay in order for it to trip within the acceptable range.  No matter how hard I tried, I couldn’t adjust the relay so that it would even be close to the desired range for the longer time trip times…. like the 2 second to 25 second range.  It wasn’t even close to the range that was on the form from the last test.

The form from the last test showed that the relay was in the right range for all the levels of test.  When I tested it, like I said, it wasn’t even close.  So, I went to the diagram in the instruction manual for this type of relay.  The diagram looks similar to this one used for thermal overloads:

Thermal Overload Tripping curves

Thermal Overload Tripping curves

See all those red lines?  Well, when you setup a relay, you have a dial where you set the range depending on the needs for the type of motor you are trying to trip. Each red line represents each setting on the dial.  Most of the relays were set on the same number, so we would be using the same red line on the diagram to figure out at different currents how long it should take for a relay to trip….

Here is the clincher….The time range that was written on the previous form wasn’t for the correct relay setting.  The person that tested the relay had accidentally looked at the wrong red line.  — That in itself is understandable, since it could be easy to get on the wrong line… The only thing is that as soon as you test the relay, you would know that something is wrong, because the relay wouldn’t trip in that range, just like I had found.

I double and triple checked everything to make sure we were looking at the same thing.  The previous form indicated the same settings on the relay as now, yet, the time ranges were for a different line! — Ok.  I know.  I have bored you to tears with all this stuff about time curves and overcurrent trips… so I will just tell you what this means…

This meant that when the person completed the forms the last time, they didn’t test the relays at all.  They just filled out the paperwork.  They put in random values that were in the acceptable range and sat around in the air conditioned lab during the entire overhaul smoking his pipe. — Actually, I don’t remember if he smoked a pipe or not.  He was the Electrical Specialist for the plant.  I remembered seeing him sitting in the lab with a relay hooked up to the test set throughout the entire overhaul when I had been there during previous overhauls, but I realized finally that he never tested the relays.  He didn’t even go so far as try to operate them.

I went back through the records to when the plant was first “checked out”.  Doyle Fullen had done the check out on the relays and the test after that.  Doyle had written the correct values from the manual on his forms.  I could see where he had actually performed the tests on the relays and was getting the same values I was finding when I tested the relays, so I was certain that I wasn’t overlooking anything.

As I tested each of the relays, I kept shaking my head in disbelief.  It was so unbelievable.  How could someone do such a thing?  Someone could have been killed because a protective relay wasn’t working correctly.  This was serious stuff.

One day while Charles and I were working away on the relays, Jack Coffman, the Superintendent of all the Power Plants came walking through the lab.  He asked us how we were doing.  I swiveled around in my chair to face him and I said, “Pretty good, except for this pain in my neck” as I rubbed the back of my neck.

Jack stopped and asked me what happened.  I told him that I had been shaking my head in disbelief for the last two weeks, and it gave me a pain in the neck.  Of course, I knew this would get his attention, so he asked, “Why?”  I went through all the details of what I had found.

I showed him how since the time that Doyle Fullen last tested the relays more than 10 years earlier, these relays hadn’t been tested at all.  I showed him how the main bus relays were so bad that it would take over 100,000 amps to have tripped the 7100 KV switchgear bus or 710 Megawatts!  More power than the entire generator could generate.  It was only rated at about 550 Megawatts at the most.

Jack stood there looking off into space for a few seconds, and then walked out the door…. I thought I saw him shaking his head as he left.  Maybe he was just looking both ways for safety reasons, but to me, it looked like a shake of disbelief.  I wonder if I had given him the same pain in the neck.

That is really the end of the relay story, but I do want to say a few words about Charles Lay.  He was a hard working electrician that was nearing retirement.  People would come around to hear us discussing religion.  I am Catholic, and he went to a Fundamental Christian Church.  We would debate the differences between our beliefs and just Christian beliefs in general.  We respected each other during our time together, even though he was sure I am going to hell when I die.

People would come in just to hear our discussion for a while as we were cleaning and calibrating the relays.  One day Charles asked me if I could help him figure out how much he was going to receive from his retirement from the electric company.  He had only been working there for three years.  Retirement at that time was determined by your years of service.  So, three years didn’t give him too much.

When I calculated his amount, he was upset.  He said, “Am I going to have to work until I die?”  I said, “Well, there’s always your 401k and Social Security.”  He replied that he can’t live on Social Security.  I said, “Well, there’s your 401k.”  He asked, “What’s that?” (oh.  not a good sign).

I explained that it was a retirement plan where you are able to put money in taxed deferred until you take it out when you retire.  He said, “Oh.  I never put anything in something like that.”  My heart just sank as I looked in his eyes.  He had suddenly realized that he wasn’t going to receive a retirement like those around him who had spent 35 years working in the Power Plant.

When I left the plant after teaching Charles Lay how to test the relays, that was the last time I ever saw him.  I don’t know what became of Charles.  I figure he would be 83 years old today.  I wonder if he finally retired when he reached the 80 points for your age and years of service.  He would have never reached enough years of service to receive a decent amount of retirement from the Electric Company since he didn’t start working there until he was 60 years old.  That is, unless he’s still working there now.

As I said earlier in this post, Charles Lay was a very good worker.  He always struck me as the “Hardworking type”.  I often think about the time we spent together, especially when I hear about a power blackout somewhere.  — A word of caution to Power Companies…. keep your protective relays in proper working condition.  Don’t slack off on the Preventative Maintenance.  — I guess that’s true for all of us… isn’t it?  Don’t slack off on Preventative Maintenance in all aspects of your life.

Added note:  On 7/6/2019, 3 weeks after re-posting this story, look what happened:  Con Edison says cause of NYC blackout was substation’s faulty relay protection system

Relay Tests and Radio Quizzes with Ben Davis

Originally posted January 24, 2014:

Reorganizations naturally shuffle things around.  People are generally resistant to change and don’t like to find that their routine has been changed without having their input on how to make things better.  When the coal-fired power plant in North Central Oklahoma went through a downsizing and reorganization in the latter part of 1987, my job changed slightly.  Personally, I was grateful for the changes.

Before the reorganization, I had inherited both the precipitators (the large boxes at a power plant that take the ash out of the exhaust from the boiler).  This meant that every overhaul, I knew what I was doing.  I was working on and in the precipitator.  This was generally a dirty and thankless job.

The plant has a similar electrostatic precipitator, only it is twice as long

The plant has a similar electrostatic precipitator, only it is twice as long

After the reorganization, however, Terry Blevins was assigned to work on the Unit 2 precipitator, while I worked on Unit 1.  I will go into this in more detail later, but for this post, I’ll just point out that this meant that when Unit 2 was on an overhaul (that means the unit is taken offline for one to three months in order to fix and repair things that can only be done while it is offline) I wasn’t automatically assigned to the precipitator.  So, I could work on other things.

Before the reorganization, Sonny Kendrick had the title “Electric Specialist”.  After the reorganization we no longer had a specialist.  I’m not sure exactly why.  I know that at Muskogee, they still had a specialist in the electric shop.  — I will talk about him next year (the specialist at Muskogee).  Anyway, I know that Sonny, at the time, was not too happy about his change in job title.  I don’t blame him.  I would be too.

One of the things that the Electric Specialist did during overhauls was test tripping relays.  Now that we no longer had a specialist, that was left up to whomever…. The first electricians, besides Sonny, that were assigned to relay testing was Ben Davis and myself.  I had started doing it on my own and after about a week, Ben Davis was assigned to help me out.

Ben Davis

Ben Davis

We were on a major overhaul on Unit 2 and it had been decided that we were not only going to test the regular super-high voltage breaker relays, we were also going to test all the 480 volt switchgear relays for Unit 2, as well as the intake and coalyard switchgears.  I don’t remember if we made it to the river pump switchgear, but I wouldn’t be surprised.  Once we started, there was no stopping us.

When I first was told to test the relays, Bill Bennett (our A foreman) told me to have Sonny tell me how to do them.  So, I walked into the lab and told Sonny that Bill had told me to ask him to help me learn how to test the protective relays on the switchgear.  Sonny, not looking too happy, grabbed a small stack of manuals, walked out into the main switchgear with me, and said, “Here is the relay test set.  Here are the manuals that tell you how to hook up the test set and test them.”  He turned and walked away…. I was sort of hoping for a more intimate lesson…

I knew the reason Sonny was so upset.  Later I learned why he would be as upset as he was to not be able to test the protective relays.  It was because when you test, clean and adjust protective relays you have an immediate rush of satisfaction that you have just done something very important.  Let me just say quickly (because in another post I will expound upon this), a protective relay is what keeps motors from blowing up.  It is what prevents blackouts from happening across the nation.  Without properly calibrated protective relays, a power company is just asking for a disaster (or… well….. their insurance company is, because they are the ones that usually end up paying for the damage — which I will also talk about in a later post).

I thought the relay test set that Sonny showed me was the neatest thing I had seen so far in the electric shop.  There were two boxes that hooked together with an umbilical cord.  They had dials, switches, connectors, meters and a digital readout down to the millisecond.  That is, you can read the time to trip a relay down to the one thousandth of a second.

AVO Multi-Amp SR-76 Relay Test Set

AVO Multi-Amp SR-76 Relay Test Set

I only wish that I had a bigger picture of this relay test set so that you could admire it as much as I did.  Even today it gives me goosebumps!  Ok.  I can imagine those relay technicians that read this blog are looking at this and thinking…. “What kind of piece of junk is this?”  Hey (as Mark Fielder used to say), this was my “baby” (only he was referring to the precipitator).

So, back to the story at hand…

Even though I was having a heck of a fun time trying to figure out how to perform these relay tests by reading these manuals about the different kinds of relays, I was glad when Ben Davis was assigned to work with me.  I don’t know if he had worked on relays before, but he seemed to know just what to do to hook up the test set and make things easier.

A panel of Protective Relays

A panel of Protective Relays

The best suggestion that Ben had right off the bat was that we should be listening to the radio while we were working.  This might have been a preventative measure after the first couple of days to prevent the same situation from occurring that happened to Ed Shiever when he and I were trapped inside a confined space for a couple of weeks (See the post:  “Ed Shiever Trapped in a Confined Space with a disciple of Ramblin’ Ann“).  Either way, it was a great idea.

You wouldn’t think that inside a switchgear 20 miles from the nearest town with a radio station, that we would have any reception on a little transistor radio, but we were able to manage.  It seemed that we had to be a little creative at times with the antenna in certain locations, but, like I said.  We managed.

My perception of Ben Davis up to this point was that he was a “Good-ol’ boy”.   That is, a country music type Oklahoman that had grown up in Shidler, Oklahoma where the major attraction in the town was the High School.  To my surprise, I quickly found out that he was a connoisseur of Rock and Roll.

It wasn’t until I was in college before I realized that the easy listening station I had been listening to on our family radio at home while I was growing up was playing rock and roll songs using an orchestra with violins and clarinets instead of electric guitars.  I learned from my dorm mates all about groups like Simon and Garfunkel, The Beatles (yeah… can you believe it?  I mean.  I knew “Hey Jude”, “Let it Be” and a few others, but most of the Beatles I thought were instrumentals normally played on violins with a man waving a wand) and many others.  When I found out about “Rock and Roll”, I had to go out and buy dozens of 8-track tapes, as fast as I could find them.

A stack of 8 Track Tapes

A stack of 8 Track Tapes

So, here was Ben Davis.  Even better than the “Good Ol’ Boy” that I already thought he was.  And he loved classical rock and roll.  I can only say that the next month and a half while we tested relays all over the plant, were one of the best times I have ever spent in my life!  He knew all the 60’s and 70’s rock and roll bands.

As each song would come on the radio, we would guess (well, I was guessing most of the time…. most of the time Ben already knew), what the name of the song was and the name of the band.  So, not only were we doing one of the most satisfying jobs at a power plant, but I was also have a lot of fun with Ben listening to the radio!  Who would have thought it?  No wonder Sonny was upset he wasn’t testing relays this overhaul.

I could go on about all the different bands and their backgrounds that I learned from Ben during that overhaul, but (unlike me), you probably already know all that stuff.  It never ceases to amaze me how many holes I have in my education until one is staring at me in the face.

This reminds me of a side story, and I apologize if I have told this before…. I don’t think I have….

After the Reorganization, and after I moved to Stillwater from Ponca City, Scott Hubbard (and Toby O’Brien) and I began carpooling.  One morning as we were listening to NPR, Scott Hubbard mentioned something about a “cur”.  I asked him, “What’s a cur?”  Well, he had the exact same reaction when 11 years earlier I had asked my friends in college at Oklahoma University, Tim Flowers and Kirby Davis, “What’s an orgasm?”  —  See how little holes in your education can make a big impact?

Just so you don’t get caught in the same predicament…  A “Cur” is a mongrel dog.  Scott Hubbard couldn’t believe that someone that read the dictionary for fun wouldn’t know what a “cur” was.  What the heck?  I didn’t grow up in Oklahoma!  — end of side story… which really isn’t a side story, since it was about a Power Plant Man — Scott Hubbard.  He probably knew what a “cur” was before he could walk.  — I know I haven’t told that story before!  I would have remembered that.

I’m not going to go on about all the fun that I had with Ben Davis testing protective relays.  I enjoy my memories, but I’m sure you don’t want to hear all about how much I looked up to this Power Plant Hero.  The only thing I will add is that the time I spent with Ben during that overhaul has been etched into my memory as one of the most enjoyable times of my life.  So, I’ll go onto the next step in our Protective Relay story….

A few years later, in 1993, Sonny Kendrick and Ben Davis and I were sent to “Advanced Protective Relay Maintenance” training in Dallas, Texas.  I remember this time so well, I remember the address where we were went.  It was at 4271 Bronze Way, Dallas, Texas.  It was hosted by the same company that made that wonderful test set I pictured above.  The AVO Multi-Amp Corporation.

I brought my wife Kelly and my three year old daughter Elizabeth with me.  They stayed at the hotel during the day and played in the swimming pool, while I went to class.

The classes lasted four days, Monday through Thursday.  That was where I learned that even though I thought our relay test set was the coolest piece of equipment in the electric shop, it turned out to be archaic by “Protective Relay Maintenance” standards.  Not that it didn’t do the job….   So, in order to train us properly, they let us use our own old test set during the training so that we could see how to properly test really advanced relays such as Distant Relays, Syncro-verifier relays, Negative Sequence Relays,directional distance relays and Pilot Wire relays.  — These are relays that are found in a large substation that trips high voltage lines that run long distances across the country.  — I can tell you’re jealous.  — Well.. I imagine it anyway.  Knowing what I know now.

This is the book we used in class

So, why drag you all the way to Dallas for this story?  There’s a reason.

time for a second side story:

You see. Tim Flowers, whom I mentioned above, knew not too long after he met me that I have the knack of running into people that I know (or should have known in this case), would love this story.  You see, I met Tim and Kirby at Oklahoma University and they drove with me to Columbia Missouri in 1979 (along with my brother Greg) when I went to register for classes at Missouri University when I decided to go back to school in my home town.

When we arrived in the town, we were hungry after driving for 8 hours straight from Stillwater, Oklahoma to Columbia, Missouri… so we stopped at Godfather’s Pizza.  As we walked in, there was a girl and a guy standing at the counter ordering a pizza.  The pretty girl (Pamela Ramsey) with long red hair turned and saw me.  She immediately came toward me saying “Kevin Breazile!!!!  You owe Me!!!  Slightly shocked and pleased, I said, “What for?” She reminded me that I never gave her the pictures that were taken during the Senior Prom.  You see.  I had taken her to the Senior Prom.

Later I explained that this happens to me a lot.  I meet people that I know in the oddest places (even though this wasn’t so odd, since I had grown up in Columbia). It was just that this was the first person we had seen since we entered town.  From that point on, Tim (who later worked as a summer help at the power plant) expected that everywhere we went we would run into someone I knew….

End of the second side story.  I’m sorry that this is making the post a little longer than usual.  I know you have to get back to work….

So, back to the relay training course in 1993 that Ben Davis, Sonny Kendrick and I were taking in Dallas…. On Wednesday night during the training there was a dinner held in a small banquet room in the hotel.  Well… of course I had to take my wife and my daughter.   So here we were sitting around this table at dinner with the rest of the class of about 10 other non-Sooner Plant employees….

I decided to talk to the guy next to me.  He said something back and my wife Kelly asked him, “Where in New Jersey are you from?”  She had picked up on a New Jersey accent.  He said, Well..  I work in the east for a company called Ebasco, but I’m really from the Midwest.  (oh.  That was my territory).  So I asked a follow-up question.  “Where in the Midwest are you from?”  He said, “From Missouri.”  — Oh.  I thought.   This is interesting. So was I.

I asked a follow-up question.  “Where in Missouri are you from?”  He answered…. “Columbia, Missouri.”  (What?   Where I had grown up?)….  So, I asked a second follow-up Question…. “What High School did you go to?”  With a curious look the man answered….. “Rockbridge High School…”   (Man!!!  the same one as me!!!)…. The third follow-up question….. “What year did you graduate?”  Now, looking really suspicious… he said, “1978”.   Trying to contain my excitement… I replied….. “Oh… so, you graduated from Rockbridge High School the same year I did….”

What are the odds?  There were 254 students in our graduating class.  This guy who currently lived somewhere in the east is sitting next to me at a dinner of about 10 people attending Advanced Protective Relay Training in Dallas, Texas where neither of us are from, and we both graduated from the same school back in Columbia, Missouri 15 years earlier!  His name is Randy Loesing.  He was working for a company called Ebasco at the time.  He said, “I thought I recognized you!  I just wasn’t sure.”  I didn’t recognize him at all until I went back home and looked in my yearbook.

It turned out that he kept in touch with two of my oldest friends from the second grade, Mark Schlemper and Brent Stewart.   So we talked about them.  What an incredible coincidence.   Like I may have mentioned before.   It happens to me all the time.  It turns out that an old friend of mine from the 3rd grade in Columbia, Missouri that I used to go to his house when we were stamp collectors and had a stamp collecting club, lives 5 miles south of me today in Round Rock Texas (He’s in Pflugerville).

Russell Somers lives in the  same direction and just about the same number of miles as when we were kids.  Not only that, but he worked at Dell while I was working at Dell (though I didn’t know it at the time).  He has an older daughter and a younger son, just like me only younger.  The same is true for another 3rd grade friend that I  graduated from Rockbridge Highschool and the University of Missouri with, Caryn Lile (now Caryn Iber) who lives in Wisconsin.  She has a daughter and a son the same age as my kids.  She was living in Tulsa when I was living in Stillwater, Oklahoma.  — Like I said… happens to me all the time.

Tim Flowers realized this odd phenomenon  in college.  I had told him earlier that my father told me that if I was every stranded somewhere that I could look up the local Veterinarian and tell him that I was the son of Dr. James Edward Breazile, and they would help me.  So, when we were hiking in the mountains in Colorado and we met a man walking along a trail in the middle of nowhere above Estes Park near the Great Divide, when I told him who I was, he gave us a curious look…. then divulged his most intimate secrets of his life and where he had stashed his most values possessions, Tim told me later.  “I really thought he was going to know who you were when he gave us that funny look.”  I replied.  “I think he did..”

I again apologize for the length of this post.  It is rare that I ramble on this long.  I can thank Ramblin’ Ann for the ability to Ramble so well.  I can thank Ben Davis for recognizing a rambling situation and replacing it with a rock and roll learning opportunity.  As I said earlier.   One of the most enjoyable times I have spent in my entire life is the time I spent with Ben Davis testing Protective Relays!  Bless you Ben and I pray for you, your wife, your son and your daughter on the way to work each morning.

Today when I hear any of the hundreds of roll and roll songs come on the radio that we listened to that month and a half, I can see us testing the relays, looking off into space saying, “Rolling Stones?”  “No.   Steve Miller Band?”  Really?  I thought Browneyed Girl was sung by the Rolling Stone!  It turned out that the version that we listened to was from the creator of the song, Van Morrison. Who would have thought that he would sound so much like Mick Jagger.  I can see Ben saying… I see what you mean…  it kind of sounds like Mick Jagger.

As an add on to this story…

I now work at General Motors in Austin Texas.  My best friend in High School was a guy named Jesse Cheng (I have mentioned him in other posts, especially in reference to the phrase “Jesse!  Come get your Chili!).  He was two years older than me, and throughout the years we would lose track of each other and then reconnect.  He went to Yale to become an Engineer, then to the University of Missouri to become a Medical Doctor, then to Harvard to earn a Masters in Public Health and Epidemiology.

It turns out that we both now work at General Motors where he works in Arlington Texas as a Medical Director and I work in IT in Austin.  We can IM (Instant Message) each other whenever we want, and we talk now at least once every week.

Pain in the Neck Muskogee Power Plant Relay Testing

Don’t let the title fool you.  I love testing Power Plant Protective Relays.  There is a sense of satisfaction when you have successfully cleaned, calibrated and tested a relay that is going to protect the equipment you have to work on every day.  With that said, I was hit with such an unbelievable situation when testing Muskogee Relays in 1995 that I was left with a serious pain in the neck.

On August 14, 2003 the electric power in the Northeast United States and Canada went out.  The Blackout lasted long enough to be a major annoyance for those in the that region of the United States.

 

Map of the power blackout in 2003

Map of the power blackout in 2003

When I heard about how the blackout had moved across the region, I immediately knew what had happened.  I was quickly reminded of the following story.  I told my wife Kelly, “I know exactly why such a large area lost power!  They hadn’t done proper preventative maintenance on the Protective Relays in the substations!  Just like….”  Well…. I’ll tell you that part now:

I have mentioned in a couple of earlier posts that something always seemed a little “off” at the Muskogee Power Plant.  I had decided early on that while working there I would stick to drinking sodas instead of water.  See the post:  “Something’s In the Water at the Muskogee Power Plant“.  Even with that knowledge, I was still shocked at what I found while testing relays at the plant.

This story really begins one Sunday at Muskogee when one of the Auxiliary Operators was making his rounds inspecting equipment.  He was driving his truck around the south edge of the Unit 6 parking lot on the service road.  He glanced over at a pump next to the road, and at first, he thought he was just seeing things.  After stopping the truck and backing up for a second glance, he was sure he wasn’t dreaming.  It’s just that what he was seeing seemed so strange, he wasn’t sure what was happening.

The operator could see what appeared to be silver paint chips popping off of the large pump motor in all directions.  After closer examination, he figured out that the motor was burning up.  It was still running, but it had become so hot that the paint was literally burning off of the motor.

Horizontal pump

Like this Horizontal pump only much bigger and painted silver

A motor like this would get hot if the bearings shell out.  Before the motor is destroyed, the protective relays on the breaker in the 4,000 Volt switchgear shuts the motor off.  In this case, the relay hadn’t tripped the motor, so, it had become extremely hot and could have eventually exploded if left running.  The operator shut the motor down and wrote a work order for the electricians.

Doyle Fullen was the foreman in the electric shop that received the work order.  When he looked into what had happened, he realized that the protective relay had not been inspected for a couple of years for this motor.

I couldn’t find a picture of Doyle.  In his youth he reminds me of a very smart Daryl in Walking Dead:

Norman Reedus from Walking Dead

Norman Reedus from Walking Dead

In fact, since before the downsizing in 1994, none of the Protective Relays at the plant had been inspected.  The person that had been inspecting the relays for many years had moved to another job or retired in 1994.  This was just a warning shot across the bow that could have had major consequences.

No one at Muskogee had been trained to test Protective Relays since the downsizing, so they reached out to our plant in North Central Oklahoma for help.  That was when I was told that I was going to be going to Muskogee during the next overhaul (outage).  I had been formally trained to inspect, clean, calibrate and test Protective Relays with two of my Power Plant Heroes, Ben Davis and Sonny Kendrick years earlier.  See the post:  “Relay Tests and Radio Quizzes with Ben Davis“.

My Protective Relay Maintenance course book

My Protective Relay Maintenance course book

Without going into too much detail about the actual tests we performed as I don’t want to make this a long rambling post (like… well…. like most of my posts…..I can already tell this is going to be a long one), I will just say that I took our antiquated relay tester down to Muskogee to inspect their relays and teach another electrician Charles Lay, how to perform those tests in the future.  Muskogee had a similar Relay Test Set.  These were really outdated, but they did everything we needed, and it helped you understand exactly what was going on when you don’t have a newfangled Relay Test Set.

AVO Multi-Amp SR-76 Relay Test Set

AVO Multi-Amp SR-76 Relay Test Set

You need to periodically test both mechanical and electronic protective relays.  In the electronic relays the components change their properties slightly over time, changing the time it takes to trip a breaker under a given circumstance (we’re talking about milliseconds).  In the mechanical relays (which I have always found to be more reliable), they sit inside a black box all the time, heating up and cooling as the equipment is used.  Over time, the varnish on the copper coils evaporates and settles on all the components.  This becomes sticky so that the relay won’t operate at the point where it should.

A panel of Protective Relays

A panel of Protective Relays

In the picture above, the black boxes on the top, middle and right are mechanical relays.  This means that something actually has to turn or pick up in order to trip the equipment.  The electronic relays may have a couple of small relays, but for the most part, they are made up of transistors, resistors, capacitors and diodes.

So, with all that said, let me start the real story…. gee…. It’s about time…

So, here I am sitting in the electric shop lab just off of the Unit 6 T-G floor.  We set up all the equipment and had taken a couple of OverCurrent relays out of some high voltage breakers in the switchgear.  I told Charles that before you actually start testing the relays, you need to have the test documents from the previous test and we also needed the instruction manuals for each of the relays because the manuals will have the diagrams that you use to determine the exact time that the relays should trip for each of the tests.  So, we went up to the print room to find the old tests and manuals.  Since they weren’t well organized, we just grabbed the entire folder where all the relays tests were kept since Unit 6 had been in operation.

When we began testing the relays at first I thought that the relay test set wasn’t working correctly.  Here I was trying to impress my new friend, Charles Lay, a 63 year old highly religious fundamental Christian that I knew what I was doing, and I couldn’t even make a relay trip.  I was trying to find the “As Found” tripping level.  That is, before you clean up the relay.  Just like you found it.  Only, it wouldn’t trip.

It turned out that the relay was stuck from the varnish as I explained above.  It appeared as if the relay hadn’t been tested or even operated for years.  The paperwork showed that it had been tested three years earlier.  Protective Relays should be tested at least every two years, but I wouldn’t have thought that the relay would be in such a bad condition in just three years.  It had been sitting in a sealed container to keep out dust.  But it was what it was.

I told Charles that in order to find the “As Found” point where the relay would trip, we would need to crank up the test set as high as needed to find when it actually did trip.  It turned out that the relay which should have instantaneously tripped somewhere around 150 amps wouldn’t have tripped until the motor was pulling over 4,000 amps.  I could tell right away why the Auxiliary Operator found that motor burning up without tripping.  The protective relays were stuck.

As it turned out… almost all of the 125 or so relays were in the same condition.  We cleaned them all up and made them operational.

There is an overcurrent relay for the main bus on each section of a main switchgear.

A picture of a clean switchgear. Picture 6 rows of switchgear like this

A picture of a clean switchgear. Picture 6 rows of switchgear like this

When I tested the “As Found” instantaneous trip for the main bus relay, I found that it was so high that the Unit 6 Main Turbine Generator would have melted down before the protective relay would have tripped the power to that one section of switchgear.  The entire electric bus would have been nothing but molten metal by that time.

As I tested each of these relays, I kept shaking my head in disbelief.  But that wasn’t the worst of it.  The mystery as to why these relays were all glued shut by varnish was finally solved, and that reason was even more unbelievable.

Here is what I found…..  The first thing you do when you are going to test a relay is that you fill out a form that includes all the relay information, such as, what it is for, what are the settings on the relay, and what are the levels of tests that you are going to perform on it.  You also include a range of milliseconds that are acceptable for the relay for each of the tests.  Normally, you just copy what was used in the previous test, because you need to include the time it took for the Previous “As Left” test on your form.  That is why we needed the forms from the previous test.

So, I had copied the information from the previous test form and began testing the relay (one of the first overcurrent relays we tested)…  Again… I was a 34 year old teacher trying to impress my 63 year old student.  So, I was showing him how you mechanically adjust the relay in order for it to trip within the acceptable range.  No matter how hard I tried, I couldn’t adjust the relay so that it would even be close to the desired range for the longer time trip times…. like the 2 second to 25 second range.  It wasn’t even close to the range that was on the form from the last test.

The form from the last test showed that the relay was in the right range for all the levels of test.  When I tested it, like I said, it wasn’t even close.  So, I went to the diagram in the instruction manual for this type of relay.  The diagram looks similar to this one used for thermal overloads:

Thermal Overload Tripping curves

Thermal Overload Tripping curves

See all those red lines?  Well, when you setup a relay, you have a dial where you set the range depending on the needs for the type of motor you are trying to trip. Each red line represents each setting on the dial.  Most of the relays were set on the same number, so we would be using the same red line on the diagram to figure out at different currents how long it should take for a relay to trip….

Here is the clincher….The time range that was written on the previous form wasn’t for the correct relay setting.  The person that tested the relay had accidentally looked at the wrong red line.  — That in itself is understandable, since it could be easy to get on the wrong line… The only thing is that as soon as you test the relay, you would know that something is wrong, because the relay wouldn’t trip in that range, just like I had found.

I double and triple checked everything to make sure we were looking at the same thing.  The previous form indicated the same settings on the relay as now, yet, the time ranges were for a different line! — Ok.  I know.  I have bored you to tears with all this stuff about time curves and overcurrent trips… so I will just tell you what this means…

This meant that when the person completed the forms the last time, they didn’t test the relays at all.  They just filled out the paperwork.  They put in random values that were in the acceptable range and sat around in the air conditioned lab during the entire overhaul smoking his pipe. — Actually, I don’t remember if he smoked a pipe or not.  He was the Electrical Specialist for the plant.  I remembered seeing him sitting in the lab with a relay hooked up to the test set throughout the entire overhaul, but I realized finally that he never tested the relays.  He didn’t even go so far as try to operate them.

I went back through the records to when the plant was first “checked out”.  Doyle Fullen had done the check out on the relays and the test after that.  Doyle had written the correct values from the manual on his forms.  I could see where he had actually performed the tests on the relays and was getting the same values I was finding when I tested the relays, so I was certain that I wasn’t overlooking anything.

As I tested each of the relays, I kept shaking my head in disbelief.  It was so unbelievable.  How could someone do such a thing?  Someone could have been killed because a protective relay wasn’t working correctly.  This was serious stuff.

One day while Charles and I were working away on the relays, Jack Coffman, the Superintendent of all the Power Plants came walking through the lab.  He asked us how we were doing.  I swiveled around in my chair to face him and I said, “Pretty good, except for this pain in my neck” as I rubbed the back of my neck.

Jack stopped and asked me what happened.  I told him that I had been shaking my head in disbelief for the last two weeks, and it gave me a pain in the neck.  Of course, I knew this would get his attention, so he asked, “Why?”  I went through all the details of what I had found.

I showed him how since the time that Doyle Fullen last tested the relays more than 10 years earlier, these relays hadn’t been tested at all.  I showed him how the main bus relays were so bad that it would take over 100,000 amps to have tripped the 7100 KV switchgear bus or 710 Megawatts!  More power than the entire generator could generate.  It was only rated at about 550 Megawatts at the most.

Jack stood there looking off into space for a few seconds, and then walked out the door…. I thought I saw him shaking his head as he left.  Maybe he was just looking both ways for safety reasons, but to me, it looked like a shake of disbelief.  I wonder if I had given him the same pain in the neck.

That is really the end of the relay story, but I do want to say a few words about Charles Lay.  He was a hard working electrician that was nearing retirement.  People would come around to hear us discussing religion.  I am Catholic, and he went to a Fundamental Christian Church.  We would debate the differences between our beliefs and just Christian beliefs in general.  We respected each other during our time together, even though he was sure I am going to hell when I die.

People would come in just to hear our discussion for a while as we were cleaning and calibrating the relays.  One day Charles asked me if I could help him figure out how much he was going to receive from his retirement from the electric company.  He had only been working there for three years.  Retirement at that time was determined by your years of service.  So, three years didn’t give him too much.

When I calculated his amount, he was upset.  He said, “Am I going to have to work until I die?”  I said, “Well, there’s always your 401k and Social Security.”  He replied that he can’t live on Social Security.  I said, “Well, there’s your 401k.”  He asked, “What’s that?” (oh.  not a good sign).

I explained that it was a retirement plan where you are able to put money in taxed deferred until you take it out when you retire.  He said, “Oh.  I never put anything in something like that.”  My heart just sank as I looked in his eyes.  He had suddenly realized that he wasn’t going to receive a retirement like those around him who had spent 35 years working in the Power Plant.

When I left the plant after teaching Charles Lay how to test the relays, that was the last time I ever saw him.  I don’t know what became of Charles.  I figure he would be 83 years old today.  I wonder if he finally retired when he reached the 80 points for your age and years of service.  He would have never reached enough years of service to receive a decent amount of retirement from the Electric Company since he didn’t start working there until he was 60 years old.  That is, unless he’s still working there now.

As I said earlier in this post, Charles Lay was a very good worker.  He always struck me as the “Hardworking type”.  I often think about the time we spent together, especially when I hear about a power blackout somewhere.  — A word of caution to Power Companies…. keep your protective relays in proper working condition.  Don’t slack off on the Preventative Maintenance.  — I guess that’s true for all of us… isn’t it?  Don’t slack off on Preventative Maintenance in all aspects of your life.

Pain in the Neck Muskogee Power Plant Relay Testing

RapDon’t let the title fool you.  I love testing Power Plant Protective Relays.  There is a sense of satisfaction when you have successfully cleaned, calibrated and tested a relay that is going to protect the equipment you have to work on every day.  With that said, I was hit with such an unbelievable situation when testing Muskogee Relays in 1995 that I was left with a serious pain in the neck.

On August 14, 2003 the electric power in the Northeast United States and Canada went out.  The Blackout lasted long enough to be a major annoyance for those in the that region of the United States.

 

Map of the power blackout in 2003

Map of the power blackout in 2003

When I heard about how the blackout had moved across the region, I immediately knew what had happened.  I was quickly reminded of the following story.  I told my wife Kelly, “I know exactly why such a large area lost power!  They hadn’t done proper preventative maintenance on the Protective Relays in the substations!  Just like….”  Well…. I’ll tell you that part now:

I have mentioned in a couple of earlier posts that something always seemed a little “off” at the Muskogee Power Plant.  I had decided early on that while working there I would stick to drinking sodas instead of water.  See the post:  “Something’s In the Water at the Muskogee Power Plant“.  Even with that knowledge, I was still shocked at what I found while testing relays at the plant.

This story really begins when one Sunday at Muskogee when one of the Auxiliary Operators was making his rounds inspecting equipment.  He was driving his truck around the south edge of the Unit 6 parking lot on the service road.  He glanced over at a pump next to the road, and at first, he thought he was just seeing things.  After stopping the truck and backing up for a second glance, he was sure he wasn’t dreaming.  It’s just that what he was seeing seemed so strange, he wasn’t sure what was happening.

The operator could see what appeared to be silver paint chips popping off of the large pump motor in all directions.  After closer examination, he figured out that the motor was burning up.  It was still running, but it had become so hot that the paint was literally burning off of the motor.

Horizontal pump

Like this Horizontal pump only much bigger and painted silver

A motor like this would get hot if the bearings shell out.  Before the motor is destroyed, the protective relays on the breaker in the 4,000 Volt switchgear shuts the motor off.  In this case, the relay hadn’t tripped the motor, so, it had become extremely hot and could have eventually exploded if left running.  The operator shut the motor down and wrote a work order for the electricians.

Doyle Fullen was the foreman in the electric shop that received the work order.  When he looked into what had happened, he realized that the protective relay had no been inspected for a couple of years for this motor.

I couldn’t find a picture of Doyle.  In his youth he reminds me of a very smart Daryl in Walking Dead:

Norman Reedus from Walking Dead

Norman Reedus from Walking Dead

In fact, since before the downsizing in 1994, none of the Protective Relays at the plant had been inspected.  The person that had been inspecting the relays for many years had moved to another job or retired in 1994.  This was just a warning shot across the bow that could have had major consequences.

No one at Muskogee had been trained to test Protective Relays since the downsizing, so they reached out to our plant in North Central Oklahoma for help.  That was when I was told that I was going to be going to Muskogee during the next overhaul (outage).  I had been formally trained to inspect, clean, calibrate and test Protective Relays with two of my Power Plant Heroes, Ben Davis and Sonny Kendrick years earlier.  See the post:  “Relay Tests and Radio Quizzes with Ben Davis“.

My Protective Relay Maintenance course book

My Protective Relay Maintenance course book

Without going into too much detail about the actual tests we performed as I don’t want to make this a long rambling post (like… well…. like most of my posts…..I can already tell this is going to be a long one), I will just say that I took our antiquated relay tester down to Muskogee to inspect their relays and teach another electrician Charles Lay, how to perform those tests in the future.  Muskogee had a similar Relay Test Set.  These were really outdated, but they did everything we needed, and it helped you understand exactly what was going on when you don’t have a newfangled Relay Test Set.

AVO Multi-Amp SR-76 Relay Test Set

AVO Multi-Amp SR-76 Relay Test Set

You need to periodically test both mechanical and electronic protective relays.  In the electronic relays the components change their properties slightly over time, changing the time it takes to trip a breaker under a given circumstance (we’re talking about milliseconds).  In the mechanical relays (which I have always found to be more reliable), they sit inside a black box all the time, heating up and cooling as the equipment is used.  Over time, the varnish on the copper coils evaporates and settles on all the components.  This becomes sticky so that the relay won’t operate at the point where it should.

A panel of Protective Relays

A panel of Protective Relays

In the picture above, the black boxes on the top, middle and right are mechanical relays.  This means that something actually has to turn or pick up in order to trip the equipment.  The electronic relays may have a couple of small relays, but for the most part, they are made up of transistors, resistors, capacitors and diodes.

So, with all that said, let me start the real story…. gee…. It’s about time…

So, here I am sitting in the electric shop lab just off of the Unit 6 T-G floor.  We set up all the equipment and had taken a couple of OverCurrent relays out of some high voltage breakers in the switchgear.  I told Charles that before you actually start testing the relays, you need to have the test documents from the previous test and we also needed the instruction manuals for each of the relays because the manuals will have the diagrams that you use to determine the exact time that the relays should trip for each of the tests.  So, we went up to the print room to find the old tests and manuals.  Since they weren’t well organized, we just grabbed the entire folder where all the relays tests were kept since Unit 6 had been in operation.

When we began testing the relays at first I thought that the relay test set wasn’t working correctly.  Here I was trying to impress my new friend, Charles Lay, a 63 year old highly religious fundamental Christian that I knew what I was doing, and I couldn’t even make a relay trip.  I was trying to find the “As Found” tripping level.  That is, before you clean up the relay.  Just like you found it.  Only, it wouldn’t trip.

It turned out that the relay was stuck from the varnish as I explained above.  It appeared as if the relay hadn’t been tested or even operated for years.  The paperwork showed that it had been tested three years earlier.  Protective Relays should be tested at least every two years, but I wouldn’t have thought that the relay would be in such a bad condition in just three years.  It had been sitting in a sealed container to keep out dust.  But it was what it was.

I told Charles that in order to find the “As Found” point where the relay would trip, we would need to crank up the test set as high as needed to find when it actually did trip.  It turned out that the relay which should have instantaneously tripped somewhere around 150 amps wouldn’t have tripped until the motor was pulling over 4,000 amps.  I could tell right away why the Auxiliary Operator found that motor burning up without tripping.  The protective relays were stuck.

As it turned out… almost all of the 125 or so relays were in the same condition.  We cleaned them all up and made them operational.

There is an overcurrent relay for the main bus on each section of a main switchgear.

A picture of a clean switchgear. Picture 6 rows of switchgear like this

A picture of a clean switchgear. Picture 6 rows of switchgear like this

When I tested the “As Found” instantaneous trip for the main bus relay, I found that it was so high that the Unit 6 Main Turbine Generator would have melted down before the protective relay would have tripped the power to that one section of switchgear.  The entire electric bus would have been nothing but molten metal by that time.

As I tested each of these relays, I kept shaking my head in disbelief.  But that wasn’t the worst of it.  The mystery as to why these relays were all glued shut by varnish was finally solved, and that reason was even more unbelievable.

Here is what I found…..  The first thing you do when you are going to test a relay is that you fill out a form that includes all the relay information, such as, what it is for, what are the settings on the relay, and what are the levels of tests that you are going to perform on it.  You also include a range of milliseconds that are acceptable for the relay for each of the tests.  Normally, you just copy what was used in the previous test, because you need to include the time it took for the Previous “As Left” test on your form.  That is why we needed the forms from the previous test.

So, I had copied the information from the previous test form and began testing the relay (one of the first overcurrent relays we tested)…  Again… I was a 34 year old teacher trying to impress my 63 year old student.  So, I was showing him how you mechanically adjust the relay in order for it to trip within the acceptable range.  No matter how hard I tried, I couldn’t adjust the relay so that it would even be close to the desired range for the longer time trip times…. like the 2 second to 25 second range.  It wasn’t even close to the range that was on the form from the last test.

The form from the last test showed that the relay was in the right range for all the levels of test.  When I tested it, like I said, it wasn’t even close.  So, I went to the diagram in the instruction manual for this type of relay.  The diagram looks similar to this one used for thermal overloads:

Thermal Overload Tripping curves

Thermal Overload Tripping curves

See all those red lines?  Well, when you setup a relay, you have a dial where you set the range depending on the needs for the type of motor you are trying to trip. Each red line represents each setting on the dial.  Most of the relays were set on the same number, so we would be using the same red line on the diagram to figure out at different currents how long it should take for a relay to trip….

Here is the clincher….The time range that was written on the previous form wasn’t for the correct relay setting.  The person that tested the relay had accidentally looked at the wrong red line.  — That in itself is understandable, since it could be easy to get on the wrong line… The only thing is that as soon as you test the relay, you would know that something is wrong, because the relay wouldn’t trip in that range, just like I had found.

I double and triple checked everything to make sure we were looking at the same thing.  The previous form indicated the same settings on the relay as now, yet, the time ranges were for a different line! — Ok.  I know.  I have bored you to tears with all this stuff about time curves and overcurrent trips… so I will just tell you what this means…

This meant that when the person completed the forms the last time, they didn’t test the relays at all.  They just filled out the paperwork.  They put in random values that were in the acceptable range and sat around in the air conditioned lab during the entire overhaul smoking his pipe. — Actually, I don’t remember if he smoked a pipe or not.  He was the Electrical Specialist for the plant.  I remembered seeing him sitting in the lab with a relay hooked up to the test set throughout the entire overhaul, but I realized finally that he never tested the relays.  He didn’t even go so far as try to operate them.

I went back through the records to when the plant was first “checked out”.  Doyle Fullen had done the check out on the relays and the test after that.  Doyle had written the correct values from the manual on his forms.  I could see where he had actually performed the tests on the relays and was getting the same values I was finding when I tested the relays, so I was certain that I wasn’t overlooking anything.

As I tested each of the relays, I kept shaking my head in disbelief.  It was so unbelievable.  How could someone do such a thing?  Someone could have been killed because a protective relay wasn’t working correctly.  This was serious stuff.

One day while Charles and I were working away on the relays, Jack Coffman, the Superintendent of all the Power Plants came walking through the lab.  He asked us how we were doing.  I swiveled around in my chair to face him and I said, “Pretty good, except for this pain in my neck” as I rubbed the back of my neck.

Jack stopped and asked me what happened.  I told him that I had been shaking my head in disbelief for the last two weeks, and it gave me a pain in the neck.  Of course, I knew this would get his attention, so he asked, “Why?”  I went through all the details of what I had found.

I showed him how since the time that Doyle Fullen last tested the relays more than 10 years earlier, these relays hadn’t been tested at all.  I showed him how the main bus relays were so bad that it would take over 100,000 amps to have tripped the 7100 KV switchgear bus.

Jack stood there looking off into space for a few seconds, and then walked out the door…. I thought I saw him shaking his head as he left.  Maybe he was just looking both ways for safety reasons, but to me, it looked like a shake of disbelief.  I wonder if I had given him the same pain in the neck.

That is really the end of the relay story, but I do want to say a few words about Charles Lay.  He was a hard working electrician that was nearing retirement.  People would come around to hear us discussing religion.  I am Catholic, and he went to a Fundamental Christian Church.  We would debate the differences between our beliefs and just Christian beliefs in general.  We respected each other during our time together, even though he was sure I am going to hell when I die.

People would come in just to hear our discussion for a while as we were cleaning and calibrating the relays.  One day Charles asked me if I could help him figure out how much he was going to receive from his retirement from the electric company.  He had only been working there for three years.  Retirement at that time was determined by your years of service.  So, three years didn’t give him too much.

When I calculated his amount, he was upset.  He said, “Am I going to have to work until I die?”  I said, “Well, there’s always your 401k and Social Security.”  He replied that he can’t live on Social Security.  I said, “Well, there’s your 401k.”  He asked, “What’s that?” (oh.  not a good sign).

I explained that it was a retirement plan where you are able to put money in taxed deferred until you take it out when you retire.  He said, “Oh.  I never put anything in something like that.”  My heart just sank as I looked in his eyes.  He had suddenly realized that he wasn’t going to receive a retirement like those around him who had spent 35 years working in the Power Plant.

When I left the plant after teaching Charles Lay how to test the relays, that was the last time I ever saw him.  I don’t know what became of Charles.  I figure he would be 83 years old today.  I wonder if he finally retired when he reached the 80 points for your age and years of service.  He would have never reached enough years of service to receive a decent amount of retirement from the Electric Company since he didn’t start working there until he was 60 years old.  That is, unless he’s still working there now.

As I said earlier in this post, Charles Lay was a very good worker.  He always struck me as the “Hardworking type”.  I often think about the time we spent together, especially when I hear about a power blackout somewhere.  — A word of caution to Power Companies…. keep your protective relays in proper working condition.  Don’t slack off on the Preventative Maintenance.  — I guess that’s true for all of us… isn’t it?  Don’t slack off on Preventative Maintenance in all aspects of your life.

Relay Tests and Radio Quizzes with Ben Davis

Originally posted January 24, 2014:

Reorganizations naturally shuffle things around.  People are generally resistant to change and don’t like to find that their routine has been changed without having their input on how to make things better.  When the coal-fired power plant in North Central Oklahoma went through a downsizing and reorganization in the latter part of 1987, my job changed slightly.  Personally, I was grateful for the changes.

Before the reorganization, I had inherited both the precipitators (the large boxes at a power plant that take the ash out of the exhaust from the boiler).  This meant that every overhaul, I knew what I was doing.  I was working on and in the precipitator.  This was generally a dirty and thankless job.

The plant has a similar electrostatic precipitator, only it is twice as long

The plant has a similar electrostatic precipitator, only it is twice as long

After the reorganization, however, Terry Blevins was assigned to work on the Unit 2 precipitator, while I worked on Unit 1.  I will go into this in more detail later, but for this post, I’ll just point out that this meant that when Unit 2 was on an overhaul (that means the unit is taken offline for one to three months in order to fix and repair things that can only be done while it is offline) I wasn’t automatically assigned to the precipitator.  So, I could work on other things.

Before the reorganization, Sonny Kendrick had the title “Electric Specialist”.  After the reorganization we no longer had a specialist.  I’m not sure exactly why.  I know that at Muskogee, they still had a specialist in the electric shop.  — I will talk about him next year (the specialist at Muskogee).  Anyway, I know that Sonny, at the time, was not too happy about his change in job title.  I don’t blame him.  I would be too.

One of the things that the Electric Specialist did during overhauls was test tripping relays.  Now that we no longer had a specialist, that was left up to whomever…. The first electricians, besides Sonny, that were assigned to relay testing was Ben Davis and myself.  I had started doing it on my own and after about a week, Ben Davis was assigned to help me out.

Ben Davis

Ben Davis

We were on a major overhaul on Unit 2 and it had been decided that we were not only going to test the regular super-high voltage breaker relays, we were also going to test all the 480 volt switchgear relays for Unit 2, as well as the intake and coalyard switchgears.  I don’t remember if we made it to the river pump switchgear, but I wouldn’t be surprised.  Once we started, there was no stopping us.

When I first was told to test the relays, Bill Bennett (our A foreman) told me to have Sonny tell me how to do them.  So, I walked into the lab and told Sonny that Bill had told me to ask him to help me learn how to test the protective relays on the switchgear.  Sonny, not looking too happy, grabbed a small stack of manuals, walked out into the main switchgear with me, and said, “Here is the relay set.  Here are the manuals that tell you how to hook up the test set and test them.”  He turned and walked away…. I was sort of hoping for a more intimate lesson…

I knew the reason Sonny was so upset.  Later I learned why he would be as upset as he was to not be able to test the protective relays.  It was because when you test, clean and adjust protective relays you have an immediate rush of satisfaction that you have just done something very important.  Let me just say quickly (because in another post I will expound upon this), a protective relay is what keeps motors from blowing up.  It is what prevents blackouts from happening across the nation.  Without properly calibrated protective relays, a power company is just asking for a disaster (or… well….. their insurance company is, because they are the ones that usually end up paying for the damage — which I will also talk about in a later post).

I thought the relay test set that Sonny showed me was the neatest thing I had seen so far in the electric shop.  There were two boxes that hooked together with an umbilical cord.  They had dials, switches, connectors, meters and a digital readout down to the millisecond.  That is, you can read the time to trip a relay down to the one thousandth of a second.

AVO Multi-Amp SR-76 Relay Test Set

AVO Multi-Amp SR-76 Relay Test Set

I only wish that I had a bigger picture of this relay test set so that you could admire it as much as I did.  Even today it gives me goosebumps!  Ok.  I can imagine those relay technicians that read this blog are looking at this and thinking…. “What kind of piece of junk is this?”  Hey (as Mark Fielder used to say), this was my “baby” (only he was referring to the precipitator).

So, back to the story at hand…

Even though I was having a heck of a fun time trying to figure out how to perform these relay tests by reading these manuals about the different kinds of relays, I was glad when Ben Davis was assigned to work with me.  I don’t know if he had worked on relays before, but he seemed to know just what to do to hook up the test set and make things easier.

A panel of Protective Relays

A panel of Protective Relays

The best suggestion that Ben had right off the bat was that we should be listening to the radio while we were working.  This might have been a preventative measure after the first couple of days to prevent the same situation from occurring that happened to Ed Shiever when he and I were trapped inside a confined space for a couple of weeks (See the post:  “Ed Shiever Trapped in a Confined Space with a disciple of Ramblin’ Ann“).  Either way, it was a great idea.

You wouldn’t think that inside a switchgear 20 miles from the nearest town with a radio station, that we would have any reception on a little transistor radio, but we were able to manage.  It seemed that we had to be a little creative at times with the antenna in certain locations, but, like I said.  We managed.

My perception of Ben Davis up to this point was that he was a “Good-ol’ boy”.   That is, a country music type Oklahoman that had grown up in Shidler, Oklahoma where the major attraction in the town was the High School.  To my surprise, I quickly found out that he was a connoisseur of Rock and Roll.

It wasn’t until I was in college before I realized that the easy listening station I had been listening to on our family radio at home while I was growing up was playing rock and roll songs using an orchestra with violins and clarinets instead of electric guitars.  I learned from my dorm mates all about groups like Simon and Garfunkel, The Beatles (yeah… can you believe it?  I mean.  I knew “Hey Jude”, “Let it Be” and a few others, but most of the Beatles I thought were instrumentals normally played on violins with a man waving a wand) and many others.  When I found out about “Rock and Roll”, I had to go out and buy dozens of 8-track tapes, as fast as I could find them.

A stack of 8 Track Tapes

A stack of 8 Track Tapes

So, here was Ben Davis.  Even better than the “Good Ol’ Boy” that I already thought he was.  And he loved classical rock and roll.  I can only say that the next month and a half while we tested relays all over the plant, were one of the best times I have ever spent in my life!  He knew all the 60’s and 70’s rock and roll bands.

As each song would come on the radio, we would guess (well, I was guessing most of the time…. most of the time Ben already knew), what the name of the song was and the name of the band.  So, not only were we doing one of the most satisfying jobs at a power plant, but I was also have a lot of fun with Ben listening to the radio!  Who would have thought it?  No wonder Sonny was upset he wasn’t testing relays this overhaul.

I could go on about all the different bands and their backgrounds that I learned from Ben during that overhaul, but (unlike me), you probably already know all that stuff.  It never ceases to amaze me how many holes I have in my education until one is staring at me in the face.

This reminds me of a side story, and I apologize if I have told this before…. I don’t think I have….

After the Reorganization, and after I moved to Stillwater from Ponca City, Scott Hubbard (and Toby O’Brien) and I began carpooling.  One morning as we were listening to NPR, Scott Hubbard mentioned something about a “cur”.  I asked him, “What’s a cur?”  Well, he had the exact same reaction when 11 years earlier I had asked my friends in college at Oklahoma University, Tim Flowers and Kirby Davis, “What’s an orgasm?”  —  See how little holes in your education can make a big impact?

Just so you don’t get caught in the same predicament…  A “Cur” is a mongrel dog.  Scott Hubbard couldn’t believe that someone that read the dictionary for fun wouldn’t know what a “cur” was.  What the heck?  I didn’t grow up in Oklahoma!  — end of side story… which really isn’t a side story, since it was about a Power Plant Man — Scott Hubbard.  He probably knew what a “cur” was before he could walk.  — I know I haven’t told that story before!  I would have remembered that.

I’m not going to go on about all the fun that I had with Ben Davis testing protective relays.  I enjoy my memories, but I’m sure you don’t want to hear all about how much I looked up to this Power Plant Hero.  The only thing I will add is that the time I spent with Ben during that overhaul has been etched into my memory as one of the most enjoyable times of my life.  So, I’ll go onto the next step in our Protective Relay story….

A few years later, in 1993, Sonny Kendrick and Ben Davis and I were sent to “Advanced Protective Relay Maintenance” training in Dallas, Texas.  I remember this time so well, I remember the address where we were went.  It was at 4271 Bronze Way, Dallas, Texas.  It was hosted by the same company that made that wonderful test set I pictured above.  The AVO Multi-Amp Corporation.

I brought my wife Kelly and my three year old daughter Elizabeth with me.  They stayed at the hotel during the day and played in the swimming pool, while I went to class.

The classes lasted four days, Monday through Thursday.  That was where I learned that even though I thought our relay test set was the coolest piece of equipment in the electric shop, it turned out to be archaic by “Protective Relay Maintenance” standards.  Not that it didn’t do the job….   So, in order to train us properly, they let us use our own old test set during the training so that we could see how to properly test really advanced relays such as Distant Relays, Syncro-verifier relays, Negative Sequence Relays,directional distance relays and Pilot Wire relays.  — These are relays that are found in a large substation that trips high voltage lines that run long distances across the country.  — I can tell you’re jealous.  — Well.. I imagine it anyway.  Knowing what I know now.

So, why drag you all the way to Dallas for this story?  There’s a reason.

time for a second side story:

You see. Tim Flowers, whom I mentioned above, knew not too long after he met me that I have the knack of running into people that I know (or should have known in this case), would love this story.  You see, I met Tim and Kirby at Oklahoma University and they drove with me to Columbia Missouri in 1979 (along with my brother Greg) when I went to register for classes at Missouri University when I decided to go back to school in my home town.

When we arrived in the town, we were hungry after driving for 8 hours straight from Stillwater, Oklahoma to Columbia, Missouri… so we stopped at Godfather’s Pizza.  As we walked in, there was a girl and a guy standing at the counter ordering a pizza.  The pretty girl (Pamela Ramsey) with long red hair turned and saw me.  She immediately came toward me saying “Kevin Breazile!!!!  You owe Me!!!  Slightly shocked and pleased, I said, “What for?” She reminded me that I never gave her the pictures that were taken during the Senior Prom.  You see.  I had taken her to the Senior Prom.

Later I explained that this happens to me a lot.  I meet people that I know in the oddest places (even though this wasn’t so odd, since I had grown up in Columbia). It was just that this was the first person we had seen since we entered town.  From that point on, Tim (who later worked as a summer help at the power plant) expected that everywhere we went we would run into someone I knew….

End of the second side story.  I’m sorry that this is making the post a little longer than usual.  I know you have to get back to work….

So, back to the relay training course in 1993 that Ben Davis, Sonny Kendrick and I were taking in Dallas…. On Wednesday night during the training there was a dinner held in a small banquet room in the hotel.  Well… of course I had to take my wife and my daughter.   So here we were sitting around this table at dinner with the rest of the class of about 10 other non-Sooner Plant employees….

I decided to talk to the guy next to me.  He said something back and my wife Kelly asked him, “Where in New Jersey are you from?”  She had picked up on a New Jersey accent.  He said, Well..  I work in the east for a company called Ebasco, but I’m really from the Midwest.  (oh.  That was my territory).  So I asked a follow-up question.  “Where in the Midwest are you from?”  He said, “From Missouri.”  — Oh.  I thought.   This is interesting. So am I.

I asked a follow-up question.  “Where in Missouri are you from?”  He answered…. “Columbia, Missouri.”  (What?   Where I had grown up?)….  So, I asked a second follow-up Question…. “What High School did you go to?”  With a curious look the man answered….. “Rockbridge High School…”   (Man!!!  the same one as me!!!)…. The third follow-up question….. “What year did you graduate?”  Now, looking really suspicious… he said, “1978”.   Trying to maintain my excitement… I replied….. “Oh… so, you graduated from Rockbridge High School the same year I did….”

What are the odds?  There were 254 students in our graduating class.  This guy who currently lived somewhere in the east is sitting next to me at a dinner of about 10 people attending Advanced Protective Relay Training in Dallas, Texas where neither of us are from, and we both graduated from the same school back in Columbia, Missouri 15 years earlier!  His name is Randy Loesing.  He was working for a company called Ebasco at the time.  He said, “I thought I recognized you!  I just wasn’t sure.”  I didn’t recognize him at all.

It turned out that he kept in touch with two of my oldest friends from the second grade, Mark Schlemper and Brent Stewart.   So we talked about them.  What an incredible coincidence.   Like I may have mentioned before.   It happens to me all the time.  It turns out that an old friend of mine from the 3rd grade in Columbia, Missouri that I used to go to his house when we were stamp collectors and had a stamp collecting club, lives 5 miles south of me today in Round Rock Texas (He’s in Pflugerville).

Russell Somers lives in the  same direction and just about the same number of miles as when we were kids.  Not only that, but he worked at Dell while I was working at Dell (though I didn’t know it at the time).  He has an older daughter and a younger son, just like me only younger.  The same is true for another 3rd grade friend that I  graduated from Rockbridge Highschool and the University of Missouri with, Caryn Lile (now Caryn Iber) who lives in Wisconsin.  She has a daughter and a son the same age as my kids.  She was living in Tulsa when I was living in Stillwater, Oklahoma.  — Like I said… happens to me all the time.

Tim Flowers realized this odd phenomenon  in college.  I had told him earlier that my father told me that if I was every stranded somewhere that I could look up the local Veterinarian and tell him that I was the son of Dr. James Edward Breazile, and they would help me.  So, when we were hiking in the mountains in Colorado and we met a man walking along a trail in the middle of nowhere above Estes Park near the Great Divide, when I told him who I was, he gave us a curious look…. then divulged his most intimate secrets of his life and where he had stashed his most values possessions, Tim told me later.  “I really thought he was going to know who you were when he gave us that funny look.”  I replied.  “I think he did..”

I again apologize for the length of this post.  It is rare that I ramble on this long.  I can thank Ramblin’ Ann for the ability to Ramble so well.  I can thank Ben Davis for recognizing a rambling situation and replacing it with a rock and roll learning opportunity.  As I said earlier.   One of the most enjoyable times I have spent in my entire life is the time I spent with Ben Davis testing Protective Relays!  Bless you Ben and I pray for you, your wife, your son and your daughter on the way to work each morning.

Today when I hear any of the hundreds of roll and roll songs come on the radio that we listened to that month and a half, I can see us testing the relays, looking off into space saying, “Rolling Stones?”  “No.   Steve Miller Band?”  Really?  I thought Browneyed Girl was sung by the Rolling Stone!  It turned out that the version that we listened to was from the creator of the song, Van Morrison. It was like a “One Hit Wonder”.   Who would have thought that he would sound so much like Mick Jagger.  I can see Ben saying… I see what you mean…  it kind of sounds like Mick Jagger.

Relay Tests and Radio Quizzes with Ben Davis

Reorganizations naturally shuffle things around.  People are generally resistant to change and don’t like to find that their routine has been changed without having their input on how to make things better.  When the coal-fired power plant in North Central Oklahoma went through a downsizing and reorganization in the latter part of 1987, my job changed slightly.  Personally, I was grateful for the changes.

Before the reorganization, I had inherited both the precipitators (the large boxes at a power plant that take the ash out of the exhaust from the boiler).  This meant that every overhaul, I knew what I was doing.  I was working on and in the precipitator.  This was generally a dirty and thankless job.

The plant has a similar electrostatic precipitator, only it is twice as long

The plant has a similar electrostatic precipitator, only it is twice as long

After the reorganization, however, Terry Blevins was assigned to work on the Unit 2 precipitator, while I worked on Unit 1.  I will go into this in more detail later, but for this post, I’ll just point out that this meant that when Unit 2 was on an overhaul (that means the unit is taken offline for one to three months in order to fix and repair things that can only be done while it is offline) I wasn’t automatically assigned to the precipitator.  So, I could work on other things.

Before the reorganization, Sonny Kendrick had the title “Electric Specialist”.  After the reorganization we no longer had a specialist.  I’m not sure exactly why.  I know that at Muskogee, they still had a specialist in the electric shop.  — I will talk about him next year (the specialist at Muskogee).  Anyway, I know that Sonny, at the time, was not too happy about his change in job title.  I don’t blame him.  I would be too.

One of the things that the Electric Specialist did during overhauls was test tripping relays.  Now that we no longer had a specialist, that was left up to whomever…. The first electricians, besides Sonny, that were assigned to relay testing was Ben Davis and myself.  I had started doing it on my own and after about a week, Ben Davis was assigned to help me out.

We were on a major overhaul on Unit 2 and it had been decided that we were not only going to test the regular super-high voltage breaker relays, we were also going to test all the 480 volt switchgear relays for Unit 2, as well as the intake and coalyard switchgears.  I don’t remember if we made it to the river pump switchgear, but I wouldn’t be surprised.  Once we started, there was no stopping us.

When I first was told to test the relays, Bill Bennett (our A foreman) told me to have Sonny tell me how to do them.  So, I walked into the lab and told Sonny that Bill had told me to ask him to help me learn how to test the protective relays on the switchgear.  Sonny, not looking too happy, grabbed a small stack of manuals, walked out into the main switchgear with me, and said, “Here is the relay set.  Here are the manuals that tell you how to hook up the test set and test them.”  He turned and walked away…. I was sort of hoping for a more intimate lesson…

I knew the reason Sonny was so upset.  Later I learned why he would be as upset as he was to not be able to test the protective relays.  It was because when you test, clean and adjust protective relays you have an immediate rush of satisfaction that you have just done something very important.  Let me just say quickly (because in another post I will expound upon this), a protective relay is what keeps motors from blowing up.  It is what prevents blackouts from happening across the nation.  Without properly calibrated protective relays, a power company is just asking for a disaster (or… well….. their insurance company is, because they are the ones that usually end up paying for the damage — which I will also talk about in a later post).

I thought the relay test set that Sonny showed me was the neatest thing I had seen so far in the electric shop.  There were two boxes that hooked together with an umbilical cord.  They had dials, switches, connectors, meters and a digital readout down to the millisecond.  That is, you can read the time to trip a relay down to the one thousandth of a second.

AVO Multi-Amp SR-76 Relay Test Set

AVO Multi-Amp SR-76 Relay Test Set

I only wish that I had a bigger picture of this relay test set so that you could admire it as much as I did.  Even today it gives me goosebumps!  Ok.  I can imagine those relay technicians that read this blog are looking at this and thinking…. “What kind of piece of junk is this?”  Hey (as Mark Fielder used to say), this was my “baby” (only he was referring to the precipitator).

So, back to the story at hand…

Even though I was having a heck of a fun time trying to figure out how to perform these relay tests by reading these manuals about the different kinds of relays, I was glad when Ben Davis was assigned to work with me.  I don’t know if he had worked on relays before, but he seemed to know just what to do to hook up the test set and make things easier.

A panel of Protective Relays

A panel of Protective Relays

The best suggestion that Ben had right off the bat was that we should be listening to the radio while we were working.  This might have been a preventative measure after the first couple of days to prevent the same situation from occurring that happened to Ed Shiever when he and I were trapped inside a confined space for a couple of weeks (See the post:  “Ed Shiever Trapped in a Confined Space with a disciple of Ramblin’ Ann“).  Either way, it was a great idea.

You wouldn’t think that inside a switchgear 20 miles from the nearest town with a radio station, that we would have any reception on a little transistor radio, but we were able to manage.  It seemed that we had to be a little creative at times with the antenna in certain locations, but, like I said.  We managed.

My perception of Ben Davis up to this point was that he was a “Good-ol’ boy”.   That is, a country music type Oklahoman that had grown up in Shidler, Oklahoma where the major attraction in the town was the High School.  To my surprise, I quickly found out that he was a connoisseur of Rock and Roll.

It wasn’t until I was in college before I realized that the easy listening station I had been listening to on our family radio at home while I was growing up was playing rock and roll songs using an orchestra with violins and clarinets instead of electric guitars.  I learned from my dorm mates all about groups like Simon and Garfunkel, The Beatles (yeah… can you believe it?  I mean.  I knew “Hey Jude”, “Let it Be” and a few others, but most of the Beatles I thought were instrumentals normally played on violins with a man waving a wand) and many others.  When I found out about “Rock and Roll”, I had to go out and buy dozens of 8-track tapes, as fast as I could find them.

A stack of 8 Track Tapes

A stack of 8 Track Tapes

So, here was Ben Davis.  Even better than the “Good Ol’ Boy” that I already thought he was.  And he loved classical rock and roll.  I can only say that the next month and a half while we tested relays all over the plant, were one of the best times I have ever spent in my life!  He knew all the 60’s and 70’s rock and roll bands.

As each song would come on the radio, we would guess (well, I was guessing most of the time…. most of the time Ben already knew), what the name of the song was and the name of the band.  So, not only were we doing one of the most satisfying jobs at a power plant, but I was also have a lot of fun with Ben listening to the radio!  Who would have thought it?  No wonder Sonny was upset he wasn’t testing relays this overhaul.

I could go on about all the different bands and their backgrounds that I learned from Ben during that overhaul, but (unlike me), you probably already know all that stuff.  It never ceases to amaze me how many holes I have in my education until one is staring at me in the face.

This reminds me of a side story, and I apologize if I have told this before…. I don’t think I have….

After the Reorganization, and after I moved to Stillwater from Ponca City, Scott Hubbard (and Toby O’Brien) and I began carpooling.  One morning as we were listening to NPR, Scott Hubbard mentioned something about a “cur”.  I asked him, “What’s a cur?”  Well, he had the exact same reaction when 11 years earlier I had asked my friends in college at Oklahoma University, Tim Flowers and Kirby Davis, “What’s an orgasm?”  —  See how little holes in your education can make a big impact?

Just so you don’t get caught in the same predicament…  A “Cur” is a mongrel dog.  Scott Hubbard couldn’t believe that someone that read the dictionary for fun wouldn’t know what a “cur” was.  What the heck?  I didn’t grow up in Oklahoma!  — end of side story… which really isn’t a side story, since it was about a Power Plant Man — Scott Hubbard.  He probably knew what a “cur” was before he could walk.  — I know I haven’t told that story before!  I would have remembered that.

I’m not going to go on about all the fun that I had with Ben Davis testing protective relays.  I enjoy my memories, but I’m sure you don’t want to hear all about how much I looked up to this Power Plant Hero.  The only thing I will add is that the time I spent with Ben during that overhaul has been etched into my memory as one of the most enjoyable times of my life.  So, I’ll go onto the next step in our Protective Relay story….

A few years later, in 1993, Sonny Kendrick and Ben Davis and I were sent to “Advanced Protective Relay Maintenance” training in Dallas, Texas.  I remember this time so well, I remember the address where we were went.  It was at 4271 Bronze Way, Dallas, Texas.  It was hosted by the same company that made that wonderful test set I pictured above.  The AVO Multi-Amp Corporation.

I brought my wife Kelly and my three year old daughter Elizabeth with me.  They stayed at the hotel during the day and played in the swimming pool, while I went to class.

The classes lasted four days, Monday through Thursday.  That was where I learned that even though I thought our relay test set was the coolest piece of equipment in the electric shop, it turned out to be archaic by “Protective Relay Maintenance” standards.  Not that it didn’t do the job….   So, in order to train us properly, they let us use our own old test set during the training so that we could see how to properly test really advanced relays such as Distant Relays, Syncro-verifier relays, Negative Sequence Relays,directional distance relays and Pilot Wire relays.  — These are relays that are found in a large substation that trips high voltage lines that run long distances across the country.  — I can tell you’re jealous.  — Well.. I imagine it anyway.  Knowing what I know now.

So, why drag you all the way to Dallas for this story?  There’s a reason.

time for a second side story:

You see. Tim Flowers, whom I mentioned above, knew not too long after he met me that I have the knack of running into people that I know (or should have known in this case), would love this story.  You see, I met Tim and Kirby at Oklahoma University and they drove with me to Columbia Missouri in 1979 (along with my brother Greg) when I went to register for classes at Missouri University when I decided to go back to school in my home town.

When we arrived in the town, we were hungry after driving for 8 hours straight from Stillwater, Oklahoma to Columbia, Missouri… so we stopped at Godfather’s Pizza.  As we walked in, there was a girl and a guy standing at the counter ordering a pizza.  The pretty girl (Pamela Ramsey) with long red hair turned and saw me.  She immediately came toward me saying “Kevin Breazile!!!!  You owe Me!!!  Slightly shocked and pleased, I said, “What for?” She reminded me that I never gave her the pictures that were taken during the Senior Prom.  You see.  I had taken her to the Senior Prom.

Later I explained that this happens to me a lot.  I meet people that I know in the oddest places (even though this wasn’t so odd, since I had grown up in Columbia). It was just that this was the first person we had seen since we entered town.  From that point on, Tim (who later worked as a summer help at the power plant) expected that everywhere we went we would run into someone I knew….

End of the second side story.  I’m sorry that this is making the post a little longer than usual.  I know you have to get back to work….

So, back to the relay training course in 1993 that Ben Davis, Sonny Kendrick and I were taking in Dallas…. On Wednesday night during the training there was a dinner held in a small banquet room in the hotel.  Well… of course I had to take my wife and my daughter.   So here we were sitting around this table at dinner with the rest of the class of about 10 other non-Sooner Plant employees….

I decided to talk to the guy next to me.  He said something back and my wife Kelly asked him, “Where in New Jersey are you from?”  She had picked up on a New Jersey accent.  He said, Well..  I work in the east for a company called Ebasco, but I’m really from the Midwest.  (oh.  That was my territory).  So I asked a follow-up question.  “Where in the Midwest are you from?”  He said, “From Missouri.”  — Oh.  I thought.   This is interesting. So am I.

I asked a follow-up question.  “Where in Missouri are you from?”  He answered…. “Columbia, Missouri.”  (What?   Where I had grown up?)….  So, I asked a second follow-up Question…. “What High School did you go to?”  With a curious look the man answered….. “Rockbridge High School…”   (Man!!!  the same one as me!!!)…. The third follow-up question….. “What year did you graduate?”  Now, looking really suspicious… he said, “1978”.   Trying to maintain my excitement… I replied….. “Oh… so, you graduated from Rockbridge High School the same year I did….”

What are the odds?  There were 254 students in our graduating class.  This guy who currently lived somewhere in the east is sitting next to me at a dinner of about 10 people attending Advanced Protective Relay Training in Dallas, Texas where neither of us are from, and we both graduated from the same school back in Columbia, Missouri 15 years earlier!  His name is Randy Loesing.  He was working for a company called Ebasco at the time.  He said, “I thought I recognized you!  I just wasn’t sure.”  I didn’t recognize him at all.

It turned out that he kept in touch with two of my oldest friends from the second grade, Mark Schlemper and Brent Stewart.   So we talked about them.  What an incredible coincidence.   Like I may have mentioned before.   It happens to me all the time.  It turns out that an old friend of mine from the 3rd grade in Columbia, Missouri that I used to go to his house when we were stamp collectors and had a stamp collecting club, lives 5 miles south of me today in Round Rock Texas (He’s in Pflugerville).

Russell Somers lives in the  same direction and just about the same number of miles as when we were kids.  Not only that, but he worked at Dell while I was working at Dell (though I didn’t know it at the time).  He has an older daughter and a younger son, just like me only younger.  The same is true for another 3rd grade friend that I  graduated from Rockbridge Highschool and the University of Missouri with, Caryn Lile (now Caryn Iber) who lives in Wisconsin.  She has a daughter and a son the same age as my kids.  She was living in Tulsa when I was living in Stillwater, Oklahoma.  — Like I said… happens to me all the time.

Tim Flowers realized this odd phenomenon  in college.  I had told him earlier that my father told me that if I was every stranded somewhere that I could look up the local Veterinarian and tell him that I was the son of Dr. James Edward Breazile, and they would help me.  So, when we were hiking in the mountains in Colorado and we met a man walking along a trail in the middle of nowhere above Estes Park near the Great Divide, when I told him who I was, he gave us a curious look…. then divulged his most intimate secrets of his life and where he had stashed his most values possessions, Tim told me later.  “I really thought he was going to know who you were when he gave us that funny look.”  I replied.  “I think he did..”

I again apologize for the length of this post.  It is rare that I ramble on this long.  I can thank Ramblin’ Ann for the ability to Ramble so well.  I can thank Ben Davis for recognizing a rambling situation and replacing it with a rock and roll learning opportunity.  As I said earlier.   One of the most enjoyable times I have spent in my entire life is the time I spent with Ben Davis testing Protective Relays!  Bless you Ben and I pray for you, your wife, your son and your daughter on the way to work each morning.

Today when I hear any of the hundreds of roll and roll songs come on the radio that we listened to that month and a half, I can see us testing the relays, looking off into space saying, “Rolling Stones?”  “No.   Steve Miller Band?”  Really?  I thought Browneyed Girl was sung by the Rolling Stone!  It turned out that the version that we listened to was from the creator of the song, Van Morrison. It was like a “One Hit Wonder”.   Who would have thought that he would sound so much like Mick Jagger.  I can see Ben saying… I see what you mean…  it kind of sounds like Mick Jagger.