Tag Archives: relay

Relay Tests and Radio Quizzes with Ben Davis

Favorites Post #67

Originally posted January 24, 2014:

Reorganizations naturally shuffle things around.  People are generally resistant to change and don’t like to find that their routine has been changed without having their input on how to make things better.  When the coal-fired power plant in North Central Oklahoma went through a downsizing and reorganization in the latter part of 1987, my job changed slightly.  Personally, I was grateful for the changes.

Before the reorganization, I had inherited both the precipitators (the large boxes at a power plant that take the ash out of the exhaust from the boiler).  This meant that every overhaul, I knew what I was doing.  I was working on and in the precipitator.  This was generally a dirty and thankless job.

The plant has a similar electrostatic precipitator, only it is twice as long

The plant has a similar electrostatic precipitator, only it is twice as long

After the reorganization, however, Terry Blevins was assigned to work on the Unit 2 precipitator, while I worked on Unit 1.  I will go into this in more detail later, but for this post, I’ll just point out that this meant that when Unit 2 was on an overhaul (that means the unit is taken offline for one to three months in order to fix and repair things that can only be done while it is offline) I wasn’t automatically assigned to the precipitator.  So, I could work on other things.

Terry Blevins

Before the reorganization, Sonny Kendrick had the title “Electric Specialist”.  After the reorganization we no longer had a specialist.  I’m not sure exactly why.  I know that at Muskogee, they still had a specialist in the electric shop.  — I will talk about him next year (the specialist at Muskogee).  Anyway, I know that Sonny, at the time, was not too happy about his change in job title.  I don’t blame him.  I would be too.  — As you can see by the picture below, he eventually learned to be happy again:

Sonny as he is today

One of the things that the Electric Specialist did during overhauls was test tripping relays.  Now that we no longer had a specialist, that was left up to whomever…. The first electricians, besides Sonny, that were assigned to relay testing was Ben Davis and myself.  I had started doing it on my own and after about a week, Ben Davis was assigned to help me out.

Ben Davis

Ben Davis

We were on a major overhaul on Unit 2 and it had been decided that we were not only going to test the regular super-high voltage breaker relays, we were also going to test all the 480 volt switchgear relays for Unit 2, as well as the intake and coalyard switchgears.  I seem to remember making it to the river pump switchgear.  Once we started, there was no stopping us.

When I first was told to test the relays, Bill Bennett (our A foreman) told me to have Sonny tell me how to do them.  So, I walked into the lab and told Sonny that Bill had told me to ask him to help me learn how to test the protective relays on the switchgear.  Sonny, not looking too happy, grabbed a small stack of manuals, walked out into the main switchgear with me, and said, “Here is the relay test set.  Here are the manuals that tell you how to hook up the test set and test them.”  He turned and walked away…. I was sort of hoping for a more intimate lesson…

I knew the reason Sonny was so upset.  Later I learned why he would be as upset as he was to not be able to test the protective relays.  It was because when you test, clean and adjust protective relays you have an immediate rush of satisfaction that you have just done something very important.  Let me just say quickly (because in another post I will expound upon this), a protective relay is what keeps motors from blowing up.  It is what prevents blackouts from happening across the nation.  Without properly calibrated protective relays, a power company is just asking for a disaster (or… well….. their insurance company is, because they are the ones that usually end up paying for the damage — which I will also talk about in a later post).

I thought the relay test set that Sonny showed me was the neatest thing I had seen so far in the electric shop.  There were two boxes that hooked together with an umbilical cord.  They had dials, switches, connectors, meters and a digital readout down to the millisecond.  That is, you can read the time to trip a relay down to the one thousandth of a second.

AVO Multi-Amp SR-76 Relay Test Set

AVO Multi-Amp SR-76 Relay Test Set

I only wish that I had a bigger picture of this relay test set so that you could admire it as much as I did.  Even today it gives me goosebumps!  Ok.  I can imagine those relay technicians that read this blog are looking at this and thinking…. “What kind of piece of junk is this?”  Hey (as Mark Fielder used to say), this was my “baby” (only he was referring to the precipitator).

So, back to the story at hand…

Even though I was having a heck of a fun time trying to figure out how to perform these relay tests by reading these manuals about the different kinds of relays, I was glad when Ben Davis was assigned to work with me.  I don’t know if he had worked on relays before, but he seemed to know just what to do to hook up the test set and make things easier.

A panel of Protective Relays

A panel of Protective Relays

The best suggestion that Ben had right off the bat was that we should be listening to the radio while we were working.  This might have been a preventative measure after the first couple of days to prevent the same situation from occurring that happened to Ed Shiever when he and I were trapped inside a confined space for a couple of weeks (See the post:  “Ed Shiever Trapped in a Confined Space with a disciple of Ramblin’ Ann“).  Either way, it was a great idea.

You wouldn’t think that inside a switchgear 20 miles from the nearest town with a radio station, that we would have any reception on a little transistor radio, but we were able to manage.  It seemed that we had to be a little creative at times with the antenna in certain locations, but, like I said.  We managed.

My perception of Ben Davis up to this point was that he was a “Good-ol’ boy”.  That is, a country music type Oklahoman that had grown up in Shidler, Oklahoma where the major attraction in the town was the High School.  To my surprise, I quickly found out that he was a connoisseur of Rock and Roll.

It wasn’t until I was in college before I realized that the easy listening station I had been listening to on our family radio at home while I was growing up was playing rock and roll songs using an orchestra with violins and clarinets instead of electric guitars.  I learned from my dorm mates all about groups like Simon and Garfunkel, The Beatles (yeah… can you believe it?  I mean.  I knew “Hey Jude”, “Let it Be” and a few others, but most of the Beatles I thought were instrumentals normally played on violins with a man waving a wand) and many others.  When I found out about “Rock and Roll”, I had to go out and buy dozens of 8-track tapes, as fast as I could find them.

A stack of 8 Track Tapes

A stack of 8 Track Tapes

So, here was Ben Davis.  Even better than the “Good Ol’ Boy” that I already thought he was.  And he loved classical rock and roll.  I can only say that the next month and a half while we tested relays all over the plant, were one of the best times I have ever spent in my life!  He knew all the 60’s and 70’s rock and roll bands.

As each song would come on the radio, we would guess (well, I was guessing most of the time…. most of the time Ben already knew), what the name of the song was and the name of the band.  So, not only were we doing one of the most satisfying jobs at a power plant, but I was also have a lot of fun with Ben listening to the radio!  Who would have thought it?  No wonder Sonny was upset he wasn’t testing relays this overhaul.

I could go on about all the different bands and their backgrounds that I learned from Ben during that overhaul, but (unlike me), you probably already know all that stuff.  It never ceases to amaze me how many holes I have in my education until one is staring at me in the face.

This reminds me of a side story, and I apologize if I have told this before…. I don’t think I have….

After the Reorganization, and after I moved to Stillwater from Ponca City, Scott Hubbard (and Toby O’Brien) and I began carpooling.  One morning as we were listening to NPR, Scott Hubbard mentioned something about a “cur”.  I asked him, “What’s a cur?”  Well, he had the exact same reaction when 11 years earlier I had asked my friends in college at Oklahoma University, Tim Flowers and Kirby Davis, “What’s an orgasm?”  —  See how little holes in your education can make a big impact?

Just so you don’t get caught in the same predicament…  A “Cur” is a mongrel dog.  Scott Hubbard couldn’t believe that someone that read the dictionary for fun wouldn’t know what a “cur” was.  What the heck?  I didn’t grow up in Oklahoma!  I checked and the Dictionary in the electric shop didn’t include the word “Cur” — end of side story… which really isn’t a side story, since it was about a Power Plant Man — Scott Hubbard.  He probably knew what a “cur” was before he could walk.  — I know I haven’t told that story before!  I would have remembered that.

I’m not going to go on about all the fun that I had with Ben Davis testing protective relays.  I enjoy my memories, but I’m sure you don’t want to hear all about how much I looked up to this Power Plant Hero.  The only thing I will add is that the time I spent with Ben during that overhaul has been etched into my memory as one of the most enjoyable times of my life.  So, I’ll go onto the next step in our Protective Relay story….

A few years later, in 1993, Sonny Kendrick and Ben Davis and I were sent to “Advanced Protective Relay Maintenance” training in Dallas, Texas.  I remember this time so well, I remember the address where we were went.  It was at 4271 Bronze Way, Dallas, Texas.  It was hosted by the same company that made that wonderful test set I pictured above.  The AVO Multi-Amp Corporation.

I brought my wife Kelly and my three year old daughter Elizabeth with me.  They stayed at the hotel during the day and played in the swimming pool, while I went to class.

The classes lasted four days, Monday through Thursday.  That was where I learned that even though I thought our relay test set was the coolest piece of equipment in the electric shop, it turned out to be archaic by “Protective Relay Maintenance” standards.  Not that it didn’t do the job….   So, in order to train us properly, they let us use our own old test set during the training so that we could see how to properly test really advanced relays such as Distant Relays, Syncro-verifier relays, Negative Sequence Relays,directional distance relays and Pilot Wire relays.  — These are relays that are found in a large substation that trips high voltage lines that run long distances across the country.  — I can tell you’re jealous.  — Well.. I imagine it anyway.  Knowing what I know now.

This is the book we used in class

So, why drag you all the way to Dallas for this story?  There’s a reason.

time for a second side story:

You see. Tim Flowers, whom I mentioned above, knew not too long after he met me that I have the knack of running into people that I know (or should have known in this case), would love this story.  You see, I met Tim and Kirby at Oklahoma University and they drove with me to Columbia Missouri in 1979 (along with my brother Greg) when I went to register for classes at Missouri University when I decided to go back to school in my home town.

When we arrived in the town, we were hungry after driving for 8 hours straight from Stillwater, Oklahoma to Columbia, Missouri… so we stopped at Godfather’s Pizza.  As we walked in, there was a girl and a guy standing at the counter ordering a pizza.  The pretty girl (Pamela Ramsey) with long red hair turned and saw me.  She immediately came toward me saying “Kevin Breazile!!!!  You owe Me!!!  Slightly shocked and pleased, I said, “What for?” She reminded me that I never gave her the pictures that were taken during the Senior Prom.  You see.  I had taken her to the Senior Prom.

Later I explained that this happens to me a lot.  I meet people that I know in the oddest places (even though this wasn’t so odd, since I had grown up in Columbia). It was just that this was the first person we had seen since we entered town.  From that point on, Tim (who later worked as a summer help at the power plant) expected that everywhere we went we would run into someone I knew….

End of the second side story.  I’m sorry that this is making the post a little longer than usual.  I know you have to get back to work….

So, back to the relay training course in 1993 that Ben Davis, Sonny Kendrick and I were taking in Dallas…. On Wednesday night during the training there was a dinner held in a small banquet room in the hotel.  Well… of course I had to take my wife and my daughter.   So here we were sitting around this table at dinner with the rest of the class of about 10 other non-Sooner Plant employees….

I decided to talk to the guy next to me.  He said something back and my wife Kelly asked him, “Where in New Jersey are you from?”  She had picked up on a New Jersey accent.  He said, Well..  I work in the east for a company called Ebasco, but I’m really from the Midwest.  (oh.  That was my territory).  So I asked a follow-up question.  “Where in the Midwest are you from?”  He said, “From Missouri.”  — Oh.  I thought.   This is interesting. So was I.

I asked a follow-up question.  “Where in Missouri are you from?”  He answered…. “Columbia, Missouri.”  (What?   Where I had grown up?)….  So, I asked a second follow-up Question…. “What High School did you go to?”  With a curious look the man answered….. “Rockbridge High School…”   (Man!!!  the same one as me!!!)…. The third follow-up question….. “What year did you graduate?”  Now, looking really suspicious… he said, “1978”.   Trying to contain my excitement… I replied….. “Oh… so, you graduated from Rockbridge High School the same year I did….”

What are the odds?  There were 254 students in our graduating class.  This guy who currently lived somewhere in the east is sitting next to me at a dinner of about 10 people attending Advanced Protective Relay Training in Dallas, Texas where neither of us are from, and we both graduated from the same school back in Columbia, Missouri 15 years earlier!  His name is Randy Loesing.  He was working for a company called Ebasco at the time.  He said, “I thought I recognized you!  I just wasn’t sure.”  I didn’t recognize him at all until I went back home and looked in my yearbook.

It turned out that he kept in touch with two of my oldest friends from the second grade, Mark Schlemper and Brent Stewart.   So we talked about them.  What an incredible coincidence.   Like I may have mentioned before.   It happens to me all the time.  It turns out that an old friend of mine from the 3rd grade in Columbia, Missouri that I used to go to his house when we were stamp collectors and had a stamp collecting club, lives 5 miles south of me today in Round Rock Texas (He’s in Pflugerville).

Russell Somers lives in the  same direction and just about the same number of miles as when we were kids.  Not only that, but he worked at Dell while I was working at Dell (though I didn’t know it at the time).  He has an older daughter and a younger son, just like me only younger.  The same is true for another 3rd grade friend that I  graduated from Rockbridge Highschool and the University of Missouri with, Caryn Lile (now Caryn Iber) who lives in Wisconsin.  She has a daughter and a son the same age as my kids.  She was living in Tulsa when I was living in Stillwater, Oklahoma.  — Like I said… happens to me all the time.

Tim Flowers realized this odd phenomenon  in college.  I had told him earlier that my father told me that if I was ever stranded somewhere that I could look up the local Veterinarian and tell him that I was the son of Dr. James Edward Breazile, and they would help me.  So, when we were hiking in the mountains in Colorado and we met a man walking along a trail in the middle of nowhere above Estes Park near the Great Divide, when I told him who I was, he gave us a curious look…. then divulged his most intimate secrets of his life and where he had stashed his most values possessions, Tim told me later.  “I really thought he was going to know who you were when he gave us that funny look.”  I replied.  “I think he did..”

I again apologize for the length of this post.  It is rare that I ramble on this long.  I can thank Ramblin’ Ann for the ability to Ramble so well.  I can thank Ben Davis for recognizing a rambling situation and replacing it with a rock and roll learning opportunity.  As I said earlier.   One of the most enjoyable times I have spent in my entire life is the time I spent with Ben Davis testing Protective Relays!  Bless you Ben and I pray for you, your wife, your son and your daughter on the way to work each morning.

Today when I hear any of the hundreds of rock and roll songs come on the radio that we listened to that month and a half, I can see us testing the relays, looking off into space saying, “Rolling Stones?”  “No.   Steve Miller Band?”  Really?  I thought Browneyed Girl was sung by the Rolling Stone!  It turned out that the version that we listened to was from the creator of the song, Van Morrison. Who would have thought that he would sound so much like Mick Jagger.  I can see Ben saying… I see what you mean…  it kind of sounds like Mick Jagger.

As an add on to this story…

I now work at General Motors in Austin Texas.  My best friend in High School was a guy named Jesse Cheng (I have mentioned him in other posts, especially in reference to the phrase “Jesse!  Come get your Chili!).  He was two years older than me, and throughout the years we would lose track of each other and then reconnect.  He went to Yale to become an Engineer, then to the University of Missouri to become a Medical Doctor, then to Harvard to earn a Masters in Public Health and Epidemiology.

It turns out that we both now work at General Motors where he works in Arlington Texas as a Medical Director and I work in IT in Austin.  We can IM (Instant Message) each other whenever we want, and we talk now at least once every week.

Pain in the Neck Muskogee Power Plant Relay Testing

Favorites Post #23

Originally post 2/21/2015

Don’t let the title fool you.  I love testing Power Plant Protective Relays.  There is a sense of satisfaction when you have successfully cleaned, calibrated and tested a relay that is going to protect the equipment you have to work on every day.  With that said, I was hit with such an unbelievable situation when testing Muskogee Relays in 1995 that I was left with a serious pain in the neck.

On August 14, 2003 the electric power in the Northeast United States and Canada went out.  The Blackout lasted long enough to be a major annoyance for those in the that region of the United States.

Map of the power blackout in 2003

Map of the power blackout in 2003

When I heard about how the blackout had moved across the region, I immediately knew what had happened.  I was quickly reminded of the following story.  I told my wife Kelly, “I know exactly why such a large area lost power!  They hadn’t done proper preventative maintenance on the Protective Relays in the substations!  Just like….”  Well…. I’ll tell you that part now:

I have mentioned in a couple of earlier posts that something always seemed a little “off” at the Muskogee Power Plant.  I had decided early on that while working there I would stick to drinking sodas instead of water.  See the post:  “Something is in the Water at the Muskogee Power Plant“.  Even with that knowledge, I was still shocked at what I found while testing relays at the plant.

This story really begins one Sunday at Muskogee when one of the Auxiliary Operators was making his rounds inspecting equipment.  He was driving his truck around the south edge of the Unit 6 parking lot on the service road.  He glanced over at a pump next to the road, and at first, he thought he was just seeing things.  After stopping the truck and backing up for a second glance, he was sure he wasn’t dreaming.  It’s just that what he was seeing seemed so strange, he wasn’t sure what was happening.

The operator could see what appeared to be silver paint chips popping off of the large pump motor in all directions.  After closer examination, he figured out that the motor was burning up.  It was still running, but it had become so hot that the paint was literally burning off of the motor.

Horizontal pump

Like this Horizontal pump only much bigger and painted silver

A motor like this would get hot if the bearings shell out.  Before the motor is destroyed, the protective relays on the breaker in the 4,000 Volt switchgear shuts the motor off.  In this case, the relay hadn’t tripped the motor, so, it had become extremely hot and could have eventually exploded if left running.  The operator shut the motor down and wrote a work order for the electricians.

Doyle Fullen was the foreman in the electric shop that received the work order.  When he looked into what had happened, he realized that the protective relay had not been inspected for a couple of years for this motor.

I couldn’t find a picture of Doyle.  In his youth he reminds me of a very smart Daryl in Walking Dead:

Norman Reedus from Walking Dead

Norman Reedus from Walking Dead

In fact, since before the downsizing in 1994, none of the Protective Relays at the plant had been inspected.  The person that had been inspecting the relays for many years had moved to another job or retired in 1994.  This was just a warning shot across the bow that could have had major consequences.

No one at Muskogee had been trained to test Protective Relays since the downsizing, so they reached out to our plant in North Central Oklahoma for help.  That was when I was told that I was going to be going to Muskogee during the next overhaul (outage).  I had been formally trained to inspect, clean, calibrate and test Protective Relays with two of my Power Plant Heroes, Ben Davis and Sonny Kendrick years earlier.  See the post:  “Relay Tests and Radio Quizzes with Ben Davis“.

My Protective Relay Maintenance course book

My Protective Relay Maintenance course book

Without going into too much detail about the actual tests we performed as I don’t want to make this a long rambling post (like… well…. like most of my posts…..I can already tell this is going to be a long one), I will just say that I took our antiquated relay tester down to Muskogee to inspect their relays and teach another electrician Charles Lay, how to perform those tests in the future.  Muskogee had a similar Relay Test Set.  These were very outdated, but they did everything we needed, and it helped you understand exactly what was going on when you don’t have a newfangled Relay Test Set.

AVO Multi-Amp SR-76 Relay Test Set

AVO Multi-Amp SR-76 Relay Test Set

You need to periodically test both mechanical and electronic protective relays.  In the electronic relays the components change their properties slightly over time, changing the time it takes to trip a breaker under a given circumstance (we’re talking about milliseconds).  In the mechanical relays (which I have always found to be more reliable), they sit inside a black box all the time, heating up and cooling as the equipment is used.  Over time, the varnish on the copper coils evaporates and settles on all the components.  This becomes sticky so that the relay won’t operate at the point where it should.

A panel of Protective Relays

A panel of Protective Relays

In the picture above, the black boxes on the top, middle and right are mechanical relays.  This means that something actually has to turn or pick up in order to trip the equipment.  The electronic relays may have a couple of small relays, but for the most part, they are made up of transistors, resistors, capacitors and diodes.

So, with all that said, let me start the real story…. gee…. It’s about time…

So, here I am sitting in the electric shop lab just off of the Unit 6 T-G floor.  We set up all the equipment and had taken a couple of OverCurrent relays out of some high voltage breakers in the switchgear.  I told Charles that before you actually start testing the relays, you need to have the test documents from the previous test and we also needed the instruction manuals for each of the relays because the manuals will have the diagrams that you use to determine the exact time that the relays should trip for each of the tests.  So, we went up to the print room to find the old tests and manuals.  Since they weren’t well organized, we just grabbed the entire folder where all the relays tests were kept since Unit 6 had been in operation.

When we began testing the relays at first I thought that the relay test set wasn’t working correctly.  Here I was trying to impress my new friend, Charles Lay, a 63 year old highly religious fundamental Christian that I knew what I was doing, and I couldn’t even make a relay trip.  I was trying to find the “As Found” tripping level.  That is, before you clean up the relay.  Just like you found it.  Only, it wouldn’t trip.

It turned out that the relay was stuck from the varnish as I explained above.  It appeared as if the relay hadn’t been tested or even operated for years.  The paperwork showed that it had been tested three years earlier.  Protective Relays should be tested at least every two years, but I wouldn’t have thought that the relay would be in such a bad condition in just three years.  It had been sitting in a sealed container to keep out dust.  But it was what it was.

I told Charles that in order to find the “As Found” point where the relay would trip, we would need to crank up the test set as high as needed to find when it actually did trip.  It turned out that the relay which should have instantaneously tripped somewhere around 150 amps wouldn’t have tripped until the motor was pulling over 4,000 amps.  I could tell right away why the Auxiliary Operator found that motor burning up without tripping.  The protective relays were stuck.

As it turned out… almost all of the 125 or so relays were in the same condition.  We cleaned them all up and made them operational.

There is an overcurrent relay for the main bus on each section of a main switchgear.

A picture of a clean switchgear. Picture 6 rows of switchgear like this

A picture of a clean switchgear. Picture 6 rows of switchgear like this

When I tested the “As Found” instantaneous trip for the main bus relay, I found that it was so high that the Unit 6 Main Turbine Generator would have melted down before the protective relay would have tripped the power to that one section of switchgear.  The entire electric bus would have been nothing but molten metal by that time.

As I tested each of these relays, I kept shaking my head in disbelief.  But that wasn’t the worst of it.  The mystery as to why these relays were all glued shut by varnish was finally solved, and that reason was even more unbelievable.

Here is what I found…..  The first thing you do when you are going to test a relay is that you fill out a form that includes all the relay information, such as, what it is for, what are the settings on the relay, and what are the levels of tests that you are going to perform on it.  You also include a range of milliseconds that are acceptable for the relay for each of the tests.  Normally, you just copy what was used in the previous test, because you need to include the time it took for the Previous “As Left” test on your form.  That is why we needed the forms from the previous test.

So, I had copied the information from the previous test form and began testing the relay (one of the first overcurrent relays we tested)…  Again… I was a 34 year old teacher trying to impress my 63 year old student.  So, I was showing him how you mechanically adjust the relay in order for it to trip within the acceptable range.  No matter how hard I tried, I couldn’t adjust the relay so that it would even be close to the desired range for the longer time trip times…. like the 2 second to 25 second range.  It wasn’t even close to the range that was on the form from the last test.

The form from the last test showed that the relay was in the right range for all the levels of test.  When I tested it, like I said, it wasn’t even close.  So, I went to the diagram in the instruction manual for this type of relay.  The diagram looks similar to this one used for thermal overloads:

Thermal Overload Tripping curves

Thermal Overload Tripping curves

See all those red lines?  Well, when you setup a relay, you have a dial where you set the range depending on the needs for the type of motor you are trying to trip. Each red line represents each setting on the dial.  Most of the relays were set on the same number, so we would be using the same red line on the diagram to figure out at different currents how long it should take for a relay to trip….

Here is the clincher….The time range that was written on the previous form wasn’t for the correct relay setting.  The person that tested the relay had accidentally looked at the wrong red line.  — That in itself is understandable, since it could be easy to get on the wrong line… The only thing is that as soon as you test the relay, you would know that something is wrong, because the relay wouldn’t trip in that range, just like I had found.

I double and triple checked everything to make sure we were looking at the same thing.  The previous form indicated the same settings on the relay as now, yet, the time ranges were for a different line! — Ok.  I know.  I have bored you to tears with all this stuff about time curves and overcurrent trips… so I will just tell you what this means…

This meant that when the person completed the forms the last time, they didn’t test the relays at all.  They just filled out the paperwork.  They put in random values that were in the acceptable range and sat around in the air conditioned lab during the entire overhaul smoking his pipe. — Actually, I don’t remember if he smoked a pipe or not.  He was the Electrical Specialist for the plant.  I remembered seeing him sitting in the lab with a relay hooked up to the test set throughout the entire overhaul when I had been there during previous overhauls, but I realized finally that he never tested the relays.  He didn’t even go so far as try to operate them.

I went back through the records to when the plant was first “checked out”.  Doyle Fullen had done the check out on the relays and the test after that.  Doyle had written the correct values from the manual on his forms.  I could see where he had actually performed the tests on the relays and was getting the same values I was finding when I tested the relays, so I was certain that I wasn’t overlooking anything.

As I tested each of the relays, I kept shaking my head in disbelief.  It was so unbelievable.  How could someone do such a thing?  Someone could have been killed because a protective relay wasn’t working correctly.  This was serious stuff.

One day while Charles and I were working away on the relays, Jack Coffman, the Superintendent of all the Power Plants came walking through the lab.  He asked us how we were doing.  I swiveled around in my chair to face him and I said, “Pretty good, except for this pain in my neck” as I rubbed the back of my neck.

Jack stopped and asked me what happened.  I told him that I had been shaking my head in disbelief for the last two weeks, and it gave me a pain in the neck.  Of course, I knew this would get his attention, so he asked, “Why?”  I went through all the details of what I had found.

I showed him how since the time that Doyle Fullen last tested the relays more than 10 years earlier, these relays hadn’t been tested at all.  I showed him how the main bus relays were so bad that it would take over 100,000 amps to have tripped the 7100 KV switchgear bus or 710 Megawatts!  More power than the entire generator could generate.  The main generator was only rated at about 550 Megawatts at the most.

Jack stood there looking off into space for a few seconds, and then walked out the door…. I thought I saw him shaking his head as he left.  Maybe he was just looking both ways for safety reasons, but to me, it looked like a shake of disbelief.  I wonder if I had given him the same pain in the neck.

That is really the end of the relay story, but I do want to say a few words about Charles Lay.  He was a hard working electrician that was nearing retirement.  People would come around to hear us discussing religion.  I am Catholic, and he went to a Fundamental Christian Church.  We would debate the differences between our beliefs and just Christian beliefs in general.  We respected each other during our time together, even though he was sure I am going to hell when I die.

People would come in just to hear our discussion for a while as we were cleaning and calibrating the relays.  One day Charles asked me if I could help him figure out how much he was going to receive from his retirement from the electric company.  He had only been working there for three years.  Retirement at that time was determined by your years of service.  So, three years didn’t give him too much.

When I calculated his amount, he was upset.  He said, “Am I going to have to work until I die?”  I said, “Well, there’s always your 401k and Social Security.”  He replied that he can’t live on Social Security.  I said, “Well, there’s your 401k.”  He asked, “What’s that?” (oh.  not a good sign).

I explained that it was a retirement plan where you are able to put money in taxed deferred until you take it out when you retire.  He said, “Oh.  I never put anything in something like that.”  My heart just sank as I looked in his eyes.  He had suddenly realized that he wasn’t going to receive a retirement like those around him who had spent 35 years working in the Power Plant.

When I left the plant after teaching Charles Lay how to test the relays, that was the last time I ever saw him.  I don’t know what became of Charles.  I figure he would be 83 years old today.  I wonder if he finally retired when he reached the 80 points for your age and years of service.  He would have never reached enough years of service to receive a decent amount of retirement from the Electric Company since he didn’t start working there until he was 60 years old.  That is, unless he’s still working there now.

As I said earlier in this post, Charles Lay was a very good worker.  He always struck me as the “Hardworking type”.  I often think about the time we spent together, especially when I hear about a power blackout somewhere.  — A word of caution to Power Companies…. keep your protective relays in proper working condition.  Don’t slack off on the Preventative Maintenance.  — I guess that’s true for all of us… isn’t it?  Don’t slack off on Preventative Maintenance in all aspects of your life.

Added note:  On 7/6/2019, 3 weeks after re-posting this story, look what happened:  Con Edison says cause of NYC blackout was substation’s faulty relay protection system

 

Comments from previous posts:

Ron Kilman 

Great story! And I don’t mind the lengthy technical stuff. I like it.

We bought our current house in 2004. It was built in 1983. About 4 years ago we lost power to the GFI outlet in our “guest” bathroom and to an outlet in the “Florida room”. I replaced the GFI, but that didn’t fix anything. The power to the GFI was off. The affected bathroom was used very little, so it wasn’t a big deal. In the “Florida room” we just ran an extension cord from a different outlet and went on. I just assumed that a mouse or squirrel in the attack had messed with the wiring. Maybe I had pulled some wiring lose when I was crawling in the attic. Last December, I finally “bit the bullet” and called an electrician. What he found was a GFI in another bathroom had been wired backwards. It was wired in series with the GFI in the other bathroom. The mis-wired GFI had tripped but it didn’t cut the power to it’s own outlet. It only cut power to the other bathroom GFI and the “Florida room” outlet. It cost me $80 to re-learn what I had learned long ago – “Never assume previous work was does correctly – even if done by a “professional””.

Dave Tarver 

KB,
I suppose the old timers knew more about this! not sure as in they had a saying at relief time between shifts No sparks out of any motors, no paint burning off any bearings, the truck is in the parking spot and filled with gas, all in all had a smooth shift! LOL! What would fascinate me was the mill fires in the early days – you go down there and here is this whole thing cherry red glowing! unbelievable! I have been over to said plant a few times and their switch gears were pretty dirty typically ours were aways very clean and the breakers always worked very well, very reliable and dependable.

  1. Plant Electrician 

    Yeah. That was the first thing I noticed about Muskogee. Our plant took more pride in keeping things clean. To be fair, the Muskogee plant was close to the coal pile and the plant was down wind. So coal dust would blow right into the plant covering everything.

    Andy Tubbs 

    Charles came to our Plant and Calibrated Relays on Unit #1 in Sept. of 2011. Brought his wife with him and called it a working vacation. Still a very nice and intelligent man.

    Mark Lanesbury 

    This is bringing back the memories. Worked in a coal fired plant for many years as a T.A. (Technicians Assistant) doing the protection maintenance and even the commissioning of a new power station for a couple of years in Australia.
    And I agree, it is a place that gets in your blood. And as you are attesting here, some of the stories are unbelievable.
    And I must put in my bit about those AVO multi Amp test sets. The day that we received one they were affectionately known as the ‘T.A.’s’. Apparently they could be relied on to not make a mistake (unlike the occasional human 🙂 ), and give correct settings. Us human T.A.’s were told, ‘if they could have made coffee they would have sacked the lot of us!’ 😀
    Great story Plant Electrician, a good throwback to yesteryear. I wonder with all the computer gear they now have what has become of the technicians. It’s all probably digitised and in constant readouts to a computer test panel somewhere. One technician probably does all testing from a clean, air-conditioned room somewhere 🙂

Learnin’ ’bout ‘lectricity with Andy Tubbs

Originally posted January 18, 2013:

The day I became an electrician at the coal-fired power plant, I suddenly became an expert in electricity. I think it was on Tuesday, just one day after joining the electric shop that I was walking through the welding shop when someone stopped me and asked me how they would wire their living room with different light switches at different corners and make it work correctly. As if I had been an electrician for years. Luckily I was just finishing a house wiring course at the Indian Meridian Vo-Tech in Stillwater, Oklahoma and they had us figure out problems just like those.

Within the first week, George Alley brought a ceiling fan to the shop that he had picked up somewhere and was wondering if we could get it to work. My foreman Charles Foster thought it would be a good small project for me to work on to help me learn about electrical circuits.

After all, this ceiling fan could go slow, medium and fast, and it could go forward or reverse. Only at the moment, all it would do was sit there and hum when you hooked up the power. — So that was my first “unofficial” project, since the main goal was to make George happy so that he would help us out when we needed something special from the mechanics.

When I was a janitor, I had observed the electricians preparing to go to work in the morning, and often, one of them would go to the print cabinets at one end of the shop and pull out a blueprint and lay it across the work table and study it for a while. Then they would either put it back or fold it and put it in their tool bucket and head out the door to go do a job. Now, it was my turn.

Andy Tubbs was one of the two people that played the best jokes on me when I was a janitor. Larry Burns was the other person, and he was the person I was replacing as he had moved to another plant. Andy was the one that had taken the handle off of my push broom the moment I had my back turned so that when I turned around to grab my broom, only the broom head was on the floor, while the broom handle was across the counter by the lab, and Andy was across the other side of the room trying to act like he wasn’t paying attention, but with an expression like he had just played a darn good joke. — I actually had to go back into the bathroom I was cleaning so that I could laugh out loud. I was really impressed by Andy’s ability to play a good joke.

While I’m on the subject, shortly after I became an electrician, I was sitting in the electric shop office talking to Charles when he stopped and said, “Wait…. Listen….” We paused, waiting for something…. A few seconds later, the sound of a hoot owl came over the PA system (what we called the “Gray Phone”). Charles said, It’s an interesting coincidence that the only time the perfect sound of a hoot owl comes over the Gray Phone is when Andy Tubbs is riding in an elevator by himself or with a close friend.

Gaitronics Gray Phone

Gaitronics Gray Phone

I had been sent with Andy Tubbs and Diana Lucas (later Diana Brien), to go to the coal yard and figure out why some circuit for the train gate was not working. Andy had pulled out the blueprints and was studying them. I came up alongside him and looked at all the blue lines running here and there with circles with letters and numbers, and what I recognized as open and closed switches….

Here is a simple electric circuit drawn in the style of many of the electric blueprint drawings

Here is a simple electric circuit drawn in the style of many of the electric blueprint drawings

Andy stopped and gave me a momentary lecture on the nature of electricity. It was so perfectly summed up, that for years whenever I thought about the nature of electricity, I always began with remembering what Andy told me. He said this:

“Think of electricity like water in a hose. Voltage is the water pressure. Amperage is the amount of water going through the hose. You can have the nozzle on the end of the hose shut off so that no water is coming out and then you have no amperage, but you will still have the pressure as long as it is turned on at the source so you will still have voltage.”

“In these diagrams, you just have to figure out how the water is going to get from one side to the other. These circles are things like relays or lights or motors. When the electricity makes it through them, they turn on as long as the electricity can make it all the way to the other side.”

That was it! That was my lesson in ‘lectricity. All I needed to know. The blueprints were big puzzles. I loved working puzzles. You just had to figure out how you were going to get something to run, and that meant that certain relays had to pickup to close switches that might pick up other relays to close other switches. I found that most of the electricians in the shop were good at working all sorts of puzzles.

Andy went to the cabinet and grabbed one of the Simpson multimeters and a handset for a telephone that had red and black wires wrapped around it.

A telephone handset that looked like this, only it had a battery taped to it, and two leads coming out the bottom

A telephone handset that looked like this, only it had a battery taped to it, and two leads coming out the bottom

I was puzzled by this at first. I thought I would just wait to see what we did with it instead of ask what it was for. We grabbed our tool buckets (which also doubled as a stool and tripled as a trash can as needed), and put them in the substation truck. The other truck was being manned by the designated electrician truck driver for that week. We needed a truck that we could drive around in without having to hold up the truck driver.

We drove to the coalyard and went into the dumper switchgear. Andy and Diane opened up a large junction box that was full of terminal blocks with wires going every which way in an orderly fashion. They located a couple of wires, and Andy unwrapped the wires from the handset while Diane removed the screws holding the wires to the terminal block. Then Andy clipped one wire from the telephone handset to each of the two wires and handed me the phone.

Diane told me that they were going to drive down toward the train gate where the railroad tracks come into the plant and try to find these wires on the other end. So, what they needed me to do was to talk on the phone so when they find my voice, they will know that they have the right wires. Diane said, “Just say anything.” Then they left the switchgear and I could hear them drive away in the truck.

Well. This was my opportunity to just talk to no one for a while without interruption. How many times do you get to do that in one day? Probably only when you are on the way to work and back again if you aren’t carpooling with anyone. Or you’re sittin’ on your “thinkin’ chair” in a single occupant restroom. So, I just kicked into Ramblin’ Ann mode and let myself go. I believe my monologue went something like this:

“The other day I was walking through a field, and who should I run across, but my old friend Fred. I said, ‘Well, Hi Fred, how is it going?’ and Fred told me that he was doing just fine, but that he had lost his cow and was wondering if I could help him look for it. I told him I couldn’t right now because I was helping some people find a wire at the moment, and if I became distracted, we might not only lose the cow, but we might lose the wires as well, so I better just keep on talking so that my friends on the other end can find the wires they are looking for. After that I went to the store and I picked up three cans of peas. I thought about getting four cans of peas but settled on three and brought them to the checkout counter, and while I was waiting in line I noticed that the little boy in front of me with his mom was looking at me as if he wanted to have one of my cans of peas, so I quickly made it clear to him that I was buying these cans of peas for myself by sliding them further away from him and glaring at him. Luckily the boy wasn’t persistent otherwise I would have broken down and given him a can of peas because he was looking kind of hungry and I was feeling sorry for him, though, I didn’t want him to know how I was feeling, so I put on a grim expression….”

Needless to say… My monologue went on for another 15 minutes. Yes… .15 minutes. I had expected Andy and Diane to have returned earlier, but I didn’t know how hard it was going to be to find the other end of the wires, so I just kept on ramblin’ to the best of my ability. It’s like what it says in the Bible. If we wrote down everything I said, it would have filled many volumes. Being a Disciple of Ramblin’ Ann came in handy that day. For more about Ramblin’ Ann, you can read the following post:

Ed Shiever Trapped in a Confined Space With A Disciple of Ramblin’ Ann

When Andy and Diane returned they said that they had found the wires right away, but that they had sat there for a while just listening to me ramble. They said I was cracking them up. They also mentioned that they thought I was completely crazy. Well. I was glad that they found the wires and that my rambling abilities had come in handy.

Five months after I had joined the electric shop, Andy and I were sent to Oklahoma City to learn about a new kind of electric troubleshooting. It was called “Digital Electronics”. I had just finished my electronics class at the Vo-Tech, and so I was eager to put it into practice. Andy and I went to a two day seminar where we learned to troubleshoot what was basically a PC motherboard of 1984. We used a special tool called a digital probe and learned how the processor worked with the memory chips and the bios. It wasn’t like a motherboard is today. It was simple.

A simple Motherboard like this

A simple Motherboard like this

It was just designed for the class so that we could use the digital probe to follow the different leads from the chips as the electric pulses turned on and off.

We were using digital probes similar to this

We were using digital probes similar to this

At the time I was thinking that this was a waste of time. I had been learning all about troubleshooting electronic circuits from Bill Rivers and Sonny Kendrick. I couldn’t see how this was going to be useful. I didn’t know that within a couple of years, most of our electronic circuits in the precipitator controls were all going to be replaced with digital controls, and this was exactly what I was going to need to know.

So, Andy and I spent two days learning all the basics of how new computers were going to be working. This was the same year that Michael Dell was beginning his new computer company further down I-35 in Austin Texas. Who would have thought that 18 years later I would be working for Dell. But that’s another lifetime away…

Comments from the original post:

Ron Kilman January 19, 2013:

Early in my career at the Seminole Plant I learned when someone paged you on the gray phone, you should always check the earpiece of the phone before you put it on your ear – it might be full of clear silicone calk (or worse). Also, at the end of the day when you reach to pick up your lunch box, you should pick it up gently. Someone could have slipped a full bottle of mercury (like 20 pounds) in it. This prevents you from pulling the handle off your lunch box or hearing it crash to the floor, smashing everything in its path. It’s amazing what Power Plant Men are capable of doing.

  1. Plant Electrician January 19, 2013:

    We used hand lotion in the electric shop for the gray phone trick. I remember Andy catching an unsuspecting operator in the main switchgear more than once.

    1. Ron Kilman January 20, 2013:

      Hand lotion is much nicer than silicone caulk!

Power Plant Lock Out – Tag Out or Just Tag Out

Originally Posted July 12, 2013:

All safe electricians worth their salt know about OSHA regulation 1910.147(c)(3). Only Power Plant electricians have learned more about OSHA regulation 1910.147(a)(1)(ii)(C). Section 147 has to do with locking out and tagging a power source in order to protect the employees working on the circuit. 147(a)(1)(ii) says that Power plants are exempt from section 147. In other words, if you are working in a power plant it is all right to have a less stringent lock-out/tag-out procedure in place than if you didn’t work in a power plant.

One of the first things I learned from Charles Foster, my foreman when I became an electrician was how to remove the “heaters” from a breaker relay in order to protect myself from an “unauthorized” operation of the breaker. That means…. in case someone accidentally turned on the breaker and started up the motor or whatever else I was working on. “Heaters” are what we called the overloads that trip a 480 volt breaker when the circuit uses more power than it is supposed to be using. They are called heaters, because they literally “heat up” in order to trip the breaker.

typical 480 volt overload heaters

typical 480 volt overload heaters

Charles Foster told me the following story about my bucket buddy Diana Lucas (later Diana Brien):

Dee was wiring up a sump pump at the bottom of the coal dumper. The motor had been taken out while the pump had been repaired. Once back in place Dee was sent to wire it back up. The proper clearance had been taken to work on the motor. That is, she had gone to the Shift Supervisor’s office in the Control Room to request a clearance on the motor. Then later she had witnessed the operator opening the 480 volt breaker and place the clearance tag on the breaker.

A typical Clearance Tag. Our tags had the word

A typical Clearance Tag. Our tags had the word “Clearance” at the top. We called them a “Hold Tag”

The tag is signed by the Shift Supervisor and is only to be removed by an operator sent by the Shift Supervisor. It is placed through a slot in the handle on the breaker that keeps the breaker from closing unless the tag is removed first…. well… that’s the theory anyway.

Dee had just finished hooking the three leads in the junction box together with the cable coming into the box using two wrenches. She reached down into her tool bucket that she was using as her stool to get some rubber tape to begin wrapping the connections. The three bare connections were sticking out in front of her face.

A large vertical pump motor

A large vertical pump motor

The Junction box is the box on the right side of this motor. At this point the cover would be off and the wires would be sticking straight out. As she reached into the bucket, the motor turned on and began running.

Startled, Dee stopped what she was doing. I suppose she also pinched herself to make sure she wasn’t dreaming. Then I suppose she checked her diaper to make sure it was still dry. Then I suppose she may have said a few choice words whether anyone was around to hear them or not. Maybe not all in that order.

For those of you who don’t realize what this meant. It meant that if the motor had started running about 5 to 10 seconds later, someone, some time later may have made their way down to the west end of the dumper sump only to find one charred Diana Lucas (who never would have later become Diana Brien). They might not have recognized her at first. I can assure you. It wouldn’t have been pretty.

You see… someone had removed the Hold Tag and purposely started up the motor totally disregarding the clearance. I won’t mention any names, but his initials were Jerry Osborn.

So, after Charles told me this story, he showed me what to do to prevent this from ever happening to me.

Charles and I went to the Shift Supervisor’s office to take a clearance on a motor. Then we followed the operator to the breaker and watched him open the breaker and put the tag on the handle. Then we signed something and the operator left.

After the operator left, Charles told me to open the breaker and slip the hold tag through the slot in the door so that the door could open without removing the tag. I followed his directions.

Once the door was open, he told me to remove the three heaters on the bottom of the relay and hide them at the bottom of the breaker box.

480 volt relay with the 3 heaters at the bottom. That white square is an overload reset button. in front of the first heater

480 volt relay with the 3 heaters at the bottom. That white square is an overload reset button. in front of the first heater.  The black squares with the W 34 on them are the heaters.

You see… with the heaters removed, even if someone were to close the breaker and try to start the motor, the electricity would never leave the breaker box because I had just created an open circuit between the relay and the wires going to the motor.

Well… If you don’t learn from history you are bound to repeat it. No. I’m not going to change subjects and start talking about making it illegal to own guns.

Anyway, there is always a chance for something to go wrong. The Peter Principle demands it. So, at one point, someone forgot to replace the heaters in the relay before returning their clearance. When the motor was tested for rotation, it didn’t work. At that point the electrician knew that they had forgotten to re-insert the heaters. So, they had to return to the breaker to install the heaters before the motor would run.

This didn’t set well with the Shift Supervisor, who has supreme power at the power plant…. well… besides the janitor who had total control over the toilet paper supply.

Technically we were not going around the hold tag by removing the heaters because they were downstream from the breaker handle which cut off the power to the relay. The Shift Supervisor on the other hand believed that the hold tag included everything in the breaker box, including the relay and heaters (which really was stretching it).

An argument ensued that pitted the shift supervisors and the supervisor of operations, Ted Holdges with the electricians. Ted argued that we should not be removing the heaters to keep ourselves from becoming electrocuted accidentally when someone inadvertently removes a hold tag and turns the breaker on and starts up a motor. Electricians on the other hand argued that if we were going to be exposed to the possibility of being electrocuted, we would rather not work on any circuit. Without being completely assured that we would not occasionally be blown to pieces when someone or something accidentally caused the circuit to become hot, we concluded it wasn’t worth it.

So, a compromise was reached. We could remove the heaters, but they had to be put in a plastic bag and attached to the hold tag on the outside of the breaker. That way, when the clearance was returned, not only were the heaters readily available the operator would know to contact the electrician to re-install the heaters. The electricians didn’t really like this alternative, but we agreed. We were assured that there wasn’t any way that a breaker was going to be turned on and operated with the heaters in them when someone was actually working on a circuit.

Fast forward three years. 1992.

Bill Ennis and Ted Riddle were working on replacing a large electric junction box on the stack out tower. The Stack Out Tower is the tower that pours the coal out on the coal pile. Halfway up this tower there is a large junction box where most of the electric cables passed through going to the top of the tower. Bill Ennis had taken a clearance on a number of motor and control breakers.

Bill returned from lunch one day to work on the junction box, removing the old cables. Putting new lugs on them and placing them in the new junction box. As he began working, he decided to take out his multimeter and check the wires he was about to work on….

We had a couple of these Simpson Analog Multimeters in the shop

We had a couple of these Simpson Analog Multimeters in the shop

Bill was surprised to find that one set of cables were hot. They had 480 volts on them. Everything in this box should have been dead. I suppose he pinched himself to make sure he wasn’t dreaming. Then I suppose he checked his diaper to make sure it was still dry. Then I suppose he said a few choice words that Ted may have heard if he was standing close by.  Maybe not all in that order.

What had happened was that there had been two clearances on this one particular motor. One of the electricians had returned his clearance, and had installed the heaters that were in the plastic bag on the front of the breaker and the motor had been tested for rotation and put back in service. The operator had taken both clearances off of the breaker by mistake.

Ok. It was time for another meeting. Something had gone wrong. If it had not been for the guardian angels of both Diana Brien and Bill Ennis, at this point we would have had at least two dead electricians, and believe me…. I know that when an operator had later climbed the stack out tower to check the equipment, if he had run across the body of Bill Ennis… it definitely wouldn’t have been pretty (even on a good day).

I attended this meeting with Ted Holdges as did most of the electricians. I began by telling Ted that when we had met three years earlier I was newly married and wouldn’t have minded so much if I was killed by being electrocuted because I was young and only had a wife who knew how to take care of herself. But now it was different. I had a little girl at home and I need to be around to help her grow up.

Ted looked surprised by my remark. I had just told him the way I felt about this whole situation. The argument that we were making was that we should be able to place locks on the breakers just like OSHA demanded from the other industries. We had demonstrated that we didn’t have a system that would protect us from human error. We needed something that definitely kept us safe.

We told Ted that even if we had locks, and for some reason the breaker just had to be closed and the electrician had forgotten to remove his lock, the shift supervisor could keep a master key in his office to remove the lock. He finally agreed. His problem was a loss of control. The thought was that the Shift Supervisor had ultimate power.

If you don’t learn from history you are bound to repeat it. No. I’m not going to change subjects and talk about socialized healthcare and how it destroys all concepts of quality and privacy.

So, as electricians, we weren’t really happy with this situation. We had a secret weapon against human error. Sure we would place a lock on the breaker. But after the operator would leave, before we placed our lock on the breaker, we might just open up the breaker box and remove the entire face off of the relay. It was similar to removing the heaters only it was bigger. It completely opened the circuit no matter what.

I hadn’t really planned on talking about this next story for a couple more years, but I’ll tell it now because it fits with this story.

In the month of May, 2001. I had already given my notice to leave the plant to work for Dell as a software developer. I was asked to work on a job with my old bucket buddy Diana Brien.

Diana-Brien

My Bucket Buddy Diana Brien

The problem was that there was a grounded three phase circuit up on the Surge Bin tower. It had been tracked down to the dust collectors located below the surge bin conveyor floor.

Dee and I walked up to the Gravimetric feeder deck to look at the breaker to make sure it was turned off. It had a Danger tag on it that had been placed by the Shift Supervisor.

A typical Danger tag used at the plant

A typical Danger tag used at the plant

The breaker was open and the message on the tag said “Do not close this breaker. The circuit is grounded”.

Ok. We walked up to the surge bin tower through the counter weight room for belts 18 and 19. We opened up the big junction box that fed the power to the two large dust collector motors on the landing behind us. After taking the cover off of the box, I took out my multimeter and checked the circuit.

Like this. Ok. So the multimeters had become more sophisticated over the years.

Like this. Ok. So the multimeters had become more sophisticated over the years.

The big copper bus was dead (that means, there was no electricity present).

So, Dee and I worked on locating the grounded circuit. I had just removed the cover to the junction box on one of the motors while Dee was removing some wires from the control panel when Larry Tapp arrived on the landing through the same route we had taken from the gravimetric feeder deck.

Larry asked us what we were doing. We told him we were tracking down the ground on the Dust Collectors. Larry looked surprised.

You see… Larry explained that he had just come from the Gravimetric feeder deck where he had just closed the breaker for the dust collectors. This particular breaker didn’t have a relay, as it was controlled by the control panel where Dee had been working.

So, I rechecked the copper bus with my multimeter and it was hot. 480 volts hot.

I had just been looking through my tool bucket for two wrenches to remove a piece of the bus work just to make sure the ground wasn’t in the box itself when Larry had arrived. In other words, if Larry had arrived 5 to 10 seconds later, he would have probably arrived to find Dee looking down at my body, stunned that I had just been electrocuted by a circuit that we had just tested and found dead.

If you don’t learn by history you are bound to repeat it.

You see… there is a difference between a Hold Tag and a Danger Tag. A hold tag is placed on a breaker after someone has requested a clearance by signing a form in the Shift Supervisor’s office in the control room. A Danger tag can be placed and removed at anytime by the person that placed the tag on the breaker.

So, I personally wrote this up as a “near” accident. We could have wiped our brow, pinched ourselves to make sure we weren’t dreaming. We could have checked our diaper to make sure it was still dry and then Dee could have said a few choice words that Larry Tapp would have agreed with (I have always had a mental block against expressing myself in that manner…. I found other ways). And we could have left this incident as a secret between Larry, Dee and I.

I thought it was a good time to remind the electricians throughout power production to follow the clearance procedures when working on high voltage circuits. Sure. Dee, Bill Ennis and I have powerful guardian Angels looking out for us…. but gee… I think we should be expected to look out for ourselves. So, I wrote up this incident to warn the rest of the team….. If we don’t learn from history, we are bound to repeat it.

I met with my roomie Steven Trammell, a month and a half later in Perkins Restaurant in Stillwater, Oklahoma to discuss his performance plan. I was a 360 Degree Assessment Counselor and my favorite roommate from 17 years earlier had chosen me to review his performance appraisal. During this meeting I asked Steven, who had driven from Harrah, Oklahoma from another power plant to meet with me, if he had read the near accident report about the dust collector at our plant.

My roomie told me that he had, and that he thought it seemed to unduly blame the electrician. I told him I was the electrician and that I wrote the report. After 18 years of being an electrician, I had become so relaxed in my job that I had become dangerous to myself and others. So, after I did a cause-effect analysis of the near accident, most of the cause had come from my own belief that I could circumvent clearance procedures and save time and still believe that I was being safe.

On my drive back to the plant after the meeting with my favorite roomie of all time, I had time to think about this…. I was going to be leaving the power plant in a little over a month to work for Dell as a programmer. I knew this when I had been negligent with the Danger tag. I could have caused the death of both Dee and I. I will sure be glad to be in Texas. — Only.. I will miss my friends most of all.

I leave the Power Plant with this one thought…. If you don’t learn from history, you are bound to repeat it. I mean it… This time I really do.

Comments from the previous posts:

  1. Ron Kilman July 13, 2013:

    This is a great story. I thank God for His guardian angels and I thank you for taking responsibility. It was always difficult investigating accidents because people are a little reluctant to share their mistakes with the world. But a wise man knows it’s better to have a bruised ego than a fried friend.

Larry McCurry July 13, 2013:

Kevin,

  1. As an old time operator and having follows in my fathers footsteps as a Shift Supervisor, The answer to all of these problems to add steps to the clearance procedure to make sure the heaters were removed and then replaced. The second was definitely an operator error, and I agree with you about it, The Shift supervisors did argue for it however the hubris of certain power hungry people managed to intimidate and control the situation. You do not ever work on equipment without your own clearance or a plan that includes the SS, as you mentioned He is the operating authority, or was until a person by the initials of Jim Arnold rewrote the procedures and made himself the Authority.

Jack Curtis August 12, 2013:
Good Story Indeed.
There is, I’m sure, a gene in human DNA labeled: “Murphy” that assures that anything that can happen, does. And in total ignorance, I’ll bet that some constant percentage of plant electricians were fried. I suppose the onset of computer controls has reduced that but that it still happens at a lower rate. The way you and your friends sort of automatically reached for your multimeters is a clue! These are shocking, highly-charged stories…
NEO July 16, 2014
Yep, and even c.3 rules fail on occasion. Most places I’ve worked the other key was at superintendent level, and required a veritable mountain of paperwork to acquire. G-d help anybody who lost his key! 🙂
Traditionally on distribution lines, we left our shotgun hanging on the (grounded) stinger to notify everyone since there is no (effective) way to LOTO a power line. Imagine my vocabulary one fine day up in Montana when I drove up to reenergize a tap to find it energized and a crew I had never seen putting my shotgun in their truck. That day, I got a profuse apology (and assurances) from the line superintendent in person, 50 miles from the office. I never worked dead again without three point grounding, and I require my people to use it as well, I don’t like funerals. Hot line is actually safer for most operations.
Dan Antion July 17, 2014
I have great respect for electricity, power tools and the threat of human error. That was a close call. Glad you had the presence of mind and experience to think to check again.

Relay Tests and Radio Quizzes with Ben Davis

Originally posted January 24, 2014:

Reorganizations naturally shuffle things around.  People are generally resistant to change and don’t like to find that their routine has been changed without having their input on how to make things better.  When the coal-fired power plant in North Central Oklahoma went through a downsizing and reorganization in the latter part of 1987, my job changed slightly.  Personally, I was grateful for the changes.

Before the reorganization, I had inherited both the precipitators (the large boxes at a power plant that take the ash out of the exhaust from the boiler).  This meant that every overhaul, I knew what I was doing.  I was working on and in the precipitator.  This was generally a dirty and thankless job.

The plant has a similar electrostatic precipitator, only it is twice as long

The plant has a similar electrostatic precipitator, only it is twice as long

After the reorganization, however, Terry Blevins was assigned to work on the Unit 2 precipitator, while I worked on Unit 1.  I will go into this in more detail later, but for this post, I’ll just point out that this meant that when Unit 2 was on an overhaul (that means the unit is taken offline for one to three months in order to fix and repair things that can only be done while it is offline) I wasn’t automatically assigned to the precipitator.  So, I could work on other things.

Before the reorganization, Sonny Kendrick had the title “Electric Specialist”.  After the reorganization we no longer had a specialist.  I’m not sure exactly why.  I know that at Muskogee, they still had a specialist in the electric shop.  — I will talk about him next year (the specialist at Muskogee).  Anyway, I know that Sonny, at the time, was not too happy about his change in job title.  I don’t blame him.  I would be too.

One of the things that the Electric Specialist did during overhauls was test tripping relays.  Now that we no longer had a specialist, that was left up to whomever…. The first electricians, besides Sonny, that were assigned to relay testing was Ben Davis and myself.  I had started doing it on my own and after about a week, Ben Davis was assigned to help me out.

Ben Davis

Ben Davis

We were on a major overhaul on Unit 2 and it had been decided that we were not only going to test the regular super-high voltage breaker relays, we were also going to test all the 480 volt switchgear relays for Unit 2, as well as the intake and coalyard switchgears.  I don’t remember if we made it to the river pump switchgear, but I wouldn’t be surprised.  Once we started, there was no stopping us.

When I first was told to test the relays, Bill Bennett (our A foreman) told me to have Sonny tell me how to do them.  So, I walked into the lab and told Sonny that Bill had told me to ask him to help me learn how to test the protective relays on the switchgear.  Sonny, not looking too happy, grabbed a small stack of manuals, walked out into the main switchgear with me, and said, “Here is the relay test set.  Here are the manuals that tell you how to hook up the test set and test them.”  He turned and walked away…. I was sort of hoping for a more intimate lesson…

I knew the reason Sonny was so upset.  Later I learned why he would be as upset as he was to not be able to test the protective relays.  It was because when you test, clean and adjust protective relays you have an immediate rush of satisfaction that you have just done something very important.  Let me just say quickly (because in another post I will expound upon this), a protective relay is what keeps motors from blowing up.  It is what prevents blackouts from happening across the nation.  Without properly calibrated protective relays, a power company is just asking for a disaster (or… well….. their insurance company is, because they are the ones that usually end up paying for the damage — which I will also talk about in a later post).

I thought the relay test set that Sonny showed me was the neatest thing I had seen so far in the electric shop.  There were two boxes that hooked together with an umbilical cord.  They had dials, switches, connectors, meters and a digital readout down to the millisecond.  That is, you can read the time to trip a relay down to the one thousandth of a second.

AVO Multi-Amp SR-76 Relay Test Set

AVO Multi-Amp SR-76 Relay Test Set

I only wish that I had a bigger picture of this relay test set so that you could admire it as much as I did.  Even today it gives me goosebumps!  Ok.  I can imagine those relay technicians that read this blog are looking at this and thinking…. “What kind of piece of junk is this?”  Hey (as Mark Fielder used to say), this was my “baby” (only he was referring to the precipitator).

So, back to the story at hand…

Even though I was having a heck of a fun time trying to figure out how to perform these relay tests by reading these manuals about the different kinds of relays, I was glad when Ben Davis was assigned to work with me.  I don’t know if he had worked on relays before, but he seemed to know just what to do to hook up the test set and make things easier.

A panel of Protective Relays

A panel of Protective Relays

The best suggestion that Ben had right off the bat was that we should be listening to the radio while we were working.  This might have been a preventative measure after the first couple of days to prevent the same situation from occurring that happened to Ed Shiever when he and I were trapped inside a confined space for a couple of weeks (See the post:  “Ed Shiever Trapped in a Confined Space with a disciple of Ramblin’ Ann“).  Either way, it was a great idea.

You wouldn’t think that inside a switchgear 20 miles from the nearest town with a radio station, that we would have any reception on a little transistor radio, but we were able to manage.  It seemed that we had to be a little creative at times with the antenna in certain locations, but, like I said.  We managed.

My perception of Ben Davis up to this point was that he was a “Good-ol’ boy”.  That is, a country music type Oklahoman that had grown up in Shidler, Oklahoma where the major attraction in the town was the High School.  To my surprise, I quickly found out that he was a connoisseur of Rock and Roll.

It wasn’t until I was in college before I realized that the easy listening station I had been listening to on our family radio at home while I was growing up was playing rock and roll songs using an orchestra with violins and clarinets instead of electric guitars.  I learned from my dorm mates all about groups like Simon and Garfunkel, The Beatles (yeah… can you believe it?  I mean.  I knew “Hey Jude”, “Let it Be” and a few others, but most of the Beatles I thought were instrumentals normally played on violins with a man waving a wand) and many others.  When I found out about “Rock and Roll”, I had to go out and buy dozens of 8-track tapes, as fast as I could find them.

A stack of 8 Track Tapes

A stack of 8 Track Tapes

So, here was Ben Davis.  Even better than the “Good Ol’ Boy” that I already thought he was.  And he loved classical rock and roll.  I can only say that the next month and a half while we tested relays all over the plant, were one of the best times I have ever spent in my life!  He knew all the 60’s and 70’s rock and roll bands.

As each song would come on the radio, we would guess (well, I was guessing most of the time…. most of the time Ben already knew), what the name of the song was and the name of the band.  So, not only were we doing one of the most satisfying jobs at a power plant, but I was also have a lot of fun with Ben listening to the radio!  Who would have thought it?  No wonder Sonny was upset he wasn’t testing relays this overhaul.

I could go on about all the different bands and their backgrounds that I learned from Ben during that overhaul, but (unlike me), you probably already know all that stuff.  It never ceases to amaze me how many holes I have in my education until one is staring at me in the face.

This reminds me of a side story, and I apologize if I have told this before…. I don’t think I have….

After the Reorganization, and after I moved to Stillwater from Ponca City, Scott Hubbard (and Toby O’Brien) and I began carpooling.  One morning as we were listening to NPR, Scott Hubbard mentioned something about a “cur”.  I asked him, “What’s a cur?”  Well, he had the exact same reaction when 11 years earlier I had asked my friends in college at Oklahoma University, Tim Flowers and Kirby Davis, “What’s an orgasm?”  —  See how little holes in your education can make a big impact?

Just so you don’t get caught in the same predicament…  A “Cur” is a mongrel dog.  Scott Hubbard couldn’t believe that someone that read the dictionary for fun wouldn’t know what a “cur” was.  What the heck?  I didn’t grow up in Oklahoma!  — end of side story… which really isn’t a side story, since it was about a Power Plant Man — Scott Hubbard.  He probably knew what a “cur” was before he could walk.  — I know I haven’t told that story before!  I would have remembered that.

I’m not going to go on about all the fun that I had with Ben Davis testing protective relays.  I enjoy my memories, but I’m sure you don’t want to hear all about how much I looked up to this Power Plant Hero.  The only thing I will add is that the time I spent with Ben during that overhaul has been etched into my memory as one of the most enjoyable times of my life.  So, I’ll go onto the next step in our Protective Relay story….

A few years later, in 1993, Sonny Kendrick and Ben Davis and I were sent to “Advanced Protective Relay Maintenance” training in Dallas, Texas.  I remember this time so well, I remember the address where we were went.  It was at 4271 Bronze Way, Dallas, Texas.  It was hosted by the same company that made that wonderful test set I pictured above.  The AVO Multi-Amp Corporation.

I brought my wife Kelly and my three year old daughter Elizabeth with me.  They stayed at the hotel during the day and played in the swimming pool, while I went to class.

The classes lasted four days, Monday through Thursday.  That was where I learned that even though I thought our relay test set was the coolest piece of equipment in the electric shop, it turned out to be archaic by “Protective Relay Maintenance” standards.  Not that it didn’t do the job….   So, in order to train us properly, they let us use our own old test set during the training so that we could see how to properly test really advanced relays such as Distant Relays, Syncro-verifier relays, Negative Sequence Relays,directional distance relays and Pilot Wire relays.  — These are relays that are found in a large substation that trips high voltage lines that run long distances across the country.  — I can tell you’re jealous.  — Well.. I imagine it anyway.  Knowing what I know now.

This is the book we used in class

So, why drag you all the way to Dallas for this story?  There’s a reason.

time for a second side story:

You see. Tim Flowers, whom I mentioned above, knew not too long after he met me that I have the knack of running into people that I know (or should have known in this case), would love this story.  You see, I met Tim and Kirby at Oklahoma University and they drove with me to Columbia Missouri in 1979 (along with my brother Greg) when I went to register for classes at Missouri University when I decided to go back to school in my home town.

When we arrived in the town, we were hungry after driving for 8 hours straight from Stillwater, Oklahoma to Columbia, Missouri… so we stopped at Godfather’s Pizza.  As we walked in, there was a girl and a guy standing at the counter ordering a pizza.  The pretty girl (Pamela Ramsey) with long red hair turned and saw me.  She immediately came toward me saying “Kevin Breazile!!!!  You owe Me!!!  Slightly shocked and pleased, I said, “What for?” She reminded me that I never gave her the pictures that were taken during the Senior Prom.  You see.  I had taken her to the Senior Prom.

Later I explained that this happens to me a lot.  I meet people that I know in the oddest places (even though this wasn’t so odd, since I had grown up in Columbia). It was just that this was the first person we had seen since we entered town.  From that point on, Tim (who later worked as a summer help at the power plant) expected that everywhere we went we would run into someone I knew….

End of the second side story.  I’m sorry that this is making the post a little longer than usual.  I know you have to get back to work….

So, back to the relay training course in 1993 that Ben Davis, Sonny Kendrick and I were taking in Dallas…. On Wednesday night during the training there was a dinner held in a small banquet room in the hotel.  Well… of course I had to take my wife and my daughter.   So here we were sitting around this table at dinner with the rest of the class of about 10 other non-Sooner Plant employees….

I decided to talk to the guy next to me.  He said something back and my wife Kelly asked him, “Where in New Jersey are you from?”  She had picked up on a New Jersey accent.  He said, Well..  I work in the east for a company called Ebasco, but I’m really from the Midwest.  (oh.  That was my territory).  So I asked a follow-up question.  “Where in the Midwest are you from?”  He said, “From Missouri.”  — Oh.  I thought.   This is interesting. So was I.

I asked a follow-up question.  “Where in Missouri are you from?”  He answered…. “Columbia, Missouri.”  (What?   Where I had grown up?)….  So, I asked a second follow-up Question…. “What High School did you go to?”  With a curious look the man answered….. “Rockbridge High School…”   (Man!!!  the same one as me!!!)…. The third follow-up question….. “What year did you graduate?”  Now, looking really suspicious… he said, “1978”.   Trying to contain my excitement… I replied….. “Oh… so, you graduated from Rockbridge High School the same year I did….”

What are the odds?  There were 254 students in our graduating class.  This guy who currently lived somewhere in the east is sitting next to me at a dinner of about 10 people attending Advanced Protective Relay Training in Dallas, Texas where neither of us are from, and we both graduated from the same school back in Columbia, Missouri 15 years earlier!  His name is Randy Loesing.  He was working for a company called Ebasco at the time.  He said, “I thought I recognized you!  I just wasn’t sure.”  I didn’t recognize him at all until I went back home and looked in my yearbook.

It turned out that he kept in touch with two of my oldest friends from the second grade, Mark Schlemper and Brent Stewart.   So we talked about them.  What an incredible coincidence.   Like I may have mentioned before.   It happens to me all the time.  It turns out that an old friend of mine from the 3rd grade in Columbia, Missouri that I used to go to his house when we were stamp collectors and had a stamp collecting club, lives 5 miles south of me today in Round Rock Texas (He’s in Pflugerville).

Russell Somers lives in the  same direction and just about the same number of miles as when we were kids.  Not only that, but he worked at Dell while I was working at Dell (though I didn’t know it at the time).  He has an older daughter and a younger son, just like me only younger.  The same is true for another 3rd grade friend that I  graduated from Rockbridge Highschool and the University of Missouri with, Caryn Lile (now Caryn Iber) who lives in Wisconsin.  She has a daughter and a son the same age as my kids.  She was living in Tulsa when I was living in Stillwater, Oklahoma.  — Like I said… happens to me all the time.

Tim Flowers realized this odd phenomenon  in college.  I had told him earlier that my father told me that if I was every stranded somewhere that I could look up the local Veterinarian and tell him that I was the son of Dr. James Edward Breazile, and they would help me.  So, when we were hiking in the mountains in Colorado and we met a man walking along a trail in the middle of nowhere above Estes Park near the Great Divide, when I told him who I was, he gave us a curious look…. then divulged his most intimate secrets of his life and where he had stashed his most values possessions, Tim told me later.  “I really thought he was going to know who you were when he gave us that funny look.”  I replied.  “I think he did..”

I again apologize for the length of this post.  It is rare that I ramble on this long.  I can thank Ramblin’ Ann for the ability to Ramble so well.  I can thank Ben Davis for recognizing a rambling situation and replacing it with a rock and roll learning opportunity.  As I said earlier.   One of the most enjoyable times I have spent in my entire life is the time I spent with Ben Davis testing Protective Relays!  Bless you Ben and I pray for you, your wife, your son and your daughter on the way to work each morning.

Today when I hear any of the hundreds of rock and roll songs come on the radio that we listened to that month and a half, I can see us testing the relays, looking off into space saying, “Rolling Stones?”  “No.   Steve Miller Band?”  Really?  I thought Browneyed Girl was sung by the Rolling Stone!  It turned out that the version that we listened to was from the creator of the song, Van Morrison. Who would have thought that he would sound so much like Mick Jagger.  I can see Ben saying… I see what you mean…  it kind of sounds like Mick Jagger.

As an add on to this story…

I now work at General Motors in Austin Texas.  My best friend in High School was a guy named Jesse Cheng (I have mentioned him in other posts, especially in reference to the phrase “Jesse!  Come get your Chili!).  He was two years older than me, and throughout the years we would lose track of each other and then reconnect.  He went to Yale to become an Engineer, then to the University of Missouri to become a Medical Doctor, then to Harvard to earn a Masters in Public Health and Epidemiology.

It turns out that we both now work at General Motors where he works in Arlington Texas as a Medical Director and I work in IT in Austin.  We can IM (Instant Message) each other whenever we want, and we talk now at least once every week.

Pain in the Neck Muskogee Power Plant Relay Testing

Don’t let the title fool you.  I love testing Power Plant Protective Relays.  There is a sense of satisfaction when you have successfully cleaned, calibrated and tested a relay that is going to protect the equipment you have to work on every day.  With that said, I was hit with such an unbelievable situation when testing Muskogee Relays in 1995 that I was left with a serious pain in the neck.

On August 14, 2003 the electric power in the Northeast United States and Canada went out.  The Blackout lasted long enough to be a major annoyance for those in the that region of the United States.

Map of the power blackout in 2003

Map of the power blackout in 2003

When I heard about how the blackout had moved across the region, I immediately knew what had happened.  I was quickly reminded of the following story.  I told my wife Kelly, “I know exactly why such a large area lost power!  They hadn’t done proper preventative maintenance on the Protective Relays in the substations!  Just like….”  Well…. I’ll tell you that part now:

I have mentioned in a couple of earlier posts that something always seemed a little “off” at the Muskogee Power Plant.  I had decided early on that while working there I would stick to drinking sodas instead of water.  See the post:  “Something’s In the Water at the Muskogee Power Plant“.  Even with that knowledge, I was still shocked at what I found while testing relays at the plant.

This story really begins one Sunday at Muskogee when one of the Auxiliary Operators was making his rounds inspecting equipment.  He was driving his truck around the south edge of the Unit 6 parking lot on the service road.  He glanced over at a pump next to the road, and at first, he thought he was just seeing things.  After stopping the truck and backing up for a second glance, he was sure he wasn’t dreaming.  It’s just that what he was seeing seemed so strange, he wasn’t sure what was happening.

The operator could see what appeared to be silver paint chips popping off of the large pump motor in all directions.  After closer examination, he figured out that the motor was burning up.  It was still running, but it had become so hot that the paint was literally burning off of the motor.

Horizontal pump

Like this Horizontal pump only much bigger and painted silver

A motor like this would get hot if the bearings shell out.  Before the motor is destroyed, the protective relays on the breaker in the 4,000 Volt switchgear shuts the motor off.  In this case, the relay hadn’t tripped the motor, so, it had become extremely hot and could have eventually exploded if left running.  The operator shut the motor down and wrote a work order for the electricians.

Doyle Fullen was the foreman in the electric shop that received the work order.  When he looked into what had happened, he realized that the protective relay had not been inspected for a couple of years for this motor.

I couldn’t find a picture of Doyle.  In his youth he reminds me of a very smart Daryl in Walking Dead:

Norman Reedus from Walking Dead

Norman Reedus from Walking Dead

In fact, since before the downsizing in 1994, none of the Protective Relays at the plant had been inspected.  The person that had been inspecting the relays for many years had moved to another job or retired in 1994.  This was just a warning shot across the bow that could have had major consequences.

No one at Muskogee had been trained to test Protective Relays since the downsizing, so they reached out to our plant in North Central Oklahoma for help.  That was when I was told that I was going to be going to Muskogee during the next overhaul (outage).  I had been formally trained to inspect, clean, calibrate and test Protective Relays with two of my Power Plant Heroes, Ben Davis and Sonny Kendrick years earlier.  See the post:  “Relay Tests and Radio Quizzes with Ben Davis“.

My Protective Relay Maintenance course book

My Protective Relay Maintenance course book

Without going into too much detail about the actual tests we performed as I don’t want to make this a long rambling post (like… well…. like most of my posts…..I can already tell this is going to be a long one), I will just say that I took our antiquated relay tester down to Muskogee to inspect their relays and teach another electrician Charles Lay, how to perform those tests in the future.  Muskogee had a similar Relay Test Set.  These were really outdated, but they did everything we needed, and it helped you understand exactly what was going on when you don’t have a newfangled Relay Test Set.

AVO Multi-Amp SR-76 Relay Test Set

AVO Multi-Amp SR-76 Relay Test Set

You need to periodically test both mechanical and electronic protective relays.  In the electronic relays the components change their properties slightly over time, changing the time it takes to trip a breaker under a given circumstance (we’re talking about milliseconds).  In the mechanical relays (which I have always found to be more reliable), they sit inside a black box all the time, heating up and cooling as the equipment is used.  Over time, the varnish on the copper coils evaporates and settles on all the components.  This becomes sticky so that the relay won’t operate at the point where it should.

A panel of Protective Relays

A panel of Protective Relays

In the picture above, the black boxes on the top, middle and right are mechanical relays.  This means that something actually has to turn or pick up in order to trip the equipment.  The electronic relays may have a couple of small relays, but for the most part, they are made up of transistors, resistors, capacitors and diodes.

So, with all that said, let me start the real story…. gee…. It’s about time…

So, here I am sitting in the electric shop lab just off of the Unit 6 T-G floor.  We set up all the equipment and had taken a couple of OverCurrent relays out of some high voltage breakers in the switchgear.  I told Charles that before you actually start testing the relays, you need to have the test documents from the previous test and we also needed the instruction manuals for each of the relays because the manuals will have the diagrams that you use to determine the exact time that the relays should trip for each of the tests.  So, we went up to the print room to find the old tests and manuals.  Since they weren’t well organized, we just grabbed the entire folder where all the relays tests were kept since Unit 6 had been in operation.

When we began testing the relays at first I thought that the relay test set wasn’t working correctly.  Here I was trying to impress my new friend, Charles Lay, a 63 year old highly religious fundamental Christian that I knew what I was doing, and I couldn’t even make a relay trip.  I was trying to find the “As Found” tripping level.  That is, before you clean up the relay.  Just like you found it.  Only, it wouldn’t trip.

It turned out that the relay was stuck from the varnish as I explained above.  It appeared as if the relay hadn’t been tested or even operated for years.  The paperwork showed that it had been tested three years earlier.  Protective Relays should be tested at least every two years, but I wouldn’t have thought that the relay would be in such a bad condition in just three years.  It had been sitting in a sealed container to keep out dust.  But it was what it was.

I told Charles that in order to find the “As Found” point where the relay would trip, we would need to crank up the test set as high as needed to find when it actually did trip.  It turned out that the relay which should have instantaneously tripped somewhere around 150 amps wouldn’t have tripped until the motor was pulling over 4,000 amps.  I could tell right away why the Auxiliary Operator found that motor burning up without tripping.  The protective relays were stuck.

As it turned out… almost all of the 125 or so relays were in the same condition.  We cleaned them all up and made them operational.

There is an overcurrent relay for the main bus on each section of a main switchgear.

A picture of a clean switchgear. Picture 6 rows of switchgear like this

A picture of a clean switchgear. Picture 6 rows of switchgear like this

When I tested the “As Found” instantaneous trip for the main bus relay, I found that it was so high that the Unit 6 Main Turbine Generator would have melted down before the protective relay would have tripped the power to that one section of switchgear.  The entire electric bus would have been nothing but molten metal by that time.

As I tested each of these relays, I kept shaking my head in disbelief.  But that wasn’t the worst of it.  The mystery as to why these relays were all glued shut by varnish was finally solved, and that reason was even more unbelievable.

Here is what I found…..  The first thing you do when you are going to test a relay is that you fill out a form that includes all the relay information, such as, what it is for, what are the settings on the relay, and what are the levels of tests that you are going to perform on it.  You also include a range of milliseconds that are acceptable for the relay for each of the tests.  Normally, you just copy what was used in the previous test, because you need to include the time it took for the Previous “As Left” test on your form.  That is why we needed the forms from the previous test.

So, I had copied the information from the previous test form and began testing the relay (one of the first overcurrent relays we tested)…  Again… I was a 34 year old teacher trying to impress my 63 year old student.  So, I was showing him how you mechanically adjust the relay in order for it to trip within the acceptable range.  No matter how hard I tried, I couldn’t adjust the relay so that it would even be close to the desired range for the longer time trip times…. like the 2 second to 25 second range.  It wasn’t even close to the range that was on the form from the last test.

The form from the last test showed that the relay was in the right range for all the levels of test.  When I tested it, like I said, it wasn’t even close.  So, I went to the diagram in the instruction manual for this type of relay.  The diagram looks similar to this one used for thermal overloads:

Thermal Overload Tripping curves

Thermal Overload Tripping curves

See all those red lines?  Well, when you setup a relay, you have a dial where you set the range depending on the needs for the type of motor you are trying to trip. Each red line represents each setting on the dial.  Most of the relays were set on the same number, so we would be using the same red line on the diagram to figure out at different currents how long it should take for a relay to trip….

Here is the clincher….The time range that was written on the previous form wasn’t for the correct relay setting.  The person that tested the relay had accidentally looked at the wrong red line.  — That in itself is understandable, since it could be easy to get on the wrong line… The only thing is that as soon as you test the relay, you would know that something is wrong, because the relay wouldn’t trip in that range, just like I had found.

I double and triple checked everything to make sure we were looking at the same thing.  The previous form indicated the same settings on the relay as now, yet, the time ranges were for a different line! — Ok.  I know.  I have bored you to tears with all this stuff about time curves and overcurrent trips… so I will just tell you what this means…

This meant that when the person completed the forms the last time, they didn’t test the relays at all.  They just filled out the paperwork.  They put in random values that were in the acceptable range and sat around in the air conditioned lab during the entire overhaul smoking his pipe. — Actually, I don’t remember if he smoked a pipe or not.  He was the Electrical Specialist for the plant.  I remembered seeing him sitting in the lab with a relay hooked up to the test set throughout the entire overhaul when I had been there during previous overhauls, but I realized finally that he never tested the relays.  He didn’t even go so far as try to operate them.

I went back through the records to when the plant was first “checked out”.  Doyle Fullen had done the check out on the relays and the test after that.  Doyle had written the correct values from the manual on his forms.  I could see where he had actually performed the tests on the relays and was getting the same values I was finding when I tested the relays, so I was certain that I wasn’t overlooking anything.

As I tested each of the relays, I kept shaking my head in disbelief.  It was so unbelievable.  How could someone do such a thing?  Someone could have been killed because a protective relay wasn’t working correctly.  This was serious stuff.

One day while Charles and I were working away on the relays, Jack Coffman, the Superintendent of all the Power Plants came walking through the lab.  He asked us how we were doing.  I swiveled around in my chair to face him and I said, “Pretty good, except for this pain in my neck” as I rubbed the back of my neck.

Jack stopped and asked me what happened.  I told him that I had been shaking my head in disbelief for the last two weeks, and it gave me a pain in the neck.  Of course, I knew this would get his attention, so he asked, “Why?”  I went through all the details of what I had found.

I showed him how since the time that Doyle Fullen last tested the relays more than 10 years earlier, these relays hadn’t been tested at all.  I showed him how the main bus relays were so bad that it would take over 100,000 amps to have tripped the 7100 KV switchgear bus or 710 Megawatts!  More power than the entire generator could generate.  It was only rated at about 550 Megawatts at the most.

Jack stood there looking off into space for a few seconds, and then walked out the door…. I thought I saw him shaking his head as he left.  Maybe he was just looking both ways for safety reasons, but to me, it looked like a shake of disbelief.  I wonder if I had given him the same pain in the neck.

That is really the end of the relay story, but I do want to say a few words about Charles Lay.  He was a hard working electrician that was nearing retirement.  People would come around to hear us discussing religion.  I am Catholic, and he went to a Fundamental Christian Church.  We would debate the differences between our beliefs and just Christian beliefs in general.  We respected each other during our time together, even though he was sure I am going to hell when I die.

People would come in just to hear our discussion for a while as we were cleaning and calibrating the relays.  One day Charles asked me if I could help him figure out how much he was going to receive from his retirement from the electric company.  He had only been working there for three years.  Retirement at that time was determined by your years of service.  So, three years didn’t give him too much.

When I calculated his amount, he was upset.  He said, “Am I going to have to work until I die?”  I said, “Well, there’s always your 401k and Social Security.”  He replied that he can’t live on Social Security.  I said, “Well, there’s your 401k.”  He asked, “What’s that?” (oh.  not a good sign).

I explained that it was a retirement plan where you are able to put money in taxed deferred until you take it out when you retire.  He said, “Oh.  I never put anything in something like that.”  My heart just sank as I looked in his eyes.  He had suddenly realized that he wasn’t going to receive a retirement like those around him who had spent 35 years working in the Power Plant.

When I left the plant after teaching Charles Lay how to test the relays, that was the last time I ever saw him.  I don’t know what became of Charles.  I figure he would be 83 years old today.  I wonder if he finally retired when he reached the 80 points for your age and years of service.  He would have never reached enough years of service to receive a decent amount of retirement from the Electric Company since he didn’t start working there until he was 60 years old.  That is, unless he’s still working there now.

As I said earlier in this post, Charles Lay was a very good worker.  He always struck me as the “Hardworking type”.  I often think about the time we spent together, especially when I hear about a power blackout somewhere.  — A word of caution to Power Companies…. keep your protective relays in proper working condition.  Don’t slack off on the Preventative Maintenance.  — I guess that’s true for all of us… isn’t it?  Don’t slack off on Preventative Maintenance in all aspects of your life.

Added note:  On 7/6/2019, 3 weeks after re-posting this story, look what happened:  Con Edison says cause of NYC blackout was substation’s faulty relay protection system

Learnin’ ’bout ‘lectricity with Andy Tubbs

Originally posted January 18, 2013:

The day I became an electrician at the coal-fired power plant, I suddenly became an expert in electricity. I think it was on Tuesday, just one day after joining the electric shop that I was walking through the welding shop when someone stopped me and asked me how they would wire their living room with different light switches at different corners and make it work correctly. As if I had been an electrician for years. Luckily I was just finishing a house wiring course at the Indian Meridian Vo-Tech in Stillwater, Oklahoma and they had us figure out problems just like those.

Within the first week, George Alley brought a ceiling fan to the shop that he had picked up somewhere and was wondering if we could get it to work. My foreman Charles Foster thought it would be a good small project for me to work on to help me learn about electrical circuits.

After all, this ceiling fan could go slow, medium and fast, and it could go forward or reverse. Only at the moment, all it would do was sit there and hum when you hooked up the power. — So that was my first “unofficial” project, since the main goal was to make George happy so that he would help us out when we needed something special from the mechanics.

When I was a janitor, I had observed the electricians preparing to go to work in the morning, and often, one of them would go to the print cabinets at one end of the shop and pull out a blueprint and lay it across the work table and study it for a while. Then they would either put it back or fold it and put it in their tool bucket and head out the door to go do a job. Now, it was my turn.

Andy Tubbs was one of the two people that played the best jokes on me when I was a janitor. Larry Burns was the other person, and he was the person I was replacing as he had moved to another plant. Andy was the one that had taken the handle off of my push broom the moment I had my back turned so that when I turned around to grab my broom, only the broom head was on the floor, while the broom handle was across the counter by the lab, and Andy was across the other side of the room trying to act like he wasn’t paying attention, but with an expression like he had just played a darn good joke. — I actually had to go back into the bathroom I was cleaning so that I could laugh out loud. I was really impressed by Andy’s ability to play a good joke.

While I’m on the subject, shortly after I became an electrician, I was sitting in the electric shop office talking to Charles when he stopped and said, “Wait…. Listen….” We paused, waiting for something…. A few seconds later, the sound of a hoot owl came over the PA system (what we called the “Gray Phone”). Charles said, It’s an interesting coincidence that the only time the perfect sound of a hoot owl comes over the Gray Phone is when Andy Tubbs is riding in an elevator by himself or with a close friend.

Gaitronics Gray Phone

Gaitronics Gray Phone

I had been sent with Andy Tubbs and Diana Lucas (later Diana Brien), to go to the coal yard and figure out why some circuit for the train gate was not working. Andy had pulled out the blueprints and was studying them. I came up alongside him and looked at all the blue lines running here and there with circles with letters and numbers, and what I recognized as open and closed switches….

Here is a simple electric circuit drawn in the style of many of the electric blueprint drawings

Here is a simple electric circuit drawn in the style of many of the electric blueprint drawings

Andy stopped and gave me a momentary lecture on the nature of electricity. It was so perfectly summed up, that for years whenever I thought about the nature of electricity, I always began with remembering what Andy told me. He said this:

“Think of electricity like water in a hose. Voltage is the water pressure. Amperage is the amount of water going through the hose. You can have the nozzle on the end of the hose shut off so that no water is coming out and then you have no amperage, but you will still have the pressure as long as it is turned on at the source so you will still have voltage.”

“In these diagrams, you just have to figure out how the water is going to get from one side to the other. These circles are things like relays or lights or motors. When the electricity makes it through them, they turn on as long as the electricity can make it all the way to the other side.”

That was it! That was my lesson in ‘lectricity. All I needed to know. The blueprints were big puzzles. I loved working puzzles. You just had to figure out how you were going to get something to run, and that meant that certain relays had to pickup to close switches that might pick up other relays to close other switches. I found that most of the electricians in the shop were good at working all sorts of puzzles.

Andy went to the cabinet and grabbed one of the Simpson multimeters and a handset for a telephone that had red and black wires wrapped around it.

A telephone handset that looked like this, only it had a battery taped to it, and two leads coming out the bottom

A telephone handset that looked like this, only it had a battery taped to it, and two leads coming out the bottom

I was puzzled by this at first. I thought I would just wait to see what we did with it instead of ask what it was for. We grabbed our tool buckets (which also doubled as a stool and tripled as a trash can as needed), and put them in the substation truck. The other truck was being manned by the designated electrician truck driver for that week. We needed a truck that we could drive around in without having to hold up the truck driver.

We drove to the coalyard and went into the dumper switchgear. Andy and Diane opened up a large junction box that was full of terminal blocks with wires going every which way in an orderly fashion. They located a couple of wires, and Andy unwrapped the wires from the handset while Diane removed the screws holding the wires to the terminal block. Then Andy clipped one wire from the telephone handset to each of the two wires and handed me the phone.

Diane told me that they were going to drive down toward the train gate where the railroad tracks come into the plant and try to find these wires on the other end. So, what they needed me to do was to talk on the phone so when they find my voice, they will know that they have the right wires. Diane said, “Just say anything.” Then they left the switchgear and I could hear them drive away in the truck.

Well. This was my opportunity to just talk to no one for a while without interruption. How many times do you get to do that in one day? Probably only when you are on the way to work and back again if you aren’t carpooling with anyone. Or you’re sittin’ on your “thinkin’ chair” in a single occupant restroom. So, I just kicked into Ramblin’ Ann mode and let myself go. I believe my monologue went something like this:

“The other day I was walking through a field, and who should I run across, but my old friend Fred. I said, ‘Well, Hi Fred, how is it going?’ and Fred told me that he was doing just fine, but that he had lost his cow and was wondering if I could help him look for it. I told him I couldn’t right now because I was helping some people find a wire at the moment, and if I became distracted, we might not only lose the cow, but we might lose the wires as well, so I better just keep on talking so that my friends on the other end can find the wires they are looking for. After that I went to the store and I picked up three cans of peas. I thought about getting four cans of peas but settled on three and brought them to the checkout counter, and while I was waiting in line I noticed that the little boy in front of me with his mom was looking at me as if he wanted to have one of my cans of peas, so I quickly made it clear to him that I was buying these cans of peas for myself by sliding them further away from him and glaring at him. Luckily the boy wasn’t persistent otherwise I would have broken down and given him a can of peas because he was looking kind of hungry and I was feeling sorry for him, though, I didn’t want him to know how I was feeling, so I put on a grim expression….”

Needless to say… My monologue went on for another 15 minutes. Yes… .15 minutes. I had expected Andy and Diane to have returned earlier, but I didn’t know how hard it was going to be to find the other end of the wires, so I just kept on ramblin’ to the best of my ability. It’s like what it says in the Bible. If we wrote down everything I said, it would have filled many volumes. Being a Disciple of Ramblin’ Ann came in handy that day. For more about Ramblin’ Ann, you can read the following post:

Ed Shiever Trapped in a Confined Space With A Disciple of Ramblin’ Ann

When Andy and Diane returned they said that they had found the wires right away, but that they had sat there for a while just listening to me ramble. They said I was cracking them up. They also mentioned that they thought I was completely crazy. Well. I was glad that they found the wires and that my rambling abilities had come in handy.

Five months after I had joined the electric shop, Andy and I were sent to Oklahoma City to learn about a new kind of electric troubleshooting. It was called “Digital Electronics”. I had just finished my electronics class at the Vo-Tech, and so I was eager to put it into practice. Andy and I went to a two day seminar where we learned to troubleshoot what was basically a PC motherboard of 1984. We used a special tool called a digital probe and learned how the processor worked with the memory chips and the bios. It wasn’t like a motherboard is today. It was simple.

A simple Motherboard like this

A simple Motherboard like this

It was just designed for the class so that we could use the digital probe to follow the different leads from the chips as the electric pulses turned on and off.

We were using digital probes similar to this

We were using digital probes similar to this

At the time I was thinking that this was a waste of time. I had been learning all about troubleshooting electronic circuits from Bill Rivers and Sonny Kendrick. I couldn’t see how this was going to be useful. I didn’t know that within a couple of years, most of our electronic circuits in the precipitator controls were all going to be replaced with digital controls, and this was exactly what I was going to need to know.

So, Andy and I spent two days learning all the basics of how new computers were going to be working. This was the same year that Michael Dell was beginning his new computer company further down I-35 in Austin Texas. Who would have thought that 18 years later I would be working for Dell. But that’s another lifetime away…

Comments from the original post:

Ron Kilman January 19, 2013:

Early in my career at the Seminole Plant I learned when someone paged you on the gray phone, you should always check the earpiece of the phone before you put it on your ear – it might be full of clear silicone calk (or worse). Also, at the end of the day when you reach to pick up your lunch box, you should pick it up gently. Someone could have slipped a full bottle of mercury (like 20 pounds) in it. This prevents you from pulling the handle off your lunch box or hearing it crash to the floor, smashing everything in its path. It’s amazing what Power Plant Men are capable of doing.

  1. Plant Electrician January 19, 2013:

    We used hand lotion in the electric shop for the gray phone trick. I remember Andy catching an unsuspecting operator in the main switchgear more than once.

    1. Ron Kilman January 20, 2013:

      Hand lotion is much nicer than silicone caulk!

Power Plant Lock Out – Tag Out or Just Tag Out

Originally Posted July 12, 2013:

All safe electricians worth their salt know about OSHA regulation 1910.147(c)(3). Only Power Plant electricians have learned more about OSHA regulation 1910.147(a)(1)(ii)(C). Section 147 has to do with locking out and tagging a power source in order to protect the employees working on the circuit. 147(a)(1)(ii) says that Power plants are exempt from section 147. In other words, if you are working in a power plant it is all right to have a less stringent lock-out/tag-out procedure in place than if you didn’t work in a power plant.

One of the first things I learned from Charles Foster, my foreman when I became an electrician was how to remove the “heaters” from a breaker relay in order to protect myself from an “unauthorized” operation of the breaker. That means…. in case someone accidentally turned on the breaker and started up the motor or whatever else I was working on. “Heaters” are what we called the overloads that trip a 480 volt breaker when the circuit uses more power than it is supposed to be using. They are called heaters, because they literally “heat up” in order to trip the breaker.

typical 480 volt overload heaters

typical 480 volt overload heaters

Charles Foster told me the following story about my bucket buddy Diana Lucas (later Diana Brien):

Dee was wiring up a sump pump at the bottom of the coal dumper. The motor had been taken out while the pump had been repaired. Once back in place Dee was sent to wire it back up. The proper clearance had been taken to work on the motor. That is, she had gone to the Shift Supervisor’s office in the Control Room to request a clearance on the motor. Then later she had witnessed the operator opening the 480 volt breaker and place the clearance tag on the breaker.

A typical Clearance Tag.  Our tags had the word

A typical Clearance Tag. Our tags had the word “Clearance” at the top. We called them a “Hold Tag”

The tag is signed by the Shift Supervisor and is only to be removed by an operator sent by the Shift Supervisor. It is placed through a slot in the handle on the breaker that keeps the breaker from closing unless the tag is removed first…. well… that’s the theory anyway.

Dee had just finished hooking the three leads in the junction box together with the cable coming into the box using two wrenches. She reached down into her tool bucket that she was using as her stool to get some rubber tape to begin wrapping the connections. The three bare connections were sticking out in front of her face.

A large vertical pump motor

A large vertical pump motor

The Junction box is the box on the right side of this motor. At this point the cover would be off and the wires would be sticking straight out. As she reached into the bucket, the motor turned on and began running.

Startled, Dee stopped what she was doing. I suppose she also pinched herself to make sure she wasn’t dreaming. Then I suppose she checked her diaper to make sure it was still dry. Then I suppose she may have said a few choice words whether anyone was around to hear them or not. Maybe not all in that order.

For those of you who don’t realize what this meant. It meant that if the motor had started running about 5 to 10 seconds later, someone, some time later may have made their way down to the west end of the dumper sump only to find one charred Diana Lucas (who never would have later become Diana Brien). They might not have recognized her at first. I can assure you. It wouldn’t have been pretty.

You see… someone had removed the Hold Tag and purposely started up the motor totally disregarding the clearance. I won’t mention any names, but his initials were Jerry Osborn.

So, after Charles told me this story, he showed me what to do to prevent this from ever happening to me.

Charles and I went to the Shift Supervisor’s office to take a clearance on a motor. Then we followed the operator to the breaker and watched him open the breaker and put the tag on the handle. Then we signed something and the operator left.

After the operator left, Charles told me to open the breaker and slip the hold tag through the slot in the door so that the door could open without removing the tag. I followed his directions.

Once the door was open, he told me to remove the three heaters on the bottom of the relay and hide them at the bottom of the breaker box.

480 volt relay with the 3 heaters at the bottom.  That white square is an overload reset button. in front of the first heater

480 volt relay with the 3 heaters at the bottom. That white square is an overload reset button. in front of the first heater

You see… with the heaters removed, even if someone were to close the breaker and try to start the motor, the electricity would never leave the breaker box because I had just created an open circuit between the relay and the wires going to the motor.

Well… If you don’t learn from history you are bound to repeat it. No. I’m not going to change subjects and start talking about making it illegal to own guns.

Anyway, there is always a chance for something to go wrong. The Peter Principle demands it. So, at one point, someone forgot to replace the heaters in the relay before returning their clearance. When the motor was tested for rotation, it didn’t work. At that point the electrician knew that they had forgotten to re-insert the heaters. So, they had to return to the breaker to install the heaters before the motor would run.

This didn’t set well with the Shift Supervisor, who has supreme power at the power plant…. well… besides the janitor who had total control over the toilet paper supply.

Technically we were not going around the hold tag by removing the heaters because they were downstream from the breaker handle which cut off the power to the relay. The Shift Supervisor on the other hand believed that the hold tag included everything in the breaker box, including the relay and heaters (which really was stretching it).

An argument ensued that pitted the shift supervisors and the supervisor of operations, Ted Holdges with the electricians. Ted argued that we should not be removing the heaters to keep ourselves from becoming electrocuted accidentally when someone inadvertently removes a hold tag and turns the breaker on and starts up a motor. Electricians on the other hand argued that if we were going to be exposed to the possibility of being electrocuted, we would rather not work on any circuit. Without being completely assured that we would not occasionally be blown to pieces when someone or something accidentally caused the circuit to become hot, we concluded it wasn’t worth it.

So, a compromise was reached. We could remove the heaters, but they had to be put in a plastic bag and attached to the hold tag on the outside of the breaker. That way, when the clearance was returned, not only were the heaters readily available the operator would know to contact the electrician to re-install the heaters. The electricians didn’t really like this alternative, but we agreed. We were assured that there wasn’t any way that a breaker was going to be turned on and operated with the heaters in them when someone was actually working on a circuit.

Fast forward three years. 1992.

Bill Ennis and Ted Riddle were working on replacing a large electric junction box on the stack out tower. The Stack Out Tower is the tower that pours the coal out on the coal pile. Halfway up this tower there is a large junction box where most of the electric cables passed through going to the top of the tower. Bill Ennis had taken a clearance on a number of motor and control breakers.

Bill returned from lunch one day to work on the junction box, removing the old cables. Putting new lugs on them and placing them in the new junction box. As he began working, he decided to take out his multimeter and check the wires he was about to work on….

We had a couple of these Simpson Analog Multimeters in the shop

We had a couple of these Simpson Analog Multimeters in the shop

Bill was surprised to find that one set of cables were hot. They had 480 volts on them. Everything in this box should have been dead. I suppose he pinched himself to make sure he wasn’t dreaming. Then I suppose he checked his diaper to make sure it was still dry. Then I suppose he said a few choice words that Ted may have heard if he was standing close by.

What had happened was that there had been two clearances on this one particular motor. One of the electricians had returned his clearance, and had installed the heaters that were in the plastic bag on the front of the breaker and the motor had been tested for rotation and put back in service. The operator had taken both clearances off of the breaker by mistake.

Ok. It was time for another meeting. Something had gone wrong. If it had not been for the guardian angels of both Diana Brien and Bill Ennis, at this point we would have had at least two dead electricians, and believe me…. I know that when an operator had later climbed the stack out tower to check the equipment, if he had run across the body of Bill Ennis… it definitely wouldn’t have been pretty (even on a good day).

I attended this meeting with Ted Holdges as did most of the electricians. I began by telling Ted that when we had met three years earlier I was newly married and wouldn’t have minded so much if I was killed by being electrocuted because I was young and only had a wife who knew how to take care of herself. But now it was different. I had a little girl at home and I need to be around to help her grow up.

Ted looked surprised by my remark. I had just told him the way I felt about this whole situation. The argument that we were making was that we should be able to place locks on the breakers just like OSHA demanded from the other industries. We had demonstrated that we didn’t have a system that would protect us from human error. We needed something that definitely kept us safe.

We told Ted that even if we had locks, and for some reason the breaker just had to be closed and the electrician had forgotten to remove his lock, the shift supervisor could keep a master key in his office to remove the lock. He finally agreed. His problem was a loss of control. The thought was that the Shift Supervisor had ultimate power.

If you don’t learn from history you are bound to repeat it. No. I’m not going to change subjects and talk about socialized healthcare and how it destroys all concepts of quality and privacy.

So, as electricians, we weren’t really happy with this situation. We had a secret weapon against human error. Sure we would place a lock on the breaker. But after the operator would leave, before we placed our lock on the breaker, we might just open up the breaker box and remove the entire face off of the relay. It was similar to removing the heaters only it was bigger. It completely opened the circuit no matter what.

I hadn’t really planned on talking about this next story for a couple more years, but I’ll tell it now because it fits with this story.

In the month of May, 2001. I had already given my notice to leave the plant to work for Dell as a software developer. I was asked to work on a job with my old bucket buddy Diana Brien.

Diana-Brien

My Bucket Buddy Diana Brien

The problem was that there was a grounded three phase circuit up on the Surge Bin tower. It had been tracked down to the dust collectors located below the surge bin conveyor floor.

Dee and I walked up to the Gravimetric feeder deck to look at the breaker to make sure it was turned off. It had a Danger tag on it that had been placed by the Shift Supervisor.

A typical Danger tag used at the plant

A typical Danger tag used at the plant

The breaker was open and the message on the tag said “Do not close this breaker. The circuit is grounded”.

Ok. We walked up to the surge bin tower through the counter weight room for belts 18 and 19. We opened up the big junction box that fed the power to the two large dust collector motors on the landing behind us. After taking the cover off of the box, I took out my multimeter and checked the circuit.

Like this.  Ok.  So the multimeters had become more sophisticated over the years.

Like this. Ok. So the multimeters had become more sophisticated over the years.

The big copper bus was dead (that means, there was no electricity present).

So, Dee and I worked on locating the grounded circuit. I had just removed the cover to the junction box on one of the motors while Dee was removing some wires from the control panel when Larry Tapp arrived on the landing through the same route we had taken from the gravimetric feeder deck.

Larry asked us what we were doing. We told him we were tracking down the ground on the Dust Collectors. Larry looked surprised.

You see… Larry explained that he had just come from the Gravimetric feeder deck where he had just closed the breaker for the dust collectors. This particular breaker didn’t have a relay, as it was controlled by the control panel where Dee had been working.

So, I rechecked the copper bus with my multimeter and it was hot. 480 volts hot.

I had just been looking through my tool bucket for two wrenches to remove a piece of the bus work just to make sure the ground wasn’t in the box itself when Larry had arrived. In other words, if Larry had arrived 5 to 10 seconds later, he would have probably arrived to find Dee looking down at my body, stunned that I had just been electrocuted by a circuit that we had just tested and found dead.

If you don’t learn by history you are bound to repeat it.

You see… there is a difference between a Hold Tag and a Danger Tag. A hold tag is placed on a breaker after someone has requested a clearance by signing a form in the Shift Supervisor’s office in the control room. A Danger tag can be placed and removed at anytime by the person that placed the tag on the breaker.

So, I personally wrote this up as a “near” accident. We could have wiped our brow, pinched ourselves to make sure we weren’t dreaming. We could have checked our diaper to make sure it was still dry and then Dee could have said a few choice words that Larry Tapp would have agreed with (I have always had a mental block against expressing myself in that manner…. I found other ways). And we could have left this incident as a secret between Larry, Dee and I.

I thought it was a good time to remind the electricians throughout power production to follow the clearance procedures when working on high voltage circuits. Sure. Dee, Bill Ennis and I have powerful guardian Angels looking out for us…. but gee… I think we should be expected to look out for ourselves. So, I wrote up this incident to warn the rest of the team….. If we don’t learn from history, we are bound to repeat it.

I met with my roomie Steven Trammell, a month and a half later in Perkins Restaurant in Stillwater, Oklahoma to discuss his performance plan. I was a 360 Degree Assessment Counselor and my favorite roommate from 17 years earlier had chosen me to review his performance appraisal. During this meeting I asked Steven, who had driven from Harrah, Oklahoma from another power plant to meet with me, if he had read the near accident report about the dust collector at our plant.

My roomie told me that he had, and that he thought it seemed to unduly blame the electrician. I told him I was the electrician and that I wrote the report. After 18 years of being an electrician, I had become so relaxed in my job that I had become dangerous to myself and others. So, after I did a cause-effect analysis of the near accident, most of the cause had come from my own belief that I could circumvent clearance procedures and save time and still believe that I was being safe.

On my drive back to the plant after the meeting with my favorite roomie of all time, I had time to think about this…. I was going to be leaving the power plant in a little over a month to work for Dell as a programmer. I knew this when I had been negligent with the Danger tag. I could have caused the death of both Dee and I. I will sure be glad to be in Texas. — Only.. I will miss my friends most of all.

I leave the Power Plant with this one thought…. If you don’t learn from history, you are bound to repeat it. I mean it… This time I really do.

Comments from the previous posts:

  1. Ron Kilman July 13, 2013:

    This is a great story. I thank God for His guardian angels and I thank you for taking responsibility. It was always difficult investigating accidents because people are a little reluctant to share their mistakes with the world. But a wise man knows it’s better to have a bruised ego than a fried friend.

Larry McCurry July 13, 2013:

Kevin,

  1. As an old time operator and having follows in my fathers footsteps as a Shift Supervisor, The answer to all of these problems to add steps to the clearance procedure to make sure the heaters were removed and then replaced. The second was definitely an operator error, and I agree with you about it, The Shift supervisors did argue for it however the hubris of certain power hungry people managed to intimidate and control the situation. You do not ever work on equipment without your own clearance or a plan that includes the SS, as you mentioned He is the operating authority, or was until a person by the initials of Jim Arnold rewrote the procedures and made himself the Authority.

Jack Curtis August 12, 2013:
Good Story Indeed.
There is, I’m sure, a gene in human DNA labeled: “Murphy” that assures that anything that can happen, does. And in total ignorance, I’ll bet that some constant percentage of plant electricians were fried. I suppose the onset of computer controls has reduced that but that it still happens at a lower rate. The way you and your friends sort of automatically reached for your multimeters is a clue! These are shocking, highly-charged stories…
NEO July 16, 2014
Yep, and even c.3 rules fail on occasion. Most places I’ve worked the other key was at superintendent level, and required a veritable mountain of paperwork to acquire. G-d help anybody who lost his key! 🙂
Traditionally on distribution lines, we left our shotgun hanging on the (grounded) stinger to notify everyone since there is no (effective) way to LOTO a power line. Imagine my vocabulary one fine day up in Montana when I drove up to reenergize a tap to find it energized and a crew I had never seen putting my shotgun in their truck. That day, I got a profuse apology (and assurances) from the line superintendent in person, 50 miles from the office. I never worked dead again without three point grounding, and I require my people to use it as well, I don’t like funerals. Hot line is actually safer for most operations.
Dan Antion July 17, 2014
I have great respect for electricity, power tools and the threat of human error. That was a close call. Glad you had the presence of mind and experience to think to check again.

Relay Tests and Radio Quizzes with Ben Davis

Originally posted January 24, 2014:

Reorganizations naturally shuffle things around.  People are generally resistant to change and don’t like to find that their routine has been changed without having their input on how to make things better.  When the coal-fired power plant in North Central Oklahoma went through a downsizing and reorganization in the latter part of 1987, my job changed slightly.  Personally, I was grateful for the changes.

Before the reorganization, I had inherited both the precipitators (the large boxes at a power plant that take the ash out of the exhaust from the boiler).  This meant that every overhaul, I knew what I was doing.  I was working on and in the precipitator.  This was generally a dirty and thankless job.

The plant has a similar electrostatic precipitator, only it is twice as long

The plant has a similar electrostatic precipitator, only it is twice as long

After the reorganization, however, Terry Blevins was assigned to work on the Unit 2 precipitator, while I worked on Unit 1.  I will go into this in more detail later, but for this post, I’ll just point out that this meant that when Unit 2 was on an overhaul (that means the unit is taken offline for one to three months in order to fix and repair things that can only be done while it is offline) I wasn’t automatically assigned to the precipitator.  So, I could work on other things.

Before the reorganization, Sonny Kendrick had the title “Electric Specialist”.  After the reorganization we no longer had a specialist.  I’m not sure exactly why.  I know that at Muskogee, they still had a specialist in the electric shop.  — I will talk about him next year (the specialist at Muskogee).  Anyway, I know that Sonny, at the time, was not too happy about his change in job title.  I don’t blame him.  I would be too.

One of the things that the Electric Specialist did during overhauls was test tripping relays.  Now that we no longer had a specialist, that was left up to whomever…. The first electricians, besides Sonny, that were assigned to relay testing was Ben Davis and myself.  I had started doing it on my own and after about a week, Ben Davis was assigned to help me out.

Ben Davis

Ben Davis

We were on a major overhaul on Unit 2 and it had been decided that we were not only going to test the regular super-high voltage breaker relays, we were also going to test all the 480 volt switchgear relays for Unit 2, as well as the intake and coalyard switchgears.  I don’t remember if we made it to the river pump switchgear, but I wouldn’t be surprised.  Once we started, there was no stopping us.

When I first was told to test the relays, Bill Bennett (our A foreman) told me to have Sonny tell me how to do them.  So, I walked into the lab and told Sonny that Bill had told me to ask him to help me learn how to test the protective relays on the switchgear.  Sonny, not looking too happy, grabbed a small stack of manuals, walked out into the main switchgear with me, and said, “Here is the relay test set.  Here are the manuals that tell you how to hook up the test set and test them.”  He turned and walked away…. I was sort of hoping for a more intimate lesson…

I knew the reason Sonny was so upset.  Later I learned why he would be as upset as he was to not be able to test the protective relays.  It was because when you test, clean and adjust protective relays you have an immediate rush of satisfaction that you have just done something very important.  Let me just say quickly (because in another post I will expound upon this), a protective relay is what keeps motors from blowing up.  It is what prevents blackouts from happening across the nation.  Without properly calibrated protective relays, a power company is just asking for a disaster (or… well….. their insurance company is, because they are the ones that usually end up paying for the damage — which I will also talk about in a later post).

I thought the relay test set that Sonny showed me was the neatest thing I had seen so far in the electric shop.  There were two boxes that hooked together with an umbilical cord.  They had dials, switches, connectors, meters and a digital readout down to the millisecond.  That is, you can read the time to trip a relay down to the one thousandth of a second.

AVO Multi-Amp SR-76 Relay Test Set

AVO Multi-Amp SR-76 Relay Test Set

I only wish that I had a bigger picture of this relay test set so that you could admire it as much as I did.  Even today it gives me goosebumps!  Ok.  I can imagine those relay technicians that read this blog are looking at this and thinking…. “What kind of piece of junk is this?”  Hey (as Mark Fielder used to say), this was my “baby” (only he was referring to the precipitator).

So, back to the story at hand…

Even though I was having a heck of a fun time trying to figure out how to perform these relay tests by reading these manuals about the different kinds of relays, I was glad when Ben Davis was assigned to work with me.  I don’t know if he had worked on relays before, but he seemed to know just what to do to hook up the test set and make things easier.

A panel of Protective Relays

A panel of Protective Relays

The best suggestion that Ben had right off the bat was that we should be listening to the radio while we were working.  This might have been a preventative measure after the first couple of days to prevent the same situation from occurring that happened to Ed Shiever when he and I were trapped inside a confined space for a couple of weeks (See the post:  “Ed Shiever Trapped in a Confined Space with a disciple of Ramblin’ Ann“).  Either way, it was a great idea.

You wouldn’t think that inside a switchgear 20 miles from the nearest town with a radio station, that we would have any reception on a little transistor radio, but we were able to manage.  It seemed that we had to be a little creative at times with the antenna in certain locations, but, like I said.  We managed.

My perception of Ben Davis up to this point was that he was a “Good-ol’ boy”.   That is, a country music type Oklahoman that had grown up in Shidler, Oklahoma where the major attraction in the town was the High School.  To my surprise, I quickly found out that he was a connoisseur of Rock and Roll.

It wasn’t until I was in college before I realized that the easy listening station I had been listening to on our family radio at home while I was growing up was playing rock and roll songs using an orchestra with violins and clarinets instead of electric guitars.  I learned from my dorm mates all about groups like Simon and Garfunkel, The Beatles (yeah… can you believe it?  I mean.  I knew “Hey Jude”, “Let it Be” and a few others, but most of the Beatles I thought were instrumentals normally played on violins with a man waving a wand) and many others.  When I found out about “Rock and Roll”, I had to go out and buy dozens of 8-track tapes, as fast as I could find them.

A stack of 8 Track Tapes

A stack of 8 Track Tapes

So, here was Ben Davis.  Even better than the “Good Ol’ Boy” that I already thought he was.  And he loved classical rock and roll.  I can only say that the next month and a half while we tested relays all over the plant, were one of the best times I have ever spent in my life!  He knew all the 60’s and 70’s rock and roll bands.

As each song would come on the radio, we would guess (well, I was guessing most of the time…. most of the time Ben already knew), what the name of the song was and the name of the band.  So, not only were we doing one of the most satisfying jobs at a power plant, but I was also have a lot of fun with Ben listening to the radio!  Who would have thought it?  No wonder Sonny was upset he wasn’t testing relays this overhaul.

I could go on about all the different bands and their backgrounds that I learned from Ben during that overhaul, but (unlike me), you probably already know all that stuff.  It never ceases to amaze me how many holes I have in my education until one is staring at me in the face.

This reminds me of a side story, and I apologize if I have told this before…. I don’t think I have….

After the Reorganization, and after I moved to Stillwater from Ponca City, Scott Hubbard (and Toby O’Brien) and I began carpooling.  One morning as we were listening to NPR, Scott Hubbard mentioned something about a “cur”.  I asked him, “What’s a cur?”  Well, he had the exact same reaction when 11 years earlier I had asked my friends in college at Oklahoma University, Tim Flowers and Kirby Davis, “What’s an orgasm?”  —  See how little holes in your education can make a big impact?

Just so you don’t get caught in the same predicament…  A “Cur” is a mongrel dog.  Scott Hubbard couldn’t believe that someone that read the dictionary for fun wouldn’t know what a “cur” was.  What the heck?  I didn’t grow up in Oklahoma!  — end of side story… which really isn’t a side story, since it was about a Power Plant Man — Scott Hubbard.  He probably knew what a “cur” was before he could walk.  — I know I haven’t told that story before!  I would have remembered that.

I’m not going to go on about all the fun that I had with Ben Davis testing protective relays.  I enjoy my memories, but I’m sure you don’t want to hear all about how much I looked up to this Power Plant Hero.  The only thing I will add is that the time I spent with Ben during that overhaul has been etched into my memory as one of the most enjoyable times of my life.  So, I’ll go onto the next step in our Protective Relay story….

A few years later, in 1993, Sonny Kendrick and Ben Davis and I were sent to “Advanced Protective Relay Maintenance” training in Dallas, Texas.  I remember this time so well, I remember the address where we were went.  It was at 4271 Bronze Way, Dallas, Texas.  It was hosted by the same company that made that wonderful test set I pictured above.  The AVO Multi-Amp Corporation.

I brought my wife Kelly and my three year old daughter Elizabeth with me.  They stayed at the hotel during the day and played in the swimming pool, while I went to class.

The classes lasted four days, Monday through Thursday.  That was where I learned that even though I thought our relay test set was the coolest piece of equipment in the electric shop, it turned out to be archaic by “Protective Relay Maintenance” standards.  Not that it didn’t do the job….   So, in order to train us properly, they let us use our own old test set during the training so that we could see how to properly test really advanced relays such as Distant Relays, Syncro-verifier relays, Negative Sequence Relays,directional distance relays and Pilot Wire relays.  — These are relays that are found in a large substation that trips high voltage lines that run long distances across the country.  — I can tell you’re jealous.  — Well.. I imagine it anyway.  Knowing what I know now.

This is the book we used in class

So, why drag you all the way to Dallas for this story?  There’s a reason.

time for a second side story:

You see. Tim Flowers, whom I mentioned above, knew not too long after he met me that I have the knack of running into people that I know (or should have known in this case), would love this story.  You see, I met Tim and Kirby at Oklahoma University and they drove with me to Columbia Missouri in 1979 (along with my brother Greg) when I went to register for classes at Missouri University when I decided to go back to school in my home town.

When we arrived in the town, we were hungry after driving for 8 hours straight from Stillwater, Oklahoma to Columbia, Missouri… so we stopped at Godfather’s Pizza.  As we walked in, there was a girl and a guy standing at the counter ordering a pizza.  The pretty girl (Pamela Ramsey) with long red hair turned and saw me.  She immediately came toward me saying “Kevin Breazile!!!!  You owe Me!!!  Slightly shocked and pleased, I said, “What for?” She reminded me that I never gave her the pictures that were taken during the Senior Prom.  You see.  I had taken her to the Senior Prom.

Later I explained that this happens to me a lot.  I meet people that I know in the oddest places (even though this wasn’t so odd, since I had grown up in Columbia). It was just that this was the first person we had seen since we entered town.  From that point on, Tim (who later worked as a summer help at the power plant) expected that everywhere we went we would run into someone I knew….

End of the second side story.  I’m sorry that this is making the post a little longer than usual.  I know you have to get back to work….

So, back to the relay training course in 1993 that Ben Davis, Sonny Kendrick and I were taking in Dallas…. On Wednesday night during the training there was a dinner held in a small banquet room in the hotel.  Well… of course I had to take my wife and my daughter.   So here we were sitting around this table at dinner with the rest of the class of about 10 other non-Sooner Plant employees….

I decided to talk to the guy next to me.  He said something back and my wife Kelly asked him, “Where in New Jersey are you from?”  She had picked up on a New Jersey accent.  He said, Well..  I work in the east for a company called Ebasco, but I’m really from the Midwest.  (oh.  That was my territory).  So I asked a follow-up question.  “Where in the Midwest are you from?”  He said, “From Missouri.”  — Oh.  I thought.   This is interesting. So was I.

I asked a follow-up question.  “Where in Missouri are you from?”  He answered…. “Columbia, Missouri.”  (What?   Where I had grown up?)….  So, I asked a second follow-up Question…. “What High School did you go to?”  With a curious look the man answered….. “Rockbridge High School…”   (Man!!!  the same one as me!!!)…. The third follow-up question….. “What year did you graduate?”  Now, looking really suspicious… he said, “1978”.   Trying to contain my excitement… I replied….. “Oh… so, you graduated from Rockbridge High School the same year I did….”

What are the odds?  There were 254 students in our graduating class.  This guy who currently lived somewhere in the east is sitting next to me at a dinner of about 10 people attending Advanced Protective Relay Training in Dallas, Texas where neither of us are from, and we both graduated from the same school back in Columbia, Missouri 15 years earlier!  His name is Randy Loesing.  He was working for a company called Ebasco at the time.  He said, “I thought I recognized you!  I just wasn’t sure.”  I didn’t recognize him at all until I went back home and looked in my yearbook.

It turned out that he kept in touch with two of my oldest friends from the second grade, Mark Schlemper and Brent Stewart.   So we talked about them.  What an incredible coincidence.   Like I may have mentioned before.   It happens to me all the time.  It turns out that an old friend of mine from the 3rd grade in Columbia, Missouri that I used to go to his house when we were stamp collectors and had a stamp collecting club, lives 5 miles south of me today in Round Rock Texas (He’s in Pflugerville).

Russell Somers lives in the  same direction and just about the same number of miles as when we were kids.  Not only that, but he worked at Dell while I was working at Dell (though I didn’t know it at the time).  He has an older daughter and a younger son, just like me only younger.  The same is true for another 3rd grade friend that I  graduated from Rockbridge Highschool and the University of Missouri with, Caryn Lile (now Caryn Iber) who lives in Wisconsin.  She has a daughter and a son the same age as my kids.  She was living in Tulsa when I was living in Stillwater, Oklahoma.  — Like I said… happens to me all the time.

Tim Flowers realized this odd phenomenon  in college.  I had told him earlier that my father told me that if I was every stranded somewhere that I could look up the local Veterinarian and tell him that I was the son of Dr. James Edward Breazile, and they would help me.  So, when we were hiking in the mountains in Colorado and we met a man walking along a trail in the middle of nowhere above Estes Park near the Great Divide, when I told him who I was, he gave us a curious look…. then divulged his most intimate secrets of his life and where he had stashed his most values possessions, Tim told me later.  “I really thought he was going to know who you were when he gave us that funny look.”  I replied.  “I think he did..”

I again apologize for the length of this post.  It is rare that I ramble on this long.  I can thank Ramblin’ Ann for the ability to Ramble so well.  I can thank Ben Davis for recognizing a rambling situation and replacing it with a rock and roll learning opportunity.  As I said earlier.   One of the most enjoyable times I have spent in my entire life is the time I spent with Ben Davis testing Protective Relays!  Bless you Ben and I pray for you, your wife, your son and your daughter on the way to work each morning.

Today when I hear any of the hundreds of roll and roll songs come on the radio that we listened to that month and a half, I can see us testing the relays, looking off into space saying, “Rolling Stones?”  “No.   Steve Miller Band?”  Really?  I thought Browneyed Girl was sung by the Rolling Stone!  It turned out that the version that we listened to was from the creator of the song, Van Morrison. Who would have thought that he would sound so much like Mick Jagger.  I can see Ben saying… I see what you mean…  it kind of sounds like Mick Jagger.

As an add on to this story…

I now work at General Motors in Austin Texas.  My best friend in High School was a guy named Jesse Cheng (I have mentioned him in other posts, especially in reference to the phrase “Jesse!  Come get your Chili!).  He was two years older than me, and throughout the years we would lose track of each other and then reconnect.  He went to Yale to become an Engineer, then to the University of Missouri to become a Medical Doctor, then to Harvard to earn a Masters in Public Health and Epidemiology.

It turns out that we both now work at General Motors where he works in Arlington Texas as a Medical Director and I work in IT in Austin.  We can IM (Instant Message) each other whenever we want, and we talk now at least once every week.

Pain in the Neck Muskogee Power Plant Relay Testing

Don’t let the title fool you.  I love testing Power Plant Protective Relays.  There is a sense of satisfaction when you have successfully cleaned, calibrated and tested a relay that is going to protect the equipment you have to work on every day.  With that said, I was hit with such an unbelievable situation when testing Muskogee Relays in 1995 that I was left with a serious pain in the neck.

On August 14, 2003 the electric power in the Northeast United States and Canada went out.  The Blackout lasted long enough to be a major annoyance for those in the that region of the United States.

 

Map of the power blackout in 2003

Map of the power blackout in 2003

When I heard about how the blackout had moved across the region, I immediately knew what had happened.  I was quickly reminded of the following story.  I told my wife Kelly, “I know exactly why such a large area lost power!  They hadn’t done proper preventative maintenance on the Protective Relays in the substations!  Just like….”  Well…. I’ll tell you that part now:

I have mentioned in a couple of earlier posts that something always seemed a little “off” at the Muskogee Power Plant.  I had decided early on that while working there I would stick to drinking sodas instead of water.  See the post:  “Something’s In the Water at the Muskogee Power Plant“.  Even with that knowledge, I was still shocked at what I found while testing relays at the plant.

This story really begins one Sunday at Muskogee when one of the Auxiliary Operators was making his rounds inspecting equipment.  He was driving his truck around the south edge of the Unit 6 parking lot on the service road.  He glanced over at a pump next to the road, and at first, he thought he was just seeing things.  After stopping the truck and backing up for a second glance, he was sure he wasn’t dreaming.  It’s just that what he was seeing seemed so strange, he wasn’t sure what was happening.

The operator could see what appeared to be silver paint chips popping off of the large pump motor in all directions.  After closer examination, he figured out that the motor was burning up.  It was still running, but it had become so hot that the paint was literally burning off of the motor.

Horizontal pump

Like this Horizontal pump only much bigger and painted silver

A motor like this would get hot if the bearings shell out.  Before the motor is destroyed, the protective relays on the breaker in the 4,000 Volt switchgear shuts the motor off.  In this case, the relay hadn’t tripped the motor, so, it had become extremely hot and could have eventually exploded if left running.  The operator shut the motor down and wrote a work order for the electricians.

Doyle Fullen was the foreman in the electric shop that received the work order.  When he looked into what had happened, he realized that the protective relay had not been inspected for a couple of years for this motor.

I couldn’t find a picture of Doyle.  In his youth he reminds me of a very smart Daryl in Walking Dead:

Norman Reedus from Walking Dead

Norman Reedus from Walking Dead

In fact, since before the downsizing in 1994, none of the Protective Relays at the plant had been inspected.  The person that had been inspecting the relays for many years had moved to another job or retired in 1994.  This was just a warning shot across the bow that could have had major consequences.

No one at Muskogee had been trained to test Protective Relays since the downsizing, so they reached out to our plant in North Central Oklahoma for help.  That was when I was told that I was going to be going to Muskogee during the next overhaul (outage).  I had been formally trained to inspect, clean, calibrate and test Protective Relays with two of my Power Plant Heroes, Ben Davis and Sonny Kendrick years earlier.  See the post:  “Relay Tests and Radio Quizzes with Ben Davis“.

My Protective Relay Maintenance course book

My Protective Relay Maintenance course book

Without going into too much detail about the actual tests we performed as I don’t want to make this a long rambling post (like… well…. like most of my posts…..I can already tell this is going to be a long one), I will just say that I took our antiquated relay tester down to Muskogee to inspect their relays and teach another electrician Charles Lay, how to perform those tests in the future.  Muskogee had a similar Relay Test Set.  These were really outdated, but they did everything we needed, and it helped you understand exactly what was going on when you don’t have a newfangled Relay Test Set.

AVO Multi-Amp SR-76 Relay Test Set

AVO Multi-Amp SR-76 Relay Test Set

You need to periodically test both mechanical and electronic protective relays.  In the electronic relays the components change their properties slightly over time, changing the time it takes to trip a breaker under a given circumstance (we’re talking about milliseconds).  In the mechanical relays (which I have always found to be more reliable), they sit inside a black box all the time, heating up and cooling as the equipment is used.  Over time, the varnish on the copper coils evaporates and settles on all the components.  This becomes sticky so that the relay won’t operate at the point where it should.

A panel of Protective Relays

A panel of Protective Relays

In the picture above, the black boxes on the top, middle and right are mechanical relays.  This means that something actually has to turn or pick up in order to trip the equipment.  The electronic relays may have a couple of small relays, but for the most part, they are made up of transistors, resistors, capacitors and diodes.

So, with all that said, let me start the real story…. gee…. It’s about time…

So, here I am sitting in the electric shop lab just off of the Unit 6 T-G floor.  We set up all the equipment and had taken a couple of OverCurrent relays out of some high voltage breakers in the switchgear.  I told Charles that before you actually start testing the relays, you need to have the test documents from the previous test and we also needed the instruction manuals for each of the relays because the manuals will have the diagrams that you use to determine the exact time that the relays should trip for each of the tests.  So, we went up to the print room to find the old tests and manuals.  Since they weren’t well organized, we just grabbed the entire folder where all the relays tests were kept since Unit 6 had been in operation.

When we began testing the relays at first I thought that the relay test set wasn’t working correctly.  Here I was trying to impress my new friend, Charles Lay, a 63 year old highly religious fundamental Christian that I knew what I was doing, and I couldn’t even make a relay trip.  I was trying to find the “As Found” tripping level.  That is, before you clean up the relay.  Just like you found it.  Only, it wouldn’t trip.

It turned out that the relay was stuck from the varnish as I explained above.  It appeared as if the relay hadn’t been tested or even operated for years.  The paperwork showed that it had been tested three years earlier.  Protective Relays should be tested at least every two years, but I wouldn’t have thought that the relay would be in such a bad condition in just three years.  It had been sitting in a sealed container to keep out dust.  But it was what it was.

I told Charles that in order to find the “As Found” point where the relay would trip, we would need to crank up the test set as high as needed to find when it actually did trip.  It turned out that the relay which should have instantaneously tripped somewhere around 150 amps wouldn’t have tripped until the motor was pulling over 4,000 amps.  I could tell right away why the Auxiliary Operator found that motor burning up without tripping.  The protective relays were stuck.

As it turned out… almost all of the 125 or so relays were in the same condition.  We cleaned them all up and made them operational.

There is an overcurrent relay for the main bus on each section of a main switchgear.

A picture of a clean switchgear. Picture 6 rows of switchgear like this

A picture of a clean switchgear. Picture 6 rows of switchgear like this

When I tested the “As Found” instantaneous trip for the main bus relay, I found that it was so high that the Unit 6 Main Turbine Generator would have melted down before the protective relay would have tripped the power to that one section of switchgear.  The entire electric bus would have been nothing but molten metal by that time.

As I tested each of these relays, I kept shaking my head in disbelief.  But that wasn’t the worst of it.  The mystery as to why these relays were all glued shut by varnish was finally solved, and that reason was even more unbelievable.

Here is what I found…..  The first thing you do when you are going to test a relay is that you fill out a form that includes all the relay information, such as, what it is for, what are the settings on the relay, and what are the levels of tests that you are going to perform on it.  You also include a range of milliseconds that are acceptable for the relay for each of the tests.  Normally, you just copy what was used in the previous test, because you need to include the time it took for the Previous “As Left” test on your form.  That is why we needed the forms from the previous test.

So, I had copied the information from the previous test form and began testing the relay (one of the first overcurrent relays we tested)…  Again… I was a 34 year old teacher trying to impress my 63 year old student.  So, I was showing him how you mechanically adjust the relay in order for it to trip within the acceptable range.  No matter how hard I tried, I couldn’t adjust the relay so that it would even be close to the desired range for the longer time trip times…. like the 2 second to 25 second range.  It wasn’t even close to the range that was on the form from the last test.

The form from the last test showed that the relay was in the right range for all the levels of test.  When I tested it, like I said, it wasn’t even close.  So, I went to the diagram in the instruction manual for this type of relay.  The diagram looks similar to this one used for thermal overloads:

Thermal Overload Tripping curves

Thermal Overload Tripping curves

See all those red lines?  Well, when you setup a relay, you have a dial where you set the range depending on the needs for the type of motor you are trying to trip. Each red line represents each setting on the dial.  Most of the relays were set on the same number, so we would be using the same red line on the diagram to figure out at different currents how long it should take for a relay to trip….

Here is the clincher….The time range that was written on the previous form wasn’t for the correct relay setting.  The person that tested the relay had accidentally looked at the wrong red line.  — That in itself is understandable, since it could be easy to get on the wrong line… The only thing is that as soon as you test the relay, you would know that something is wrong, because the relay wouldn’t trip in that range, just like I had found.

I double and triple checked everything to make sure we were looking at the same thing.  The previous form indicated the same settings on the relay as now, yet, the time ranges were for a different line! — Ok.  I know.  I have bored you to tears with all this stuff about time curves and overcurrent trips… so I will just tell you what this means…

This meant that when the person completed the forms the last time, they didn’t test the relays at all.  They just filled out the paperwork.  They put in random values that were in the acceptable range and sat around in the air conditioned lab during the entire overhaul smoking his pipe. — Actually, I don’t remember if he smoked a pipe or not.  He was the Electrical Specialist for the plant.  I remembered seeing him sitting in the lab with a relay hooked up to the test set throughout the entire overhaul, but I realized finally that he never tested the relays.  He didn’t even go so far as try to operate them.

I went back through the records to when the plant was first “checked out”.  Doyle Fullen had done the check out on the relays and the test after that.  Doyle had written the correct values from the manual on his forms.  I could see where he had actually performed the tests on the relays and was getting the same values I was finding when I tested the relays, so I was certain that I wasn’t overlooking anything.

As I tested each of the relays, I kept shaking my head in disbelief.  It was so unbelievable.  How could someone do such a thing?  Someone could have been killed because a protective relay wasn’t working correctly.  This was serious stuff.

One day while Charles and I were working away on the relays, Jack Coffman, the Superintendent of all the Power Plants came walking through the lab.  He asked us how we were doing.  I swiveled around in my chair to face him and I said, “Pretty good, except for this pain in my neck” as I rubbed the back of my neck.

Jack stopped and asked me what happened.  I told him that I had been shaking my head in disbelief for the last two weeks, and it gave me a pain in the neck.  Of course, I knew this would get his attention, so he asked, “Why?”  I went through all the details of what I had found.

I showed him how since the time that Doyle Fullen last tested the relays more than 10 years earlier, these relays hadn’t been tested at all.  I showed him how the main bus relays were so bad that it would take over 100,000 amps to have tripped the 7100 KV switchgear bus or 710 Megawatts!  More power than the entire generator could generate.  It was only rated at about 550 Megawatts at the most.

Jack stood there looking off into space for a few seconds, and then walked out the door…. I thought I saw him shaking his head as he left.  Maybe he was just looking both ways for safety reasons, but to me, it looked like a shake of disbelief.  I wonder if I had given him the same pain in the neck.

That is really the end of the relay story, but I do want to say a few words about Charles Lay.  He was a hard working electrician that was nearing retirement.  People would come around to hear us discussing religion.  I am Catholic, and he went to a Fundamental Christian Church.  We would debate the differences between our beliefs and just Christian beliefs in general.  We respected each other during our time together, even though he was sure I am going to hell when I die.

People would come in just to hear our discussion for a while as we were cleaning and calibrating the relays.  One day Charles asked me if I could help him figure out how much he was going to receive from his retirement from the electric company.  He had only been working there for three years.  Retirement at that time was determined by your years of service.  So, three years didn’t give him too much.

When I calculated his amount, he was upset.  He said, “Am I going to have to work until I die?”  I said, “Well, there’s always your 401k and Social Security.”  He replied that he can’t live on Social Security.  I said, “Well, there’s your 401k.”  He asked, “What’s that?” (oh.  not a good sign).

I explained that it was a retirement plan where you are able to put money in taxed deferred until you take it out when you retire.  He said, “Oh.  I never put anything in something like that.”  My heart just sank as I looked in his eyes.  He had suddenly realized that he wasn’t going to receive a retirement like those around him who had spent 35 years working in the Power Plant.

When I left the plant after teaching Charles Lay how to test the relays, that was the last time I ever saw him.  I don’t know what became of Charles.  I figure he would be 83 years old today.  I wonder if he finally retired when he reached the 80 points for your age and years of service.  He would have never reached enough years of service to receive a decent amount of retirement from the Electric Company since he didn’t start working there until he was 60 years old.  That is, unless he’s still working there now.

As I said earlier in this post, Charles Lay was a very good worker.  He always struck me as the “Hardworking type”.  I often think about the time we spent together, especially when I hear about a power blackout somewhere.  — A word of caution to Power Companies…. keep your protective relays in proper working condition.  Don’t slack off on the Preventative Maintenance.  — I guess that’s true for all of us… isn’t it?  Don’t slack off on Preventative Maintenance in all aspects of your life.