Tag Archives: Horizontal pump

Crossfunctional Power Plant Dysfunction

The coal-fired Power Plant in North Central Oklahoma had gone from 360 employees in 1987 down to 124 employees on August 1, 1994 after the second downsizing.  Monday morning when we arrived at work, the maintenance department met in the main break room to be told how we were going to survive the loss of 100 employees.  With only 7 electricians left, I kept trying to add up on my fingers how we could possibly keep up with all the work we had to do.

Jasper Christensen stood up and after saying that he understood how we must feel about our present situation, he told us that we will have to each work harder.  I shook my head in disbelief (inside my head only… I didn’t really shake my head, as it was frozen with the same blank stare everyone else was wearing).  I knew we weren’t going to be working harder.  — What does that really mean anyway.  I thought he should have said, “We will each have to work “smarter” because we can’t really work “harder”.  Jasper was a nice person, but he never really was much for words so I gave him a pass on this one.  After all, he never really took a course in motivational speaking.

Jasper Christensen

Jasper Christensen

Interestingly, the three people in charge at the plant, Jasper, Jim Arnold and Bill Green were all 53 years old, and only within 4 months in age from each other.  They all belonged to the “old school way of doing things” (see the post:  “From Pioneers to Power Plant Managers“).  As Jasper continued in his speech I noticed that gone was any talk of working together to achieve our goals.  I immediately felt that we had just rolled back our management to a time before our first downsizing in 1987 when the Evil Plant Manager used to rule the plant with an iron fist.

I felt this way because we were being told how we were going to change everything we do without giving any of our own input.  For instance, we would no longer have a Quality Action Team.  That was disbanded immediately.  We would no longer hold Quality Team meetings (we were also told that the Quality process was not going away, though we couldn’t see how it was going to work).  The Safety Task Force did survive.

We were also told that we would no longer fill out any forms unless they are requested by someone.  It seems that we had over 1,300 forms that were being filled out at the plant and most of them were never being used for anything, so, unless someone requested a form, we wouldn’t just fill them out for the sake of filling them out.  This was actually a good idea.  I know we filled out forms in triplicate each week when we did transformer and substation inspections.  Most of those were never looked at, I’m sure.

It turned out later that we needed only about 400 of the 1300 forms our plant was churning out each month.

We were told we wouldn’t be doing Substation inspections.  That was not our responsibility.  It would be done by the Transmission and Distribution division instead.  I was beginning to see how management was trying to figure out how 7 electricians were going to “work harder”.  The answer at the moment was that we were going to do less.  The purpose of the Substation and Transformer checks each week was to look for problems while they were minor instead of waiting for a catastrophe to happen.

We were told that we were not going to “Gold Plate” our work.  We were going to just do what it took to complete the task without worrying about polishing it up to make it “perfect” (which is what real Power Plant Men do).  Instead we were going to “Farm Fix it”.  I’ll go more into this subject with a separate post.

We were then told that we would no longer have an Electric Shop and an Instrument and Controls shop.  We would from then on all meet in the Mechanical Maintenance shop.  We were not supposed to go to the Electric Shop or the Instrument and Controls shops for breaks because we were all going to be cross-functional.  We are all Maintenance now.  No longer specialized (sort of).

We were going to have four Maintenance teams.  Each one will have mechanics, welders, machinists, electricians and Instrument and controls people.  Each member on each team would learn to do each other’s jobs to a degree.

An electrician will learn how to tack weld.  A mechanic will learn how to run conduit and pull wire.  An instrument and controls person will learn how to use the lathe.  We would each learn enough about each job in order to perform minor tasks in each area without having to call the expert in that skill.

When the meeting was over, we each met with our own foremen.  Alan Kramer was my new foreman.  He used to be a foreman in the Instrument and Controls shop.

Alan Kramer

Alan Kramer

It became apparent that even though Jasper had come across as if everything had already been decided and that this was the way it was going to be, things hadn’t really been ironed out yet.  Actually, this was just a first pass.  The main goal was for us to figure out how to get all the work done that needed to be done.  I was still an electrician and I was still responsible for working on electrical jobs.

One really good part of the new situation was that I was now on the same team as Charles Foster.  We had always been very good friends, but I hadn’t worked on the same team as Charles since my first year as an electrician in 1984, ten years earlier when he was my first foreman in the electric shop (See the post:  “New Home in the Power Plant Electric Shop“).  We were the two electricians on Alan Kramer’s team.

Charles Foster

Charles Foster

Besides the fact that everyone was very bitter over the despicable treatment of our fellow Power Plant Men that were laid off the previous Friday (see the post: “Power Plant Downsizing Disaster and the Left Behinds“), we knew that we had to figure out how to make this new arrangement work.  We knew our upper management was using the old tyrannical style of management, but we also knew that at this point, they needed every one of us.  They couldn’t go around firing us just because we spoke our mind (which was good for me, because, I was still in the process of learning how to keep my mouth shut when that was the most beneficial course of action).

As Dysfunctional as our upper management seemed to be at the moment, our new teams embraced the idea of our new Cross-Functional teams with some minor changes.  First, we still needed to see ourselves as electricians, instrument and controls, machinists, welders and mechanics.  We each had our own “certifications” and expertise that only a person with that trade could perform.

Charles and I would still go to the electric shop in the morning before work began, and during lunch and breaks.  Our electric equipment to perform our job was there, and we still needed to maintain a stock of electric supplies.  The same was true for the Instrument and Controls crew members.

Even today, after having been gone from the Power Plant for 13 1/2 years, the electric shop office phone still has my voice on the voice mail message.  I know, because a couple of years ago, when it was accidentally erased, Tim Foster (Charles Foster’s son), asked me to record a new message so they could put it back on the phone.  I considered that a great honor to be asked by True Power Plant Men to record their voice mail message on the electric shop phone.  The Phone number by the way is:  (405) 553-29??.  Oh.  I can’t remember the last two digits.  🙂

Once the kinks were worked out of the cross-functional team structure, it worked really well.  I just kept thinking…. Boy, if we only had a group of supportive upper management that put their plant first over their own personal power needs, this would be great.  The True Power Plant Men figured out how to work around them, so that in spite of the obstacles, within about 4 years, we had hit our stride.

Let me give you an example of how well the cross-functional teams worked compared to the old conventional way we used to work.  I will start by describing how we used to do things….  Let’s say that a pump breaks down at the coal yard…

Horizontal pump

Horizontal pump

— start here —

An operator creates the Maintenance Order (M.O.).  It is eventually assigned to a crew of mechanics.  (start the clock here).  When they have time, they go to the coal yard to look over the problem.  Yep.  The pump is not working.  They will have to take it back to the shop to fix it.

A Maintenance Order is created for the electricians to unwire the motor.  The electricians receive the maintenance order and prioritize it.  They finally assign it to a team to go work on it.  Say, in one week from the time they received the M.O.  The electrician goes to the control room to request a clearance on the pump.  The next day the electrician unwires the motor.  They complete the maintenance order at the end of the day and send it back up to the A Foreman.

The completed electric maintenance order is sent back to the mechanics letting them know that the motor for the pump has been unwired.  When they receive it, a couple of days later, they schedule some time that week to go work on the pump.  At that time, they bring the motor to the electric shop so that it can be worked on at the same time.

The motor and the pump is worked on some time during the next week.

A machinist is needed to re-sleeve a bearing housing on either the motor or the pump or both.  So, an M.O. is created for the machinist to work on creating a sleeve in an end bell of the motor or the pump.

Gary (Stick) McCain

Gary (Stick) McCain — Machinist Extraordinaire

The electricians inform the mechanics when the motor is ready.  When they are done with the pump, and they have put it back in place, they put the motor back.  Then they create an M.O. for the Machinist to line up the motor and the pump before the coupling is installed.

The Machinists prioritize their work and at some point, let’s say a couple of days, they make it up to the motor and work on aligning the pump and the motor.

During the re-installation, it is decided that a bracket that has worn out needs to be welded back.  So, an M.O. is created for the welders to replace the bracket before the motor can be rewired.

The welders prioritize their work, and in a week (or two) they finally have time to go weld the bracket.

George Clouse

George Clouse – Welding Wizard

They return their M.O. completed to the mechanics who then tell the electricians that they can re-wire the motor.

The electricians prioritize their work and when they have time to go re-wire the motor, they wire it up.  After wiring it, they go to the control room to have the operators help them bump test the motor to make sure it runs in the right direction.  An entire day goes by until the electrician receives a call saying that the operator is ready to bump test the motor.  The electrician and/or mechanic meets the operator at the pump to bump test the motor.  Once this test is performed, the mechanic re-couples the motor.

The electrician then removes his clearance on the pump and it is put back into service.  The M.O.s are completed.

—  End here.  The time it took to repair the pump and put it back in service would commonly take one month —

Now see what happens when you have a cross-functional team working on it….(and be amazed).

— Start here —

The maintenance team receives a ticket (M.O.) from the planner that a pump is broken at the coal yard.  A mechanic goes and looks at it and determines it needs to be repaired.  He calls his Electrician Teammate and tells him that the motor needs to be unwired in order to fix the pump.  The electrician goes to the control room and takes a clearance on the pump.

The electrician then goes to the switchgear and waits for the operator to place the clearance.  When that is completed, the electrician goes to the pump and unwires the motor.  While there, he helps the mechanic pull the motor and put it aside.  The electrician determines there if the motor needs to be worked on.  If possible, it is repaired in place, or the motor is brought to the electric shop at the same time as the pump.  It is determined that the pump needs to be worked on, so they work together to bring it to the shop where the mechanics work on the pump.  Any machinist work is done at that time.

When the pump is being put back in place, the bracket is found broken, so they call the welder on their team who comes up and welds it back on.  The machinist comes with the electrician and the mechanic to align the motor.  The operators are called to bump test the motor.  As soon as the test is over, the coupling is installed.  The clearance is removed and the pump is put back in place.

— End here.  The pump can now be repaired within one week instead of four weeks.  Often the pump can be repaired in days instead of weeks. —

The reason why the cross-functional teams worked so well is that we all had the same priority.  We all had the same job and we had all the skills on our team to do all the work.  This was a fantastic change from working in silos.

This was “Working Smarter”, not “Working Harder”.  Ever since that day when we first learned that we had to “Work Harder” I always cringe when I hear that phrase.  To me, “Working Harder” means, “Working Dumber”.  Today I am a big advocate of Cross-Functional Teams.  I have seen them work successfully.  There was only one catch which I will talk about later.  This worked beautifully, but keep in mind… We had cross-functional teams made of the best Power Plant Men on the planet!  So, I may have a lopsided view of how successful they really work in the general public.

Advertisements

Pain in the Neck Muskogee Power Plant Relay Testing

Don’t let the title fool you.  I love testing Power Plant Protective Relays.  There is a sense of satisfaction when you have successfully cleaned, calibrated and tested a relay that is going to protect the equipment you have to work on every day.  With that said, I was hit with such an unbelievable situation when testing Muskogee Relays in 1995 that I was left with a serious pain in the neck.

On August 14, 2003 the electric power in the Northeast United States and Canada went out.  The Blackout lasted long enough to be a major annoyance for those in the that region of the United States.

Map of the power blackout in 2003

Map of the power blackout in 2003

When I heard about how the blackout had moved across the region, I immediately knew what had happened.  I was quickly reminded of the following story.  I told my wife Kelly, “I know exactly why such a large area lost power!  They hadn’t done proper preventative maintenance on the Protective Relays in the substations!  Just like….”  Well…. I’ll tell you that part now:

I have mentioned in a couple of earlier posts that something always seemed a little “off” at the Muskogee Power Plant.  I had decided early on that while working there I would stick to drinking sodas instead of water.  See the post:  “Something’s In the Water at the Muskogee Power Plant“.  Even with that knowledge, I was still shocked at what I found while testing relays at the plant.

This story really begins one Sunday at Muskogee when one of the Auxiliary Operators was making his rounds inspecting equipment.  He was driving his truck around the south edge of the Unit 6 parking lot on the service road.  He glanced over at a pump next to the road, and at first, he thought he was just seeing things.  After stopping the truck and backing up for a second glance, he was sure he wasn’t dreaming.  It’s just that what he was seeing seemed so strange, he wasn’t sure what was happening.

The operator could see what appeared to be silver paint chips popping off of the large pump motor in all directions.  After closer examination, he figured out that the motor was burning up.  It was still running, but it had become so hot that the paint was literally burning off of the motor.

Horizontal pump

Like this Horizontal pump only much bigger and painted silver

A motor like this would get hot if the bearings shell out.  Before the motor is destroyed, the protective relays on the breaker in the 4,000 Volt switchgear shuts the motor off.  In this case, the relay hadn’t tripped the motor, so, it had become extremely hot and could have eventually exploded if left running.  The operator shut the motor down and wrote a work order for the electricians.

Doyle Fullen was the foreman in the electric shop that received the work order.  When he looked into what had happened, he realized that the protective relay had not been inspected for a couple of years for this motor.

I couldn’t find a picture of Doyle.  In his youth he reminds me of a very smart Daryl in Walking Dead:

Norman Reedus from Walking Dead

Norman Reedus from Walking Dead

In fact, since before the downsizing in 1994, none of the Protective Relays at the plant had been inspected.  The person that had been inspecting the relays for many years had moved to another job or retired in 1994.  This was just a warning shot across the bow that could have had major consequences.

No one at Muskogee had been trained to test Protective Relays since the downsizing, so they reached out to our plant in North Central Oklahoma for help.  That was when I was told that I was going to be going to Muskogee during the next overhaul (outage).  I had been formally trained to inspect, clean, calibrate and test Protective Relays with two of my Power Plant Heroes, Ben Davis and Sonny Kendrick years earlier.  See the post:  “Relay Tests and Radio Quizzes with Ben Davis“.

My Protective Relay Maintenance course book

My Protective Relay Maintenance course book

Without going into too much detail about the actual tests we performed as I don’t want to make this a long rambling post (like… well…. like most of my posts…..I can already tell this is going to be a long one), I will just say that I took our antiquated relay tester down to Muskogee to inspect their relays and teach another electrician Charles Lay, how to perform those tests in the future.  Muskogee had a similar Relay Test Set.  These were really outdated, but they did everything we needed, and it helped you understand exactly what was going on when you don’t have a newfangled Relay Test Set.

AVO Multi-Amp SR-76 Relay Test Set

AVO Multi-Amp SR-76 Relay Test Set

You need to periodically test both mechanical and electronic protective relays.  In the electronic relays the components change their properties slightly over time, changing the time it takes to trip a breaker under a given circumstance (we’re talking about milliseconds).  In the mechanical relays (which I have always found to be more reliable), they sit inside a black box all the time, heating up and cooling as the equipment is used.  Over time, the varnish on the copper coils evaporates and settles on all the components.  This becomes sticky so that the relay won’t operate at the point where it should.

A panel of Protective Relays

A panel of Protective Relays

In the picture above, the black boxes on the top, middle and right are mechanical relays.  This means that something actually has to turn or pick up in order to trip the equipment.  The electronic relays may have a couple of small relays, but for the most part, they are made up of transistors, resistors, capacitors and diodes.

So, with all that said, let me start the real story…. gee…. It’s about time…

So, here I am sitting in the electric shop lab just off of the Unit 6 T-G floor.  We set up all the equipment and had taken a couple of OverCurrent relays out of some high voltage breakers in the switchgear.  I told Charles that before you actually start testing the relays, you need to have the test documents from the previous test and we also needed the instruction manuals for each of the relays because the manuals will have the diagrams that you use to determine the exact time that the relays should trip for each of the tests.  So, we went up to the print room to find the old tests and manuals.  Since they weren’t well organized, we just grabbed the entire folder where all the relays tests were kept since Unit 6 had been in operation.

When we began testing the relays at first I thought that the relay test set wasn’t working correctly.  Here I was trying to impress my new friend, Charles Lay, a 63 year old highly religious fundamental Christian that I knew what I was doing, and I couldn’t even make a relay trip.  I was trying to find the “As Found” tripping level.  That is, before you clean up the relay.  Just like you found it.  Only, it wouldn’t trip.

It turned out that the relay was stuck from the varnish as I explained above.  It appeared as if the relay hadn’t been tested or even operated for years.  The paperwork showed that it had been tested three years earlier.  Protective Relays should be tested at least every two years, but I wouldn’t have thought that the relay would be in such a bad condition in just three years.  It had been sitting in a sealed container to keep out dust.  But it was what it was.

I told Charles that in order to find the “As Found” point where the relay would trip, we would need to crank up the test set as high as needed to find when it actually did trip.  It turned out that the relay which should have instantaneously tripped somewhere around 150 amps wouldn’t have tripped until the motor was pulling over 4,000 amps.  I could tell right away why the Auxiliary Operator found that motor burning up without tripping.  The protective relays were stuck.

As it turned out… almost all of the 125 or so relays were in the same condition.  We cleaned them all up and made them operational.

There is an overcurrent relay for the main bus on each section of a main switchgear.

A picture of a clean switchgear. Picture 6 rows of switchgear like this

A picture of a clean switchgear. Picture 6 rows of switchgear like this

When I tested the “As Found” instantaneous trip for the main bus relay, I found that it was so high that the Unit 6 Main Turbine Generator would have melted down before the protective relay would have tripped the power to that one section of switchgear.  The entire electric bus would have been nothing but molten metal by that time.

As I tested each of these relays, I kept shaking my head in disbelief.  But that wasn’t the worst of it.  The mystery as to why these relays were all glued shut by varnish was finally solved, and that reason was even more unbelievable.

Here is what I found…..  The first thing you do when you are going to test a relay is that you fill out a form that includes all the relay information, such as, what it is for, what are the settings on the relay, and what are the levels of tests that you are going to perform on it.  You also include a range of milliseconds that are acceptable for the relay for each of the tests.  Normally, you just copy what was used in the previous test, because you need to include the time it took for the Previous “As Left” test on your form.  That is why we needed the forms from the previous test.

So, I had copied the information from the previous test form and began testing the relay (one of the first overcurrent relays we tested)…  Again… I was a 34 year old teacher trying to impress my 63 year old student.  So, I was showing him how you mechanically adjust the relay in order for it to trip within the acceptable range.  No matter how hard I tried, I couldn’t adjust the relay so that it would even be close to the desired range for the longer time trip times…. like the 2 second to 25 second range.  It wasn’t even close to the range that was on the form from the last test.

The form from the last test showed that the relay was in the right range for all the levels of test.  When I tested it, like I said, it wasn’t even close.  So, I went to the diagram in the instruction manual for this type of relay.  The diagram looks similar to this one used for thermal overloads:

Thermal Overload Tripping curves

Thermal Overload Tripping curves

See all those red lines?  Well, when you setup a relay, you have a dial where you set the range depending on the needs for the type of motor you are trying to trip. Each red line represents each setting on the dial.  Most of the relays were set on the same number, so we would be using the same red line on the diagram to figure out at different currents how long it should take for a relay to trip….

Here is the clincher….The time range that was written on the previous form wasn’t for the correct relay setting.  The person that tested the relay had accidentally looked at the wrong red line.  — That in itself is understandable, since it could be easy to get on the wrong line… The only thing is that as soon as you test the relay, you would know that something is wrong, because the relay wouldn’t trip in that range, just like I had found.

I double and triple checked everything to make sure we were looking at the same thing.  The previous form indicated the same settings on the relay as now, yet, the time ranges were for a different line! — Ok.  I know.  I have bored you to tears with all this stuff about time curves and overcurrent trips… so I will just tell you what this means…

This meant that when the person completed the forms the last time, they didn’t test the relays at all.  They just filled out the paperwork.  They put in random values that were in the acceptable range and sat around in the air conditioned lab during the entire overhaul smoking his pipe. — Actually, I don’t remember if he smoked a pipe or not.  He was the Electrical Specialist for the plant.  I remembered seeing him sitting in the lab with a relay hooked up to the test set throughout the entire overhaul when I had been there during previous overhauls, but I realized finally that he never tested the relays.  He didn’t even go so far as try to operate them.

I went back through the records to when the plant was first “checked out”.  Doyle Fullen had done the check out on the relays and the test after that.  Doyle had written the correct values from the manual on his forms.  I could see where he had actually performed the tests on the relays and was getting the same values I was finding when I tested the relays, so I was certain that I wasn’t overlooking anything.

As I tested each of the relays, I kept shaking my head in disbelief.  It was so unbelievable.  How could someone do such a thing?  Someone could have been killed because a protective relay wasn’t working correctly.  This was serious stuff.

One day while Charles and I were working away on the relays, Jack Coffman, the Superintendent of all the Power Plants came walking through the lab.  He asked us how we were doing.  I swiveled around in my chair to face him and I said, “Pretty good, except for this pain in my neck” as I rubbed the back of my neck.

Jack stopped and asked me what happened.  I told him that I had been shaking my head in disbelief for the last two weeks, and it gave me a pain in the neck.  Of course, I knew this would get his attention, so he asked, “Why?”  I went through all the details of what I had found.

I showed him how since the time that Doyle Fullen last tested the relays more than 10 years earlier, these relays hadn’t been tested at all.  I showed him how the main bus relays were so bad that it would take over 100,000 amps to have tripped the 7100 KV switchgear bus or 710 Megawatts!  More power than the entire generator could generate.  It was only rated at about 550 Megawatts at the most.

Jack stood there looking off into space for a few seconds, and then walked out the door…. I thought I saw him shaking his head as he left.  Maybe he was just looking both ways for safety reasons, but to me, it looked like a shake of disbelief.  I wonder if I had given him the same pain in the neck.

That is really the end of the relay story, but I do want to say a few words about Charles Lay.  He was a hard working electrician that was nearing retirement.  People would come around to hear us discussing religion.  I am Catholic, and he went to a Fundamental Christian Church.  We would debate the differences between our beliefs and just Christian beliefs in general.  We respected each other during our time together, even though he was sure I am going to hell when I die.

People would come in just to hear our discussion for a while as we were cleaning and calibrating the relays.  One day Charles asked me if I could help him figure out how much he was going to receive from his retirement from the electric company.  He had only been working there for three years.  Retirement at that time was determined by your years of service.  So, three years didn’t give him too much.

When I calculated his amount, he was upset.  He said, “Am I going to have to work until I die?”  I said, “Well, there’s always your 401k and Social Security.”  He replied that he can’t live on Social Security.  I said, “Well, there’s your 401k.”  He asked, “What’s that?” (oh.  not a good sign).

I explained that it was a retirement plan where you are able to put money in taxed deferred until you take it out when you retire.  He said, “Oh.  I never put anything in something like that.”  My heart just sank as I looked in his eyes.  He had suddenly realized that he wasn’t going to receive a retirement like those around him who had spent 35 years working in the Power Plant.

When I left the plant after teaching Charles Lay how to test the relays, that was the last time I ever saw him.  I don’t know what became of Charles.  I figure he would be 83 years old today.  I wonder if he finally retired when he reached the 80 points for your age and years of service.  He would have never reached enough years of service to receive a decent amount of retirement from the Electric Company since he didn’t start working there until he was 60 years old.  That is, unless he’s still working there now.

As I said earlier in this post, Charles Lay was a very good worker.  He always struck me as the “Hardworking type”.  I often think about the time we spent together, especially when I hear about a power blackout somewhere.  — A word of caution to Power Companies…. keep your protective relays in proper working condition.  Don’t slack off on the Preventative Maintenance.  — I guess that’s true for all of us… isn’t it?  Don’t slack off on Preventative Maintenance in all aspects of your life.

Added note:  On 7/6/2019, 3 weeks after re-posting this story, look what happened:  Con Edison says cause of NYC blackout was substation’s faulty relay protection system

Crossfunctional Power Plant Dysfunction

The coal-fired Power Plant in North Central Oklahoma had gone from 360 employees in 1987 down to 124 employees on August 1, 1994 after the second downsizing.  Monday morning when we arrived at work, the maintenance department met in the main break room to be told how we were going to survive the loss of 100 employees.  With only 7 electricians left, I kept trying to add up on my fingers how we could possibly keep up with all the work we had to do.

Jasper Christensen stood up and after saying that he understood how we must feel about our present situation, he told us that we will have to each work harder.  I shook my head in disbelief (inside my head only… I didn’t really shake my head, as it was frozen with the same blank stare everyone else was wearing).  I knew we weren’t going to be working harder.  — What does that really mean anyway.  I thought he should have said, “We will each have to work “smarter” because we can’t really work “harder”.  Jasper was a nice person, but he never really was much for words so I gave him a pass on this one.

Jasper Christensen

Jasper Christensen

Interestingly, the three people in charge at the plant, Jasper, Jim Arnold and Bill Green were all 53 years old, and only within 4 months in age from each other.  They all belonged to the “old school way of doing things” (see the post:  “From Pioneers to Power Plant Managers“).  As Jasper continued in his speech I noticed that gone was any talk of working together to achieve our goals.  I immediately felt that we had just rolled back our management to a time before our first downsizing in 1987 when the Evil Plant Manager used to rule the plant with an iron fist.

I felt this way because we were being told how we were going to change everything we do without giving any of our own input.  For instance, we would no longer have a Quality Action Team.  That was disbanded immediately.  We would no longer hold Quality Team meetings (we were also told that the Quality process was not going away, though we couldn’t see how it was going to work).  The Safety Task Force did survive.

We were also told that we would no longer fill out any forms unless they are requested by someone.  It seems that we had over 1,300 forms that were being filled out at the plant and most of them were never being used for anything, so, unless someone requested a form, we wouldn’t just fill them out for the sake of filling them out.  This was actually a good idea.  I know we filled out forms in triplicate each week when we did transformer and substation inspections.  Most of those were never looked at, I’m sure.

It turned out later that we needed only about 400 of the 1300 forms our plant was churning out each month.

We were told we wouldn’t be doing Substation inspections.  That was not our responsibility.  It would be done by the Transmission and Distribution division instead.  I was beginning to see how management was trying to figure out how 7 electricians were going to “work harder”.  The answer at the moment was that we were going to do less.  The purpose of the Substation and Transformer checks each week was to look for problems while they were minor instead of waiting for a catastrophe to happen.

We were told that we were not going to “Gold Plate” our work.  We were going to just do what it took to complete the task without worrying about polishing it up to make it “perfect” (which is what real Power Plant Men do).  Instead we were going to “Farm Fix it”.  I’ll go more into this subject with a separate post.

We were then told that we would no longer have an Electric Shop and an Instrument and Controls shop.  We would from then on all meet in the Mechanical Maintenance shop.  We were not supposed to go to the Electric Shop or the Instrument and Controls shops for breaks because we were all going to be cross-functional.  We are all Maintenance now.  No longer specialized (sort of).

We were going to have four Maintenance teams.  Each one will have mechanics, welders, machinists, electricians and Instrument and controls people.  Each member on each team would learn to do each other’s jobs to a degree.

An electrician will learn how to tack weld.  A mechanic will learn how to run conduit and pull wire.  An instrument and controls person will learn how to use the lathe.  We would each learn enough about each job in order to perform minor tasks in each area without having to call the expert in that skill.

When the meeting was over, we each met with our own foremen.  Alan Kramer was my new foreman.  He used to be a foreman in the Instrument and Controls shop.

Alan Kramer

Alan Kramer

It became apparent that even though Jasper had come across as if everything had already been decided and that this was the way it was going to be, things hadn’t really been ironed out yet.  Actually, this was just a first pass.  The main goal was for us to figure out how to get all the work done that needed to be done.  I was still an electrician and I was still responsible for working on electrical jobs.

One really good part of the new situation was that I was now on the same team as Charles Foster.  We had always been very good friends, but I hadn’t worked on the same team as Charles since my first year as an electrician in 1984, ten years earlier when he was my first foreman in the electric shop (See the post:  “New Home in the Power Plant Electric Shop“).  We were the two electricians on Alan Kramer’s team.

Charles Foster

Charles Foster

Besides the fact that everyone was very bitter over the despicable treatment of our fellow Power Plant Men that were laid off the previous Friday (see the post: “Power Plant Downsizing Disaster and the Left Behinds“), we knew that we had to figure out how to make this new arrangement work.  We knew our upper management was using the old tyrannical style of management, but we also knew that at this point, they needed every one of us.  They couldn’t go around firing us just because we spoke our mind (which was good for me, because, I was still in the process of learning how to keep my mouth shut when that was the most beneficial course of action).

As Dysfunctional as our upper management seemed to be at the moment, our new teams embraced the idea of our new Cross-Functional teams with some minor changes.  First, we still needed to see ourselves as electricians, instrument and controls, machinists, welders and mechanics.  We each had our own “certifications” and expertise that only a person with that trade could perform.

Charles and I would still go to the electric shop in the morning before work began, and during lunch and breaks.  Our electric equipment to perform our job was there, and we still needed to maintain a stock of electric supplies.  The same was true for the Instrument and Controls crew members.

Even today, after having been gone from the Power Plant for 13 1/2 years, the electric shop office phone still has my voice on the voice mail message.  I know, because a couple of years ago, when it was accidentally erased, Tim Foster (Charles Foster’s son), asked me to record a new message so they could put it back on the phone.  I considered that a great honor to be asked by True Power Plant Men to record their voice mail message on the electric shop phone.  The Phone number by the way is:  (405) 553-29??.  Oh.  I can’t remember the last two digits.  🙂

Once the kinks were worked out of the cross-functional team structure, it worked really well.  I just kept thinking…. Boy, if we only had a group of supportive upper management that put their plant first over their own personal power needs, this would be great.  The True Power Plant Men figured out how to work around them, so that in spite of the obstacles, within about 4 years, we had hit our stride.

Let me give you an example of how well the cross-functional teams worked compared to the old conventional way we used to work.  I will start by describing how we used to do things….  Let’s say that a pump breaks down at the coal yard…

Horizontal pump

Horizontal pump

— start here —

An operator creates the Maintenance Order (M.O.).  It is eventually assigned to a crew of mechanics.  (start the clock here).  When they have time, they go to the coal yard to look over the problem.  Yep.  The pump is not working.  They will have to take it back to the shop to fix it.

A Maintenance Order is created for the electricians to unwire the motor.  The electricians receive the maintenance order and prioritize it.  They finally assign it to a team to go work on it.  Say, in one week from the time they received the M.O.  The electrician goes to the control room to request a clearance on the pump.  The next day the electrician unwires the motor.  They complete the maintenance order at the end of the day and send it back up to the A Foreman.

The completed electric maintenance order is sent back to the mechanics letting them know that the motor for the pump has been unwired.  When they receive it, a couple of days later, they schedule some time that week to go work on the pump.  At that time, they bring the motor to the electric shop so that it can be worked on at the same time.

The motor and the pump is worked on some time during the next week.

A machinist is needed to re-sleeve a bearing housing on either the motor or the pump or both.  So, an M.O. is created for the machinist to work on creating a sleeve in an end bell of the motor or the pump.

Gary (Stick) McCain

Gary (Stick) McCain — Machinist Extraordinaire

The electricians inform the mechanics when the motor is ready.  When they are done with the pump, and they have put it back in place, they put the motor back.  Then they create an M.O. for the Machinist to line up the motor and the pump before the coupling is installed.

The Machinists prioritize their work and at some point, let’s say a couple of days, they make it up to the motor and work on aligning the pump and the motor.

During the re-installation, it is decided that a bracket that has worn out needs to be welded back.  So, an M.O. is created for the welders to replace the bracket before the motor can be rewired.

The welders prioritize their work, and in a week (or two) they finally have time to go weld the bracket.

George Clouse

George Clouse – Welding Wizard

They return their M.O. completed to the mechanics who then tell the electricians that they can re-wire the motor.

The electricians prioritize their work and when they have time to go re-wire the motor, they wire it up.  After wiring it, they go to the control room to have the operators help them bump test the motor to make sure it runs in the right direction.  An entire day goes by until the electrician receives a call saying that the operator is ready to bump test the motor.  The electrician and/or mechanic meets the operator at the pump to bump test the motor.  Once this test is performed, the mechanic re-couples the motor.

The electrician then removes his clearance on the pump and it is put back into service.  The M.O.s are completed.

—  End here.  The time it took to repair the pump and put it back in service would commonly take one month —

Now see what happens when you have a cross-functional team working on it….(and be amazed).

— Start here —

The maintenance team receives a ticket (M.O.) from the planner that a pump is broken at the coal yard.  A mechanic goes and looks at it and determines it needs to be repaired.  He calls his Electrician Teammate and tells him that the motor needs to be unwired in order to fix the pump.  The electrician goes to the control room and takes a clearance on the pump.

The electrician then goes to the switchgear and waits for the operator to place the clearance.  When that is completed, the electrician goes to the pump and unwires the motor.  While there, he helps the mechanic pull the motor and put it aside.  The electrician determines there if the motor needs to be worked on.  If possible, it is repaired in place, or the motor is brought to the electric shop at the same time as the pump.  It is determined that the pump needs to be worked on, so they work together to bring it to the shop where the mechanics work on the pump.  Any machinist work is done at that time.

When the pump is being put back in place, the bracket is found broken, so they call the welder on their team who comes up and welds it back on.  The machinist comes with the electrician and the mechanic to align the motor.  The operators are called to bump test the motor.  As soon as the test is over, the coupling is installed.  The clearance is removed and the pump is put back in place.

— End here.  The pump can now be repaired within one week instead of four weeks.  Often the pump can be repaired in days instead of weeks. —

The reason why the cross-functional teams worked so well is that we all had the same priority.  We all had the same job and we had all the skills on our team to do all the work.  This was a fantastic change from working in silos.

This was “Working Smarter”, not “Working Harder”.  Ever since that day when we first learned that we had to “Work Harder” I always cringe when I hear that phrase.  To me, “Working Harder” means, “Working Dumber”.  Today I am a big advocate of Cross-Functional Teams.  I have seen them work successfully.  There was only one catch which I will talk about later.  This worked beautifully, but keep in mind… We had cross-functional teams made of the best Power Plant Men on the planet!  So, I may have a lopsided view of how successful they really work in the general public.

Pain in the Neck Muskogee Power Plant Relay Testing

Don’t let the title fool you.  I love testing Power Plant Protective Relays.  There is a sense of satisfaction when you have successfully cleaned, calibrated and tested a relay that is going to protect the equipment you have to work on every day.  With that said, I was hit with such an unbelievable situation when testing Muskogee Relays in 1995 that I was left with a serious pain in the neck.

On August 14, 2003 the electric power in the Northeast United States and Canada went out.  The Blackout lasted long enough to be a major annoyance for those in the that region of the United States.

 

Map of the power blackout in 2003

Map of the power blackout in 2003

When I heard about how the blackout had moved across the region, I immediately knew what had happened.  I was quickly reminded of the following story.  I told my wife Kelly, “I know exactly why such a large area lost power!  They hadn’t done proper preventative maintenance on the Protective Relays in the substations!  Just like….”  Well…. I’ll tell you that part now:

I have mentioned in a couple of earlier posts that something always seemed a little “off” at the Muskogee Power Plant.  I had decided early on that while working there I would stick to drinking sodas instead of water.  See the post:  “Something’s In the Water at the Muskogee Power Plant“.  Even with that knowledge, I was still shocked at what I found while testing relays at the plant.

This story really begins one Sunday at Muskogee when one of the Auxiliary Operators was making his rounds inspecting equipment.  He was driving his truck around the south edge of the Unit 6 parking lot on the service road.  He glanced over at a pump next to the road, and at first, he thought he was just seeing things.  After stopping the truck and backing up for a second glance, he was sure he wasn’t dreaming.  It’s just that what he was seeing seemed so strange, he wasn’t sure what was happening.

The operator could see what appeared to be silver paint chips popping off of the large pump motor in all directions.  After closer examination, he figured out that the motor was burning up.  It was still running, but it had become so hot that the paint was literally burning off of the motor.

Horizontal pump

Like this Horizontal pump only much bigger and painted silver

A motor like this would get hot if the bearings shell out.  Before the motor is destroyed, the protective relays on the breaker in the 4,000 Volt switchgear shuts the motor off.  In this case, the relay hadn’t tripped the motor, so, it had become extremely hot and could have eventually exploded if left running.  The operator shut the motor down and wrote a work order for the electricians.

Doyle Fullen was the foreman in the electric shop that received the work order.  When he looked into what had happened, he realized that the protective relay had not been inspected for a couple of years for this motor.

I couldn’t find a picture of Doyle.  In his youth he reminds me of a very smart Daryl in Walking Dead:

Norman Reedus from Walking Dead

Norman Reedus from Walking Dead

In fact, since before the downsizing in 1994, none of the Protective Relays at the plant had been inspected.  The person that had been inspecting the relays for many years had moved to another job or retired in 1994.  This was just a warning shot across the bow that could have had major consequences.

No one at Muskogee had been trained to test Protective Relays since the downsizing, so they reached out to our plant in North Central Oklahoma for help.  That was when I was told that I was going to be going to Muskogee during the next overhaul (outage).  I had been formally trained to inspect, clean, calibrate and test Protective Relays with two of my Power Plant Heroes, Ben Davis and Sonny Kendrick years earlier.  See the post:  “Relay Tests and Radio Quizzes with Ben Davis“.

My Protective Relay Maintenance course book

My Protective Relay Maintenance course book

Without going into too much detail about the actual tests we performed as I don’t want to make this a long rambling post (like… well…. like most of my posts…..I can already tell this is going to be a long one), I will just say that I took our antiquated relay tester down to Muskogee to inspect their relays and teach another electrician Charles Lay, how to perform those tests in the future.  Muskogee had a similar Relay Test Set.  These were really outdated, but they did everything we needed, and it helped you understand exactly what was going on when you don’t have a newfangled Relay Test Set.

AVO Multi-Amp SR-76 Relay Test Set

AVO Multi-Amp SR-76 Relay Test Set

You need to periodically test both mechanical and electronic protective relays.  In the electronic relays the components change their properties slightly over time, changing the time it takes to trip a breaker under a given circumstance (we’re talking about milliseconds).  In the mechanical relays (which I have always found to be more reliable), they sit inside a black box all the time, heating up and cooling as the equipment is used.  Over time, the varnish on the copper coils evaporates and settles on all the components.  This becomes sticky so that the relay won’t operate at the point where it should.

A panel of Protective Relays

A panel of Protective Relays

In the picture above, the black boxes on the top, middle and right are mechanical relays.  This means that something actually has to turn or pick up in order to trip the equipment.  The electronic relays may have a couple of small relays, but for the most part, they are made up of transistors, resistors, capacitors and diodes.

So, with all that said, let me start the real story…. gee…. It’s about time…

So, here I am sitting in the electric shop lab just off of the Unit 6 T-G floor.  We set up all the equipment and had taken a couple of OverCurrent relays out of some high voltage breakers in the switchgear.  I told Charles that before you actually start testing the relays, you need to have the test documents from the previous test and we also needed the instruction manuals for each of the relays because the manuals will have the diagrams that you use to determine the exact time that the relays should trip for each of the tests.  So, we went up to the print room to find the old tests and manuals.  Since they weren’t well organized, we just grabbed the entire folder where all the relays tests were kept since Unit 6 had been in operation.

When we began testing the relays at first I thought that the relay test set wasn’t working correctly.  Here I was trying to impress my new friend, Charles Lay, a 63 year old highly religious fundamental Christian that I knew what I was doing, and I couldn’t even make a relay trip.  I was trying to find the “As Found” tripping level.  That is, before you clean up the relay.  Just like you found it.  Only, it wouldn’t trip.

It turned out that the relay was stuck from the varnish as I explained above.  It appeared as if the relay hadn’t been tested or even operated for years.  The paperwork showed that it had been tested three years earlier.  Protective Relays should be tested at least every two years, but I wouldn’t have thought that the relay would be in such a bad condition in just three years.  It had been sitting in a sealed container to keep out dust.  But it was what it was.

I told Charles that in order to find the “As Found” point where the relay would trip, we would need to crank up the test set as high as needed to find when it actually did trip.  It turned out that the relay which should have instantaneously tripped somewhere around 150 amps wouldn’t have tripped until the motor was pulling over 4,000 amps.  I could tell right away why the Auxiliary Operator found that motor burning up without tripping.  The protective relays were stuck.

As it turned out… almost all of the 125 or so relays were in the same condition.  We cleaned them all up and made them operational.

There is an overcurrent relay for the main bus on each section of a main switchgear.

A picture of a clean switchgear. Picture 6 rows of switchgear like this

A picture of a clean switchgear. Picture 6 rows of switchgear like this

When I tested the “As Found” instantaneous trip for the main bus relay, I found that it was so high that the Unit 6 Main Turbine Generator would have melted down before the protective relay would have tripped the power to that one section of switchgear.  The entire electric bus would have been nothing but molten metal by that time.

As I tested each of these relays, I kept shaking my head in disbelief.  But that wasn’t the worst of it.  The mystery as to why these relays were all glued shut by varnish was finally solved, and that reason was even more unbelievable.

Here is what I found…..  The first thing you do when you are going to test a relay is that you fill out a form that includes all the relay information, such as, what it is for, what are the settings on the relay, and what are the levels of tests that you are going to perform on it.  You also include a range of milliseconds that are acceptable for the relay for each of the tests.  Normally, you just copy what was used in the previous test, because you need to include the time it took for the Previous “As Left” test on your form.  That is why we needed the forms from the previous test.

So, I had copied the information from the previous test form and began testing the relay (one of the first overcurrent relays we tested)…  Again… I was a 34 year old teacher trying to impress my 63 year old student.  So, I was showing him how you mechanically adjust the relay in order for it to trip within the acceptable range.  No matter how hard I tried, I couldn’t adjust the relay so that it would even be close to the desired range for the longer time trip times…. like the 2 second to 25 second range.  It wasn’t even close to the range that was on the form from the last test.

The form from the last test showed that the relay was in the right range for all the levels of test.  When I tested it, like I said, it wasn’t even close.  So, I went to the diagram in the instruction manual for this type of relay.  The diagram looks similar to this one used for thermal overloads:

Thermal Overload Tripping curves

Thermal Overload Tripping curves

See all those red lines?  Well, when you setup a relay, you have a dial where you set the range depending on the needs for the type of motor you are trying to trip. Each red line represents each setting on the dial.  Most of the relays were set on the same number, so we would be using the same red line on the diagram to figure out at different currents how long it should take for a relay to trip….

Here is the clincher….The time range that was written on the previous form wasn’t for the correct relay setting.  The person that tested the relay had accidentally looked at the wrong red line.  — That in itself is understandable, since it could be easy to get on the wrong line… The only thing is that as soon as you test the relay, you would know that something is wrong, because the relay wouldn’t trip in that range, just like I had found.

I double and triple checked everything to make sure we were looking at the same thing.  The previous form indicated the same settings on the relay as now, yet, the time ranges were for a different line! — Ok.  I know.  I have bored you to tears with all this stuff about time curves and overcurrent trips… so I will just tell you what this means…

This meant that when the person completed the forms the last time, they didn’t test the relays at all.  They just filled out the paperwork.  They put in random values that were in the acceptable range and sat around in the air conditioned lab during the entire overhaul smoking his pipe. — Actually, I don’t remember if he smoked a pipe or not.  He was the Electrical Specialist for the plant.  I remembered seeing him sitting in the lab with a relay hooked up to the test set throughout the entire overhaul, but I realized finally that he never tested the relays.  He didn’t even go so far as try to operate them.

I went back through the records to when the plant was first “checked out”.  Doyle Fullen had done the check out on the relays and the test after that.  Doyle had written the correct values from the manual on his forms.  I could see where he had actually performed the tests on the relays and was getting the same values I was finding when I tested the relays, so I was certain that I wasn’t overlooking anything.

As I tested each of the relays, I kept shaking my head in disbelief.  It was so unbelievable.  How could someone do such a thing?  Someone could have been killed because a protective relay wasn’t working correctly.  This was serious stuff.

One day while Charles and I were working away on the relays, Jack Coffman, the Superintendent of all the Power Plants came walking through the lab.  He asked us how we were doing.  I swiveled around in my chair to face him and I said, “Pretty good, except for this pain in my neck” as I rubbed the back of my neck.

Jack stopped and asked me what happened.  I told him that I had been shaking my head in disbelief for the last two weeks, and it gave me a pain in the neck.  Of course, I knew this would get his attention, so he asked, “Why?”  I went through all the details of what I had found.

I showed him how since the time that Doyle Fullen last tested the relays more than 10 years earlier, these relays hadn’t been tested at all.  I showed him how the main bus relays were so bad that it would take over 100,000 amps to have tripped the 7100 KV switchgear bus or 710 Megawatts!  More power than the entire generator could generate.  It was only rated at about 550 Megawatts at the most.

Jack stood there looking off into space for a few seconds, and then walked out the door…. I thought I saw him shaking his head as he left.  Maybe he was just looking both ways for safety reasons, but to me, it looked like a shake of disbelief.  I wonder if I had given him the same pain in the neck.

That is really the end of the relay story, but I do want to say a few words about Charles Lay.  He was a hard working electrician that was nearing retirement.  People would come around to hear us discussing religion.  I am Catholic, and he went to a Fundamental Christian Church.  We would debate the differences between our beliefs and just Christian beliefs in general.  We respected each other during our time together, even though he was sure I am going to hell when I die.

People would come in just to hear our discussion for a while as we were cleaning and calibrating the relays.  One day Charles asked me if I could help him figure out how much he was going to receive from his retirement from the electric company.  He had only been working there for three years.  Retirement at that time was determined by your years of service.  So, three years didn’t give him too much.

When I calculated his amount, he was upset.  He said, “Am I going to have to work until I die?”  I said, “Well, there’s always your 401k and Social Security.”  He replied that he can’t live on Social Security.  I said, “Well, there’s your 401k.”  He asked, “What’s that?” (oh.  not a good sign).

I explained that it was a retirement plan where you are able to put money in taxed deferred until you take it out when you retire.  He said, “Oh.  I never put anything in something like that.”  My heart just sank as I looked in his eyes.  He had suddenly realized that he wasn’t going to receive a retirement like those around him who had spent 35 years working in the Power Plant.

When I left the plant after teaching Charles Lay how to test the relays, that was the last time I ever saw him.  I don’t know what became of Charles.  I figure he would be 83 years old today.  I wonder if he finally retired when he reached the 80 points for your age and years of service.  He would have never reached enough years of service to receive a decent amount of retirement from the Electric Company since he didn’t start working there until he was 60 years old.  That is, unless he’s still working there now.

As I said earlier in this post, Charles Lay was a very good worker.  He always struck me as the “Hardworking type”.  I often think about the time we spent together, especially when I hear about a power blackout somewhere.  — A word of caution to Power Companies…. keep your protective relays in proper working condition.  Don’t slack off on the Preventative Maintenance.  — I guess that’s true for all of us… isn’t it?  Don’t slack off on Preventative Maintenance in all aspects of your life.

Pain in the Neck Muskogee Power Plant Relay Testing

RapDon’t let the title fool you.  I love testing Power Plant Protective Relays.  There is a sense of satisfaction when you have successfully cleaned, calibrated and tested a relay that is going to protect the equipment you have to work on every day.  With that said, I was hit with such an unbelievable situation when testing Muskogee Relays in 1995 that I was left with a serious pain in the neck.

On August 14, 2003 the electric power in the Northeast United States and Canada went out.  The Blackout lasted long enough to be a major annoyance for those in the that region of the United States.

 

Map of the power blackout in 2003

Map of the power blackout in 2003

When I heard about how the blackout had moved across the region, I immediately knew what had happened.  I was quickly reminded of the following story.  I told my wife Kelly, “I know exactly why such a large area lost power!  They hadn’t done proper preventative maintenance on the Protective Relays in the substations!  Just like….”  Well…. I’ll tell you that part now:

I have mentioned in a couple of earlier posts that something always seemed a little “off” at the Muskogee Power Plant.  I had decided early on that while working there I would stick to drinking sodas instead of water.  See the post:  “Something’s In the Water at the Muskogee Power Plant“.  Even with that knowledge, I was still shocked at what I found while testing relays at the plant.

This story really begins when one Sunday at Muskogee when one of the Auxiliary Operators was making his rounds inspecting equipment.  He was driving his truck around the south edge of the Unit 6 parking lot on the service road.  He glanced over at a pump next to the road, and at first, he thought he was just seeing things.  After stopping the truck and backing up for a second glance, he was sure he wasn’t dreaming.  It’s just that what he was seeing seemed so strange, he wasn’t sure what was happening.

The operator could see what appeared to be silver paint chips popping off of the large pump motor in all directions.  After closer examination, he figured out that the motor was burning up.  It was still running, but it had become so hot that the paint was literally burning off of the motor.

Horizontal pump

Like this Horizontal pump only much bigger and painted silver

A motor like this would get hot if the bearings shell out.  Before the motor is destroyed, the protective relays on the breaker in the 4,000 Volt switchgear shuts the motor off.  In this case, the relay hadn’t tripped the motor, so, it had become extremely hot and could have eventually exploded if left running.  The operator shut the motor down and wrote a work order for the electricians.

Doyle Fullen was the foreman in the electric shop that received the work order.  When he looked into what had happened, he realized that the protective relay had no been inspected for a couple of years for this motor.

I couldn’t find a picture of Doyle.  In his youth he reminds me of a very smart Daryl in Walking Dead:

Norman Reedus from Walking Dead

Norman Reedus from Walking Dead

In fact, since before the downsizing in 1994, none of the Protective Relays at the plant had been inspected.  The person that had been inspecting the relays for many years had moved to another job or retired in 1994.  This was just a warning shot across the bow that could have had major consequences.

No one at Muskogee had been trained to test Protective Relays since the downsizing, so they reached out to our plant in North Central Oklahoma for help.  That was when I was told that I was going to be going to Muskogee during the next overhaul (outage).  I had been formally trained to inspect, clean, calibrate and test Protective Relays with two of my Power Plant Heroes, Ben Davis and Sonny Kendrick years earlier.  See the post:  “Relay Tests and Radio Quizzes with Ben Davis“.

My Protective Relay Maintenance course book

My Protective Relay Maintenance course book

Without going into too much detail about the actual tests we performed as I don’t want to make this a long rambling post (like… well…. like most of my posts…..I can already tell this is going to be a long one), I will just say that I took our antiquated relay tester down to Muskogee to inspect their relays and teach another electrician Charles Lay, how to perform those tests in the future.  Muskogee had a similar Relay Test Set.  These were really outdated, but they did everything we needed, and it helped you understand exactly what was going on when you don’t have a newfangled Relay Test Set.

AVO Multi-Amp SR-76 Relay Test Set

AVO Multi-Amp SR-76 Relay Test Set

You need to periodically test both mechanical and electronic protective relays.  In the electronic relays the components change their properties slightly over time, changing the time it takes to trip a breaker under a given circumstance (we’re talking about milliseconds).  In the mechanical relays (which I have always found to be more reliable), they sit inside a black box all the time, heating up and cooling as the equipment is used.  Over time, the varnish on the copper coils evaporates and settles on all the components.  This becomes sticky so that the relay won’t operate at the point where it should.

A panel of Protective Relays

A panel of Protective Relays

In the picture above, the black boxes on the top, middle and right are mechanical relays.  This means that something actually has to turn or pick up in order to trip the equipment.  The electronic relays may have a couple of small relays, but for the most part, they are made up of transistors, resistors, capacitors and diodes.

So, with all that said, let me start the real story…. gee…. It’s about time…

So, here I am sitting in the electric shop lab just off of the Unit 6 T-G floor.  We set up all the equipment and had taken a couple of OverCurrent relays out of some high voltage breakers in the switchgear.  I told Charles that before you actually start testing the relays, you need to have the test documents from the previous test and we also needed the instruction manuals for each of the relays because the manuals will have the diagrams that you use to determine the exact time that the relays should trip for each of the tests.  So, we went up to the print room to find the old tests and manuals.  Since they weren’t well organized, we just grabbed the entire folder where all the relays tests were kept since Unit 6 had been in operation.

When we began testing the relays at first I thought that the relay test set wasn’t working correctly.  Here I was trying to impress my new friend, Charles Lay, a 63 year old highly religious fundamental Christian that I knew what I was doing, and I couldn’t even make a relay trip.  I was trying to find the “As Found” tripping level.  That is, before you clean up the relay.  Just like you found it.  Only, it wouldn’t trip.

It turned out that the relay was stuck from the varnish as I explained above.  It appeared as if the relay hadn’t been tested or even operated for years.  The paperwork showed that it had been tested three years earlier.  Protective Relays should be tested at least every two years, but I wouldn’t have thought that the relay would be in such a bad condition in just three years.  It had been sitting in a sealed container to keep out dust.  But it was what it was.

I told Charles that in order to find the “As Found” point where the relay would trip, we would need to crank up the test set as high as needed to find when it actually did trip.  It turned out that the relay which should have instantaneously tripped somewhere around 150 amps wouldn’t have tripped until the motor was pulling over 4,000 amps.  I could tell right away why the Auxiliary Operator found that motor burning up without tripping.  The protective relays were stuck.

As it turned out… almost all of the 125 or so relays were in the same condition.  We cleaned them all up and made them operational.

There is an overcurrent relay for the main bus on each section of a main switchgear.

A picture of a clean switchgear. Picture 6 rows of switchgear like this

A picture of a clean switchgear. Picture 6 rows of switchgear like this

When I tested the “As Found” instantaneous trip for the main bus relay, I found that it was so high that the Unit 6 Main Turbine Generator would have melted down before the protective relay would have tripped the power to that one section of switchgear.  The entire electric bus would have been nothing but molten metal by that time.

As I tested each of these relays, I kept shaking my head in disbelief.  But that wasn’t the worst of it.  The mystery as to why these relays were all glued shut by varnish was finally solved, and that reason was even more unbelievable.

Here is what I found…..  The first thing you do when you are going to test a relay is that you fill out a form that includes all the relay information, such as, what it is for, what are the settings on the relay, and what are the levels of tests that you are going to perform on it.  You also include a range of milliseconds that are acceptable for the relay for each of the tests.  Normally, you just copy what was used in the previous test, because you need to include the time it took for the Previous “As Left” test on your form.  That is why we needed the forms from the previous test.

So, I had copied the information from the previous test form and began testing the relay (one of the first overcurrent relays we tested)…  Again… I was a 34 year old teacher trying to impress my 63 year old student.  So, I was showing him how you mechanically adjust the relay in order for it to trip within the acceptable range.  No matter how hard I tried, I couldn’t adjust the relay so that it would even be close to the desired range for the longer time trip times…. like the 2 second to 25 second range.  It wasn’t even close to the range that was on the form from the last test.

The form from the last test showed that the relay was in the right range for all the levels of test.  When I tested it, like I said, it wasn’t even close.  So, I went to the diagram in the instruction manual for this type of relay.  The diagram looks similar to this one used for thermal overloads:

Thermal Overload Tripping curves

Thermal Overload Tripping curves

See all those red lines?  Well, when you setup a relay, you have a dial where you set the range depending on the needs for the type of motor you are trying to trip. Each red line represents each setting on the dial.  Most of the relays were set on the same number, so we would be using the same red line on the diagram to figure out at different currents how long it should take for a relay to trip….

Here is the clincher….The time range that was written on the previous form wasn’t for the correct relay setting.  The person that tested the relay had accidentally looked at the wrong red line.  — That in itself is understandable, since it could be easy to get on the wrong line… The only thing is that as soon as you test the relay, you would know that something is wrong, because the relay wouldn’t trip in that range, just like I had found.

I double and triple checked everything to make sure we were looking at the same thing.  The previous form indicated the same settings on the relay as now, yet, the time ranges were for a different line! — Ok.  I know.  I have bored you to tears with all this stuff about time curves and overcurrent trips… so I will just tell you what this means…

This meant that when the person completed the forms the last time, they didn’t test the relays at all.  They just filled out the paperwork.  They put in random values that were in the acceptable range and sat around in the air conditioned lab during the entire overhaul smoking his pipe. — Actually, I don’t remember if he smoked a pipe or not.  He was the Electrical Specialist for the plant.  I remembered seeing him sitting in the lab with a relay hooked up to the test set throughout the entire overhaul, but I realized finally that he never tested the relays.  He didn’t even go so far as try to operate them.

I went back through the records to when the plant was first “checked out”.  Doyle Fullen had done the check out on the relays and the test after that.  Doyle had written the correct values from the manual on his forms.  I could see where he had actually performed the tests on the relays and was getting the same values I was finding when I tested the relays, so I was certain that I wasn’t overlooking anything.

As I tested each of the relays, I kept shaking my head in disbelief.  It was so unbelievable.  How could someone do such a thing?  Someone could have been killed because a protective relay wasn’t working correctly.  This was serious stuff.

One day while Charles and I were working away on the relays, Jack Coffman, the Superintendent of all the Power Plants came walking through the lab.  He asked us how we were doing.  I swiveled around in my chair to face him and I said, “Pretty good, except for this pain in my neck” as I rubbed the back of my neck.

Jack stopped and asked me what happened.  I told him that I had been shaking my head in disbelief for the last two weeks, and it gave me a pain in the neck.  Of course, I knew this would get his attention, so he asked, “Why?”  I went through all the details of what I had found.

I showed him how since the time that Doyle Fullen last tested the relays more than 10 years earlier, these relays hadn’t been tested at all.  I showed him how the main bus relays were so bad that it would take over 100,000 amps to have tripped the 7100 KV switchgear bus.

Jack stood there looking off into space for a few seconds, and then walked out the door…. I thought I saw him shaking his head as he left.  Maybe he was just looking both ways for safety reasons, but to me, it looked like a shake of disbelief.  I wonder if I had given him the same pain in the neck.

That is really the end of the relay story, but I do want to say a few words about Charles Lay.  He was a hard working electrician that was nearing retirement.  People would come around to hear us discussing religion.  I am Catholic, and he went to a Fundamental Christian Church.  We would debate the differences between our beliefs and just Christian beliefs in general.  We respected each other during our time together, even though he was sure I am going to hell when I die.

People would come in just to hear our discussion for a while as we were cleaning and calibrating the relays.  One day Charles asked me if I could help him figure out how much he was going to receive from his retirement from the electric company.  He had only been working there for three years.  Retirement at that time was determined by your years of service.  So, three years didn’t give him too much.

When I calculated his amount, he was upset.  He said, “Am I going to have to work until I die?”  I said, “Well, there’s always your 401k and Social Security.”  He replied that he can’t live on Social Security.  I said, “Well, there’s your 401k.”  He asked, “What’s that?” (oh.  not a good sign).

I explained that it was a retirement plan where you are able to put money in taxed deferred until you take it out when you retire.  He said, “Oh.  I never put anything in something like that.”  My heart just sank as I looked in his eyes.  He had suddenly realized that he wasn’t going to receive a retirement like those around him who had spent 35 years working in the Power Plant.

When I left the plant after teaching Charles Lay how to test the relays, that was the last time I ever saw him.  I don’t know what became of Charles.  I figure he would be 83 years old today.  I wonder if he finally retired when he reached the 80 points for your age and years of service.  He would have never reached enough years of service to receive a decent amount of retirement from the Electric Company since he didn’t start working there until he was 60 years old.  That is, unless he’s still working there now.

As I said earlier in this post, Charles Lay was a very good worker.  He always struck me as the “Hardworking type”.  I often think about the time we spent together, especially when I hear about a power blackout somewhere.  — A word of caution to Power Companies…. keep your protective relays in proper working condition.  Don’t slack off on the Preventative Maintenance.  — I guess that’s true for all of us… isn’t it?  Don’t slack off on Preventative Maintenance in all aspects of your life.

Crossfunctional Power Plant Dysfunction

The coal-fired Power Plant in North Central Oklahoma had gone from 360 employees in 1987 down to 124 employees on August 1, 1994 after the second downsizing.  Monday morning when we arrived at work, the maintenance department met in the main break room to be told how we were going to survive the loss of 100 employees.  With only 7 electricians left, I kept trying to add up on my fingers how we could possibly keep up with all the work we had to do.

Jasper Christensen stood up and after saying that he understood how we must feel about our present situation, he told us that we will have to each work harder.  I shook my head in disbelief (inside my head only… I didn’t really shake my head, as it was frozen with the same blank stare everyone else was wearing).  I knew we weren’t going to be working harder.  — What does that really mean anyway.  I thought he should have said, “We will each have to work “smarter” because we can’t really work “harder”.  Jasper was a nice person, but he never really was much for words so I gave him a pass on this one.

Jasper Christensen

Jasper Christensen

Interestingly, the three people in charge at the plant, Jasper, Jim Arnold and Bill Green were all 53 years old, and only within 4 months in age from each other.  They all belonged to the “old school way of doing things” (see the post:  “From Pioneers to Power Plant Managers“).  As Jasper continued in his speech I noticed that gone was any talk of working together to achieve our goals.  I immediately felt that we had just rolled back our management to a time before our first downsizing in 1987 when the Evil Plant Manager used to rule the plant with an iron fist.

I felt this way because we were being told how we were going to change everything we do without giving any of our own input.  For instance, we would no longer have a Quality Action Team.  That was disbanded immediately.  We would no longer hold Quality Team meetings (we were also told that the Quality process was not going away, though we couldn’t see how it was going to work).  The Safety Task Force did survive.

We were also told that we would no longer fill out any forms unless they are requested by someone.  It seems that we had over 1,300 forms that were being filled out at the plant and most of them were never being used for anything, so, unless someone requested a form, we wouldn’t just fill them out for the sake of filling them out.  This was actually a good idea.  I know we filled out forms in triplicate each week when we did transformer and substation inspections.  Most of those were never looked at, I’m sure.

It turned out later that we needed only about 400 of the 1300 forms our plant was churning out each month.

We were told we wouldn’t be doing Substation inspections.  That was not our responsibility.  It would be done by the Transmission and Distribution division instead.  I was beginning to see how management was trying to figure out how 7 electricians were going to “work harder”.  The answer at the moment was that we were going to do less.  The purpose of the Substation and Transformer checks each week was to look for problems while they were minor instead of waiting for a catastrophe to happen.

We were told that we were not going to “Gold Plate” our work.  We were going to just do what it took to complete the task without worrying about polishing it up to make it “perfect” (which is what real Power Plant Men do).  Instead we were going to “Farm Fix it”.  I’ll go more into this subject with a separate post.

We were then told that we would no longer have an Electric Shop and an Instrument and Controls shop.  We would from then on all meet in the Mechanical Maintenance shop.  We were not supposed to go to the Electric Shop or the Instrument and Controls shops for breaks because we were all going to be cross-functional.  We are all Maintenance now.  No longer specialized (sort of).

We were going to have four Maintenance teams.  Each one will have mechanics, welders, machinists, electricians and Instrument and controls people.  Each member on each team would learn to do each other’s jobs to a degree.

An electrician will learn how to tack weld.  A mechanic will learn how to run conduit and pull wire.  An instrument and controls person will learn how to use the lathe.  We would each learn enough about each job in order to perform minor tasks in each area without having to call the expert in that skill.

When the meeting was over, we each met with our own foremen.  Alan Kramer was my new foreman.  He used to be a foreman in the Instrument and Controls shop.

Alan Kramer

Alan Kramer

It became apparent that even though Jasper had come across as if everything had already been decided and that this was the way it was going to be, things hadn’t really been ironed out yet.  Actually, this was just a first pass.  The main goal was for us to figure out how to get all the work done that needed to be done.  I was still an electrician and I was still responsible for working on electrical jobs.

One really good part of the new situation was that I was now on the same team as Charles Foster.  We had always been very good friends, but I hadn’t worked on the same team as Charles since my first year as an electrician in 1984, ten years earlier when he was my first foreman in the electric shop (See the post:  “New Home in the Power Plant Electric Shop“).  We were the two electricians on Alan Kramer’s team.

Charles Foster

Charles Foster

Besides the fact that everyone was very bitter over the despicable treatment of our fellow Power Plant Men that were laid off the previous Friday (see the post: “Power Plant Downsizing Disaster and the Left Behinds“), we knew that we had to figure out how to make this new arrangement work.  We knew our upper management was using the old tyrannical style of management, but we also knew that at this point, they needed every one of us.  They couldn’t go around firing us just because we spoke our mind (which was good for me, because, I was still in the process of learning how to keep my mouth shut when that was the most beneficial course of action).

As Dysfunctional as our upper management seemed to be at the moment, our new teams embraced the idea of our new Cross-Functional teams with some minor changes.  First, we still needed to see ourselves as electricians, instrument and controls, machinists, welders and mechanics.  We each had our own “certifications” and expertise that only a person with that trade could perform.

Charles and I would still go to the electric shop in the morning before work began, and during lunch and breaks.  Our electric equipment to perform our job was there, and we still needed to maintain a stock of electric supplies.  The same was true for the Instrument and Controls crew members.

Even today, after having been gone from the Power Plant for 13 1/2 years, the electric shop office phone still has my voice on the voice mail message.  I know, because a couple of years ago, when it was accidentally erased, Tim Foster (Charles Foster’s son), asked me to record a new message so they could put it back on the phone.  I considered that a great honor to be asked by True Power Plant Men to record their voice mail message on the electric shop phone.  The Phone number by the way is:  (405) 553-29??.  Oh.  I can’t remember the last two digits.  🙂

Once the kinks were worked out of the cross-functional team structure, it worked really well.  I just kept thinking…. Boy, if we only had a group of supportive upper management that put their plant first over their own personal power needs, this would be great.  The True Power Plant Men figured out how to work around them, so that in spite of the obstacles, within about 4 years, we had hit our stride.

Let me give you an example of how well the cross-functional teams worked compared the old conventional way we used to work.  I will start by describing how we used to do things….  Let’s say that a pump breaks down at the coal yard…

Horizontal pump

Horizontal pump

— start here —

An operator creates the Maintenance Order (M.O.).  It is eventually assigned to a crew of mechanics.  (start the clock here).  When they have time, they go to the coal yard to look over the problem.  Yep.  The pump is not working.  They will have to take it back to the shop to fix it.

A Maintenance Order is created for the electricians to unwire the motor.  The electricians receive the maintenance order and prioritize it.  They finally assign it to a team to go work on it.  Say, in one week from the time they received the M.O.  The electrician goes to the control room to request a clearance on the pump.  The next day the electrician unwires the motor.  They complete the maintenance order at the end of the day and send it back up to the A Foreman.

The completed electric maintenance order is sent back to the mechanics letting them know that the motor for the pump has been unwired.  When they receive it, a couple of days later, they schedule some time that week to go work on the pump.  At that time, they bring the motor to the electric shop so that it can be worked on at the same time.

The motor and the pump is worked on some time during the next week.

A machinist is needed to re-sleeve a bearing housing on either the motor or the pump or both.  So, an M.O. is created for the machinist to work on creating a sleeve in an end bell of the motor or the pump.

Gary (Stick) McCain

Gary (Stick) McCain — Machinist Extraordinaire

The electricians inform the mechanics when the motor is ready.  When they are done with the pump, and they have put it back in place, they put the motor back.  Then they create an M.O. for the Machinist to line up the motor and the pump before the coupling is installed.

The Machinists prioritize their work and at some point, let’s say a couple of days, they make it up to the motor and work on aligning the pump and the motor.

During the re-installation, it is decided that a bracket that has worn out needs to be welded back.  So, an M.O. is created for the welders to replace the bracket before the motor can be rewired.

The welders prioritize their work, and in a week (or two) they finally have time to go weld the bracket.

George Clouse

George Clouse – Welding Wizard

They return their M.O. completed to the mechanics who then tell the electricians that they can re-wire the motor.

The electricians prioritize their work and when they have time to go re-wire the motor, they wire it up.  After wiring it, they go to the control room to have the operators help them bump test the motor to make sure it runs in the right direction.  An entire day goes by until the electrician receives a call saying that the operator is ready to bump test the motor.  The electrician and/or mechanic meets the operator at the pump to bump test the motor.  Once this test is performed, the mechanic re-couples the motor.

The electrician then removes his clearance on the pump and it is put back into service.  The M.O.s are completed.

—  End here.  The time it took to repair the pump and put it back in service would commonly take one month —

Now see what happens when you have a cross-functional team working on it….(and be amazed).

— Start here —

The maintenance team receives a ticket (M.O.) from the planner that a pump is broken at the coal yard.  A mechanic goes and looks at it and determines it needs to be repaired.  He calls his Electrician Teammate and tells him that the motor needs to be unwired in order to fix the pump.  The electrician goes to the control room and takes a clearance on the pump.

The electrician then goes to the switchgear and waits for the operator to place the clearance.  When that is completed, the electrician goes to the pump and unwires the motor.  While there, he helps the mechanic pull the motor and put it aside.  The electrician determines there if the motor needs to be worked on.  If possible, it is repaired in place, or the motor is brought to the electric shop at the same time as the pump.  It is determined that the pump needs to be worked on, so they work together to bring it to the shop where the mechanics work on the pump.  Any machinist work is done at that time.

When the pump is being put back in place, the bracket is found broken, so they call the welder on their team who comes up and welds it back on.  The machinist comes with the electrician and the mechanic to align the motor.  The operators are called to bump test the motor.  As soon as the test is over, the coupling is installed.  The clearance is removed and the pump is put back in place.

— End here.  The pump can now be repaired within one week instead of four weeks.  Often the pump can be repaired in days instead of weeks. —

The reason why the cross-functional teams worked so well is that we all had the same priority.  We all had the same job and we had all the skills on our team to do all the work.  This was a fantastic change from working in silos.

This was “Working Smarter”, not “Working Harder”.  Ever since that day when we first learned that we had to “Work Harder” I always cringe when I hear that phrase.  To me, “Working Harder” means, “Working Dumber”.  Today I am a big advocate of Cross-Functional Teams.  I have seen them work successfully.  There was only one catch which I will talk about later.  This worked beautifully, but keep in mind… We had cross-functional teams made of the best Power Plant Men on the planet!  So, I may have a lopsided view of how successful they really work in the general public.