Crossfunctional Power Plant Dysfunction
The coal-fired Power Plant in North Central Oklahoma had gone from 360 employees in 1987 down to 124 employees on August 1, 1994 after the second downsizing. Monday morning when we arrived at work, the maintenance department met in the main break room to be told how we were going to survive the loss of 100 employees. With only 7 electricians left, I kept trying to add up on my fingers how we could possibly keep up with all the work we had to do.
Jasper Christensen stood up and after saying that he understood how we must feel about our present situation, he told us that we will have to each work harder. I shook my head in disbelief (inside my head only… I didn’t really shake my head, as it was frozen with the same blank stare everyone else was wearing). I knew we weren’t going to be working harder. — What does that really mean anyway. I thought he should have said, “We will each have to work “smarter” because we can’t really work “harder”. Jasper was a nice person, but he never really was much for words so I gave him a pass on this one. After all, he never really took a course in motivational speaking.
Interestingly, the three people in charge at the plant, Jasper, Jim Arnold and Bill Green were all 53 years old, and only within 4 months in age from each other. They all belonged to the “old school way of doing things” (see the post: “From Pioneers to Power Plant Managers“). As Jasper continued in his speech I noticed that gone was any talk of working together to achieve our goals. I immediately felt that we had just rolled back our management to a time before our first downsizing in 1987 when the Evil Plant Manager used to rule the plant with an iron fist.
I felt this way because we were being told how we were going to change everything we do without giving any of our own input. For instance, we would no longer have a Quality Action Team. That was disbanded immediately. We would no longer hold Quality Team meetings (we were also told that the Quality process was not going away, though we couldn’t see how it was going to work). The Safety Task Force did survive.
We were also told that we would no longer fill out any forms unless they are requested by someone. It seems that we had over 1,300 forms that were being filled out at the plant and most of them were never being used for anything, so, unless someone requested a form, we wouldn’t just fill them out for the sake of filling them out. This was actually a good idea. I know we filled out forms in triplicate each week when we did transformer and substation inspections. Most of those were never looked at, I’m sure.
It turned out later that we needed only about 400 of the 1300 forms our plant was churning out each month.
We were told we wouldn’t be doing Substation inspections. That was not our responsibility. It would be done by the Transmission and Distribution division instead. I was beginning to see how management was trying to figure out how 7 electricians were going to “work harder”. The answer at the moment was that we were going to do less. The purpose of the Substation and Transformer checks each week was to look for problems while they were minor instead of waiting for a catastrophe to happen.
We were told that we were not going to “Gold Plate” our work. We were going to just do what it took to complete the task without worrying about polishing it up to make it “perfect” (which is what real Power Plant Men do). Instead we were going to “Farm Fix it”. I’ll go more into this subject with a separate post.
We were then told that we would no longer have an Electric Shop and an Instrument and Controls shop. We would from then on all meet in the Mechanical Maintenance shop. We were not supposed to go to the Electric Shop or the Instrument and Controls shops for breaks because we were all going to be cross-functional. We are all Maintenance now. No longer specialized (sort of).
We were going to have four Maintenance teams. Each one will have mechanics, welders, machinists, electricians and Instrument and controls people. Each member on each team would learn to do each other’s jobs to a degree.
An electrician will learn how to tack weld. A mechanic will learn how to run conduit and pull wire. An instrument and controls person will learn how to use the lathe. We would each learn enough about each job in order to perform minor tasks in each area without having to call the expert in that skill.
When the meeting was over, we each met with our own foremen. Alan Kramer was my new foreman. He used to be a foreman in the Instrument and Controls shop.
It became apparent that even though Jasper had come across as if everything had already been decided and that this was the way it was going to be, things hadn’t really been ironed out yet. Actually, this was just a first pass. The main goal was for us to figure out how to get all the work done that needed to be done. I was still an electrician and I was still responsible for working on electrical jobs.
One really good part of the new situation was that I was now on the same team as Charles Foster. We had always been very good friends, but I hadn’t worked on the same team as Charles since my first year as an electrician in 1984, ten years earlier when he was my first foreman in the electric shop (See the post: “New Home in the Power Plant Electric Shop“). We were the two electricians on Alan Kramer’s team.
Besides the fact that everyone was very bitter over the despicable treatment of our fellow Power Plant Men that were laid off the previous Friday (see the post: “Power Plant Downsizing Disaster and the Left Behinds“), we knew that we had to figure out how to make this new arrangement work. We knew our upper management was using the old tyrannical style of management, but we also knew that at this point, they needed every one of us. They couldn’t go around firing us just because we spoke our mind (which was good for me, because, I was still in the process of learning how to keep my mouth shut when that was the most beneficial course of action).
As Dysfunctional as our upper management seemed to be at the moment, our new teams embraced the idea of our new Cross-Functional teams with some minor changes. First, we still needed to see ourselves as electricians, instrument and controls, machinists, welders and mechanics. We each had our own “certifications” and expertise that only a person with that trade could perform.
Charles and I would still go to the electric shop in the morning before work began, and during lunch and breaks. Our electric equipment to perform our job was there, and we still needed to maintain a stock of electric supplies. The same was true for the Instrument and Controls crew members.
Even today, after having been gone from the Power Plant for 13 1/2 years, the electric shop office phone still has my voice on the voice mail message. I know, because a couple of years ago, when it was accidentally erased, Tim Foster (Charles Foster’s son), asked me to record a new message so they could put it back on the phone. I considered that a great honor to be asked by True Power Plant Men to record their voice mail message on the electric shop phone. The Phone number by the way is: (405) 553-29??. Oh. I can’t remember the last two digits. 🙂
Once the kinks were worked out of the cross-functional team structure, it worked really well. I just kept thinking…. Boy, if we only had a group of supportive upper management that put their plant first over their own personal power needs, this would be great. The True Power Plant Men figured out how to work around them, so that in spite of the obstacles, within about 4 years, we had hit our stride.
Let me give you an example of how well the cross-functional teams worked compared to the old conventional way we used to work. I will start by describing how we used to do things…. Let’s say that a pump breaks down at the coal yard…
— start here —
An operator creates the Maintenance Order (M.O.). It is eventually assigned to a crew of mechanics. (start the clock here). When they have time, they go to the coal yard to look over the problem. Yep. The pump is not working. They will have to take it back to the shop to fix it.
A Maintenance Order is created for the electricians to unwire the motor. The electricians receive the maintenance order and prioritize it. They finally assign it to a team to go work on it. Say, in one week from the time they received the M.O. The electrician goes to the control room to request a clearance on the pump. The next day the electrician unwires the motor. They complete the maintenance order at the end of the day and send it back up to the A Foreman.
The completed electric maintenance order is sent back to the mechanics letting them know that the motor for the pump has been unwired. When they receive it, a couple of days later, they schedule some time that week to go work on the pump. At that time, they bring the motor to the electric shop so that it can be worked on at the same time.
The motor and the pump is worked on some time during the next week.
A machinist is needed to re-sleeve a bearing housing on either the motor or the pump or both. So, an M.O. is created for the machinist to work on creating a sleeve in an end bell of the motor or the pump.
The electricians inform the mechanics when the motor is ready. When they are done with the pump, and they have put it back in place, they put the motor back. Then they create an M.O. for the Machinist to line up the motor and the pump before the coupling is installed.
The Machinists prioritize their work and at some point, let’s say a couple of days, they make it up to the motor and work on aligning the pump and the motor.
During the re-installation, it is decided that a bracket that has worn out needs to be welded back. So, an M.O. is created for the welders to replace the bracket before the motor can be rewired.
The welders prioritize their work, and in a week (or two) they finally have time to go weld the bracket.
They return their M.O. completed to the mechanics who then tell the electricians that they can re-wire the motor.
The electricians prioritize their work and when they have time to go re-wire the motor, they wire it up. After wiring it, they go to the control room to have the operators help them bump test the motor to make sure it runs in the right direction. An entire day goes by until the electrician receives a call saying that the operator is ready to bump test the motor. The electrician and/or mechanic meets the operator at the pump to bump test the motor. Once this test is performed, the mechanic re-couples the motor.
The electrician then removes his clearance on the pump and it is put back into service. The M.O.s are completed.
— End here. The time it took to repair the pump and put it back in service would commonly take one month —
Now see what happens when you have a cross-functional team working on it….(and be amazed).
— Start here —
The maintenance team receives a ticket (M.O.) from the planner that a pump is broken at the coal yard. A mechanic goes and looks at it and determines it needs to be repaired. He calls his Electrician Teammate and tells him that the motor needs to be unwired in order to fix the pump. The electrician goes to the control room and takes a clearance on the pump.
The electrician then goes to the switchgear and waits for the operator to place the clearance. When that is completed, the electrician goes to the pump and unwires the motor. While there, he helps the mechanic pull the motor and put it aside. The electrician determines there if the motor needs to be worked on. If possible, it is repaired in place, or the motor is brought to the electric shop at the same time as the pump. It is determined that the pump needs to be worked on, so they work together to bring it to the shop where the mechanics work on the pump. Any machinist work is done at that time.
When the pump is being put back in place, the bracket is found broken, so they call the welder on their team who comes up and welds it back on. The machinist comes with the electrician and the mechanic to align the motor. The operators are called to bump test the motor. As soon as the test is over, the coupling is installed. The clearance is removed and the pump is put back in place.
— End here. The pump can now be repaired within one week instead of four weeks. Often the pump can be repaired in days instead of weeks. —
The reason why the cross-functional teams worked so well is that we all had the same priority. We all had the same job and we had all the skills on our team to do all the work. This was a fantastic change from working in silos.
This was “Working Smarter”, not “Working Harder”. Ever since that day when we first learned that we had to “Work Harder” I always cringe when I hear that phrase. To me, “Working Harder” means, “Working Dumber”. Today I am a big advocate of Cross-Functional Teams. I have seen them work successfully. There was only one catch which I will talk about later. This worked beautifully, but keep in mind… We had cross-functional teams made of the best Power Plant Men on the planet! So, I may have a lopsided view of how successful they really work in the general public.
Black Ops Raid Power Plant — Power Plant Men Ignore Attackers
I don’t know if they called them “Black Ops” in 1994, but when the control room operator David Evans answered the phone that day in October, I don’t think he ever expected to have the person on the other end of the line tell him that a military special forces unit was going to stage a mock raid on the coal-fired power plant in North Central Oklahoma some time that night. I’m sure Jack Maloy, the shift supervisor, was equally surprised when David told him about the phone call. I heard later that Jack was pretty upset to find out that a military force was going to be attacking our plant in the middle of the night without his permission!
The first we heard about the call was when Jasper Christensen called a meeting of the entire maintenance department on the spur of the moment in the main break room. He told us about the phone call. He said we didn’t have any more information than that. Though the maintenance department shouldn’t be working that night, Jasper said that just in case we were called out for something, we should know that a group of commandos were going to be performing some sort of mock raid on our plant. If we encountered any soldiers sneaking around the plant in the middle of the night in full military gear, not to be alarmed. Just go on doing what you’re doing and don’t bother them.
Now that it is 21 years later (well, almost) the truth can finally come out…. Isn’t that how it goes? When we are sworn to secrecy, isn’t it 21 years before we can finally speak out? (That’s what Shadow Warriors always told me). I don’t remember us taking an oath or anything, but that’s the way it is with Power Plant Men. They just assume that if the military is staging a mock raid on our plant, it is a matter of national security. It seemed as if our plant sort of matched the layout of a power plant somewhere in Central America where the real raid was going to take place.
The main difference between our Power Plant and the one in Honduras, or wherever it was, is that our plant had recently gone through a downsizing. So, our operators at night now had to perform the duties that had before been done by the labor crew. They had to do coal cleanup throughout the conveyor system.
This meant that if one of our auxiliary operators happened to run across someone dressed in the outfit above, they would have naturally handed him either a water hose or a shovel and pointed to the nearest conveyor and said something like, “I’ll start on this end, and you can start over there.” After all. He would already be wearing his respirator.
That day on the way home, Scott Hubbard and I discussed the significance of such a raid on our Power Plant. A year and a half earlier, Janet Reno had really messed up the raid on the Branch Davidian compound in Texas when it burned down and burned everyone to death including women and children. So, it would be good to go into a situation like this more prepared.
I had often thought about the steps that could covertly be taken to single-handed destroy the power plant without using any kind of explosives. Those who understood how all the systems worked together could do it if they really wanted to. Of course, that was just how I might occupy my mind when I was doing a repetitive job, like sweeping out the main switchgear. What better place for those thoughts to drift into your mind.
Actually, now that I think about it, instead of sending in the Special Forces, just send in a few Plant Operators, Electricians and Instrument and Controls guys and they could totally destroy the plant in a matter of hours if that was their intent. The same thing could be said about putting a few incompetent people in upper management even if it isn’t their intent, only it takes longer than a couple of hours to destroy the plant in that case.
The next morning when we arrived at the plant, our foreman Alan Kramer told us the stories about the raid that happened the night before. This is what I can remember about it (if any Power Plant Men want to correct me, or add some more stories, please do in the comments below).
First he said that it appeared as if the commandos had landed in some kind of stealth helicopter out on the north side of the intake because later when the power plant men had investigated the site they could see where two wheels on the helicopter had left an impression in the mud. Dan Landes had been keeping a lookout from the top of the Unit 1 boiler, and he thought for a moment that he saw the flash of a red light…. which… thinking about it now, could have been one of those laser sites taking aim at him and mock assassinating him by shooting him in the eye from about 1/2 mile. You know how good American Snipers can be (my plug for the new movie). Good thing he was wearing his auto-tinting safety glasses.
We also heard that one of the operators, Maybe Charles Peavler (Charles is standing next to Dan wearing the pink shirt and carrying something in his lower lip) had stepped out of the office elevator on the ground floor only to come face-to-face with a soldier. When the soldier was seen by the operator, he just turned around and walked out of the door… he evidently was considered a casualty if he was seen by anyone. Either that, or he had to go do coal cleanup the rest of the night.
I think it was Jeff Meyers (front row, left in the picture above) who told us later that the Special Ops forces had left a present for the operators on the Turbine-Generator Room floor. Tracked across the clean shiny red T-G floor were muddy boot prints leading from the Unit 1 boiler entrance to the door to the control room. The tracks ended at the control room door.
The tracks were extra muddy as if someone had intentionally wanted us to see that someone had walked right up to the control room door. The tracks did not lead away from the door. They just ended right there.
So, we did have proof that the commandos had actually visited our plant that night, only because one of the operators had come face-to-face with one in the main lobby. If that hadn’t happened, then they would have come and gone and we would have been none-the-wiser… other than wondering about the strange muddy footprints and the impression left in the mud by the stealth helicopter.
I suppose it was easy for the Power Plant operators to ignore the commandos since for the most part, they never saw them coming or going. The Power Plant Men were happy to play their part in the mock raid. Of all that has been asked of these Power Plant Men over the years, this was one of the more “unique” events. How many Power Plant Men across the country can say that they took part in a Special Ops Commando Raid on their Power Plant?
All I can say is that the commandos sure picked a great bunch of Power Plant Men and Women to attack. We were all honored (even those of us who were at home in bed asleep at the time) to be able to help out the military any way we could.
Final Battle for the Illusive Power Plant Safety Pizza
The Electric Shop had tried for three years to win the Safety Slogan of the Year award. Not because we thought we were safer than any of the other teams at the coal-fired power plant in North Central Oklahoma, but because we really liked pizza (see the post: “When Power Plant Competition Turns Terribly Safe“) . When the plant was downsized in 1994, the electric shop no longer existed as it had before. We had become cross-functional teams (See the post: “Crossfunctional Power Plant Dysfunction“). It looked as if our dream of winning the Power Plant Safety Pizza was no longer in our grasp.
My carpooling buddy, Toby O’Brien had moved from our plant as a Plant Engineer to the Safety Department in Oklahoma City. He was working with Julia Bevers and Chris McAlister. Chris had also moved from our plant as a labor crew hand to the Safety Department (This was a great opportunity for Chris!).
Bill Green our new plant manager introduced a jar of beads during his first safety meeting. We each picked a bead randomly from the jar through a small hole in the top. Then Bill Green pointed out that the color of bead represented the result of doing something unsafe.
The green color meant that nothing happened. The other colors reach represented a different type of accident that occurred. The ratio of beads in the jar represented the likelihood of each type of accident happening. There was one black bead in the jar. That meant that you died when you did something unsafe. I used to keep the number of each color of marble in my wallet, but that piece of paper disintegrated over the years.
The types of accidents were something like: First Aid Case, Reportable Accident, Lost Work Day Accident, Hospitalized, and Death.
A couple of months after the downsizing, the Safety department announced that they were going to have a Safety contest. The contest would be held at each plant and it involved each of the supervisor’s computers. The prize for the contest was that the winning team would be able to eat a free lunch with complements from the safety team.
Great! Shortly after the electric shop is busted up and we were scattered to the wind, we finally had one last chance to win the ever illusive Power Plant Safety Pizza! Only, how were we going to do it? I was working on Alan Kramer’s team. My old foreman Andy Tubbs (not old in the sense that he was an old man… old in that he was my former foreman) was now one of the other supervisors with only my old bucket buddy (you know what I mean… not “old” old) Diana Brien as the electrician on his team.
Before I go further to explain my conflict during this contest, let me explain how the contest worked.
The supervisors had new computers that ran using Windows 3.1. Back then, the screensaver on the computer didn’t just shut down the monitor like most of them do today. Instead, they showed some kind of message, or picture or something animated that kept moving around so that your monitor didn’t get burned in with an image that was constantly on your screen, such as your wallpaper and your icons.
The Safety Department said that each team should come up with some way to display the idea of “Safety” using a screensaver. They suggested using the screensaver that let you type in a message that would scroll across the screen when the screensaver was turned on. That was a simple built-in screensaver that came with Windows 3.1.
Then the Safety Department would come to the plant on a particular day and judge each of the computer’s screensaver and announce the winner. Sounds simple enough.
We first heard about the Safety Slogan Screensaver contest in our Monday Morning Meeting with our team. Alan Kramer said we should come up with a good slogan that we could put on our scrolling message screensaver. I kept my mouth shut at the time, because I didn’t know exactly how to proceed. I was having a feeling of mixed loyalty since my old Electric Shop Team with Andy Tubbs as our foreman had written over 300 safety slogans and had purposely been blocked from winning the Prized Pizza each year.
Not long after the morning meeting, Andy Tubbs came up to me in the Electric Shop and said, “We have to win this contest! That Pizza should be ours! I need you to come up with the best screensaver you can that will blow the others away.” I gave him my usual answer when Andy asked me to do something (even when he was no longer my foreman). I said, “Ok, I’ll see what I can do.”
I went down our list of safety slogans looking for the best slogan I could find. Here are a few of them:
“Having an accident is never convenient, So always make Safety a key ingredient.”
“Take the time to do it right, Use your goggles, save your sight.”
“To take the lead in the ‘Safety Race’, You must pay attention to your work place.”
“Unsafe conditions can be resolved, If we all work together and get involved.”
After thumbing through the entire list, I knew we really needed something else. So, I began to think of alternate screen savers. One caught my attention. It was called “Spotlight”. It came with the “After Dark 2.0 Screensavers” (best known for the “Flying Toaster” screensaver). I had found a freeware version that did the same thing. You can see how the spotlight works at 7:15 on the video below (just slide the time bar over to 7:15):
For those who can’t view YouTube videos directly through the above picture, here is the direct link: “After Dark Screensavers“.
The spotlight screensaver basically turns your screen dark, then has a circle (or spotlight) where you can see the background screen behind it. It roams around on your desktop showing only that portion of your wallpaper at a time. You can adjust the size of the circle and the speed that it moves around the screen.
Taking our safety slogans, I began creating a wallpaper for the computer screen by filling it with little one liner safety slogans. I also added yellow flags to the wallpaper because that was a symbol for safety at our plant (for more information why see the post: “Power Plant Imps and Accident Apes“).
With the help of Charles Foster and Scott Hubbard (both Power Plant electricians), when I was finished the wallpaper looked like this:
I printed this out in black and white, but the slogans were written in different colors.
I arranged Andy’s icons on his desktop so they were around the edge of the screen. That way they didn’t cover up the safety slogans. I set the speed of the spotlight to very slow and and the size of the spotlight so that it was just big enough to see each safety slogan. The effect worked out real well. Imagine a dark screen with a spotlight moving randomly around the screen exposing each safety slogan (and yellow flag… don’t forget about those) as it went.
Besides the electricians, no one else knew that I was working on this for Andy. As far as Alan Kramer knew, I was on his side in this contest. I even kept Toby O’Brien in the dark about it, because I knew that he was going to be one of the judges and even though he knew how much winning the Safety Pizza meant to me. I didn’t want to influence his decision. Besides, this Safety Screensaver was going to win. It was the coolest screensaver around. The trick was to keep it hidden from the other teams until it was time for the Safety Department to judge it.
I had the impression from Toby that he had purposely talked the Safety Department into this contest to give me a chance to win the Safety Pizza at our plant. Scott Hubbard and I had carpooled with Toby throughout the years we were trying to win that pizza, and I think he just felt our pain enough that when he was in the position, he was trying to pay us back for our effort.
The screensaver judging was done during the morning, and was going to be announced that afternoon during the monthly safety meeting. A short time before the Safety Meeting began, Toby O’Brien came up to me and in an apologetic manner told me that the safety slogan winner probably wasn’t going to be who I thought it was. I figured that was because he thought I was hoping Alan Kramer’s team was going to win since that was my team. I just smiled back and told him that it was all right.
It was announced during the safety meeting that Andy Tubbs’ team won the contest, and all the electricians were happy. I think it was at that point that Alan Kramer realized that I had helped Andy with his screensaver. He looked at me as if I had betrayed him. I said something like, “Andy Tubbs has been trying to win a safety contest for years. It’s about time.”
The following week, when Andy’s team was given their prize for winning the safety screensaver contest, he brought two pizzas to the electric shop and we all sat around the table relishing in the pepperonis. We had finally received our Power Plant Safety Pizza! Even though I really like pizza anytime, the pizza that day tasted especially good.
I don’t know if we ever told Toby that when Andy Tubbs team won, we all won. Maybe some day he will read this story and know…. “The Rest of the Story”.
In case you can’t read all the little safety slogans on the wallpaper, here is a list of them:
Safety First. Be Safe. Safety begins here. Watch your step. Check your boundaries. Have Good Posture. Haste makes waste. Bend your knees. Avoid Shortcuts. Be Safe or Be Gone. Know your chemicals. Check O2 before Entry. Use Safety Guards. Know your limit. Report Spills. Safety is job #1. Beware of Pinch Points. Buckle up. Safety is no accident. Impatience kills. Strive to Survive. Protect your hearing. Use the right tool. Keep your back straight. Drive friendly. Keep Aisles clear. Don’t take chances. Prevention is the cure. Safety is your job. Communicate with others. Always tie off. Don’t cut corners. Wear your glasses. Act safe. Barricade Hazards. Use your respirator. Be responsible. Lock it out. Plug your ears. Stay fit. Safety never hurts. Don’t block exits. Be aware of your surroundings. Safety is top priority. Don’t be careless. Pick up your trash. Think Ahead. Slippery When Wet. Think Safety. Don’t hurry. Report Hazards. Wear your gloves. Save your eyes. No Running. Wear your Safety Belt. Plan Ahead. Avoid Backing. Use your Safety Sense. Good Housekeeping. Get Help. Keep Cylinders Chained. Protect your hands. Don’t improvise. Beware of hazards. Get the Safety Habit. Be Prepared. Gear up for Safety. Use your PPE. Do not litter. Zero Accidents. Don’t be a Bead (a reference to Bill Green’s jar of beads). Eat Right. Keep Floors clean. Watch out. Safety Pays. Drive Safely. Take Safety Home. Know Safety, use Safety. Read the MSDS. Cotton Clothes Prevents Burns. Follow the rules. Wear your hard hat. Watch out for your buddy. Test your Confined space. Remember the Yellow Flag. Safe Mind, Sound Body. Clean up your spills. Don’t take risks. Beware of Ice. Watch out for the other guy. Obey the rules. Don’t tailgate. Circle for safety. Safety Me, Safety You. Protect your Toes. Knowing is not enough. When in doubt, Check it out. Falls can kill. Be Alert! Avoid slick spots. Safety is a team event. Almost is not enough. Avoid the Noise. Give Safety your all. And finally… This Space for Rent.
Flying Leap off of a Power Plant Hot Air Duct
I was standing in the elevator on my way to the control room from the electric shop the morning of October 11, 1995 when a strange call came over the radio. It sounded like Danny Cain, one of the Instrument and Controls Technicians on my team (or crew, as we used to call them before the reorganization). Most of what he said was garbled, but from what I could catch from Danny’s broadcast was that there was a man down on a unit 2 hot air duct. Danny’s voice sounded as if he was in a panic.
About that time, the elevator door opened and I stepped out. I thought to myself… “Hot Air Duct?” Where is a hot air duct? I had been running around this plant since 1979 when they were still building the plant, and for the life of me, I couldn’t remember where a hot air duct was at that moment. I may have been panicking myself. So, I did the only thing I could think of at the time….
I walked briskly into the control room and asked the Unit 1 Control Room operator… “Where is a hot air duct?” It must have sounded like a pretty stupid question coming from someone who rarely admitted that they didn’t know everything, but I have asked my share of stupid questions in my lifetime and most of the time, after the blank stare, someone gives me the answer. This time, the answer was “In the bottom ash area under the boiler.”
I quickly left the control room without saying another word. I’m not sure why I didn’t yell something out like, “Help! Help! There’s a man down on a Unit 2 Hot Air Duct!” I guess it didn’t even occur to me. I just darted out the door and down the stairs to the ground level (six flights of stairs). I kept saying to myself… “Hot Air Duct… Hot Air Duct…” I jogged across the Turbine-Generator basement floor and into the breezeway between the T-G building and Unit 2 Boiler. I picked up my pace once I was out in the open, and quickly made my way through the door to the spot where years before in 1983 I had seen Bob Lillibridge being swallowed alive by the Boiler Ghost (See the post: “Bob Lillibridge Meets the Boiler Ghost“).
Once inside the boiler enclosure, I realized that the Hot Air Ducts were the ducts that came from the Primary Air Fans into the Bowl Mills where the coal is ground into powder and blown into the boiler. The air is heated first by going through the Air Preheater which gets it’s heat from the exhaust from the boiler. It was pretty dark around the Hot Air Ducts.
As I approached I yelled out for Danny who immediately yelled in the same stressed far off voice “Help!” His voice came from the top of the second Hot Air Duct. I could hear a struggle going on up there, so I ran over to the ladder and quickly climbed up. When I reached the top of the ladder, I saw Danny Cain and Alan Kramer (Our foreman) also, wrestling with a man that I had never seen before.
Alan was behind him with his arms wrapped around the man’s arms and his legs wrapped around his chest (I’ll call this guy Michael, since that turned out to be his name…. Michael Hyde) as the man flailed his arms kicking his feet. The three of them were rolling around on the top of the Hot Air Duct (which is a coal dusty dark place). Oh. Did I mention that it was hot? Michael was a contract worker and this was his first day on the job.
I didn’t wait to see who was winning before I picked sides. I decided that whoever this Michael Hyde person was, if he was wrestling with Danny and Alan, he wasn’t getting much sympathy from me. I grabbed the man’s legs even though I was still standing at the top of the ladder.
I wrapped one of my legs through the rungs of the ladder so that it weaved through one rung, around the next rung, and into the third rung. At this point, even if I was knocked unconscious, I wasn’t going to fall off of this ladder. We were 25 feet above the concrete floor. Danny said, “Hold him down! He’s trying to jump off the Duct!” So, while Alan and Danny wrestled with Michael’s upper torso, I decided to take care of his legs.
About that time, others started showing up down below. Jimmie Moore was one of the first to arrive. I yelled down to him that we needed our rescue gear. A stretcher and rope. Jimmie quickly coordinated getting our safety bags down to the bottom ash area. More people arrived. I remember seeing Jasper Christensen looking up at me.
About that time, Michael pulled one of his feet from my grip and gave me a swift kick on the right side of my head with the heal of his boot. I was knocked back and my hardhat went flying off into space. Jasper said something like “Don’t fall off up there!” Shaking my head to get rid of the pain, I assured Jasper that I couldn’t fall off of the ladder if I tried. I had my leg locked in the rungs.
I decided at that point that the best thing I could do to keep from being kicked in the head again was to remove this guy’s boots. So, I grabbed both of his legs and squeezed them as hard as I could so that he would feel a little bit of the pain he had just inflicted on me. At the same time, I began unlacing his work boots and dropping them to the floor.
Danny explained that Michael had had some kind of seizure. He had fallen down and was wobbling all over the place. Then when it was over and he came to, he tried to jump off of the duct.
Michael continued to struggle. He wasn’t yelling or saying anything other than grunts. It was as if he was in a total panic. His eyes were filled with fear. After I had removed his boots, I continued to have one arm wrapped around both of this legs squeezing as hard as I could to keep him from pulling one away and taking another kick at my head.
At this point, an interesting idea came into my head… I suddenly thought it would be a good idea to tickle his feet. I had two reasons for doing this… First… It was pay back for leaving a boot heel impression across the side of my face and Second…. I thought it would distract him some from his panic. Sort of as a counter-irritant.
More of the rescue team had arrived and Jimmie threw a rope up to me that I threw over a pipe so that a stretcher could be raised up. Either Jimmie Moore or Randy Dailey or both then climbed up the ladder and made their way around me to the duct so they could put Michael in the stretcher. At this point, Michael was still panicking. He was still trying to escape our grasp.
The stretcher was placed alongside Michael, and with a “one two three” we raised him up and set him down in the stretcher. We started winding the rope in and out of the stretcher to tie him down. As soon as the ropes went around his arms, Michael stopped struggling and became calm.
It appeared that the moment Michael felt safe, the struggle was over. We raised him up from the hot air duct and then lowered him to the ground. As soon as he was on the ground, the Ambulance from Ponca City, Oklahoma arrived. Evidently, as soon as the Shift Supervisor had heard “Man Down” on the radio, he called 911 in Ponca City 20 miles away to have an ambulance sent. The timing couldn’t have been better. Just as we were untying Michael, the EMTs arrived.
The EMTs from the ambulance put him on their own stretcher. I picked up his boots from the ground and placed them alongside him on the stretcher and he was carried away. After all that panic, I just wanted to go back to the Electric Shop office and calm down.
This was a happy ending to what could have been a real tragedy if Michael had been able to jump off of the air duct. We heard about an hour later that when Michael Hyde had arrived at the hospital in Ponca City, he insisted on having a drug test taken to show that he had not taken any kind of illegal drug that would have led to his bizarre behavior. He said that nothing like that had ever happened to him before. There must have been something about the heat and the dark and the smell (and maybe listening to Danny talk about donuts) that must have triggered a seizure.
To our surprise, a few months later, Michael Hyde showed up at our plant again. This time, he came with a couple of other people. They were from the Oklahoma Safety Council. During our monthly safety meeting, many of us were presented with an award for rescuing Michael. Even though my part in the rescue was rather small, I was very proud to have been recognized that day. We were each given a plaque. Here is mine:
Alan Kramer and Danny Cain had both been invited to Oklahoma City to receive their awards as they were directly involved in saving Michael’s life. Here is the plaque that Alan was given:
A plaque was also given to the plant from the Oklahoma Safety Council which as far as I know is still mounted by the office elevator on the first floor:
Michael said he wanted to thank each of us personally for rescuing him that day. I shook his hand. I told him that I was the one that was squeezing his legs so hard and tickling his feet. I don’t think he remembered much of that moment.
I have a side story to this one and I wonder if this is the place to tell it. I suppose so, as long as I keep it short….
A number of months after this incident occurred, I began to develop a sore throat on the right side of my throat. I thought at first that it was Strep Throat because it hurt quite a lot. I just waited around for it to either go away or develop into a full blown cold, so I didn’t do anything for a few months. The pain was localized at one spot in my throat.
Then one day, my right ear began to hurt and I decided it was time to go see the doctor about it. When I visited my family doctor, he took a throat culture and surprisingly it came back clean. No strep throat. I explained that it was a constant pain, and now it had moved up to my ear as well. So, he sent me to Tulsa to an Ears Nose and Throat Doctor who was supposed to be real good.
When I went to him, after a couple of examinations, he came to the conclusion that I had a TMJ problem (TMJ is Temporomandibular Joint Dysfunction). So, he sent me to a TMJ person in Stillwater. I went to him, and he x-rayed my head and gave me some medicine to try to stop the pain, but nothing worked.
At times the right side of my face would hurt real bad, and at work if we took the temperature gun and pointed it at the right side of my face, we would find that it was about 2 or 3 degrees colder than the left side.
I went to a good TMJ person in Enid Oklahoma, and he made a mouthpiece for me to wear which seemed to lower the number of times each day that I would feel intense pain up and down the right side of my face…. all along, the pain continued to get worse.
To make a longer story shorter, through the years, the pain continued to grow until each time (about 6 times each day) my head would hurt, it would hurt from the top of my head down to my throat on the right side.
I never knew for sure if this was a result of being kicked in the head during our struggle that day, along with a combination of a bad dentist in Stillwater named Doctor Moore who wrestled with my teeth one day to the point that I wanted to punch him back. After going to many types of doctors (Neurologists, Endocrinologists, Oral Surgeons, etc) and dentists, having root canals, and medications and mouthpieces, nothing seemed to help. The pain continued to grow.
I finally had a name for my condition after many years, my dad pointed out to me that according to my symptoms, in the Merck Manual, my condition would be called: Trigeminal Neuralgia. I told that to the Neurologist and he agreed. There really wasn’t any treatment for it, only medication to cut down on the pain.
After 14 years, I decided I had taken enough pain medication and just decided to let the pain happen. It would only last for 10 minutes at a time, and medication didn’t really help that much anyway.
This past summer the pain became so frequent that every hour I was having 10 minutes of terribly excruciating pain that was leaving me almost immobile. So, I called the doctor to make and appointment. I had decided that this was too unbearable. On Tuesday, July 8, 2014 I called the doctor and made an appointment for July 17, the Thursday of the following week. On July 10, the pain was so bad that for the first time since I had this pain, I actually stayed home from work because I was not able to get any sleep.
July 11, 2014, After sitting up all night in a chair, I prepared to go to work. That morning, to my surprise, the pain had stopped coming every hour. That day at work, I had no pain. When I came home that evening, as I prepared to go out to dinner with my son Anthony, I felt the pain coming back. Anthony could tell, so we just waited a few minutes, and it was over. The pain wasn’t as harsh as it had been….. That was the last time I ever had an attack of pain on my face. It just went away and never came back.
It has now been over 6 months and I have not had one attack of pain on my face (With this repost, it has now been 5 years). After 16 years of pain, it just went away in one day. Could there have been a reason? I think so… You see, my mom had been doing something on the side. She had sent my name to a group to have them ask their founder to pray for me to God.
For years, my mom had supported a Catholic mission group called: Pontifical Institute For Foreign Missions. This organization was founded 164 years ago and the group was trying to have the Bishop who founded the Mission canonized as a Saint. In order to do that, they needed to have at least three miracles associated with his intercessory prayers. His name is Bishop Angelo Ramazzotti.
What this means is, that as a group, we ask that (pray to) Bishop Angelo Ramazzotti (who we believe is in Heaven with Jesus) ask Jesus to heal us (or something miraculous for a good cause). If the person (me, in this case) appears to have been miraculously and instantly healed, then an investigation is done to determine if it can be certified as a miracle.
If three such miracles have been certified after intercessory prayers have been asked of Bishop Angelo, then he could be Canonized as a Saint. To be Canonized means that he is more or less “Recognized” as someone that is in Heaven with God. It doesn’t mean in anyway that everyone in heaven has to be canonized. It just means that if you want to ask a Saint in Heaven to pray for you to God, then you can be pretty darn sure, this guy is good friends with God. Just like asking your own family to pray for you. You would be more inclined to ask someone that you know is more “Holy” if you really need to see some results.
Well, that is what my mom had done the week before the pain suddenly went away.
If the pain that I had for the past 16 years was caused in part by the kick in the head in 1995, and this condition I had is somehow used to help Canonize Bishop Angelo Ramazzotti, then I am glad to have been a part of it. All along, I figured that something good would come out of all that pain, I just never imagined what that might be. Now that it’s gone, it’s sort of like missing an old friend…. only….. NOT.
Sometimes it just takes a good kick in the head to get things moving.
Power Plant Networks to Condor Passwords
It had been established early on at the Coal-fired Power Plant in North Central Oklahoma that I was generally a troublemaker. All three of the Plant Managers that managed the plant during my tenure can attest to that. One Plant Manager, Ron Kilman, who reads this blog has been learning over the last couple of years, just what troublemaker I really was, as he reads these posts (oh. He knew I caused trouble when he was there, but not everything).
I was the “computer guy” at the plant. Though I was an electrician, the IT Support Department in Oklahoma City, 75 miles away deferred daily computer issues to me. The IT Networking Department had me run all their networking cables all over the plant, as I have mentioned before. Anywhere someone wanted a computer installed, I was the only person at the plant that would be assigned the task.
Even though I usually tell you stories about great Power Plant Men and their outstanding achievements, today, I must confess will be solely about myself. It will illustrate why I never could categorize myself as a “True Power Plant Man” like all the Heroes of Power Plant Fame. Even though there are countless ways I can demonstrate this, I will focus on the Plant Computer network and the role I played.
I was pretty much a self-study when it came to computer networks. A few trips to Hastings Bookstore, and I had enough Networking books to be dangerous. Though I never took a test to be certified, I read the Novell Netware 4 CNA (Certified Netware Administrator) manuals.
I also read books about different ways people hack into networks, such as the book, “Hacking Exposed”.
After pulling the 100 pair telephone cable from the back of the main switchgear to the coal yard by myself for the most part, and crawling through the ceilings in the office area stringing network cable through the rafters and punched down all the wires connecting them to the switches in the telephone room, I sort of felt like I owned the computer network at our plant.
After the server rack was installed, and the Novell Netware was up and running, then suddenly, I realized that the networking people downtown didn’t want some electrician poking his nose into their network. This was different than the mainframe. When we just had the mainframe, I had free reign to reek as much havoc on the system as I wanted…. Of course…. I never wanted to do that, it just happened sometimes. I chocked those times up as learning experiences.
The networking people downtown in Corporate Headquarters at that time had one major weakness…. They couldn’t administer the network remotely (this was 1995 and Windows servers were something new). So, when something needed to be done on the server at our plant, they had two choices.
They could get in a car (or truck) and drive 75 miles to our plant, then spend 10 minutes working on the network at our plant, then drive 75 miles back to Oklahoma City during going home traffic. Or, they could call me and have me connect them to the server using a modem and PC Anywhere (A software that allowed a person to remote into a computer and take control of it). Then from a computer on our network, they could log into the computer and access the server.
Needless to say, about once each week, I would go up to the engineer’s office to a computer that they would dial into. The computer had PC Anywhere installed and I would start up it up and grant them access to take control. While they were doing this, I would be talking to them on the phone. I could watch everything they were doing.
I could see the username they were using to log in, but like today, I couldn’t see the password they were using as it just came across as asterisks. I really wanted to be able to access the network myself. I thought it would help advance my knowledge so that when I did take the Netware CNA tests, I would have some hands on experience. I really wanted to become a Network Administrator. I guess I was sort of a Network Administrator Groupie at the time. I looked up to Network Administrators like they were guru’s with special knowledge.
I talked to the networking people in Oklahoma City to see if a lowly electrician like me could have some kind of limited network administrator account on the network so that I could learn about networking. I told them I was studying to become a System Administrator. They looked into it, but never came back with anything.
I had read about how hackers would capture passwords by capturing keystrokes from the keyboard. I had done something like this, only the other way around when I was writing little DOS prank programs that changed the values on the keyboards so that when you pressed an “A” it would come out as a “B” instead. I had one that would turn your caps lock on and leave the cap lock light off. I would have it on a timer, so that it could randomly make you type everything in CAPS in the middle of your sentence. You know… just fun little things like that. I suppose today, these would be categorized as viruses, if I had made them so that they would propagate across the network.
I knew how to manipulate the keyboard using things called “Interrupts”. So, I just reversed that process and using Debug, I was able to create a small assembly language program that would capture all the keystrokes from the keyboard and log them to a file. I had learned Assembly Language from Peter Norton, the same guy who later created Norton’s Utilities and Norton’s Anti-virus. Here is my book:
So, one day when the network guy from Oklahoma City dialed into the modem I tested the program to see if it would capture keystrokes even though they weren’t coming directly from the keyboard, but from PC Anywhere. To my surprise, when he had finished doing his task, and had logged off, I opened up the log file, and sure enough, all the keystrokes were logged. I could plainly see where he logged onto the server by typing in his username and password.
The password reminded me of a friend of mine from High School, because his e-mail address was Condor… something…. The password was: condor. So, I quickly logged into the server using the username and password and created a new Network Administrator account called something like: “Admin_sa” I gave it “God” access. So, after that I could log into the network and look around to see how the system was configured.
I know this was underhanded, and today would be highly illegal, but back then, all this network stuff was new and I was learning this along with the rest of the IT department downtown. The only difference was that I was an Electrician at a Power Plant many miles away. I only used that new Administrator account a few times to look at configuration settings as I read through the Netware books. I never changed any settings or did anything devious…. at least not when we were on the Novell Netware Network. I think the thrill of capturing a password and setting up my own account was enough.
My philosophy changed later when we moved to a Windows NT Network. That had so many holes in security that it deserved to be played with. It wasn’t too long later that the Netware Network was replaced, which made all my studying for the Netware Administrator useless. I couldn’t understand at the time why we would want to move away from such a secure network to one that had such a bad design that it left itself wide open to hackers (even today, 20 years later Microsoft still has to patch their servers every month!).
I could quickly write a Word document that would reformat your hard drive just by opening it up. In fact, Charles Foster one time asked me if I could come up with a way to install AOL on his aunt’s computer in California (or some such place), who knows nothing about computers. So, I created a Word document (since she did have Word on her computer already. and added a macro to it, that installed AOL and other software, and all she had to do was double click on the Word document icon. By the time it opened up to where she could read it, it had installed all the software she needed.
Once we were on the Windows network, the attitude of the IT network people changed. They were more flexible. They could maintain the network from downtown, so they only called me when they needed someone to log directly into the front of the server, which I did for them whenever they needed it. They began to feel more comfortable with me over time, and the support people downtown sort of granted me all the access I needed at the plant.
I think the reason I finally gained the trust of the IT Support team was because I would listen to their personal problems. This was something I had learned as a kid. I used to go around the neighborhood and make friends with all the dogs. That way, when we were playing hide and seek in the middle of the night, I could creep around behind houses, and the dogs wouldn’t bark at me. They would come up to me wagging their tails. It gave me a great advantage. So, by letting the IT Support people tell me about their personal problems, they would trust me. And then when I asked them for favors, they were happy to help out.
At that point (when we were on the Windows Network), I could sit in the Electric Shop and access every computer in the plant. For a few things, I had to actually visit a computer, but for a lot of things, I could just access the computer remotely. I have a few stories that I will tell this year that will give you some insight into how I used this power to better mankind…. well, I suppose it depends on how you look at it.
Later on, when I went to work for Dell in 2001, I put away all my “trouble causing” hacking stuff and decided that now that I am working in IT, I should join the Good Side of the Force. That didn’t mean that I didn’t do some fun stuff. Actually, some of the really good hacking stuff I had learned at the plant became very useful when I was in IT and could create applications on my own using the knowledge I had gained.
There was one time at Dell that I had to hack into database files that had crashed in order to extract the data. I would never have had the confidence to even try that if I hadn’t first learned programming from the ground up at the Power Plant.
I think it was Leslie Hale, a consulting manager from Concur (an expense reporting application) ask me at a Concur conference in 2010 how I hacked all of our credit card account numbers from their database when they were encrypted. He said his team had been trying to figure out how I could have done that so quickly. They normally charged $30,000 to migrate the credit card account numbers from their on-premise system to their hosted application. Of course, they have the encryption keys. I told them, I could do it myself by tomorrow and save the $30,000. They didn’t believe me, until the next day I uploaded a file to them with all the employees and account numbers. Dell was happy they didn’t have to pay the $30,000 for something that should have been part of the migration costs already.
I know I often caused our plant supervisor’s a few mild stomach ulcers. I think they just kept me around because either they felt sorry for me, or they thought that some day I might actually come to something. I finally left the plant in 2001 to pursue a life in IT at Dell. The journey to that end is another story, to be told later. Without all the support I received at the Electric Company, I never would have been able to make that change in my life. It all began one day when the Electric Supervisor, Tom Gibson told me in 1988 that he wanted me to learn all I could about computers. I guess, that was the moment when I began “expanding my bubble.”
Pain in the Neck Muskogee Power Plant Relay Testing
Don’t let the title fool you. I love testing Power Plant Protective Relays. There is a sense of satisfaction when you have successfully cleaned, calibrated and tested a relay that is going to protect the equipment you have to work on every day. With that said, I was hit with such an unbelievable situation when testing Muskogee Relays in 1995 that I was left with a serious pain in the neck.
On August 14, 2003 the electric power in the Northeast United States and Canada went out. The Blackout lasted long enough to be a major annoyance for those in the that region of the United States.
When I heard about how the blackout had moved across the region, I immediately knew what had happened. I was quickly reminded of the following story. I told my wife Kelly, “I know exactly why such a large area lost power! They hadn’t done proper preventative maintenance on the Protective Relays in the substations! Just like….” Well…. I’ll tell you that part now:
I have mentioned in a couple of earlier posts that something always seemed a little “off” at the Muskogee Power Plant. I had decided early on that while working there I would stick to drinking sodas instead of water. See the post: “Something’s In the Water at the Muskogee Power Plant“. Even with that knowledge, I was still shocked at what I found while testing relays at the plant.
This story really begins one Sunday at Muskogee when one of the Auxiliary Operators was making his rounds inspecting equipment. He was driving his truck around the south edge of the Unit 6 parking lot on the service road. He glanced over at a pump next to the road, and at first, he thought he was just seeing things. After stopping the truck and backing up for a second glance, he was sure he wasn’t dreaming. It’s just that what he was seeing seemed so strange, he wasn’t sure what was happening.
The operator could see what appeared to be silver paint chips popping off of the large pump motor in all directions. After closer examination, he figured out that the motor was burning up. It was still running, but it had become so hot that the paint was literally burning off of the motor.
A motor like this would get hot if the bearings shell out. Before the motor is destroyed, the protective relays on the breaker in the 4,000 Volt switchgear shuts the motor off. In this case, the relay hadn’t tripped the motor, so, it had become extremely hot and could have eventually exploded if left running. The operator shut the motor down and wrote a work order for the electricians.
Doyle Fullen was the foreman in the electric shop that received the work order. When he looked into what had happened, he realized that the protective relay had not been inspected for a couple of years for this motor.
I couldn’t find a picture of Doyle. In his youth he reminds me of a very smart Daryl in Walking Dead:
In fact, since before the downsizing in 1994, none of the Protective Relays at the plant had been inspected. The person that had been inspecting the relays for many years had moved to another job or retired in 1994. This was just a warning shot across the bow that could have had major consequences.
No one at Muskogee had been trained to test Protective Relays since the downsizing, so they reached out to our plant in North Central Oklahoma for help. That was when I was told that I was going to be going to Muskogee during the next overhaul (outage). I had been formally trained to inspect, clean, calibrate and test Protective Relays with two of my Power Plant Heroes, Ben Davis and Sonny Kendrick years earlier. See the post: “Relay Tests and Radio Quizzes with Ben Davis“.
Without going into too much detail about the actual tests we performed as I don’t want to make this a long rambling post (like… well…. like most of my posts…..I can already tell this is going to be a long one), I will just say that I took our antiquated relay tester down to Muskogee to inspect their relays and teach another electrician Charles Lay, how to perform those tests in the future. Muskogee had a similar Relay Test Set. These were really outdated, but they did everything we needed, and it helped you understand exactly what was going on when you don’t have a newfangled Relay Test Set.
You need to periodically test both mechanical and electronic protective relays. In the electronic relays the components change their properties slightly over time, changing the time it takes to trip a breaker under a given circumstance (we’re talking about milliseconds). In the mechanical relays (which I have always found to be more reliable), they sit inside a black box all the time, heating up and cooling as the equipment is used. Over time, the varnish on the copper coils evaporates and settles on all the components. This becomes sticky so that the relay won’t operate at the point where it should.
In the picture above, the black boxes on the top, middle and right are mechanical relays. This means that something actually has to turn or pick up in order to trip the equipment. The electronic relays may have a couple of small relays, but for the most part, they are made up of transistors, resistors, capacitors and diodes.
So, with all that said, let me start the real story…. gee…. It’s about time…
So, here I am sitting in the electric shop lab just off of the Unit 6 T-G floor. We set up all the equipment and had taken a couple of OverCurrent relays out of some high voltage breakers in the switchgear. I told Charles that before you actually start testing the relays, you need to have the test documents from the previous test and we also needed the instruction manuals for each of the relays because the manuals will have the diagrams that you use to determine the exact time that the relays should trip for each of the tests. So, we went up to the print room to find the old tests and manuals. Since they weren’t well organized, we just grabbed the entire folder where all the relays tests were kept since Unit 6 had been in operation.
When we began testing the relays at first I thought that the relay test set wasn’t working correctly. Here I was trying to impress my new friend, Charles Lay, a 63 year old highly religious fundamental Christian that I knew what I was doing, and I couldn’t even make a relay trip. I was trying to find the “As Found” tripping level. That is, before you clean up the relay. Just like you found it. Only, it wouldn’t trip.
It turned out that the relay was stuck from the varnish as I explained above. It appeared as if the relay hadn’t been tested or even operated for years. The paperwork showed that it had been tested three years earlier. Protective Relays should be tested at least every two years, but I wouldn’t have thought that the relay would be in such a bad condition in just three years. It had been sitting in a sealed container to keep out dust. But it was what it was.
I told Charles that in order to find the “As Found” point where the relay would trip, we would need to crank up the test set as high as needed to find when it actually did trip. It turned out that the relay which should have instantaneously tripped somewhere around 150 amps wouldn’t have tripped until the motor was pulling over 4,000 amps. I could tell right away why the Auxiliary Operator found that motor burning up without tripping. The protective relays were stuck.
As it turned out… almost all of the 125 or so relays were in the same condition. We cleaned them all up and made them operational.
There is an overcurrent relay for the main bus on each section of a main switchgear.
When I tested the “As Found” instantaneous trip for the main bus relay, I found that it was so high that the Unit 6 Main Turbine Generator would have melted down before the protective relay would have tripped the power to that one section of switchgear. The entire electric bus would have been nothing but molten metal by that time.
As I tested each of these relays, I kept shaking my head in disbelief. But that wasn’t the worst of it. The mystery as to why these relays were all glued shut by varnish was finally solved, and that reason was even more unbelievable.
Here is what I found….. The first thing you do when you are going to test a relay is that you fill out a form that includes all the relay information, such as, what it is for, what are the settings on the relay, and what are the levels of tests that you are going to perform on it. You also include a range of milliseconds that are acceptable for the relay for each of the tests. Normally, you just copy what was used in the previous test, because you need to include the time it took for the Previous “As Left” test on your form. That is why we needed the forms from the previous test.
So, I had copied the information from the previous test form and began testing the relay (one of the first overcurrent relays we tested)… Again… I was a 34 year old teacher trying to impress my 63 year old student. So, I was showing him how you mechanically adjust the relay in order for it to trip within the acceptable range. No matter how hard I tried, I couldn’t adjust the relay so that it would even be close to the desired range for the longer time trip times…. like the 2 second to 25 second range. It wasn’t even close to the range that was on the form from the last test.
The form from the last test showed that the relay was in the right range for all the levels of test. When I tested it, like I said, it wasn’t even close. So, I went to the diagram in the instruction manual for this type of relay. The diagram looks similar to this one used for thermal overloads:
See all those red lines? Well, when you setup a relay, you have a dial where you set the range depending on the needs for the type of motor you are trying to trip. Each red line represents each setting on the dial. Most of the relays were set on the same number, so we would be using the same red line on the diagram to figure out at different currents how long it should take for a relay to trip….
Here is the clincher….The time range that was written on the previous form wasn’t for the correct relay setting. The person that tested the relay had accidentally looked at the wrong red line. — That in itself is understandable, since it could be easy to get on the wrong line… The only thing is that as soon as you test the relay, you would know that something is wrong, because the relay wouldn’t trip in that range, just like I had found.
I double and triple checked everything to make sure we were looking at the same thing. The previous form indicated the same settings on the relay as now, yet, the time ranges were for a different line! — Ok. I know. I have bored you to tears with all this stuff about time curves and overcurrent trips… so I will just tell you what this means…
This meant that when the person completed the forms the last time, they didn’t test the relays at all. They just filled out the paperwork. They put in random values that were in the acceptable range and sat around in the air conditioned lab during the entire overhaul smoking his pipe. — Actually, I don’t remember if he smoked a pipe or not. He was the Electrical Specialist for the plant. I remembered seeing him sitting in the lab with a relay hooked up to the test set throughout the entire overhaul when I had been there during previous overhauls, but I realized finally that he never tested the relays. He didn’t even go so far as try to operate them.
I went back through the records to when the plant was first “checked out”. Doyle Fullen had done the check out on the relays and the test after that. Doyle had written the correct values from the manual on his forms. I could see where he had actually performed the tests on the relays and was getting the same values I was finding when I tested the relays, so I was certain that I wasn’t overlooking anything.
As I tested each of the relays, I kept shaking my head in disbelief. It was so unbelievable. How could someone do such a thing? Someone could have been killed because a protective relay wasn’t working correctly. This was serious stuff.
One day while Charles and I were working away on the relays, Jack Coffman, the Superintendent of all the Power Plants came walking through the lab. He asked us how we were doing. I swiveled around in my chair to face him and I said, “Pretty good, except for this pain in my neck” as I rubbed the back of my neck.
Jack stopped and asked me what happened. I told him that I had been shaking my head in disbelief for the last two weeks, and it gave me a pain in the neck. Of course, I knew this would get his attention, so he asked, “Why?” I went through all the details of what I had found.
I showed him how since the time that Doyle Fullen last tested the relays more than 10 years earlier, these relays hadn’t been tested at all. I showed him how the main bus relays were so bad that it would take over 100,000 amps to have tripped the 7100 KV switchgear bus or 710 Megawatts! More power than the entire generator could generate. It was only rated at about 550 Megawatts at the most.
Jack stood there looking off into space for a few seconds, and then walked out the door…. I thought I saw him shaking his head as he left. Maybe he was just looking both ways for safety reasons, but to me, it looked like a shake of disbelief. I wonder if I had given him the same pain in the neck.
That is really the end of the relay story, but I do want to say a few words about Charles Lay. He was a hard working electrician that was nearing retirement. People would come around to hear us discussing religion. I am Catholic, and he went to a Fundamental Christian Church. We would debate the differences between our beliefs and just Christian beliefs in general. We respected each other during our time together, even though he was sure I am going to hell when I die.
People would come in just to hear our discussion for a while as we were cleaning and calibrating the relays. One day Charles asked me if I could help him figure out how much he was going to receive from his retirement from the electric company. He had only been working there for three years. Retirement at that time was determined by your years of service. So, three years didn’t give him too much.
When I calculated his amount, he was upset. He said, “Am I going to have to work until I die?” I said, “Well, there’s always your 401k and Social Security.” He replied that he can’t live on Social Security. I said, “Well, there’s your 401k.” He asked, “What’s that?” (oh. not a good sign).
I explained that it was a retirement plan where you are able to put money in taxed deferred until you take it out when you retire. He said, “Oh. I never put anything in something like that.” My heart just sank as I looked in his eyes. He had suddenly realized that he wasn’t going to receive a retirement like those around him who had spent 35 years working in the Power Plant.
When I left the plant after teaching Charles Lay how to test the relays, that was the last time I ever saw him. I don’t know what became of Charles. I figure he would be 83 years old today. I wonder if he finally retired when he reached the 80 points for your age and years of service. He would have never reached enough years of service to receive a decent amount of retirement from the Electric Company since he didn’t start working there until he was 60 years old. That is, unless he’s still working there now.
As I said earlier in this post, Charles Lay was a very good worker. He always struck me as the “Hardworking type”. I often think about the time we spent together, especially when I hear about a power blackout somewhere. — A word of caution to Power Companies…. keep your protective relays in proper working condition. Don’t slack off on the Preventative Maintenance. — I guess that’s true for all of us… isn’t it? Don’t slack off on Preventative Maintenance in all aspects of your life.
Added note: On 7/6/2019, 3 weeks after re-posting this story, look what happened: Con Edison says cause of NYC blackout was substation’s faulty relay protection system
Power Plant Quittin’ Time
At a Power Plant, three things are certain: Death, Taxes and Quittin’ Time. Nothing can stand in the way of any of these three activities. The only time Quittin’ time might change is on a Friday afternoon just before it is time to go home and you hear the Shift Supervisor paging one of the foremen or the Maintenance Supervisor. Then you know that Quittin’ time is likely to change at the spur of the moment. Not eliminated, but only delayed. I suppose we try doing that with Death as well. I have never tried delaying Taxes before.
After the downsizing at the Coal-fired Power Plant in North Central Oklahoma in 1994, a lot of things had changed. As an electrician, I was now working on a cross-functional team with Charles Foster as my electrical bucket buddy.
The rest of my team had different skills. Some were Instrument and Controls, others were Welders, Machinists, Mechanics and then there was Alan Kramer, our foreman.
The new way we received work orders (we called them Maintenance Orders or MOs) was from our new Planners. There were two people responsible for figuring out our work for the week. That was Ben Davis and Tony Mena. I don’t have a photo of Tony, but here is Ben.
I have talked about Ben Davis in a number of past posts, as he was my mentor when I first became an electrician. I always looked up to him as a big brother. And, well, he treated me as a younger brother… but always with more respect than I deserved. Tony on the other hand was originally hired to be on the Testing Team when I was on the Labor Crew.
I still remember Monday, July 18, 1983 watching Tony Mena and the rest of the new Testers walking around the plant following Keith Hodges around like baby quail following their mother (at least that was the way Ron Luckey described them as we watched them from the back seat of the crew cab as we drove past them).
The men and woman on Labor Crew had felt passed over when the new testing team had been formed because no one on the labor crew had been considered for the new jobs even when we met the minimum requirements (which was to have any kind of college degree). So, even though it wasn’t fair to the new testing team, we had an immediate animosity toward them.
After the first downsizing in 1988, Scott Hubbard had moved to the electric shop and I quickly learned that not all testers were rotten, job stealing chumps. Actually, none of them were. They never had anything to do with who was chosen for the Testing team. That came from above. If you are interested, you can read the post: “‘Take a Note Jan’ Said the Supervisor of Power Production“. Scott and I became like brothers when he joined our team.
After the first downsizing, the testing team was reduced down to three people, Tony Mena, Richard Allen and Doug Black. I don’t have a picture of the first two, but I do have one of Doug:
After the second downsizing, the Testing team was eliminated. Scott had become an electrician seven year earlier, Doug Black moved into the Engineering Department. Richard Allen became an Instrument and Controls person and Tony Mena became a Planner along with Ben Davis. We had two other planners Glenn Rowland and Mark Fielder (who later traded with Mike Vogle to become a foreman). Glenn and Mark spent their time planning major outages, where Tony and Ben did more of the day-to-day stuff.
Tony Mena no longer had anyone to carpool with, so he asked me if he could carpool with Scott and I. So, we agreed. We told Tony that it was important to be on time, because we didn’t want to be late arriving at the plant, and we definitely didn’t want to be late going home (which was much more important). Tony agreed that he would be on time.
Quittin’ Time at the plant is a very important and orchestrated event. It begins a half hour earlier when everyone returns to the shop and cleans up and puts their tools away. Then they go into the foremen’s office and fills out their timecards for the day. This includes adding each of the maintenance orders we have worked on during the day and how many hours on each.
The next step is to grab your lunch box and go stand by the door to wait until the exact second that it is time to leave. When that happens, a steady stream of Power Plant Men pour into the parking lot, into their pickup trucks (and other vehicles) and head either north or south down Highway 177 toward their homes. Some stopping along the way for a beverage at the corner convenience store.
The Power Plant Men have Quittin’ Time down to a honed art form. Each stroke of the brush is carefully orchestrated. Scott and I went to perform our part of the ballet where the vehicles all backed out of their parking spaces in chaotic unison and quickly perform the three lines out the end of the single lane on the south side of the parking lot.
However, when we arrived in our car, Tony was no where to be found. As we received concerned looks from Randy Dailey and Jerry Day, as they pirouetted around us, wondering why we weren’t taking our turn in the Parking Lot Tango, all we could do was shrug our shoulders and watch as the dance went on without us.
Finally about 10 minutes past Quittin’ Time, Tony came walking out of the shop apologizing for being late. We told him that was all right as long as he didn’t make a habit out of it. We were pretty peeved that day because this meant that we had 10 less minutes that day to spend with our families.
We were even more peeved when the same thing happened the next day. We didn’t wait 10 minutes. After 5 minutes we went into the maintenance foremen’s office and found Tony still working away on his computer trying to finish up his work. We told him he had to leave right now! He said he hadn’t realized it was time to go.
Nothing is worse than a delayed Quittin’ Time when it isn’t for a legitimate reason. Tony didn’t have a wife and children at home so he didn’t feel the urgency that Scott and I felt. So, I figured I was going to have to do something about this. We weren’t going to tell Tony that he could no longer ride with us, because we knew he needed the company as much as we did, so I came up with a different plan.
The next day at lunch I wrote a program on the computer called “Quittin’ Time!” Here is how it worked:
It would load up on Tony’s computer when he booted it up, so he didn’t have a choice whether it ran or not. It showed up in the Task Bar at the bottom. It said: “Quittin’ Time in: 7:45:35” for example and it would count down each second. Then it would count down all day until Quittin’ Time. There was no visible way to turn it off (Power Plant Men had yet to learn about the Task Manager as this was Windows 3.1).
You could click on Quittin Time in the Task Bar and it would open up a small box in the middle of your computer with the time ticking down, but there was no red X in the corner to shut it down. There was only a minimize underscore that would put it back in the task bar.
I had added a small feature in the dialog window. In the lower right corner, there was a little slash sort of hidden in the corner. If you clicked on that, it opened another dialog box that let you set the actual time of day for “Quittin’ Time”. So, if you had to leave early, or later, you could adjust your Quittin’ Time.
Here was the clincher with the Quittin’ Time program. It was not enough to just show Tony that it was Quittin’ Time. This program had to force Tony to shut down and go home. So, when it was 15 minutes before Quittin’ Time, a Big Yellow Window would open up on top of any other work and would flash on and off that it was “15 minutes before Quittin’ Time! Time to Finish your Work!” Tony could close this window.
Then when it was 5 minutes to Quittin’ Time, another big yellow window would open up flashing 5 minutes before Quittin’ Time! Finish your work now!” and it would beep at you 5 times. Tony could close this window.
At one minute until Quittin’ Time, all heck broke loose on the computer. A big red window would open up and the computer would start beeping continually. The flashing Window could not be closed. It would say: “Less than One Minute To Quittin’ Time! Save all your Work!” The words would continually flash as well at the red background while counting down the seconds and it could not be stopped.
At “Quittin’ Time” The Red Box would say “QUITTIN’ TIME!” and the computer would lock up beeping continuously as loud as that little beeper(the internal speaker) could beep (this was a 386 PC). At that point, the only thing you could do was hit the power button and shut your computer off. I wish I had some screen shots to show you. Maybe I’ll find my old code and recreate it and take some and add them to this post later.
Needless to say, the first day I added this program to Tony’s computer, he didn’t heed the warnings. When the computer went crazy, he tried saving his work, but ended up losing a little of it before the computer completely locked up on him. He came out to the parking lot on time, however, he wasn’t in the greatest mood. We were. Scott and I were smiling. We were going to be home on time, and best yet, that day, we were included in the “Dance of the Sugar Plum Fairies” being performed by the pickup trucks that day in the Parking Lot.
The best part of the Quittin’ Time program came later. After about a week, Tony (who now left work on time every day) asked me if I could add something to the Quittin’ Time program. He wanted to know if I could make it so that he would remember to eat lunch. He would get so involved in work that he would miss his lunch entirely. So, I added a “Lunch Time” Feature to the program as well. He could adjust his lunch time using the same option window that opened when you clicked on the little slash in the lower corner of the Quittin’ Time window.
When I added the Lunch Time feature, I also added an Internet Feature that would go out to Yahoo Stock Quotes and get the Daily Stock Quotes for all of our 401k Mutual Funds and our company stock and at 3:40pm CST would pop up a window with the day’s stocks, so you could see how the Mutual funds in your 401k did that day. — Nothing better than watching your retirement plan grow each day. Yahoo posted the Mutual Fund updates for the day around 3:30pm, so Tony would be the first person each day to get the latest Stock news for our Mutual Funds.
Tony Mena was known as Planner 4 later when we moved to SAP because that was the username he used. Ray Eberle used to say to me, “We always want to keep Planner 4 happy!” Later this year, I will go into various ways we kept Tony happy, or confused… or well… on his toes anyway.
Power Plant Farm Fixing and Risk Management
We were told at the coal-fired Power Plant in North Central Oklahoma that we were going to have to stop doing the excellent job we were used to doing. We no longer had time to make everything perfect. We just had to patch things together enough so that it was fixed and leave it at that. Jasper Christensen told us that we were going to have to “Farm Fix” things and work harder because we now only had half the employees.
Two things bothered me right away….
First, “Work Harder.” What exactly does that mean? How does one work harder? When I pick up my tool bucket to go work on a job, should I put some extra bricks in it so that it is harder to carry? What then? Think about it… Shouldn’t we be working “Smarter” instead of “Harder”? We were all hard workers (if that means, spending a good 8 hour day doing your job). Any slackers were laid off 7 years earlier.
When I heard “Farm Fixing” I took offense to the reference. Jasper had mentioned using baling wire to hold something up instead of taking the time to make our jobs look pretty. As if baling wire was somehow synonymous with “Farm Fixing”. My grandfather was a farmer…. I’ll talk about that in a bit….
Jasper also informed us that we were no longer stuck doing only our own trade. So, an electrician should expect to help out as a mechanic or a welder as long as it wasn’t too involved. Certain welding jobs, for instance, require a certified welder. If the job was just to tack weld up a bracket somewhere, then I, as an electrician, could wheel a welding machine over there and weld it up.
After that initial meeting after we had been downsized to pint-sized, we met with our own teams. Alan Kramer was my new foreman. He encouraged us to learn the different skills from our teammates.
I asked Ed Shiever to teach me how to weld. After about an hour, I decided I wasn’t too interested in melting metal using electricity. I would leave it to the experts. I was left with a sunburned chest, as I usually wore a V-Neck Tee Shirt in the summer.
Jody Morse was a mechanic on our team, who had been a friend of mine since I was a janitor. We had been on the labor crew together. He asked me if he could do some electrical work with me. He thought it would be a useful skill to learn. I happily agreed to let him work alongside me running conduit and pulling wire around the precipitator hoppers.
It wouldn’t include working on any circuits where he might accidentally come into contact with anything live. So, I thought this was a good starting point. That was one of the first skills I learned as an electrician-in-training when I was taught by Gene Roget, a master of conduit bending.
I showed Jody how to bend the conduit and have it end up being the right length with the curves in the right place (which is a little tricky at first). Then I showed Jody where the conduit needed to go, and where the wire needed to end up. He said he wanted to do this all by himself, so I left him to it and left to do something else.
A little while later, Jody came back and said he had a slight problem. He had cut the cable just a little bit too short (Yeah. I had done that myself, see the post: “When Enough Power Plant Stuff Just Ain’t Enough“). I looked at the problem with him, and he was about six inches too short.
Jody looked the job over and decided he had two options. Pull some new longer cable, or try to make the existing cable work.He figured out that if he cut off 6 inches of the conduit, and sort of bent it out so that it was no longer exactly at 90 degrees, then it would still reach where it needed to go, only the conduit wouldn’t look so pretty because the conduit would appear a little cockeyed. We figured this would be all right because Jasper had just finished telling us that we needed to make things not so pretty anymore. Jody finished the job, and filled out the Maintenance Order indicating that the job was done.
The cable and conduit job had been requested by Ron Madron, one of the Instrument and Controls guys on our team. When he went out and looked at the conduit, let’s just say that he wasn’t too impressed. He went to Alan Kramer and complained that the conduit job was disgraceful. I don’t remember his exact words, but when I heard about it, it sounded to me like he said “It was an abomination to all things electrical”.
I had always taken pride in my work, and doing a “sloppy” job was not normal for me. I didn’t want Jody to feel bad about this because he was pretty proud of having completed the job all by himself without my help. So I went and had a one-on-one with Ron and explained the situation to him. I also told him that the next time he has problem with something I did, come directly and talk to me about it instead of our foreman. We’re all on the same team now.
I think once he realized the situation, he was more receptive. Jody and I did go back out there and fix the issue by running a new cable that was long enough, with a new piece of conduit that was installed with the best of care so that it looked pretty. — None of us informed Jasper that behind his back we were still performing our jobs with great care and precision.
The more I thought about the idea of “Farm Fixing” and “Risk Management” and how it was being applied at our plant, after about a year, I wrote a letter to the Superintendent over all the Power Plants, Jack Coffman.
Here is the letter I wrote (It was titled “Farm Fixing and Risk Management” — appropriate, don’t you think?):
Dear Jack Coffman,
I went through the Root Learning Class on Friday, September 6. After the class our table remained to discuss with Bruce Scambler the situation that exists at the power plants concerning the way we maintain our equipment. We attempted to discuss our concerns with our facilitator, however, the canyon depicted in the first visual became more and more evident the further we discussed it.
My two concerns are the terms “Farm Fixing” and “Risk Management”. These are two good processes which I believe must be employed if we are to compete in an open market. I do believe, however, that our management has misunderstood their true meaning and has turned them into catch phrases that are something totally different than they were originally intended.
I come from a family of farmers. My father and grandfather were farmers. I was concerned about our use of the term “Farm-fixed”, so I discussed the way we were using it in our company with my father and I have confirmed my understanding of the term.
My grandfather as a farmer was a Welder, a Blacksmith, a Carpenter, and an Engine Mechanic. When a piece of machinery broke down while he was out harvesting or plowing a field, it is true that baling wire and a quick fix was needed to continue the work for the day. There is a small window of opportunity when harvesting and the equipment had to be running during this time or the farmer’s livelihood was at stake.
That evening, however, the piece that broke was reworked and re-machined until it was better than the original store bought item. Thus guaranteeing that it wouldn’t break down the following day. If the repairs took all night to make it right, they would stay up all night repairing it correctly. It was vital to their livelihood to have their machinery running as well as possible.
A Ford Tractor soon became my grandfather’s tractor as the original factory parts were replaced with more sturdy parts. It wasn’t repainted (gold-plated), because they weren’t planning on selling their equipment. The tractors and plows would last years longer than originally designed. All this was before farming became a subsidized industry.
We need to “Farm-Fix” our equipment. Our management however, focuses on the use of baling wire during an emergency and replaces the true meaning of Farm-Fixing with the meaning of “Jerry-Rigging”. Which is merely a temporary fix while farming and is NOT farm-fixing something. We have been maintaining our plant with quick fixes and have not been farm-fixing them. If so, our equipment would be more reliable, and would last longer than originally intended.
Risk Management is another area that has been misunderstood by our management. They have gone to school and have been trained in Risk Management. I don’t believe they are using their tools in the way that they were taught. They have taken the underlying idea that we may not need to make a change or repair a certain piece of equipment at this particular time and have made it the center of their idea of Risk Management. Risk Management is more than that. It is weighing the consequences of both actions against the cost and making an informed decision to determine the timing of maintenance.
Risk Management at our plant has become nothing more than speculation, or what I call “Wish Management”. The decision is often made based on the immediate cost and downtime to delay maintenance without properly identifying the possible damage that could occur and the cost of that scenario.
The phrase “It’s run that way this long, it will probably be all right” is used to justify not repairing the equipment. No real analysis is done. Then we cross our fingers and “Wish” that it will continue running forever.
I believe in the concepts of Risk Management and Farm-fixing. I think they are processes that should be used in our company to achieve and maintain “Best-In-Class”. I am concerned, however, that if we continue on the course that we are on where “Wishing” and “Jerry-rigging” are our processes, it will only be a matter of time before our workers get killed and our plants melt down around us.
— End of the letter. See? I was always trying to stir things up.
The first summer I worked at the Power Plant as a summer help, we had a couple of floor drain covers in the maintenance shop that were missing from the floor drains. Plywood had been used to cover the drains, which had been smashed down by the heavy equipment that traveled in and out of the shop. One day during lunch I wrote a Maintenance Order to have the floor drain covers replaced and placed it on Marlin McDaniel’s (the only A Foreman at the time) desk. I was only an 18 year old kid that was just learning my way around in the world and already stirring things up, but I figured this was an accident waiting to happen.
The very next day, a plant mechanic, Tom Dean stepped onto one of those floor drains while carrying a heavy ladder and seriously hurt his back. It was a life changing event for Tom that immediately changed his career. The next day, the drains had new covers. I talked about this in the post: “Power Plant Safety is Job Number One”
Approximately one year after I wrote the Farm-fixing and Risk Management letter to Jack Coffman, we had a major incident at the power plant that was directly caused by the decision not to replace a coupling when it was known to be faulty (risk management, they called it). It would have required extending an overhaul a day or two. Instead, after half of the T-G floor burned to the ground and the plant was offline for about 3 months. Millions of dollars of damage. That is a story for another post.