Case Study: Palo Verde Nuclear Generating Station

Paul V. Arnold
Tags: Case Studies, oil analysis

The woman behind the chest-high glass partition at the Palo Verde Nuclear Generating Station dons a fresh pair of protective gloves before beginning her next work order. She gingerly moves a fragile molded shell into place onto a ceramic dish. After acknowledging the customer’s stated instructions, she precisely measures and alights each of the requested materials into the shell. A viscous dollop is gently positioned on top. It is then presented to the customer for full testing.

In the final analysis, it is determined that the plant’s cafeteria makes an outstanding taco salad.

“Everything is done to a high degree of precision at Palo Verde,” says Bryan Johnson, a lubrication engineer at this 4.2-gigawatt electrical power plant site, located 45 miles west of Phoenix in the town of Tonopah, Ariz. “That cafeteria worker was so incredibly precise. It’s an example of just how precise everything is here. That’s our culture. She was going to make the best taco salad in the world for you, and I’m sure she is that way for everyone who places an order at her lunch station.”

Employees at the Palo Verde nuclear plant admit that they apply an uncommon (perhaps over-the-top) attention to detail, and for good reason.

“The nuclear industry puts a premium on accuracy,” says Johnson. “We have a very detailed and precise design process. We are very careful and specific in our decision process. This results in operations and practices that are more precise and exact, and in a lower risk of a mistake occurring.”

That is important because, outside of taco salads and other cafeteria fare, this is a particularly serious business.

“The risk of not doing it right the first time can be significant,” says Brad Berles, the department leader for component performance

engineering. “We are protecting the health and safety of the public within a 10-mile radius and beyond. We have a community that relies on us to do it right the first time, every time.”

Attention to detail permeates processes, practices and procedures – in core operations, electrical distribution, safety systems ... and (the focus of this article) machinery lubrication.

In-house oil analysis that is as all-encompassing as most any vendor. Testing procedures that are put under the microscope as much as the oil and grease samples. A robust intranet site that can tell you exactly how many strokes of grease were pumped into a given zerk (or how many gallons of oil were added to a sump/reservoir) at any tracked moment in time.

“Some people may argue that what we do is excessive,” says Steve Lopez, the site’s section leader for predictive maintenance engineering. “Others will find that it is right on the mark.”

Read on and you can decide.

Lube Adds Life

Unlike some of the plants that have been featured in a cover story for this publication, Palo Verde is not new to experiencing the power and benefits of lubrication excellence. The Arizona site has been on the journey for more than two decades. In 2001, it was the inaugural winner of the International Council for Machinery Lubrication’s Augustus H. Gill Award for Oil Analysis Excellence. And for much of the past decade, elements of Palo Verde’s lubrication program have been recognized as best practices in the nuclear power industry.

Johnson, who has worked at the plant for 20 years, all of those in the role of lubrication program owner, rates the overall lubrication and oil analysis program as “an 8 or a 9” on a scale of 1 (low) to 10 (high).

“I believe that a really effective lubrication program can incrementally improve the mean time between failure or service life for all of the machines that it touches, be that through using the best lubricant available, the proper amount, the right monitoring,” says Johnson. “I believe that this program will add life to the machinery, which will add value to the plant.”

Adds Lopez, a site engineering leader since 1991, “Lubrication is one of those foundational things that can have the biggest impact on equipment health and site performance. Our management recognizes that.”

The lube program – along with fellow predictive maintenance technologies such as vibration analysis, infrared thermography and motor current analysis – allows the plant to attain high levels of runtime and maximize equipment life.

“We avoid taking a machine apart by employing various technologies,” says Johnson, who works closely with the engineers who oversee application of various predictive maintenance tools. “It is equally important to use PdM data to determine that a machine is running properly as it is to find a developing failure. We rely on sensory information from others to extend the value of PdM. Our skills and tools are often invited to determine the significance of extraneous input. We have many examples where we used, for instance, lube and vibe to look at a machine and decide that it is running perfectly fine. We are able to tell the plant that there is no reason to pull the string on this other information because the machine is functioning properly. That type of a save is more common than shutting down a machine before it comes apart at the seams.”

Says Berles: “PdM allows us to minimize how frequently we go into equipment for preventive maintenance. The fewer times we have to put our hands on equipment and do maintenance on it, the better off we are, especially when we have technologies that will show us if the equipment is performing as expected in its current condition.”


Predictive maintenance technician Mary Holtzclaw, who has worked 10.5 years in the Lubrication Lab, says: “We are accessible and have a quick turnaround time. If you had to rush a sample to an off-site lab, that could cost several hundred dollars, plus rush shipping.”

Predictive maintenance technician Darlene Lang is a 23-year veteran at Palo Verde. She has worked in the Lubrication Lab since May 2006.

Plant: Palo Verde Nuclear Generating Station, located in Tonopah, Ariz. (45 miles west of Phoenix). The plant is operated by Arizona Public Service (APS) and is owned by a consortium of seven utilities in the southwestern U.S. (APS, Salt River Project, El Paso Electric, Southern California Edison, PNM Resources, Southern California Public Power Authority, and the Los Angeles Department of Water & Power). Its three reactor units took 12 years and $5.9 billion to build, and were fully operational by 1988. Units 1 and 2 went commercial in 1986, with Unit 3 following in 1988.

Site footprint: 4,000 acres.

Plant employment: Approximately 2,000 employees. The maintenance staff employs approximately 500.

Products: The site is the largest nuclear generation facility in the U.S. It is capable of generating more than 4,000 megawatts of electricity. On an annual basis, Palo Verde produces over 30 gigawatt-hours of energy.

FYI: Palo Verde is the only nuclear plant in the U.S. that does not sit on a large body of water. Instead, it uses treated effluent from several area municipalities to meet its cooling water needs. ... It won ICML’s Augustus H. Gill Award for Excellence in Oil Analysis in 2001.


Testing ... Testing

Palo Verde’s longstanding adherence to predictive maintenance and attention to detail is quite visible in its Lubrication Lab. In existence since 1992, it houses some of the most complex and intuitive analysis equipment on the market as well as a crew of trained and dedicated professionals.

“It is really something,” says Lopez. “I don’t know how many facilities have a lube lab like we do. We know it gives us an advantage.”

Laboratory instruments (including an Olympus CH2 Ferroscope IV, Stereo Star Zoom Microscope, Nicolet Magna 560 FTIR Spectrometer, Rheometric Scientific Universal Stress Rheometer SR5, Koehler K 19400 High-Temperature Dropping Point Apparatus, Standard Oil Dual Ferrograph Analyzer, Spectroil MC Spectrophotometer, Mettler DL25 Autotitrator, EM Science AquaStar C2000 Titrator, Pacific Scientific 80000A, Pacific Scientific AB52 Bottle Sampler, Koehler K70400 Oil Bath, micrOptix i-Lab S560, InfraCal Soot Meter, Koehler ASTM D-1404 and others) are used to perform the following tests:

Mike Werley, material coordinator for mechanical maintenance, examines the procedures for dispensing oil from the lubricant shed.

Palo Verde has procedures for everything, from adding lubricants to taking oil samples.

Standard tests performed for new and in-service oils are acid number, water, metals by emission spectrometer, FTIR and viscosity. Standard tests for new and in-service greases are rheology, metals by emission spectrometer and FTIR. For oils and greases, protocols are in place to trigger other tests depending upon initial test data.

“I think other utilities have this level of dedication, but I think we are among the best from a capabilities standpoint, with the lube lab that we have got and the sheer amount of analysis that we can perform on-site,” says plant manager George Andrews.

Employing such extensive lab services is not for every plant or every company. The instruments listed represent a substantial financial investment. However, Palo Verde maintenance, engineering, operations and chemistry personnel state that offering and performing these tests in-house, as opposed to contracting such work to outside suppliers, saves time and money, and allows the site to push the envelope on asset care.

“We are accessible and have a quick turnaround time,” says Mary Holtzclaw, a PdM technician for the past 10.5 years in the Lubrication Lab. “If you had to rush a sample to an off-site lab, that could cost several hundred dollars, plus rush shipping.”

Holtzclaw is one of three full-time technicians in the lab. Geana Warner has worked here since 1998 and Darlene Lang has since 2006. They are considered part of the chemistry department, reporting to chemical work management section leader Paul Koss. They take samples that have been pulled by technicians in maintenance (see sidebar on Page 12), under the direction of reliability engineering, and the reminders of the planning and scheduling organizations. They test, review and analyze, and then work hand-in-glove with Johnson, a member of the engineering team. Johnson evaluates the results and related comments in order to ensure “situation normal” status and/or formulate action plans with these and other departments.

“It is a multi-layer process involving many organizations,” says Johnson.

The techs in the lube lab take particular care to unearth any abnormalities that have the potential to impact plant operations.

“We are making visible an issue that may not be apparent,” says Warner. “We look for evidence of physical breakdown so we can catch it before it gets to a point of need.”

“It’s kind of like crime scene investigation work,” adds Lang. “You are looking for clues and bringing those to light.”

Just like crime investigators, these are highly trained professionals with an eye for minute details. Each has gone through extensive initial instruction and subsequent annual training. All of their actions are under extreme scrutiny by Koss and other plant leaders.

“I do observations and compare what they do to what the procedure states. Our procedures are the tools we use to ensure the accuracy and precision of our testing,” says Koss.

He explains that lab techs must be formally qualified to operate each instrument and test.

“A qualification card must be signed off before the individual is able to perform each test as an independent worker,” he says. “We have an individual at the independent level for all of our tests. As we bring new people in or as they expand their tool box, they start out as a dependent worker – they can do the test under the direction of an independent worker. An independent worker can then sign off on their data. Our lab has a very extensive process to track who does each of the tests and what their qualifications are.”

Lab techs must log into a computer before each and every test. That provides full traceability and accountability.

Everyone in the plant knows what the techs do and have the ability to do.

“The first thing I get – especially from the guys in the plant – is, ‘Could I bring you an oil sample from my pickup truck?’” says Warner. “We do a lot, but, no, we don’t do that.”

Palo Verde has procedures for everything, from adding lubricants to taking oil samples.

Steve Lopez (left), section leader for predictive maintenance engineering, and lubrication engineer Bryan Johnson stand on a deck that overlooks the site’s cooling towers.

A Deep Dive into Lubricant

Every lubrication-related action is indeed documented and traceable. The repository for this information is the Lube Screen, a component of the site’s computerized maintenance management system. This Web-based platform, which took 1.5 years to create, is accessible to everyone within the facility.

“I think what we have is unique and, in some ways, leading the industry,” says Johnson. “The CMMS has different screens to manage our work orders, manage our documents, manage changes to how we define equipment. Screens were developed for lubrication as well as our other predictive maintenance efforts (vibration, thermography, etc.).”

One Lube Screen feature, called Lubed Index, lets users such as maintenance technicians see what specific lubricant has been designed for (and must be fed into) each of more than 20,000 identified lube points. The report can be printed out and clipped to a work order.

Another feature, called Lube Addition Process, denotes every time that grease or oil has been added into a given component, and lists the date, the person involved in the task and how many grease gun strokes or ounces of oil were installed. Entries are categorized as planned/preventive and corrective maintenance work.

Indeed, users can access an incredible amount of historical PdM test data as well as a condition status for each monitored machine.

“We set up a system where our customer could decide how deep they want to go into the data,” says Johnson. “The first level of lubrication includes microscopic image files, where available; iron, copper and viscosity tests for the last 12 samples. That covers most of what anybody would be interested in. We have customers that do want more. For them, the tool provides all other data available in the database. They can dive deeper. If they are really a power customer and they have to do their own Excel work, they can hit an extract feature. They can say, ‘Show me the last 50 records for this machine.’ It will export the last 50 data sets for whatever they picked on that big list and drop it into Excel for plotting and trending.”

This data is driving the actions of maintenance, operations and plant leaders.

“The turbine control oil is beginning to move on us,” says section leader Lopez. “When Bryan notices from the sampling that the acid number is going up, if they replace the filters more frequently, they can bring the acid number down before it runs away from us. That can save hundreds of thousands of dollars. When the acid number gets away, there would be a big oil change – approximately 1,000 gallons. Not only that, but you would have to downpower or shut down, so you would have some lost generation time. That could run a million dollars a day. There’s a lot of potential cost savings there by recognizing it early and then being responsive to that and then putting that onto the schedule.”

Decision-makers such as plant manager Andrews can request regular information related to the pieces of machinery that are most important to them. Whenever a condition report is written against that machinery, an automated e-mail is generated and sent to the subscriber.

“I love the online system,” says Andrews. “I can go on the Web site and all of my unit issues are listed. I can go in and see the work specs, when all of them are scheduled. I follow those. This system just makes the plant run better.”

Lube Screen is one way that communication is maximized across departments and across the 4,000-acre plant.

“We want to be sure that we have our program customers tightly integrated into the process so that they can help us come up with the smartest recommendations,” says Johnson.

Sampling In-service Oil, the Palo Verde Way

How does the Palo Verde Nuclear Generating Station handle sampling of oil? Just like with any process at the Arizona power plant, it is centered around a standard operating procedure. “We have a procedure on how maintenance technicians take oil samples,” says lubrication engineer Bryan Johnson. “For every task, there’s a work order with the work package. That work package would refer back to the procedure on how to take the sample. That procedure will have steps that will tell them to look for color, loose debris – basically, do a field screening. They will write ‘rush’ on it if they think there is anything abnormal about the sample. The training that they would receive would come through the maintenance programs

Next Up? A Good, Hard Scrub

The Palo Verde Nuclear Generating Station excels at machinery lubrication and has done so for a long time. Attention to detail has gotten the plant to this point, and it is attention to detail that pushes them to continually improve.

“I don’t think anyone is ever really there,” says Johnson. “We are constantly challenging ourselves.”

Each year does indeed bring new opportunities for refinement and growth. Among the goals for 2011 are to examine the formation and impact of varnish.

“The varnish potential and creation of micro-varnish products is an area that we need to understand better, as it is a relatively new form of oil failure,” says Johnson. “We are beginning to test for this condition on site. There is some interesting work being conducted on this topic in industry. Certainly, additive depletion is an area which has contributed to the varnish potential issue in the Group II oils. I’d like to understand that correlation for our machinery.”

Also on the docket is the standardization of grease guns (brands, models, etc.) and the addition to Lube Screen of suggested greasing practices (amounts) for components such as couplings and non-bearing applications.

“We want Palo Verde to continue to be a leader in this area,” says Johnson. “In fact, in 2011, we are going to bring in the Electric Power Research Institute (EPRI) and give them an opportunity to take a really good scrub at us. They will review what we are doing and give us a critical self-assessment. We’ll take a hard look at ourselves and see where opportunities for improvement are.”

When they arrive, EPRI’s scientists and engineers will likely visit the Lubrication Lab, pore over Lube Screen data and interview the host of organizations that play a role in ensuring lubrication excellence. Who knows? They might even test out one of the cafeteria’s taco salads.

Cleanliness (But Not Moisture) Comes Standard

Does Palo Verde set target cleanliness levels for lubricants in critical machinery? Is cleanliness an issue? What about moisture targets? Bryan Johnson fills us in.

Cleanliness levels: “We don’t use ISO cleanliness codes. We use National Aerospace Standard levels on some machines. For a turbine oil, I would like to see it in the NAS 5 or NAS 6 area. Our criteria is NAS 9. For new oil when it goes into a machine, we target NAS 6 as an end criteria.”

Cleanliness, in general: “Our plant is quite clean. Our oils, in general, are quite clean. We filter all of our oils before they go into the machines. If it’s a larger volume add, we use an in-line filter to pump fluid in from the drums. If it’s a smaller volume, our secondary containers are used one time and one time only. The oil passes through a goose neck with a filter, so all of the oil coming out of the drum passes through a filter before it goes into a secondary container. .. Our plant doesn’t have issues with coal dust. Most of our equipment is indoors. We are fairly well sheltered. We are contained pretty well, so we don’t have a lot of opportunities for ingress of contaminants. Our oils are, by and large, acceptably clean. Wear issues here aren’t the result of cleanliness.”

Moisture levels: “We have a procedure for moisture. Our general number that we would post would be 500 parts per million. The reality is that we will often react to anything over 100 ppm. Our oils will generally run from 20 to 50 ppm. Our desert environment helps with that.”


Criteria for Augustus H. Gill Award Consideration

The Palo Verde Nuclear Generating Station was the 2001 recipient of the International Council for Machinery Lubrication’s Augustus H. Gill Award for Oil Analysis Excellence. What does it take to win the Gill Award? According to ICML, this award recognizes organizations that have exhibited excellence in the application of used oil analysis in machine and lubricant condition monitoring. Candidates for the Gill are measured against certain characteristics and criteria, including:

• Commitment to education

• Maintenance culture and management support

• Performance measurements used

• Proactive/predictive maintenance

• Use of standardized procedures

• Technology integration

• Contamination control

• Lubrication management

• Oil analysis methods and strategies

• Use of information technology

• Continuous improvement

Besides Palo Verde Nuclear Generating Station, past winners of this honor include J.R. Simplot, Clopay Plastics, Great River Energy and Energizer Battery.

To learn more about ICML’s awards programs, namely the Augustus H. Gill Award and the John R. Battle Award for Excellence in Machinery Lubrication, visit the ICML Web site at