View from the Top

Keck Observatory uses a robust building-automation system to increase the reliability of advanced telescopes that astronomers use to hunt planets and dissect galaxies.

When John Baldwin arrives at the W. M. Keck Observatory headquarters every day, it takes another two hours for him to drive to his office. The office sits 13,796 ft. above sea level on the summit of Mauna Kea, a dormant volcano on the island of Hawaii.

The W.M. Keck Observatory provides precise in-depth views of the universe through the two largest telescopes in the world. Photo by Andrew Richard Hara Photography © 2014

As he drives up the mountain, the altitude increases and the oxygen available to his body decreases. The altitude can cause him to feel lightheaded, fatigued, and dehydrated. It also can cause irritability and lack of concentration. These are normal working conditions for him and the two dozen other employees who work at the observatory’s summit location. The spectacular view makes it all worthwhile.

Even at almost 14,000 ft., Baldwin and the astronomical scientists from the Univ. of California and California Institute of Technology still need to look up to observe the universe with unprecedented power and precision. The Keck Observatory is home to the world’s largest and most scientifically productive optical and infrared telescopes. Each of the twin telescopes weighs 300 tons and operates with nanometer precision. The  primary mirrors are 10 meters in diameter, each composed of 36 hexagonal segments that work in concert as a single reflective surface.

As the summit superintendent, Baldwin’s job is to keep the telescopes, domes that shelter them, and the entire facility functioning. This includes chillers, air conditioners, hydraulics, pneumatics, cranes, pumps, gearboxes, motors, and other equipment. “It’s amazing how many systems need to work perfectly and in concert with each other for us to operate normally,” Baldwin said.

Originally from New Jersey, Baldwin moved to Hawaii after taking a cruise there with his family in 2003. He has worked with Keck since 2006 and daily tackles significant challenges inherent to the facility’s complexity and location.

A unique facility

The telescopes are not what most people consider to be typical telescopes.

“People don’t physically look through the telescopes,” Baldwin explained. “There are instruments on the telescope that have charge-coupled devices, like in a digital camera but much larger, and the computers do all the work. Scientists look at pictures of the galaxy, and can see the spectrum of light from the galaxy. By analyzing the spectrum, they can deduce a lot of data about a star or a galaxy. For example, they can determine what it is made of, how far away it is, how big it is, and how fast it is moving relative to the Earth. We use computers to guide the telescope to the target, then track the target for however long the exposure is needed.”

The data flow to astronomers who may not be on site. They can observe from the Kamuela headquarters location or even from California. People on site physically operate the telescopes for the astronomers. Some of the Keck observers are “planet hunters” who look at different stars and collect data to determine whether there is a planet orbiting around a star.

Special challenges

The altitude is, by far, the biggest challenge for Baldwin and his team of six maintenance technicians, he said.

“A normal task that you would complete in one hour at headquarters might take up to two and a half hours at the summit,” Baldwin explained. “That is completely normal up here. Your body tires very quickly. It’s normal to feel lightheaded. If you’ve been working here for many years and your body is acclimated, you may still feel lightheaded, but you might not notice it.”

To combat the effects of the altitude, employees are required to stop for 30 minutes every day once they reach the 9,000-ft. mark while driving up the mountain. There is a visitor center at that point where they can have breakfast and allow their bodies to acclimate. Still, the effects of the altitude can be felt.

“You may not think as clearly as you would at sea level,” Baldwin continued. “If you are doing a complex computation or calculation, it’s always a good idea to call headquarters and ask an engineer to double-check your numbers. It’s easy to forget what you are doing while you are in the middle of a project. It’s also normal to feel very fatigued and temperaments can go sour quickly.”

Since the summit employees go home very night, they must acclimate their bodies to the change in altitude twice each day. The two-hour drive counts toward the 10-hr. shifts, reducing the number of working hours for each employee and creating another challenge unique to the facility.

To counteract this efficiency loss, the organization takes care to consistently upgrade the technical and scientific capabilities. This includes the maintenance infrastructure, Baldwin said. However, it also creates additional work, so a balance must be created.

“With constant upgrades, we have the same individuals doing the plumbing, electrical, steel work, fabrication, and also installation and implementation of the upgrades,” Baldwin explained.

Baldwin ensures that his six technicians are cross-trained, as he has been throughout his career. He began as an HVAC installation mechanic and later became a facilities technician. His experience includes knowledge about controls and programming, how electricity works, hydraulics, pumps, airflow, and how different fuels are handled.

“Behind all these beautiful optics, there is very complex machinery,” he said. “The telescope control and mirror systems are super complex and really cool.”

Baldwin’s team of six includes four mechanical technicians, one senior CNC machinist, and an industrial electrician.

A look at the K2 primary mirror inside one of Keck Observatory’s unique telescopes. Photo by Andrew Cooper © 2007

Building-automation system

One of the key upgrades was the installation of a robust building-automation system, a project that began in 2009 and was spearheaded by Baldwin and his colleague, Mark Devenot.

“At the time, we had a very old Trane Tracer system on one side of the facility only,” Baldwin said. “It was archaic, and no one knew how to run it. The philosophy was, don’t touch it because we don’t know how to fix it. We couldn’t even utilize it to modify how our equipment was running.”

The company upgraded to the Alerton (Lynnwood, WA) building-automation system. Baldwin and his team were trained by a local contractor on how to install the system and how to program it. It was first rolled out on two of the air conditioners that cooled the Keck 2 dome.

“It can be monitored remotely and programmed to do exactly what you want it to do at any given time,” said Baldwin. “Our management found lots of value in this, so we were able to roll it out to the rest of the facility.” The system provides several benefits. At such an inconvenient location, the remote-monitoring capabilities are critical. Technicians can log in from home and help the on-site staff troubleshoot a problem. More convenient usability and customization also provide payback.

“For example, our domes are kept at a very low temperature, typically close to zero degrees Celsius,” Baldwin explained. “We use a nighttime forecast to tell us what temperature to use so when we open the dome at night, the mirrors and telescope structure are approximately the same temperature as the nighttime temperature. This way the mirrors don’t fog. We also need to avoid the expanding or contracting of the steel due to temperature differences which affect the optical quality of the telescope. In the past, before this system was installed, the air conditioners that live in our domes were, for the most part, off-the-shelf air conditioners—they were either on or they were off. Once they cool to the forecasted temperature, they shut down.”

Baldwin said humidity is also a concern. “At night, if we have to close the domes because of some passing weather, if our dome temperature is satisfied, then the air conditioners would not run and the humidity would spike. When the weather would clear there would be a delay opening the domes because we would be at dewpoint on our mirrors,” Baldwin further explained. “We were able to customize the actual functionality of the air conditioner, add a dehumidification mode, and stage the air conditioning to hold tighter tolerances to temperatures in the dome. Because we can be below freezing up here, the cooling coils on the air conditioners collect a lot of ice. So we have to defrost them with electric defrost. We were able to upgrade the process and control the amount of energy we were using to defrost the cooling coils for maximum efficiency.”

At this altitude, Baldwin said the weather can change quickly. “We can be 45 degrees Fahrenheit and sunny, and then an hour later we might be in a total white-out blizzard and need to evacuate the mountain for safety concerns,” he said. “I have actually shoveled more snow here than I ever did in New Jersey.”

The building-automation system allowed the observatory to use its equipment more electrically efficiently while customizing the system to meet its special needs. “Now, during times of high humidity, we are able to keep our domes around 60% humidity, whereas before it could spike up to 85% or more, depending on the outside environment,” he said. “This bodes well for our optics. We’ve since expanded to monitor with some systems like hydraulics and air compressors. In some cases, they are completely controlling some of our chillers and air conditioners, and it is very helpful especially if there is a breakdown. The more we have connected to the building-automation system, the easier it is to troubleshoot remotely.”

The system has also helped to reduce electrical costs in an area where this is a critical concern. “With this system, we’ve been able to monitor our instantaneous power demand,” Baldwin said. “We can monitor a rise in our electrical usage and stage down non-critical equipment to maintain our instantaneous demand below a preset amount to keep electrical costs down.”

Maintenance philosophy

All of the general maintenance at Keck Observatory is done in-house. Only large additions and ancillary projects, such as crane inspections, are sourced outside the infrastructure team.

“Our overall goal is to be totally reliability centered, but we are not there yet,” Baldwin said. “What we rely on is preventive and predictive maintenance. Preventive is our work-order program based on manufacturer specifications for our equipment. There is a frequency for each piece of equipment. We also have a pretty inclusive predictive-maintenance program. We do a lot of fluid analysis, thermography for electrical PMs, and we use ultrasound to give indications of broken wheel shafts that are part of our dome system. We recently started using an ultrasonic tool to monitor our bearing health and compressed-air leaks.”

Baldwin said his best practices begin with a twice-a-day facility walk-through.

“We spend about two hours walking through the entire facility, writing down pressures and temperatures, looking for leaks, and listening for odd sounds coming out of equipment,” he said. “And, because we do it so frequently, it’s one of our best indicators for something going wrong. The morning walk through is more in-depth. We use a clipboard to record pertinent information like how much water is in the facility. What are the temperatures of the chillers? What are the pressures of the pump that feeds the facility for cooling the instruments?

“Before we leave the facility, we do a visual walk through using the same route.”

He said the best advice he can give other managers driving maintenance and reliability programs is to question everything and educate your team with the most advanced tools possible. “The rewards are incredible,” he said. “If you can then educate upper management on why those tools have such value, everyone wins.”