IP Networks Rule the Plant

Taking care of your network is the key to efficiencies now and tomorrow.

By Gary Mintchell, Executive Director

Everything is networked. If not now, it will be soon.  

Ethernet has won the networking war. The most important of the protocols that we commonly, if inaccurately, label Ethernet is Internet Protocol (IP).  

You may have heard of the Internet of Things. IoT has received increasing publicity over the past couple of years. Now marketing types are rushing to coin new words and phrases. Recently, I’ve seen people discussing Industrial Internet and Industrial IP. Some traditional industrial networks such as HART, Profibus and DeviceNet have Ethernet and IP implementations—HART over IP, Profinet and EtherNet/IP.

What all of this means is that engineers, managers and technicians must become knowledgeable about the intricacies of networking. It is more important than ever that plant managers and division managers bring IT professionals into active and fruitful collaboration on building and maintaining effective plant networks. Engineers and technicians must learn enough about IP networks to recognize and troubleshoot problems early on.

Dan McGrath, industrial automation solutions manager at Panduit and a representative of the Industrial IP Website jointly operated by Panduit, Cisco Systems and Rockwell Automation, thinks we’re at a transition point of improving maintainability and uptime of processes with IP networks. 

“With IP networks coming to the plant along with control networks, it brings along people and process, mobility, energy data, sensors and video. This makes network infrastructure critical to plant success,” says McGrath. “Engineers were familiar with older networks and their rules on terminating resistors, media, number of nodes, monitors and all the other peculiarities. IP networks bring change. Some of the surface things of the past go away, since IP networks are a little like ‘plug-and-play,’ but now engineers need a vision of where the IP network is going.” 

The high-level view
Taking a high-level view of network architecture yields many benefits for performance and security. McGrath advises, “As you add automation, video and other applications, you’ll need to learn to segment your network for performance and resiliency. You need to figure out how to achieve high availability along with high reliability by looking at how some people got into trouble.” 

Industrial automation networks can easily grow quite large. McGrath notes that one machine he worked on had 500 nodes on the network. “If you don’t have structure and knowledge of designing and installing a network,” he says, “you’ll run into problems. I had a case at a food plant where the maintenance guy started plugging and unplugging network nodes trying to find a problem. He went home for the night and the machine didn’t work until he came back in to tell colleagues where he was with the troubleshooting. With IP networks, you don’t have to fall into that trap.” 

Many tools exist to guide development and work with IP networks. There are TIA standards on how to structure a network. Tools are available to assist design and documentation. McGrath advises structuring with managed switches: “Now with wireless and video connections to enterprise, managed switches help with security and safety,” he says. “A pharma company I knew had lots of unmanaged switches. A remote person could shut down the plant. The good thing is that there are solid answers. You can find a reference architecture on the Industrial IP Advantage Website.”  

Panduit, Cisco Systems and Rockwell Automation built the Industrial IP Advantage Website because they saw a need for a repository of best practices. The site contains a number of links, pages related to a variety of topics, and a plantwide reference guide. The vendor-agnostic site also contains a community angle where everyone can post questions or contribute expertise, including an IoT approach.  

“Mobility, cloud, and data analytics will change how we do maintenance and how remote experts can help,” McGrath explains. “Our site has sections for emerging topics. Currently, we have some of the high-level information and we are always adding details. In 2014, we are launching online training that should have more detail.” 

Updating network design
McGrath illustrates the importance of modern network design with an example of a large, West-Coast generating facility that uses a fiber-optic network spread over four to five acres. The network carries control-system data, which controls the operations of a coal-fired plant, which controls the ovens as well as the electrical generator. When the plant was experiencing network errors, months of troubleshooting on switches, software, sensors and controllers indicated that the problems were caused by a physical infrastructure problem. The utility company could not resume control of the facility until the issue was resolved. 

During a plant walkthrough, Panduit service professionals noticed that the plant’s fiber-optic network was approximately 12 years old and was Fiber Distributed Data Interface (FDDI)-grade from the dotcom era of the 1990s. Fiber from this era has a limited bandwidth and is unable to communicate at the data speeds of today. Panduit recommended testing the network during a maintenance window and followed with a recommendation to overhaul the fiber network. In the next maintenance window, Panduit ran new fiber and tested all connections. The problem—caused by 12-year-old fiber optics—was solved. 

Segmenting networks
Sabina Piyevsky, commercial engineering manager at Rockwell Automation, discussed network design using segmentation on the Industrial IP Website.  

There are two types of segmentation implementations. “Physical segmentation is common,” she writes, “but has been applied to an extreme. Logical segmentation is the process of outlining which endpoints need to be in the same local area network, better known as a LAN.”  

Following is from the remainder of the article: 

Segmentation is a key consideration for an Industrial Automation Control System (IACS) network. Segmentation is important to help manage the real-time communication properties of the network, and yet support the requirements as defined by the network traffic flows. 

Security is also an important consideration in making segmentation decisions. A security policy may call for limiting access of plant floor personnel (such as a vendor or contractor) to certain areas of the plant floor (such as a functional area). Segmenting these areas into distinct subnets and Virtual LANs—commonly called VLANs—greatly assists in the application of these types of security considerations.  

Subnets and VLANs are two concepts that go hand-in-hand. A VLAN is a broadcast domain within a switched network. Devices within a VLAN can communicate with each other without a Layer-3 switch or router. Devices in different VLANs need a Layer-3 switch or router to communicate the traffic. Subnets are simply a subset of IP addresses assigned to a set of devices. Subnets are Layer-3 (Network/IP) concepts and VLANs are Layer 2 (data-link/Ethernet).  

Typically, devices in a VLAN are assigned IP addresses from the same subnet and a subnet has devices in one VLAN to make routing easier and more straightforward. Best networking practices call for a one-to-one relationship between VLANs and subnets.  

When designing IACS network logical segmentation plans, there are competing objectives. On one hand, all Level 0 to 2 devices that need to communicate multicast I/O between each other must be in the same LAN. It would seem easier to put all devices in one VLAN and subnet.  

However, the smaller the VLAN, the easier it is to manage and maintain real-time communications. That’s because the broadcast traffic and multicast traffic are constrained. Real-time communications are harder to maintain as the number of switches, devices and network traffic increases in a LAN.  

Smaller VLANs also isolate devices from those that are faulty or compromised, because the negative impact only occurs within the errant devices’ VLANs. For the same reason, VLANs form the basis for setting and implementing security policy and protection. VLANs provide the broadcast isolation, policy implementation and fault-isolation benefits that are required in highly available networks.  

There are many approaches to segmenting a network. Manufacturing facility networks can be divided by functional sections of the plant floor, product lines and traffic type (for example, I/O, controller-to-controller and explicit message traffic). To achieve the goal of minimizing VLAN sizes, a mixture of all three may be used.  

Segmentation can be achieved via the following two key mechanisms in the Cell/Area IACS network: 

Physical—Using separate cabling and Layer-2 access switches. 

VLAN (802.1Q)—Using the VLAN protocol that can be implemented on the same physical infrastructure 

Physical segmentation is a highly common approach in current Ethernet implementations, but has been applied to an extreme. For example, a common approach in current Ethernet deployments is to physically separate I/O traffic from HMI traffic and not to connect the I/O traffic to any interconnected Layer-3 distribution switch. In these cases, a controller has separate network interface connections (NIC) to each network, and the only means to communicate between the two networks is over the backplane of the controller. The I/O network is, therefore, reachable only via the controller backplane that processes only CIP traffic.  

The effects of this include: 

• Devices on the I/O network are not accessible via non-CIP protocols (such as SNMP or HTTP), limiting overall interconnectivity.
• A controller was not designed to route, switch or bridge continuous network traffic, and may introduce delays when used in this manner.
• Network-based services (such as security, management, IP address allocation and so on) must either be replicated in each network or are not available.
• Increased costs occur because the available network resources in the HMI network (for example, open ports) are not available in I/O network.

Although physical segmentation dedicates network resources to these various traffic types and helps increase the level of certainty that the traffic receives sufficient network resources, best practice is that these networks be at least connected to Layer-2 or Layer-3 switches to enable interconnectivity via other methods than the controller. In this way, the networks stay interconnected and get the full benefits of the converged network.  

Ethernet is alive and well in your plant. Will you be ready to help keep it performing well? MT&AP