Expert view: The automation knowledge crisis

In fact, over the life of an automation system the total intellectual investment will come to exceed the initial hardware and software cost many times over.

This paper will discuss some of the factors contributing to the impending process industry automation knowledge crisis, present real-life industry examples, and provide a proven solution to mitigate the problems.

Essential Facts about Automation Systems

What is unique about automation system knowledge? Why is it so critical that it be captured? Consider the following facts about automation systems:

1. Automation systems are critical to operations. They are the heart (or more accurately the brain) of the plant. You cannot run a plant without them. They are also platforms for continuous improvement and change, evolving continually to embody much of what we learn about our process on a daily basis. Because they evolve continually, there is ample opportunity for the introduction of errors that can result in compromised unit operations, plant shutdowns, or possibly even an
accident.

2. Automation systems are highly interdependent. They are a collection of components from a number of suppliers integrated together to provide desired functionality. Typically these systems integrate a variety of disparate subsystems, each with unique data structures, and employing multiple interface techniques. In modern plant automation systems, a tremendous amount of data is exchanged, and interoperability between and among the subsystems increases daily. These different components need to work together seamlessly for the automation system to function properly.

3. Automation systems contain vast amounts of information. They have embodied within them knowledge from a variety of sources that have been added and refined over a period of years. This knowledge represents the collective intelligence of an accomplished group of individuals, many of whom have most likely spent a significant part of their career optimizing the functionality of that particular system and process.

Automation systems documentation capabilities are limited. Although there is a large amount of critical information contained in them, the self-documentation functionalities of even the latest versions from the leading suppliers lack the capability to really capture their inherent complexity.

Furthermore, each automation supplier provides tools that manage only their specific system, providing no understanding of the relationships and levels of interoperability between it and the various systems it is connected to. Thus, users are left with an incomplete understanding of their installed automation, forcing them to rely upon outdated, manually generated documentation, or individual recall to understand the system.

The Knowledge Crisis

Safe, reliable, and profitable plant operation requires strict protection and careful management of the knowledge that resides in automation systems. Two critical, converging factors threaten the integrity of automation systems and the ability to effectively manage them. These factors are: loss of critical automation knowledge and the escalating complexity of installed systems. Throw in ever-increasing regulatory requirements and you have a “perfect storm” on the horizon.

Loss of Critical Automation Knowledge

The United States Census Bureau considers a baby boomer to be someone born during the demographic birth boom between 1946 and 19641. In the United States alone, some project that at least three million engineering jobs will be vacated between 2004 and 2010 with no expectation of the trend slowing down. A recent ARC study revealed that a major refining company in the United States lost 2500 man-years of knowledge in just one year when 100 operators retired.

This loss of plant knowledge will take many years to regain, if it is even able to be recovered at all. This situation is not unique to the United States or the process industries; it is occurring regularly around the world. While the current economic crisis may have delayed this exodus temporarily, the clock is still ticking and the departures will be even more dramatic once the economy turns around.

This expected exodus of personnel presents the same problem relative to automation systems. Due to their insufficient self-documentation, much of the knowledge about them is contained in the minds of those who work on them. Unfortunately as they leave the company, that knowledge usually leaves with them.

This problem is exacerbated to the point of criticality when the plant has a 20-to-30-year-old obsolete control system, for which there are no training courses available and all of the system experts have retired. The options for regaining this critical skill set are few and generally inadequate. Figure 1 (Please click on ‘More Images’ above) shows an example of how dramatically group demographics can change over a decade.

This is real data from a group of automation professionals at a major global petrochemical company. The age distribution data from January of 2005 showed that the group was definitely skewed to the older end of the age spectrum. Management recognized the need to attract younger engineers into the group to maintain its viability as the baby boomers retired.

Despite their best efforts, as of January 2009, the percentage of employees age 40 and younger actually dropped by more than half! Their projection five years into the future (January 2014) shows the trend only getting worse. By 2014, only 6% of the group will be age 40 and younger and nearly half will be eligible for retirement. This puts great risk in the company’s ability to effectively manage its knowledge-intensive automation assets. Every company should perform this automation knowledge worker demographic analysis to assess their vulnerability. The results can be eye-opening.

Increasing Complexity of Installed Systems

Studies have shown that modern plants have as much as twenty-five times as many points of measurement as they did before the advent of the DCS. Each time that a price or technology breakthrough occurs with measurement technologies, more points get added to the plant.

Adding points of measurement is most often justified because they feed an integrated application that somehow improves production. Furthermore, improved human interface capabilities, the addition of multi-functional controller platforms, and the general age of the installed systems are driving upgrades to newer DCS platforms (see Figure 2: please click on ‘More Images’ above). The new DCS’ tend to be larger and are more likely to be integrated with other sub-systems than their predecessors.

This trend is likely to intensify as wireless instrumentation, lower-cost sensors, field device networks, and service oriented architectures gain acceptance in the process industries.

These technologies taken together will drive down the cost of making and integrating measurements, as well as the cost of integrating the applications that use them, therefore facilitating another trend, namely the integration of real-time systems (such as DCSs) with non-real-time systems (such as planning and scheduling). Mapping and managing all of these interdependencies becomes untenable without assistance (see Figure 3: please click on ‘More Images’ above).

In addition to the aging workforce and complexity issues described above, the bar is also being raised on regulatory performance. Allowable emission rates continue to be tightened and mandatory reporting requirements have skyrocketed. For example, Houston, Texas area petrochemical plants are required to reduce their NOX emission by 80% compared to the year 2000. In 1997, one local plant was governed by 28 published air toxicity rules.

Today, that same plant has to comply with over 125 published rules, each with additional mandatory monitoring and reporting requirements. The trend toward increased regulation is driving increased enforcement activity from regulatory agencies and the intensification of media scrutiny. The graph in Figure 4 (Please click on ‘More Images’ above) below comes from the Texas Commission on Environmental Quality’s 2008 Enforcement Report. Despite the fact that overall emissions have dropped significantly over the past five years, the amount of enforcement activity is rising. Furthermore, the penalties associated with these violations have also increased dramatically.

The Solution

With regard to automation systems, the solution to the above problems must take a number of factors into account. First, it must capture the knowledge embedded in all plant automation and related systems. This includes DCSs, SISs, PLCs, instrumentation databases, historians, and applications (both real-time and nonreal-time). It must expose automation system knowledge in context, graphically display the signal genealogy, and document the relationships among all interoperating systems.

This knowledge capture mechanism must automatically update with the latest configuration in near real-time, and track all changes. To be fully utilized, it must be easily accessible via secure plant Intranet. A solution of this nature would also facilitate the migration of legacy systems by translating their inherent knowledge to a format that provides for a quick
and easy transition to a new control system.

Integrity Software

Integrity Software from PAS meets all of these requirements. In addition to documenting the knowledge contained in more than forty different automation systems, this web-enabled application is able to collect data from multiple plant sites and display it anywhere on the company IT network.

In industry cases two and three, the inability of plant personnel to understand the genealogy of the measurement signals they were working with directly caused plant upsets. A primary justification for our purchase of Integrity Software was the avoidance of these types of incidents in the future. Integrity Software provides the ability to easily determine all references to a data point within your plant’s automation systems.

When coupled with Intergraph SmartPlant® Instrumentation software, Integrity expands the loop sheet to include a diagram of all uses of that measurement throughout the various connected automation systems, including the DCS, SIS, and plant historian (see Figure 5: please click on ‘More Images’ above). The automation portion of this diagram, which provides simple visualization of signal flow among your systems, is automatically updated in near real-time, so that users always have access to the most recent configuration data.

In the first industry case, the tragic loss of automation system expert mid-way through a migration set the project back considerably and caused significant cost and schedule overruns. Most plants cannot afford to staff very many automation system experts and hence are at risk of losing the specific knowledge that those very few experts possess. Furthermore, due to the continual changes made to automation systems, the documentation often does not match the actual configuration.

Use of Integrity Software to facilitate migrations between different releases or even different systems is an excellent means to ensure that all configuration and integration is accounted for. By capturing the “as-is” configuration of the old system, it is easy to ensure that no critical connections or parameters are omitted in the configuration of the new system.

Integrity documents the interdependencies between various systems and data structures (points, display objects, registers, etc.) and displays them in either textual or graphical format to facilitate troubleshooting during the project. It automatically identifies configuration defects such as missing references and out of range parameter settings, which dramatically reduces the time spent in factory acceptance testing. Additionally, it provides automated management of change functionality to track and display modifications.

Differences between the data versions are highlighted and time stamped. These capabilities can reduce the amount of engineering time spent on migration projects by up to 10%.

Conclusion

As seen in the industry cases presented, the loss of critical automation knowledge, the escalating complexity of installed systems, and ever-increasing regulatory compliance requirements are conspiring to create an automation knowledge crisis in the process industries. Integrity Software from PAS can help you survive this critical situation by capturing the knowledge embedded in automation systems and exposing it in context, when and where it is needed. There are tremendous benefits to be gained by accessing one’s current automation-related risks, and developing a plan to manage and preserve the priceless knowledge within the automation systems.