Metrics and Units of Observation

Go To: Advancing Competitiveness Homepage

A Conceptual Framework for Economic Decision Making in Advancing Manufacturing Industry Competitiveness: Metrics and Units of Observation

For more information, see NIST AMS 100-80.

Metrics: In identifying metrics, it is important to understand the holistic perspective. As illustrated in Fig. 2.7, the manufacturing industry change agent path to increasing long-run economic growth (i.e., real per capita gross domestic product or GDP) and increasing consumer utility (e.g., saving consumers money and/or time) starts with conducting research and development activities. These activities lead to efficiency and/or productivity innovations that are then adopted and disseminated among manufacturers. By definition, efficiency and productivity increases result in making the same things with less resources; thus, resources are freed up. Then, something new is done with these resources; for instance, a manufacturer might produce more goods or a consumer buys a new product. The new activity results in improving the quality of life and/or increasing economic security by making things better and/or cheaper.

There are three primary mechanisms through which economics achieves impact within change agents, as illustrated in Fig. 2.7:

• Motivating manufacturers to or not to adopt innovations by estimating their cost effectiveness;
• Guiding change agent R&D to high impact projects by evaluating the costs and benefits for each potential project; and
• Justifying change agent R&D budgets by evaluating if or when impact occurs.

The gold standard metric for “guiding” and “justifying” is commonly considered to be benefit cost analysis (BCA) and economic rate of return (ERR) (Boardman et al. 2018; Office of Management and Budget 2023). Conceptually, BCA is often calculated, in part, as the freed-up resources from Fig. 2.7 less the resources used to free them up (i.e., benefits less costs). The ERR is the rate of return calculated using the value of freed-up resources and resources used to free them. For “motivating” manufacturers, net present value (NPV) and internal rate of return (IRR) are the gold standard metrics (Thomas 2017), which 75 % of firms report always or almost always using for deciding which projects or acquisitions to pursue (Graham and Harvey 2001). As a result of the data requirements and the required technical knowledge, the gold standard metrics are often difficult to calculate in practice. Therefore, proxies could potentially be used as a substitute for prediction purposes, as presented in the data layers. 

Units of Observation: In addition to considering metrics, it is important to develop effective units of observation. This framework discusses developing high-impact projects, but a project can encompass a wide range of activities. When some activities are grouped together, they may obscure the ability to study the specific configurations that lead to high impact. For example, combining the effort to develop a standard for data interoperability in product design with a standard for data interoperability in production machinery blends impacts from different sources, making it harder to isolate the effects of each.

To solve this, a project is—for the purposes of this framework—defined based on three criteria:

1. It influences the same industry or adoption community.
2. It uses the same mechanism to improve productivity/efficiency (e.g., the cost item reduced for the manufacturer).
3. It groups related products and activities (e.g., a standard plus its supporting workshops and articles).

By this definition, a project is an impact-oriented grouping where, for instance, a standard for AI in manufacturing processes and a separate standard for AI in quality assurance are treated as two distinct projects. Even though both involve AI, their mechanisms of impact are different. The goal is to ensure that the grouping of activities never mask the factors that contribute to optimal results. An organization might call this a subproject or some other term; however, for simplicity this framework uses the term “project.”

Framework Components, Logic, and Implementation

Collaboration is a key component to reducing change agent costs and enabling compound learning. If you are considering adopting this framework, consider reaching out to the author Douglas Thomas, Economist: douglas.thomas [at] nist.gov (douglas[dot]thomas[at]nist[dot]gov)