Process Metrics are Key
It is well known that ITIL® describes four types of process metrics: process efficiency, process effectiveness, process progress and process compliance [see, in particular, the discussion in Service Design, §3.7.5]. However, given the ambiguities of this terminology, it is not always evident to débutants how these types may be characterized and used. If efficiency and effectiveness are relatively easy to understand, progress and compliance are less easy to comprehend. Because a good understanding of metrics is a key underpinning of good management, it is worth looking at what these types really imply. In this article, I will discuss process compliance.
Process Compliance as a Metric Type
ITIL describes the process compliance metric type as
Compliance of the process to governance requirements, regulatory requirements and compliance of people to the use of the process.
Hiding behind this all too brief enumerative description is a very wide array of controls concerning different life cycle phases and widely varying types of governance. We may eliminate the problem of using a term to define itself and generalize the description of the process compliance metric as:
The definition of how to measure the respect by a process of the controls placed on it.
The controls might be external to the service provider organization, such as governmental regulations or industry standards. They might be internal to the service provider organization, but external to the process itself. And they might be controls that are internal to the design of the process itself. And, as we shall see, we may assess the definition of the process as well as the execution of the process.
Given this definition, we see that the list of “controls” provided by ITIL consists, first of all, of an active agent that truly control a process (the process owner); next, of the references the controller uses to perform the controls (policies, objectives, documentation); and finally the measurements that are compared to the reference values (feedback). Missing from ITIL’s list are the technical agents that may control a process autonomically (typically, software tools); the measurements of the process activities themselves (and not just the output); the measurement of the inputs (whose volume can be compared to the volume of outputs); and the assessment of the outcomes at the process’ customers that were fostered by using the process. These outcomes may indicate if the objectives of the process were appropriately defined.
Compliance and Conformity
Living in an age of semantic lassitude, we should clarify what we mean by “compliance”, in an effort to overcome the indolent indifference to the meanings of words that characterizes so many people. Too, many organizations have developed idiosyncratic uses of the term “compliance”, leading to divergent understandings of what it means. It is worth distinguishing compliance from conformity as a way of clarifying our understanding.
In the English language, “to comply” has a strong nuance of a situation where one party requests a certain service or behavior from another party. When the second party responds in a way that meets the intention of the first party, the second party may be said to comply with the first party’s request. There is a further nuance that the second party may not be entirely in agreement with the request, but nonetheless acts as requested. In other words, left to its own devices, the second party may well have acted differently. We comply with a governmental requirement to pay taxes, even though we would likely not pay them if not so required.
“To conform” implies a cultural or organizational norm or standard that shapes our behavior. We conform, not because anyone in particular asked us to (that would be compliance), but because we have an intrinsic understanding of social expectations and we behave accordingly (or not). Conformity may be valued or may be despised but, like self-censorship, it is a force with which to be reckoned. We conform (or not) to our perception of social imperatives.
Although we do distinguish these nuances in English, many languages do not do so. They may well use the same terms for both compliance and conformity, with the underlying understanding of that language’s term being both richer and more ambiguous. But, as our intention here is to disambiguate, we will insist on the English language sense of the term compliance.
Process Compliance with external regulations and norms
How may a process comply with external laws and regulations? In general, these laws do not regulate the activities of the process, but regulate instead the output of the process activities. For example, the law might require that a company’s accounts be a true and faithful representation of its financial situation. However, if that company cannot demonstrate that it follows a recognized accounting process to achieve that goal, it is likely to be subject to a long and unpleasant scrutiny.
In the realm of service management, compliance is more concerned with the data manipulated by a process, rather than the process itself. Laws might require that a given process include, or exclude, certain types of data from the records handled by the process. Countries with bank secrecy laws, for example, might require that records, such as service request or incident records, not contain information about bank customers if those records are visible to people not subject to those same laws (with the egregious exception of the Internal Revenue Service and other U.S. agencies best left unnamed). Or the European Union might require that personal data of EU citizens not be visible in other countries that cannot, or will not, harbor that data safely, as per EU standards. Such safe harboring would normally be ensured by the processes used to handle those data. So, using a standardized process in a consistent way is merely a strategy for achieving compliance of the output of that process. (The same strategy is often used as a means to reduce error, increase productivity and to make a process more effective.)
Industry standards often influence the scope of application of a process. An eminent example would be the application of a compliant change management process to the IT systems used in the design and manufacture of pharmaceutical products. If the company cannot demonstrate compliance with those standards for managing changes, then it might be refused permission to market its products in the more finicky jurisdictions. If a company objects that it is perfectly capable of attaining an appropriate quality for its medicines without the overhead of following such processes, its plaint is likely to fall on deaf ears.
A third type of external compliance would be with a national or international standard, such as ISO 20000-1. But once again, the standard is more concerned with the existence of processes that achieve certain outputs, and the definition and use of clear responsibilities, rather than regulating the detailed nature of the process activities, or who might play the process roles.
Process Compliance with Organization-wide Policies
Often, a process is required, or is encouraged, to comply with certain organizational policies. The best known of such policies concern information security. As we know, the risks to information confidentiality, integrity, availability and authenticity may be managed via organizational controls, technical controls, physical controls or procedural controls. Thus, the existence of a process to handle such risks may be an acceptable way of complying with an information security policy.
Another example of a policy that directly impacts process design would be a policy regarding the segregation of responsibilities. Such a policy would require that potential conflicts of interests be identified and separate roles created to mitigate those potentialities. Furthermore, the mapping of persons or organizational entities to those roles would have to respect the policy. Other examples would be a policy requiring that high risk data require four eyes to modify it, or the segregation of roles to prevent individuals from circumventing process checks.
We may expand this concept of compliance to include the alignment of processes to organizational strategies. I cannot overemphasize the importance of this concept, as it goes to heart of the way organizations are evolving today and why certain industry frameworks are lagging behind, rather than leading, that change.
An excellent example of this would be the adoption of agile methods for the execution of the organization’s activities. Once such a strategy adopted, processes may be assessed for their compliance with that strategy. In such a context, it is questionable if a process designed on pure waterfall principles were compliant with the organization’s strategy of agility. This is perhaps one of the reasons why lean-oriented professionals might prefer to speak of value streams rather than processes.
Process Compliance with Internal Process Policies
Compliance with internal process policies may take two forms. In the first case, the process owner has preferred to articulate a policy for the regulation of the process, rather than articulating the detailed procedures to follow. This is a way of saying to the people performing the process activities, “However you perform those activities, you need to comply with the following policy.” In other words, rather than telling people what they must do, you describe the constraints or limits on what they do. A certain margin is allowed to each process actor to perform the activities, within those limits, in his or her own way.
A good example of this would be the application of a policy to user support concerning the behavior of the support personnel. Everyone might be expected to be cheerful, respectful and polite. However, the policy recognizes that there are many different ways to be polite, depending on the contexts and the personalities involved. No attempt is made to regulate how to comply with the policy.
In the second case, the process execution is assessed in terms of the degree to which it is performed as agreed. Many people mean exactly this, and this only, when they speak of process compliance. To distinguish it from the broader scope of process compliance, we may call this “process adherence”. And when 32’491 people on LinkedIn tell you they have this skill, many of them are talking about how to get people to adhere to performing a process as agreed.
To continue with the above example, support personnel might be provided with scripts telling them precisely what they must say and when, with all the please’s and thankyou’s included. Compliance is measured in terms of having followed the script, rather than in terms of the spirit underpinning it. It might be argued that such forms of compliance are more appropriate to silicon-based, rather than carbon-based, life forms.
But internal process compliance also concerns whether the process itself is indeed triggered or performed when it is agreed that it should be used. Thus, we are often concerned with whether changes are performed under the control of a change management process, or whether new applications are deployed without first having performed the relevant tests, or whether a release process was followed at all.
One might ask what role service level agreements and other types of organizational agreements play with respect to process compliance metrics. Such agreements typically define a set of reference values to which a process should comply. These reference values are, of course indicators of process performance (whether they be key or not). When making an agreement with a customer of a process, that customer is more likely to be concerned with effectiveness and efficiency metrics than with compliance metrics, although some customers might also insist on meeting standards of ethical behavior or environmental standards, for example. However, agreements with teams performing the process activities might be asked to agree to certain levels of compliance, in addition to the typical effectiveness and efficiency metrics. It is important, however, to distinguish between the definition of compliance metrics, as we have described until now, and the compliance with targets for other types of metrics, such as effectiveness and efficiency service level objectives. The latter is a completely different notion of compliance that is out of the scope of a discussion of compliance metrics.
Measuring Process Compliance
Given the breadth of the types of compliance a process might have, we might wonder what is the best way to measure process compliance. After all, we are talking about metrics—definitions of how to measure something. Compared to measuring process efficiency, measuring process compliance is something of a conundrum. It is very difficult to roll up all the different types of compliance into a single measurement, other than in a meaningless statement such as “Compliant” or “Not compliant”. Let’s look at the measurement of each type of compliance in turn.
Measuring whether a process is used or not
The metric itself is fairly straightforward. For example, if 100 changes are performed and 98 of them are done following an agreed change control process, then we might measure the compliance as 98%. The difficulty is not in defining the metric, but in making the measurement. Non-compliant actions do not usually leave traces in the process tracking tool.
However, such measurements may be made either via an auditing procedure or, in certain cases, via technical controls. An auditing procedure depends on the existence of records of process execution and adequate sampling techniques. Let’s take release management as an example. It is possible to take a random sample of the applications currently in a production environment, identifying the versions in use. By checking for the existence of testing records or release records, it might be possible to estimate the percentage of applications that have used those processes as part of their transition into production. Of course, this method is limited by the compliance of those processes in creating the requisite records!
In other cases, identifying whether a process has been used is rather a hit-or-miss affair. Take the example of changes. Only the result of a change, but not the change itself, is tangible. We can only detect unauthorized changes as a result of a laborious cause analysis, or by using technical means to automate the detection of configuration changes of system components. In such cases, the measurement of non-use of the process might have a very low precision and questionable accuracy. The measurement method itself might introduce great bias into the measurements.
Measuring whether a process follows internal process policies
In theory, this sort of compliance could be measured by comparing the cases where a policy was successfully applied to the cases where is should be applied. But this sort of measurement is also extremely difficult to obtain. Measurement systems are likely to be burdensome and unreliable. For example, customers might be asked after receiving support if the agent were polite (as a way of measuring compliance to a politeness policy). For sure, customers are more apt to complain about a lack of politeness, thereby introducing a strong bias in the measurements. Furthermore, one man’s Mede is another man’s Persian. There is hardly any objective measurement of politeness.
Another approach to such measurement is via an auditing procedure wherein the actors in a process are shadowed and their compliance with policies is assessed by the auditor. This is essentially what happens when a supervisor listens in to a conversation between an agent and a customer. Useful measurements depend on an appropriate sampling technique, else the results are heavily influenced by the auditor’s biases. Lacking such audits, wee are reduced to measuring internal policy compliance by looking at the exceptions and complaints that might arise.
Measuring process compliance with general policies, external regulations and norms
Once again, auditing is the principal means of detecting failure to comply with organization-wide policies. It depends on the existence of records that might be audited and on the auditor’s ability to sample those records such that the results may be extrapolated with a minimum of bias in sampling and analysis.
The lack of records does not mean that the process is non-compliant. It merely makes life difficult for the auditor and expensive for the service provider. Once again, the process may be audited by shadowing the actors and drawing conclusions from first hand observations. But such methods may well influence the subjects being shadowed, resulting in performance levels that are different from what might otherwise be the case [as, for example, the Hawthorne effect].
One way of measuring such compliance is to count the audit points raised and somehow factor in the importance of the different points. The “somehow” is important, insofar as most weighting factors used for ordinal statistics, such as “minor, medium, major” are completely arbitrary forms of voodoo statistics. Ideally, the impact of non-compliance issues should be translated into economic terms, which would then allow for an objective aggregation of the results. Even better would be the measurement of the cumulative impact of those issues over time.
We should avoid defining metrics as simple counts, whether they be counts of audit points, counts of complaints, or any other form of count. Counts should always be expressed relative to the key drivers of those counts. This point is easiest to exemplify using an effectiveness metric. Many organizations are tempted, for example, to count the number of incidents (or changes, or problems, etc.) as a process metric. But such counts mean nothing on their own. A very large, technically complicated organization is certain to have far more incidents than a small, simple one. So, we prefer to measure the count of incidents as a ratio to something else, such as the number of users or the number or different technologies in use, or the number of system components, etc. Applying this same concept to a compliance metric based on auditing, we would prefer to state the ratio of audit points to the total number of issues audited, on the assumption that those issues are selected in a random way or in a way that does not unduly introduce measurement bias.
We have seen that a process is subject to a wide variety of controls. Compliance with those controls may be measured, with greater or lesser precision, in a wide variety of ways. Certain controls only concern the design of the process; other controls concern the execution of the process. Controls are usually concerned with the outputs of processes, but may also concern the performance of the process activities, as well as the design of the roles and responsibilities and the mapping of people to those roles.
It can be exceedingly difficult to measure compliance. It is virtually impossible to state categorically if a process is “compliant” or not. Such an assessment would not, in any case, be very useful. More useful would be an assessment of the impact of process compliance of the economic value of performing the process. This would have the further advantage of being directly comparable to measurements of process efficiency and effectiveness (assuming, too, that effectiveness be translated into economic terms). Alas, many organizations ignore the economic cost and benefit of the policies and other controls they put in place.
In a future article, I will make a similar analysis of progress as a process metric.
The diagrams in this article by Robert S. Falkowitz are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Source of the image of the centrifugal governor: “Shaft governors, centrifugal and inertia; simple methods for the adjustment of all classes of shaft governors (1908) (14757599606)” by Collins, Hubert Edwin, 1872- – https://www.flickr.com/photos/internetarchivebookimages/14757599606/Source (no longer available there) book page: https://archive.org/stream/shaftgovernorsce00coll/shaftgovernorsce00coll#page/n15/mode/1up. Licensed under No restrictions via Wikimedia Commons – https://commons.wikimedia.org/wiki/File:Shaft_governors,_centrifugal_and_inertia;_simple_methods_for_the_adjustment_of_all_classes_of_shaft_governors_(1908)_(14757599606).jpg#/media/File:Shaft_governors,_centrifugal_and_inertia;_simple_methods_for_the_adjustment_of_all_classes_of_shaft_governors_(1908)_(14757599606).jpg