Benchmarking compares the actual performance of a company against a reference or benchmark performance (often, the actual performance of other but similar companies). It is being increasingly used by regulators around the world, particularly in relation to the incentive or CPI-X regulation of the natural monopolies such as water, electricity, and gas distribution and transmission businesses. This form of regulation provides incentives for companies to improve efficiency by reducing costs below the regulator's assumed ‘efficient’ costs and retaining the gains at least until the next price review. However, assessment of efficient levels of costs poses a challenge for a regulator who does not have robust information on such costs as the regulated business. The objective of benchmarking is to reduce such information asymmetry on the achievable efficient levels of costs by reducing reliance on the company's own costs.However, benchmarking is not without its difficulties, both conceptual and practical. The choice of comparators and techniques used to assess the efficiency are arguably the two key issues. Economic regulation if based on imperfect benchmarking measures, may create its own problems with incentives. Both, the benchmarking process and the results therefore need to be treated with care. In particular, benchmarking should be regarded as a regulatory tool complementing other approaches and not as a substitute to decision-makers and their judgments.This article discusses the benefits as well as limitations of benchmarking and highlights some important regulatory issues. Different benchmarking measures and techniques are also discussed in context of their use in regulation.</sec><sec id="sec001"><title>Rationale for using benchmarking as a regulatory tool

Natural monopolies, such as network businesses, pose a common problem to the regulators. Regulators cannot rely on direct competition to put downward pressures on the costs and hence on the profits. The problem is complicated by the fact that the regulator often needs to rely on the company for information on its costs.

The economic regulation approach which sets prices on the basis of the company's own costs has two key problems. First, regulators do not have good information on the scope of efficiency improvements that the company can make. Second, because the company knows this to be the case, it has incentives to influence the price-setting system by artificially or unnecessarily inflating its recorded and projected costs, both capital and operating. While the regulator can attempt to reduce the difference in information by conducting extensive reviews of the company's cost efficiency or its investment proposals, these types of reviews can never provide a full answer. Further, there is a risk that the regulator ends up as a de facto manager of the management of the regulated company, with major decisions by the company requiring regulatory approval, leading to a loss in accountability.

Benchmarking provides one way out of this dilemma by using the information over which the company has no influence in setting its prices, but which nonetheless provides a reasonable indication of what the company's costs should be. Setting prices of a firm according to other companies' costs rather than its own, gives strong incentives for the company to improve efficiency. This is how the firms behave in a competitive market. Benchmarking therefore helps the regulator create a regime that stimulates a competitive market for natural monopolies by setting prices in line with an exogenous benchmark or yardstick.

Setting prices purely on the basis of benchmarking would be challenging given the practical problems of finding good comparators as well as comparable data. But using benchmarking to set prices is not an all-or-nothing approach. Benchmarking could be one of the several pillars for setting prices.

Efficiency measures

In benchmarking, efficiency is considered as a relative concept. It is measured by comparing an organization's actual ratio of outputs to inputs to the ‘optimal’ ratio of outputs to inputs. For natural monopolies such as distribution and transmission businesses, operating and capital costs are often considered as the inputs and peak demand, units transmitted or distributed, and number of customers as outputs. For example, efficiency of a distribution business can be measured in terms of operating expenditure per customer or per-unit distributed.

When the production process involves a single input and a single output, this calculation is a trivial matter. However, when there is more than one input or output (which is often the case) then a method of aggregating these inputs or outputs into a single index of inputs or outputs is required to obtain a ratio measure of productivity. The total factor productivity is a measure, which can involve all inputs and outputs.

Economic efficiency has two main components: technical efficiency and allocative efficiency. Technical efficiency is the measure of an organization's ability to convert inputs into outputs, that is, to either maximize outputs for given inputs or produce the given outputs with minimal inputs. The management practices or size of an organization can however influence technical efficiency. Accordingly, technical efficiency is broken down into pure technical efficiency and scale efficiency.

Scale inefficiency is usually not attributed to managerial inefficiency because the size at which an organization operates is often outside the control of managers, at least in the short term. A scale-efficient organization can produce a similar, proportionate increase in output for a proportionate increase in input. An organization that can produce an increase in output higher or lower than the increase in input, is considered to have increasing or decreasing returns to scale. Arguably, such organizations should either increase or reduce production size until they reach the more efficient point of production at the CRS (constant return to scale).

In benchmarking, the CRS assumption is only appropriate when all organizations are operating at an optimal scale. Factors such as imperfect competition, slow demand growth, and financing constraints may cause an organization to be not operate at an optimal scale. The CRS-based comparisons can lead to larger organizations being compared to smaller ones and vice versa. However, comparisons between organisations on the VRS (variable return to scale) basis ensure that organizations with similar levels of outputs and inputs are compared with each other.

Allocative efficiency is a measure of the ability of an organization to utilize a combination of inputs to minimize the cost of production, given the input prices. To assess the allocative efficiency, a detailed decomposition of company production costs and prices is required. However, the latter is often not readily available for natural monopolies.

Benchmarking techniques

As shown in Figure 1, benchmarking techniques range in form and complexity. ‘Informal’ techniques assess the companies on one or more ratios between one input and one output, often referred to as partial productivity measures. In contrast, ‘formal’ benchmarking techniques, such as regression analysis, DEA (data envelopment analysis), and SFA (stochastic frontier analysis) use information on multiple inputs and multiple outputs of companies. Figure 1Benchmarking techniques

The OLS (Ordinary Least Square) regression analysis determines an ‘average’ efficiency benchmark for companies, whereas the DEA and SFA, and COLS (corrected OLS) regression construct an efficiency frontier showing the most efficient businesses. These techniques allow explicit recognition of different geographic conditions and circumstances in which businesses operate, and of differences in scale of operation and capital-labour ratios. Each business is then assessed in terms of its position to the efficiency frontier.

The efficiency score of a firm that lies on the efficiency frontier is always 1 (or 100%). If a firm has an efficiency score of 0.70, it means the firm must reduce its inputs (or increase its outputs) by 30% to produce the same output (or using the same input) as an efficient firm produces (or uses) and become an efficient firm.

This allows the regulator to assess the efficiency of a company in two important aspects: first, how efficient the company is at present compared to its peers; and second, what improvements in efficiency are required to move to the efficiency frontier (which can also move with time).

The DEA is based on linear programming and computes rather than estimates the relative efficiency of the firms in relation to the efficiency frontier. In contrast, the SFA and COLS are statistical techniques and hence estimate rather than compute the relative efficiencies. Both the techniques require specification of a production or cost function. Similar to the DEA, the COLS technique assumes that all deviations from the efficient frontier are due to inefficiency. On the other hand, the SFA recognizes the possibility of stochastic errors or noise in the measurement of inefficiencies.

The DEA is indeed the frontier technique most widely used by regulators as it can measure the overall efficiency of an organization and decompose it into various measures of efficiency such as pure technical and scale efficiencies to determine whether inefficiency is due to managerial capabilities of an organization, the scale of operations, or both. Further analysis can be carried out using the DEA to determine if some inefficiency is due to the operating environment (which is beyond the control of management), for example, climate, population density, and age of capital. In addition to informing managers about the efficiency of their organizations, the DEA provides the less-efficient organizations with information on an efficient mix of outputs and inputs and identifies the role models, which the organizations could consult to help improve their efficiency using the same input and output mix.

Limitations of benchmarking techniques

Partial productivity measures are widely used to assess the efficiency of organizations because they are simple to calculate and are easily understood. These comparisons can be useful when assessing the productive efficiency for very similar companies in the same situation and environment. However, these measures need to be interpreted with care as they do not take into account the differences between the company and its comparators in respect of size, operating environment, etc., and trade-off between the different inputs and outputs. Nevertheless, a combination of several partial productivity measures can provide a broader assessment of performance. However, it is difficult to determine the overall efficiency of an organization if the measures move in an opposite direction.

Like the partial productivity measures, ‘formal’ techniques are also not without difficulties and drawbacks. For example, being a non-statistical tool, the DEA has a limitation with regard to the measurement error and other noise, that may influence the shape and position of the frontier. That is, there is a limited scope of statistical tests of significance to assist the model design. Expert advice and industry forums are often relied upon while designing the model. Furthermore, the DEA results may be sensitive to outliers in the data, which may, in turn, be due to errors in the data or may reflect other random events such as climate. The outliers distort the frontier and hence reduce the apparent efficiency of similar organizations.

Another important limitation of the DEA (and SFA) is that it is sensitive to the specification of the number of inputs and outputs included and the size of sample. Increasing the sample size cannot increase the efficiency scores but can reduce them because the new organizations may push the frontier outwards. Conversely, inflating the number of inputs and/or outputs cannot decrease the efficiency scores but can increase them because the number of potential peers of each organization could decline. In case of a small sample size and many inputs and/or outputs, many companies appear on the DEA frontier. If an investigator wished to make an industry look good, he/she could reduce the sample size and increase the number of inputs and outputs in order to increase the efficiency scores.

Similar to the DEA, OLS and COLS methods attribute any deviation from the fitted line entirely to (in)efficiency. However, in reality, some of the differences between the actual and predicted best performance may be due to data and measurement errors. While the SFA recognizes the possibility of stochastic errors in measurement of inefficiencies, it requires parameters to be specified in a functional form. Any mis-specification of the production function would add distortion to the estimated efficiency. Further, most pitfalls associated with the DEA are also applicable (in varying degrees) to the SFA.

While no single benchmarking technique is exhaustive or can provide a complete picture of performance, they are also not mutually exclusive. Therefore, a sensible and prudent approach to benchmarking requires the use of several techniques.

Use of benchmarking by regulators

The use of benchmarking in regulation of utilities and transport sectors is well-established in many developed countries. However, the choice of benchmarking techniques and scope of benchmarking results in setting prices vary significantly across countries and sectors.

Partial productivity measures, DEA, and regression analyses emerge as the most widely used methods, particularly in Australia and the UK. A survey of the use of benchmarking methods for electricity transmission and distribution in 17 OECD (Organisation of Economic Co-operation and Development) and 4 non-OECD countries finds that all regulators surveyed have full discretion with regard to the choice of benchmarking method, model, and inputs and hence, a variety of methods are used by regulators. However, there is a notable preference for the non-parametric methods like the DEA, followed by the average-based benchmarking methods.

The role of benchmarking in setting prices also varies across regulators. Some electricity regulators (for example, in Abu Dhabi, Colombia, Great Britain, Norway, the Netherlands, and New South Wales) have used benchmarking as part of the price review process while for some (for example, in Finland), benchmarking has been used outside the price-setting process. In Norway with a large number of distribution businesses, the electricity regulator uses a formalized approach, converting linearly, the benchmarking efficiency scores into efficiency improvements for setting prices. In contrast, in New South Wales (Australia) and Abu Dhabi (United Arab Emirates) with just six and two distribution businesses respectively, there is no formalized procedure for conversion of efficiency scores into efficiency targets.

The scope of benchmarking is often limited to operating expenditure, and capital costs are treated separately. This is predominantly the case in Abu Dhabi, the Netherlands, New South Wales, and the UK where there are relatively fewer electricity distribution businesses. However, the Norwegian regulator uses the total controllable costs (both operating and capital) in benchmarking in light of large number of electricity distribution businesses in the sector.

International benchmarking is often used when a country has a limited number of companies in the industry. This is the case with the regulators in Abu Dhabi, Australia, and the UK. Abu Dhabi has used benchmarking based on the companies across different countries or jurisdictions in a country.

In Abu Dhabi, the water and electricity regulator has also used benchmarking to set prices of water and electricity from state-owned production plants with reference to the competitively bid prices of the independent water and power producers of similar size and technology in the sector. This is similar to the modern equivalent asset valuation technique used in the UK prior to the privatization of generating assets in the early 1990s. For certain plants which do not have suitable comparators in the Abu Dhabi sector, hypothetical benchmark plants of similar size and technology are developed using real market data or comparator plants from the overseas.

Regulatory issues

Benchmarking is a useful tool to counter information asymmetries and support a framework for economic regulation, which has better incentives to meet regulator objectives. However, use of benchmarking as a regulator tool has been subject to considerable criticisms, some of which are summarized below.

Benchmarking, by virtue of selection of inputs, outputs, benchmark companies, and design of model or cost function, is subjective in nature and creates many problems.

Similarly, the rate assumed by regulators in setting prices for transition from the current cost levels to the efficient or target levels is also considered to be subjective and without a robust methodology based on observed data.

Separate benchmarking of operating and capital costs makes no sense as it may combine the minimum operating expenditure from one company with the minimum capital expenditure from another company, and hence may set a cost target that no individual company can meet.

The use of benchmarking is also argued to be inconsistent with the conventional approaches towards estimating the cost of capital or rate of return if regulators set costs at efficient levels but allow companies to earn only a ‘normal’ rate of return derived from the average capital market data.

Use of benchmarking in regulation puts a heavy burden of proof on companies. While a benchmarking technique may assume implicitly that high costs or low efficiency scores are due to inefficiency, in fact, the companies' high costs can be due to a number of factors not captured in the analysis. The companies must find out what those factors are or else the regulator presumes the right to disallow costs because they are ‘inefficient’. The use of benchmarking will therefore deny some regulated companies the chance to recover their costs, even if they are in fact efficient.

If prices are set, which are significantly out of line with the firm's average or short-run incremental costs and long-run incremental costs, undesirable impacts can occur. For example, there will not be an incentive for the firm to meet demand at that price, the firm will go bankrupt, or incentives to invest and to provide desirable outputs over the long run would be compromised.

The use of benchmarking may introduce uncertainties in the regulatory framework. If the benchmarking results are seen as mis-specified, subject to significant uncertainty, or the firm makes either very large profits or losses due to the divergence between prices and costs, regulators would come under heavy pressure in future to adjust the methodology for setting prices.

The use of international benchmarking for industries having fewer firms is inevitably based on inconsistent or incomparable data and firms.

Benchmarking often ignores the quality of service.

While some of these criticisms are specific to a benchmarking technique or a regulatory decision at the time and hence can be addressed by refining the benchmarking models and adopting various techniques rather than relying on a single method, there are some merits in the concerns relating to the uncertainties and risks associated with the use of benchmarking.

The pro-benchmarking response to these concerns is that while incentives to make cost efficiencies are reasonably good under incentive regulation, they are not perfect and no incentive-based regulatory framework is without an uncertainty. An assessment of the achievable efficiency gains must be based on some form of benchmarking, otherwise, the regulator would have to rely entirely on the information submitted by the regulated company. Further, despite a bankruptcy risk, few regulators have simply accepted the firm's own costs at face value, and its financial viability tests and sliding-scale arrangements provide an alternative check on the bankruptcy problem.

The very objective of economic regulation is to mimic the outcome of a competitive market. It is a characteristic of competitive markets that firms' prices are set by reference to competition in the market, not necessarily by direct reference to their own costs. Therefore, there is no a priori reason for an economic regulator to focus on setting prices equal to the regulated firms' own costs, if superior information through benchmarking is available.

Similarly, the concern that separate analysis of operating and capital costs encourages intermediation between these costs also has some merit. While benchmarking should ideally apply to the total costs, this is difficult, given the heterogeneous nature of capital. Nevertheless, in arriving at a judgment on efficient operating and capital costs, regulators may find that the DEA and SFA provide useful information on the optimal mix of inputs.

Further, in practice, a regulator usually does not require the company to achieve the minimum of both, operating and capital costs achieved by different comparators. Rather, an appropriate mix of the cost levels is used and suitable glide-path targets are set to be achieved over time.

International benchmarking is more useful for countries where there are a small number of firms in the industry or where firms are likely to be inefficient. However, international benchmarking raises particular difficulties. The most notable issue is that of comparability and quality of data, which may only be improved in time and requires cooperation among regulators. The relative differences in input prices (for example, wage rates, taxes, and rates of return on capital) beyond the control of the firm may also have to be taken into consideration.

Finally, assessment of performance of a firm should be based on both, quality of service and efficiency. Otherwise, a company could improve efficiency in service delivery by sacrificing the quality of service.

Conclusion

The issue of choosing the most appropriate benchmarking methods and model specification cannot be settled on theoretical grounds. Therefore, benchmarking should not be confined to a particular technique. In each case, the regulator should use the latest techniques such as the DEA, SFA, COLS, and simple ratio-based benchmarking as well as sensitivity analysis to examine the consistency and robustness of results, as shown in Figure 2. Different approaches should use the same set of data and should rank the companies in approximately the same order, ideally identifying the same companies as the best and the worst. Further, all approaches should demonstrate stability over time. Figure 2Use of benchmarking in price setting

The decision on how far benchmarking should be used in setting the prices will need to be based upon the quality of benchmarking itself and the potential benefits that benchmarking can have as against the downside problems. Benchmarking should not be carried out in excessive detail to assess each and every cost component such that it could take the form of regulatory micro-management of the firm's activities.

Broadly, there are two possible benefits of benchmarking: first, regulators gain the confidence to set revenue exogenously rather than with reference to the firm's own costs (thus realizing superior incentives to incentive regulation) and second, regulators would set exogenously determined revenue in any case but more accurate benchmarking enables them to match the expected revenue more closely to the expected costs (avoiding windfall gains or losses for the firm).

While using benchmarking, regulators should also take account of the possible gaming or strategic behaviour of companies, which may attempt to influence the choice of benchmarking methods, model, comparators and variables, and translation of efficiency scores into efficiency targets, as shown in Figure 2. At the time of benchmarking, companies influence such choices so as to present their performance in a more favourable manner, resulting in welfare transfer from customers, or even other companies, to themselves through lower efficiency targets than the true underlying efficiency would suggest. If the same benchmarking technique is consistently used over the price reviews, companies may increase their cost base or delay efficiency improvements in periods leading to the next price review, again resulting in socially inefficient resource allocation and dead-weight loss. In case of frontier-based approaches, such behaviour can also affect the measured performance of other companies.

Finally, benchmarking methods and their raw results should not be regarded as replacements to decision-makers and their judgments. Rather, the primary function of benchmarking methods is to serve as decision-aid tools. Therefore, as in any area of public policy, regulatory decisions should ultimately be based on judgments and some discretion.