Conservation Program Evaluation
| In November, 2010, the ECO released volume 2 of its Annual Report on the progress of activities in Ontario to reduce or make more efficient use of electricity, natural gas, propane, oil and transportation fuels. Click here for more information on this report, including videos and communications materials. | |||||
|
|||||
Contents |
Introduction
The ECO’s analysis of progress on government targets (section 3.0) and conservation initiatives (section 4.0) is informed by program-specific evaluations undertaken by (or on behalf of ) conservation program administrators. At its simplest, an energy conservation program evaluation attempts to estimate the energy or peak demand savings attributable to an initiative or program.
Estimating energy savings is a two-part task. First, gross energy savings are calculated. Second, that figure is converted to net energy savings, by estimating the percentage of gross savings that is attributable to the influence of the conservation program. Net savings are almost always lower than gross savings.
| Calculating Energy Savings |
|---|
Estimating gross energy savings requires tracking program participation and estimating the energy saved by each program participant. Methods of estimating energy savings include:
To estimate gross lifetime energy savings, it is necessary to estimate persistence (the amount of time a conservation technology or behaviour can be assumed to provide savings). Converting gross energy savings to net savings is more difficult. It usually requires surveying at least a sample of program participants to understand more about their behaviour and motivation for participation. The most important element of converting gross savings to net savings is the free-ridership rate. The free-ridership rate is the percentage of program participants who would have undertaken the conservation action even in the absence of the program. For example, a program that offers a $5.00 incentive coupon to purchase an expensive, energy-efficient industrial boiler will have a very high free-ridership rate. Almost all of the program participants who redeemed the coupon and purchased the energy-efficient boiler would have bought the boiler anyway, making the net savings attributable to the program very low. The opposite of free-ridership is spillover, where an energy conservation program induces additional energy conservation actions beyond the program’s direct scope. For example, a program providing an incentive for the purchase of a programmable thermostat may lead some participants to think more about their energy consumption and undertake additional actions, such as upgrading to a more efficient furnace. Take back (leakage of conservation savings because of the tendency of some consumers to use a more efficient piece of equipment more frequently than their previous inefficient model) also has an impact on determining net savings. Understanding the effect of these variables on conservation savings is an evolving science. |
Evaluations usually do more than just estimate energy or demand savings. They often include cost-benefit analyses, using one of the tests described in “Measuring Success”, as discussed below. Evaluations may also examine the efficiency of the program’s processes or its influence on transforming the marketplace to encourage future energy conservation actions.
Depending on the program’s goals, evaluators may also be interested in measuring additional impacts, such as jobs created or greenhouse gas emissions reduced.
There is an inherent trade-off between maximizing the value and accuracy of evaluation results and minimizing the cost of evaluation. In the United States, the National Action Plan for Energy Efficiency has suggested that three to six per cent of the total cost of conservation programming should be devoted to program evaluation.
| Measuring Success |
|---|
| Most conservation evaluations use one or more quantitative tests that attempt to compare the benefits and costs of the conservation initiative. These tests may be mandated. For example, natural gas utilities4 in Ontario are required to demonstrate that the benefits of each of their conservation programs are expected to exceed the costs, using the Total Resource Cost test.
The Total Resource Cost (TRC) Test compares the net benefits of an energy conservation program (due to avoided costs of energy supply, transmission, and distribution) with the net costs (program administration costs, higher capital costs of energy efficient equipment, etc.) to all parties, including program participants and the program administrator (typically a utility or the government). The TRC test is the most commonly used test in program evaluation. The Social Cost Test is similar to the TRC test but broadens the focus to include externalities – costs or benefits not borne directly by the program administrator or participant that are typically left off the balance sheet. For example, the greenhouse gas emissions avoided through a conservation program could be treated as a benefit. Determining the appropriate value for externalities can be contentious. Neither of the above tests considers how costs and benefits are distributed. From this perspective, it does not matter whether the incremental capital cost for an energy-efficient piece of equipment is paid for by the utility or the consumer, or by some combination of the two. Other tests view costs and benefits from the perspective of the program administrator, usually a utility or agency charged with delivering conservation (Program Administrator Cost Test), a consumer participating in the program (Participant Test), or a utility customer (Ratepayer Impact Measure Test). Quantitative tests are of great value in assessing program performance, but they do have limitations. The numerical focus may provide misplaced certainty as to the true value of a program, given that there are inherent difficulties in quantifying some of the variables, such as the appropriate value for social or environmental externalities. Another point of uncertainty is that comparing costs and benefits often requires choosing a discount rate, which attempts to value how much a dollar spent (or saved) at some point in the future is worth today. Because the costs of a conservation initiative usually occur immediately and the benefits accrue over a longer timeframe, the choice of discount rate may greatly affect the program evaluation results. An energy efficiency incentive that has a high up-front cost, but provides benefits through reduced energy costs over 20 years, will fare better (using any of the tests above) if a low discount rate is used. Quantitative tests are also less useful for certain types of rograms, such as informational programs or market transformation programs, where results are inherently more difficult to measure. |
Why Evaluation is Important
Looking backward, the primary purpose of evaluation is accountability. Evaluations can assess whether ratepayer or taxpayer funds were wisely spent and whether program or policy goals were successfully met. Evaluations often have financial consequences, as program administrators (particularly regulated utilities) may receive incentives or penalties that are tied to the amount of energy saved by their programs.
Looking forward, the primary purpose of evaluation is improvement. Evaluations can determine what did or did not work in a program, and may provide information that could be used to improve the program and deliver additional energy savings. They can also be used to determine whether a program should continue at all, or whether the program funding would be better spent elsewhere.
Taking the broader view, evaluation plays an important role in measuring the value of energy conservation as a whole. Can conservation deliver reliable energy savings? Is it more cost-effective than energy supply alternatives? Are conservation programs a good use of public funds? Program evaluations provide the bedrock of information necessary to start addressing these bigger questions. They inform the debate among policymakers regarding the amount of resources that should be devoted to supporting energy conservation.
Evaluation in Ontario
The major conservation actors in Ontario – the Ontario Power Authority, the natural gas utilities, and the Ministry of Energy – have different processes in place to evaluate conservation programs.
Ontario Power Authority – Electricity Conservation Evaluation
The Ontario Power Authority (OPA) has explicit evaluation protocols in place for both conservation6 and demand response programs,7 and a dedicated internal evaluation team. The OPA intends to conduct detailed evaluations of each program within the first three years, although partial evaluations may be done more frequently to facilitate timely improvement to program design. Most evaluations are conducted by a third party, which is intended to ensure an independent and unbiased verification of results. The ECO is generally impressed with the level of rigour that has gone into these evaluations, and welcomes the OPA’s decision to make the evaluation reports public.
A strength of the OPA’s approach is that it publishes detailed measures and assumptions guides listing the estimated energy and demand savings associated with a wide range of energy efficiency measures, along with supporting references. These guides are updated annually by the OPA, and the OPA welcomes submissions (new measures or improved savings information for existing measures) from other parties. The refinement of these measures over time leads to improved estimates of energy savings and improved predictive power. This helps to assess in advance whether a new conservation program under consideration will be worthwhile. This approach to standardizing assumptions of energy savings for common conservation measures has recently been adopted in the natural gas sector as well.
Beginning in 2011, the OPA’s evaluation approach will be extended to local electricity utilities. With the exception of province-wide programs co-ordinated by the OPA, utilities will be required to submit independent evaluations of their conservation programs to the Ontario Energy Board (OEB). These evaluations must be performed by OPA-approved evaluators and use OPA’s evaluation protocols, cost-effectiveness tests, and lists of measures and assumptions. Given that utilities will be eligible for financial incentives tied to the performance of their conservation programs, there is a need for this type of rigorous evaluation process.
Enbridge and Union Gas - Natural Gas Conservation Evaluation
For natural gas conservation, the two large gas distribution utilities (Enbridge Gas Distribution and Union Gas) are also eligible for performance-based incentives, again requiring a rigorous evaluation framework. The framework for gas evaluation was developed by the OEB in 2006. This new framework has led to an improvement in the quality of gas conservation program evaluation, although some concerns remain, particularly with regard to the evaluation of customized commercial and industrial projects.
Each gas utility is required to develop an evaluation plan, and file an evaluation report annually with the OEB that summarizes the energy savings, budget and supporting evaluation work for the utility’s conservation portfolio as a whole. The utility must also submit a thirdparty audit of the evaluation report to the OEB. The primary purpose of the audit is to review the assumptions that the utility has made in determining conservation program savings (which affect the financial compensation that the utility will receive), and provide an opinion as to whether the proposed compensation to utilities is reasonable. An issue currently under consideration by the OEB is whether the auditor should be selected by the utility (as is the current practice) or the OEB.
A unique feature of conservation evaluation in the natural gas sector is the role of the Evaluation and Audit Committee. This committee is composed of the utility and three stakeholders. It has a role in contributing to the utility’s evaluation plan, reviewing and commenting on the utility’s evaluation report, recommending priorities for future evaluation subjects, and providing input on evaluation methodology.
Ontario Government - Multi-Fuel Conservation Evaluation
In contrast to the OPA and the gas utilities, the Ministry of Energy lacks a formal evaluation framework document to guide conservation program review, and has made limited efforts at program evaluation. For the Ontario Solar Thermal Heating Incentive (OSTHI) and the Municipal Eco Challenge Fund (MECF), savings are self reported by applicants. For the Home Energy Savings Program (HESP), energy and demand savings are calculated by Natural Resources Canada. For more information on these programs, see section 4.0.
To date, Ontario’s role in the OSTHI and HESP programs has been limited to providing additional incentives to complement existing federal programs. Ontario may have determined that there was limited value in devoting resources to evaluation, given the limited opportunities to use the evaluation results to influence and improve program design. However, going forward, the federal government has not indicated that it will provide further funding in these areas. This opens up a greater role for provincial autonomy in program design, and suggests that more resources should be devoted to evaluation. The Ministry of Energy has begun undertaking its own internal analysis of HESP while it considers any future programs post March 31, 2011. This is an encouraging development.
Issues and ECO Comment
A recurring theme of this report is that incomplete evaluation of government conservation programs has made it difficult to assess the benefits and energy savings that these programs are delivering. This is seen in our analyses of the electricity conservation target in government operations (section 3.4), the Home Energy Savings Program (section 4.4), the Ontario Solar Thermal Heating Incentive program (section 4.5) and the Municipal Eco Challenge Fund (section 4.6).
The lack of evaluation and verification of results makes it difficult to convince the public and decision-makers of the value of these programs (which are likely considerable), and introduces a risk that funds for conservation programs will not be made available in the future.
The ECO recommends that the Ministry of Energy establish a formal evaluation framework for its energy conservation programs and publicly report on program performance annually on a going forward basis.
Evaluation, building on the models of the Ontario Power Authority and the gas utilities, would take advantage of lessons learned, and enable meaningful comparisons of results. Ideally, programs offered by other government ministries, where energy conservation is a primary program goal could also be addressed. Through evaluation, the government would increase transparency as to how funds for conservation programs are being used. This is particularly important as the government passed a regulation in 2010 to recover part of the funds for these programs from electricity ratepayers, not from the tax base. The government would also gain knowledge of program successes and failures, facilitating the optimization of future conservation programs.
Public reporting and free information sharing will also help improve the design of conservation programs in Ontario. Many of the lessons learned as to what makes for a successful conservation program will be of value to other parties offering conservation programs.
Evaluation, building on the models of the Ontario Power Authority and the gas utilities, would take advantage of lessons learned, and enable meaningful comparisons of results. Ideally, programs offered by other government ministries, where energy conservation is a primary program goal could also be addressed. Through evaluation, the government would increase transparency as to how funds for conservation programs are being used. This is particularly important as the government passed a regulation16 in 2010 to recover part of the funds for these programs from electricity ratepayers, not from the tax base. The government would also gain knowledge of program successes and failures, facilitating the optimization of future conservation programs.
Public reporting and free information sharing will also help improve the design of conservation programs in Ontario. Many of the lessons learned as to what makes for a successful conservation program will be of value to other parties offering conservation programs.
