Draft Methods for Assessing the Cost-Effectiveness of Micronutrient Programs

Abstract

• Name: Draft Methods for Assessing the Cost-Effectiveness of Micronutrient Programs

• Purpose: To improve the efficiency of programs and make the best use of investments in controlling micronutrient deficiencies by identifying the most cost-effective program options, their costs and, coverage. The tool introduces decision-makers to the use of economic analysis for program planning and assists analysts in estimating the costs and effectiveness of micronutrient strategies.

• Type of Design: Cross sectional
• Sample Size: Each analysis or study must include at least two options and can be used for pre-program planning or for post-hoc evaluation.
• Where Used: The assessment tool was developed based on studies done in Jamaica and Guatemala in 1992-1995 under the USAID/LACHNS project, and was used by country teams in Indonesia, the Philippines, Peru, and S. Africa in the USAID/OMNI project.

Basic Information

Name:Draft Methods for Assessing the Cost-Effectiveness of Micronutrient Programs

Origin: OMNI/JSI Project, USAID.

Basic Description: Guidelines include step-by-step explanations, forms, worksheets and examples on how to set study objectives, design/identify program options, calculate costs, calculate effectiveness, develop final estimates of cost-effectiveness, and interpret the results. It is an aid to program planning and evaluation. It is currently in draft form and needs to be revised to capture the experience of cost-effectiveness studies carried out in Indonesia, the Philippines, Peru, and S. Africa during 1996-1998.

Country Applications:

• Indonesia, to evaluate a national program for supplementing women factory workers with iron/folate tablets.
• The Philippines, to identify the most cost-effective combination of fortification and supplementation programs for reaching a sizable proportion of the national population at risk of vitamin A deficiency. The study was based on existing programs.
• Peru, to identify the most cost-effective alternatives (fortification, supplementation or a combination) for reaching a sizable proportion of the national population at risk of vitamin A deficiency. The study was based on future, planned programs.
• S. Africa, to determine the costs and coverage of EPI-linked vitamin A supplementation and maize meal versus sugar fortification programs for vitamin A deficiency control.

Purpose: The overall goal is to improve the efficiency of programs and make the best use of investments in controlling micronutrient deficiencies by identifying the most cost-effective program options, their costs and coverage. Decision makers can use this information for comparing alternate strategies for delivering micronutrients. Micronutrients can be delivered through different channels such as fortification of foods, distribution of supplements and promotion of dietary changes. The costs and effectiveness of different approaches can vary due to population characteristics in a country or region, access to fortifiable foods, infrastructure, and prevailing beliefs and participation rates.

Type of Methods: Quantitative. Estimates are based on a combination of existing and new data on the costs and effectiveness of different programs.

Design: Cross sectional. The method can also be used to project changes in future costs and effectiveness based on illustrative alternate scenarios of variables that affect cost and effectiveness. The method includes identifying/developing program options, using secondary data or primary data to calculate the costs of each option, calculating the effectiveness of each option, comparing the cost-effectiveness of each option, and identifying the best option. Program alternatives consist of existing programs, modifications of existing programs, or new programs that have not been implemented yet. In addition to providing an empirical basis for selecting/designing programs, the analysis develops cost estimates that can be used for budgeting, generates information that can be used for advocacy, and can provide an assessment of the implications of changes in technical or operational components of programs.

Frequency of Administration: Usually once for planning or evaluation.

Key Users of Information

The results are expected to aid health and food/nutrition program planners in selecting program strategies to combat micronutrient deficiencies, and identifying ways to improve program efficiency.

Examples of how the results of cost-effectiveness analyses were used:

Philippines - Decision to fortify wheat flour as the main strategy, and consider a targeted supplementation strategy to reach target groups not covered through fortification.

Peru - To consider a region-specific strategy for Vitamin A deficiency control that may include wheat flour fortification and supplementation using a campaign approach for distribution.

S.Africa - To consider a region-specific fortification strategy based on preliminary results showing variability in maize meal intakes.

Indonesia - To decide whether to improve technical guidelines for the national program, consider dropping the requirement of a hemoglobin test to screen recipients of iron/folate tablets, and to include iron/folate supplementation in the national health insurance scheme for factory workers.

Guatemala - To decide whether to develop new technology to reduce losses in vitamin A during sugar fortification.

Objectives and Scope of Tool

The objectives are:

• To provide a basis for decisions about strategies for reducing micronutrient deficiencies that takes into account the costs and effectiveness of alternative delivery systems;
• To identify/develop alternate program options;
• To estimate the costs of programs;
• To determine the effectiveness of programs;
• To compare the cost-effectiveness and other attributes of alternate single and combined delivery strategies.
• In this tool, cost-effectiveness analysis methods and examples are applied to micronutrient programs, especially programs for iron and vitamin A deficiency control. The main interventions or delivery systems are:

  • Food fortification, especially sugar/cereal flour.

  • Distribution of supplements, especially prenatal iron/folate and vitamin A capsules for children and postpartum women.

Key Monitoring Needs and Evaluation Questions Tool Seeks to Address

• What are the relative costs of two or more alternative strategies that reach a given target of effectiveness?

• What is the relative coverage/effectiveness of two or more strategies that cost about the same?
• What is the cost profile or pattern of costs (i.e. comparison of different activities or program components, type of inputs, sources of funding, levels at which activities place, etc.) of a given micronutrient program?
• How can costs of a program be reduced without sacrificing effectiveness?
• How can effectiveness of a program be improved without raising costs substantially?

Key Indicators

• Cost per high-risk person protected from the risk of deficiency;
• Cost high-risk person reached with any additional micronutrient intake;
• Annual costs;
• Participant costs;
• Ratio of government to other costs;
• Costs at each administrative level;
• Costs with and without voluntary labor and donations;
• Proportion of high-risk protected against risk of deficiency;
• Proportion of high-risk population that received any additional micronutrient intake.

Research Design

Standard Protocol:
The following steps use the example of designing a cost-effectiveness study for vitamin A strategies:

STEP 1. Define target groups, and the nature and magnitude of vitamin A deficiency (VAD). Identify geographic and seasonal patterns.

STEP 2. Describe behaviors of high-risk groups.Determine possible (or actual) participation in vitamin A supplementation and fortification activities. For example, what is the utilization of health services in areas of VAD? Potential coverage through integration with routine immunizations delivered at facilities, through outreach and mobile clinics; campaign activities such as National Immunization Days (NID) for polio? Can postpartum supplements be provided through maternity services or through BCG/EPI contacts? Community-based health programs; private/public clinics; NGOs, etc.? What are the food consumption patterns of high-risk groups, particularly of vitamin A fortifiable staples, e.g., sugar, cereal flours, oil/margarine, dairy products?

STEP 3. Determine the status of current and potential delivery systems for supplements and fortification. Check convergence with VAD patterns (age groups, frequency of contact, seasonality, geographical, etc.):

• supplements: Review indicators of coverage, quality, and policies conducive to integrating vitamin A.
• fortification: status of staple foods industries, capacity, extent to which processing and packaging are centralized, across-border influx of unfortified foods, state of technology, product transport/storage/turnover, industry collaboration with government, precedents on food quality regulations and track record in monitoring/enforcement.
• dietary change/ information/IEC options: coverage, potential short- and long-term impacts, institutional capacity.

STEP 4. Develop program alternatives to meet national vitamin A program objectives (usually assuring adequacy in at least 80 percent of the high risk population). Exclude alternatives based on safety (e.g., fortification levels exceeding 3,000 RE/day by women), or feasibility. Describe the operational details of each program alternative selected. Determine annual maintenance costs of each alternative, using a combination of 'ingredients' and 'adaptation' methods.

STEP 5. Determine the effectiveness of program alternatives in terms of incremental adequacy gained. Calculate the cost-effectiveness of each alternative.

STEP 6. Identify how efficiency of each alternative can be improved through reducing costs and/or increasing effectiveness. Modify operational details, and recalculate cost-effectiveness. Conduct sensitivity analysis for variables with uncertain values.

STEP 7. Compare the program alternatives for CE and other considerations. For example, who will bear the financial costs? Is the program equitable? Is it feasible?

NOTE: This is meant to be an iterative process even though the steps appear linear.

Lessons from experience:

• Country policies and strategies should define program objectives in terms of desired coverage, against which cost-effective options can be assessed. This is important because cost-effectiveness changes as coverage changes. Economies of scale (i.e. falling average costs) may happen initially but as coverage extends to the most difficult-to-reach this trend may be reversed. Furthermore, the risk characteristics of the population are likely to change as coverage rises --effectiveness in terms of mortality gains may be several-fold higher in the hard-to-reach groups. Therefore one should not compare a highly cost-effective option (in terms of cost per year of adequacy gained) that only achieves low overall coverage with a program that covers a larger population, unless differences in coverage and scale are considered. It is important to state the coverage level at which an intervention is found to be most cost-effective.
• The measure of micronutrient adequacy used in many studies (intake relative to Recommended Dietary Allowances or RDAs) is not a perfect proxy for health outcomes. This is because: a) factors related to illness and bioavailability can modify the impact of fortification or supplementation; b) the impact on mortality can vary among sub-groups due to differences in severity of the deficiency at entry in the program and cause-specific mortality levels; c) achievement of an 'adequate' level suggests a threshold effect that is not borne out by existing studies. Further research is needed to elucidate the relationship of program outputs and coverage with health outcomes (and in the case of iron deficiency in adults, on productivity).
• Some country teams considered adequate intake by women of reproductive age as being equally important as intake by children. This resulted in significantly different cost-effectiveness results. Results of alternate assumptions about priority target groups should be presented.
• Cost estimates should be based on a representative sample of program sites. National estimates derived from sample surveys should take into account variation by scale of program operations, province/district management, infrastructure and logistics, and population characteristics.
• The careful consideration of incremental but true opportunity costs can be difficult when alternatives such as immunization-linked versus stand-alone vitamin A supplementation options are costed and compared.
• The estimation of adequacy in vitamin A intake before and after programs requires the estimation of individual vitamin A intakes at baseline, and modeling the incremental adequacy gained from fortification and supplements at a highly disaggregated (preferably individual) level. Using group averages may bias the results.
• In general, country reports placed adequate emphasis on program background and policy setting, but clear policy conclusions based on the study results required substantial discussion, broad input from various types of experts, and adequate time to take various criteria (not only cost-effectiveness) into account.
• Future studies should take into account anticipated epidemiology trends or changes in consumption patterns or health services utilization. They should project alternate scenarios and revisit the cost effectiveness estimates (and coverage of high risk groups and other decision criteria) with these projections.
• It is important to clarify that the term 'costs' does not refer to budget outlays, but the value of resources used. For example, in vitamin A supplementation programs in particular, a large proportion of 'costs' is actually the value of existing staff or volunteer time. Financial implications for budgets should be clearly separated. A similar concern about fortification costs is that industry should not be assumed to carry the full additional cost burden for higher levels of fortification. A simple calculation will show the reader that when all consumers share these costs they are very small and very likely to be passed on to the consumer by industry without any negative effects on demand for that product. Generally, government or industry statistics contain data on price increases of staples over time, and a study using this tool can show how insignificant the proposed increase in price (per kilogram) will be relative to historical price increases.

Training

Standard protocol:
Two workshops are required: one at the start of the study and one at the end. Participants are then sufficiently experienced to conduct additional studies with limited technical assistance.

Lessons from experience:
During the learning phase, it is recommended that surveyors have access to on-the-ground technical assistance in planning, implementation, analysis and interpretation of results.

Implementation

Standard protocol:
The steps include: development of program alternatives for the study based on discussions with policy-makers and implementers; collection and assessment of existing data; decision to collect new data; survey design and data collection analysis; developing preliminary and final cost and effectiveness estimates; cost-effectiveness ratios calculated; other criteria compared; workshops and discussion with decision-makers and implementers held throughout the process; and final recommendations developed and discussed with stakeholders.

Lessons from experience:
Interviewers: Two main skills groups are required. Costing data should be collected by people with costing, financial or accounting backgrounds. For the effectiveness study, surveyors should have experience with household surveys. If biological samples are collected, additional skills or practitioners are required. The number varies depending upon data needs.

Cost and Financing: Studies noted above cost $30,000 per country study plus technical assistance.

Time Requirements: Ranges from three to nine months. If secondary data on effectiveness is not available, primary data must be collected thus extending the time required to complete the study.

Analysis

Standard Protocol:
Computer Hardware and Software Requirements: Spreadsheets, modeling and multivariate analysis (e.g. SPSS).

Data Entry Techniques or Tricks: None.

Analytical Requirements: Knowledge of nutrition indicators, calculation of adequacy and reduction in risk of micronutrient deficiencies is essential. Assessment of effectiveness and attributing program effects requires knowledge of research design principles. Economic principles of opportunity costs, marginal costs, discounting, etc. are required. Also analysts should be able to project costs from a sample of program sites.

Time and Labor Requirements: At least two to three months are required to develop preliminary estimates, fill in data gaps, conduct sensitivity tests, develop alternate program scenarios and include concerns of the study audience in the analysis.

Reporting

Content: Reports contain documentation of the policy context and decisions to be made; description of program options being compared and why certain options were selected; costs, effectiveness and cost-effectiveness ratios; choice of variables for sensitivity analysis and results. Other criteria for selecting best strategies are discussed.

Flow of Information: From analysts to implementers, decision-makers and stakeholders, especially funders.

Presentation: Tables, bullets on recommendations; studying weaknesses or qualifications in interpreting the results should be noted.

Lessons from experience:
For decision makers it is important to put the findings in perspective. Reports should indicate that the actual numbers are less important than relative costs and effectiveness. The process of detailing program activities, types of inputs and target groups reached often provides insights into improving program performance even without completing the calculations. The concepts of cost-effectiveness comparisons, marginal costs and payoffs, and comparing combined/incremental strategies (rather than individual interventions before a program begin) can be difficult to comprehend. Visuals and graphics are recommended.

Dissemination of Results

Lessons from experience
Significant time should be spent on ensuring that results are well understood by those affected by the recommendations. Examples of key audiences are: industry representatives for fortification programs, EPI program managers for EPI-linked strategies, voluntary organizations and others in addition to government representatives (MOH, trade and industry, quality control/regulatory bodies, agriculture) and donors who support micronutrient programs.

Manuals and Guidelines

The guidelines exist in draft manual format. They are available from USAID/G/HN, and from John Snow Incorporated, Rosslyn, VA.

Contact Person
Tina Sanghvi
BASICS Project
1600 Wilson Blvd.
Arlington, VA 22209
703-312-6800