97
Cost and Results Analysis
9
Volume 1—Strategy
Table 2 defines and classifies some common
costing elements.
For service delivery costing, an important issue is
how to account for contact (direct labor) vs. non-
contact time of providers with clients, as well as the
proportion of time other resources such as
equipment and vehicles are in use. For personnel,
non-contact time includes room preparation,
administrative and personal time, as well as non-
service days for meetings, training activities and
vacation (Levin, McEuen, et al, PHR Project, 1999).
Decisions must be made about which cost elements
to include and how to define them. Tools and
checklists can be helpful to avoid underestimation
due to the omission of cost elements. In cases where
cost data for certain categories are insufficient, a
decision may be made to exclude those data and to
note the exclusion as a limitation of the study.
Proxy measures and estimation methods may be
applied for some cost elements if data quantity or
quality is inadequate. WHO’s Mother-Baby Costing
Spreadsheet User Guide addresses issues associated
with proxy measures and estimation methods in
comparing the WHO’s Standard Practice Model
(ideal/preferred practices) with the Current Practice
Model (optimal local practices) in a given setting
(WHO, 1998). The WHO document encompasses
18 interventions, defining their scope and the
treatment protocols at different levels of the health
system, and it lists the drugs, supplies and staff time
required for each service. The advantage of this
method is that it allows for rapid assessment through
application of the already-developed instruments.
In carrying out a study of the costs of essential
maternal health services in three African countries,
the Partnerships for Health Reform (PHR) Project
identified the need to adapt the WHO Standard
Practice Model to take into account staffing, practice
and utilization variances in each country. Since
personnel costs account for a large percentage of
most service delivery, training or supervision
activities, the methods used for determining
allocation of personnel time are important. One of
the adaptations made by the PHR team was to
develop a modified method for conducting direct
observation of health workers as a cost factor rather
than applying standard assumptions.
7
Such
adaptations help to account for the gap between
optimal and actual conditions for delivery of a
service. This method makes the results of the analysis
more realistic and accurate, and the recommenda-
tions more feasible. The downside of this method is
that it is more costly and time-consuming than, for
example, recall from provider interviews (Levin,
McEuen, et al, PHR Project, 1999).
Comparing results from different countries presents
an additional challenge. Few international standards
exist to facilitate such comparisons and there is
7
The Cost Analysis Tool (CAT), developed by PRIME II
partner EngenderHealth, also provides for direct
observation of providers to determine time allocations and
account for practice variances. The Cost Revenue Analysis
Tool (CORE) developed by MSH similarly measures
allocation of personnel time and also factors in revenues to
calculate unit costs and estimate sustainability.
Table 2
Classification of costs by inputs
Recurrent or operating costs
Personnel (all types): supervisors, health workers, administrators,
technicians, consultants, casual labor
Supplies: drugs, vaccines, syringes, small equipment
Vehicles, operation and maintenance: petrol, diesel, lubricants,
tires, spare parts, registration, insurance
Buildings, operation and maintenance: electricity, water, heating,
fuel, telephone, telex, insurance, cleaning, painting, repairs
Training: recurrent (e.g., short in-service courses)
Social mobilization: operating costs
Other operating costs not included above
Capital or investment costs
Vehicles, purchase: bicycles, motorcycles, four-wheel-drive
vehicles, trucks
Equipment, purchase: refrigerators, sterilizers, manufacturing
machinery, scales, and other equipment with a unit cost (price) of
$100 or more
Buildings, space, purchase or construction: health centers,
hospitals, training facilities, administrative offices, storage facilities
Training, nonrecurrent: training activities for health personnel that
occur only once or rarely
Social mobilization, nonrecurrent: social mobilization activities
(e.g., promotion, publicity campaign) that occur only once or rarely
Creese and Parker, 1994
96
10
Cost and Results Analysis
Volume 1—Strategy
debate on whether such standards are feasible. This
leads to variances in methods, results and interpreta-
tion, and raises questions related to reliability, val-
idity and transparency. These issues are discussed in
a Reproductive Health Costs Literature Review
prepared by the POLICY Project (Mumford, et al,
1998). This review was carried out in part to provide
more information on the costs of delivering inte-
grated RH services as advocated at the International
Conference on Population and Development in
Cairo in 1994. The authors observe that “variations
in technique [and other factors] may form the basis
for different estimates for the same intervention.”
Mumford, et al, describe and contrast “bottom-up”
costing methods, such as direct observation and
patient flow analyses, which are useful in establishing
direct costs of a service delivery or training activity,
and “top-down” methods based on cost accounting
and application of standards. “Top-down” methods
may be more likely to capture indirect costs such as
overhead, planning and administration and weaker
in terms of direct activity costs. Neither approach
is necessarily “better” than the other. Each has its
strengths and weaknesses and a situation-specific
combination of methods is likely to yield the
best result.
8
Efficiency and effectiveness considerations
Efficiency
A key concept related to costing is efficiency. In spite
of examples such as hospitals overloaded with AIDS
patients in East and Southern Africa, utilization of
services is low in many developing country health
systems, particularly at the primary level. The
reasons for this are varied and complex, but are often
related to resource constraints and the low quality of
services. This results in chronic inefficiency and
contributes to the problem of unmet need for health
services. Improving efficiency of service delivery and
implementing PI interventions that support service
delivery allows facilities to achieve the same results at
lower cost, freeing-up resources that can potentially
be invested to obtain higher-level results. In simple
terms, one can think of efficiency as “doing more
with the same or less.” In addition to improving
8
The previously mentioned Mother-Baby Package Costing
Spreadsheet User Guide and CAT and CORE tools can all
be considered “bottom-up” approaches.
efficiency, another financing goal of health sector
reform is to mobilize additional resources in order to
“do more with more,” recognizing that health
systems are also seriously under financed in most
low- and middle-income countries, even when they
may operate relatively efficiently.
The PHR maternal health study team in Africa
observed that utilization of services can have a
significant impact on unit costs for services where
there is low utilization and unused capacity in terms
of staff, facilities and equipment. Once root causes
for the performance gaps are identified through
application of the PI approach, interventions can be
chosen to increase appropriate utilization and,
thereby, reduce the unit cost of services. These
interventions might include:
1) A public information campaign to stimulate
demand for health services
2) Supportive supervision and other means designed
to increase and publicize quality and improve
utilization of services
3) Establishing incentives for providers to increase
service volume
4) Improving referral networks.
In the context of the PRIME II Project, such
interventions can be assessed in terms of their
estimated costs and anticipated results in order to
assist stakeholders in selecting only those that may
offer the greatest return on investment.
In Mali, another PHR team observed very low
utilization rates for FP/RH/PHC services in target
areas due to lack of awareness of the services,
concerns about user fees and perceptions of low
quality. A quasi-experimental approach was taken to
analyze community perceptions and behaviors and
expenditure patterns. Based on the results, PHR
identified and tested alternative ways to address
equity and ability-to-pay concerns of clients and
potential clients in order to increase utilization and
efficiency of services, and address unmet health needs
(PHR End of Project Report, January 2001). In this
case, PHR employed a consumer and community
focus similar to PRIME II’s work with the USAID
Maximizing Access and Quality (MAQ) Initiative.
85
Cost and Results Analysis
11
Volume 1—Strategy
Effectiveness
Effectiveness can be even more difficult to measure
than costs, especially at the impact level. PRIME II’s
Monitoring and Evaluation Model recognizes three
levels of results: (1) processes and outputs, (2) effects
and (3) impact. Measuring impact typically requires
a longer timeframe than measuring process and effect
results, particularly if the impact is expressed in
terms of changes in health status. Due to factors such
as the timeframe of development projects and the
difficulty of measuring changes in health status,
projects such as PRIME II generally avoid trying to
demonstrate direct health status impacts. Process and
effect results are often used as proxy measures that
can reasonably be expected to generate health status
changes. To the extent they are used, such proxy
indicators should be defined so as to incorporate as
much as possible the direct link to quality or health
status. One example of an indicator that clarifies
direct links is “births attended by trained personnel
and in which the umbilical cord is handled under
sterile conditions,” as opposed to “institutional or
attended births.”
With either health status impact or intermediate
results, before (baseline) and after (results or impact)
data are needed. If data cannot be collected on all
units or events, which is often the case, a sample or
samples need to be taken. Samples should be large
enough to detect any important changes. In
sampling terms, this is known as “statistical
significance.” Technical support may be needed to
help determine sample size for some studies. Since
many of the steps involved in analyzing costs and
results involve estimates or assumptions, it may be
useful to test variations in some of the assumptions
by modifying values within ranges that might
reasonably be expected. This is called “sensitivity
analysis.” Software tools make a sensitivity analysis
easier to conduct by enabling the user to change
values in a formula or do “modeling” more easily,
with automatic recalculation of totals. Sampling is
also addressed in the Toolkit.
In the interest of practicality and limiting costs, it is
preferable to identify results or measures of effective-
ness from existing records, known as secondary data.
Record-keeping systems may need to be improved so
that they will meet data needs. If using existing
records or improving record-keeping systems does
not seem feasible, special studies requiring collection
of newly generated data, known as primary data,
may be warranted. Applications of the PI approach
typically involve a mix of both primary and second-
ary data. Special studies or primary data collection
are more likely to be needed when quality of services
or client-provider interaction is being examined,
since data on these areas are less routinely collected.
There is a lack of consensus on which denominators
to use to compare cost-effectiveness across health and
PI interventions. Couple Years of Protection (CYP)
is a widely used (but also controversial) measure of
effectiveness for contraceptive methods that is often
promoted by donor organizations. New indicators
are needed on the broader range of FP/RH interven-
tions and efforts to improve FP/RH provider perfor-
mance. The state-of-the-art strategies, guidelines,
indicators and better practices related to PRIME II
technical leadership areas (TLAs) and sub-areas are
important to consider as we determine results
measures and denominators for Cost and Results
Analysis ratios. These approaches should consider
issues related to access, quality and integration of
services, political and other factors, and possibly
include methods for weighing these factors to arrive
at a composite measure of improved performance. PI
practitioners should strive to develop results or
improved performance indicators that are descriptive
and detailed enough to address the quality of an
activity and how it was accomplished, as well as
simply showing that it was completed.
Figure 3 juxtaposes the stages in PI and Monitoring
and Evaluation with Cost and Results Analysis to
illustrate how they relate to each other.
35
12
Cost and Results Analysis
Volume 1—Strategy
Figure 3
Relationship between PI stages, M&E stages
and Cost and Results Analysis
PI stages
Getting
project
agreement
PNA and root cause
analysis/selection of
interventions
Design and
development
of interventions
Implementation (including
monitoring)
Project evaluation
Monitoring
and
Evaluation
stages
Country situation and baseline
(diagnosis and needs assessment)
Inputs
Processes
Outputs
Effects
Impact
Cost and
Results
Analysis
Costing-----------------------------→|←------------------------------------Results
Cost and Results Analysis
95
Cost and Results Analysis
13
Volume 1—Strategy
Chapter 3
Linking Cost and Results
Introduction
The term “cost-effectiveness analysis” is often used
generically when speaking of the analysis of costs and
results of activities. Referring again to Figure 3 for
illustrative purposes, “Costing” helps analyze the In-
puts and Processes shown in the row of Monitoring
and Evaluation stages; “Cost and Results Analysis”
then links costs with “Results” over the Outputs,
Effects and Impact stages, mostly focusing on
Outputs and Effects.
Cost-benefit, cost-effectiveness, cost-utility analysis and
return on investment are all terms that could apply to
a PI process and assessment of PI interventions,
depending on the situation and alternatives being
considered. Each of these terms has a specific
meaning and therefore the terms should not be used
interchangeably. The four cost and result analysis
terms are briefly discussed here due to their central
relevance to this strategy. A glossary of costing and
health finance terms is found in Appendix 2.
9
In cost-benefit analysis (CBA), neither the cost nor the
benefit variable is fixed, and the benefit variable
(numerator) and cost (denominator) are both is
expressed in monetary terms. CBA thus compares
the monetary cost and monetary benefit of
alternatives, typically in the form of a benefits-to-
costs ratio. The term CBA is often used in describing
analyses that would more correctly be classified as
cost-effectiveness analysis (CEA) or cost-utility
analysis (CUA)—see the following definitions and
examples for these terms. Put simply, a CBA aids in
deciding whether a particular activity is worth doing
at all (i.e., whether the cost of the activity is at least
offset by its savings or financial gain). For example, if
a performance improvement activity had benefits of
$60,000 and costs of $20,000, the CBA or benefit-
to-cost ratio (BCR) would be as follows:
9 The Glossary of Costing and Health Finance Terms is an
abridged version of a “Health Reform Glossary” produced
by the USAID Partnerships for Health Reform (PHR)
Project. The definitions on this page vary slightly from the
CBA and CEA definitions shown in the Glossary.
BCR = program benefits
= $60,000
program costs $20,000
= 3, or 3 to 1
In this example, the net benefits of the activity are
$40,000 ($60,000-$20,000). This same example will
be used to illustrate the following return on
investment (ROI) example.
Cost-effectiveness analysis (CEA) is applied to
determine the costs and effectiveness of alternative
ways of achieving the same objective. A cost-
effectiveness ratio is expressed as cost divided by unit
of effectiveness for each alternative intervention. The
result or effectiveness value is not expressed in
monetary terms, but in units of results. An example
from child health might be “cost per fully
immunized child.”
CEA can help to identify the most efficient way of
achieving a specific objective. CEA gives guidance on
how to use funds most efficiently when a specified
output (or “desired performance” in PI terms) must
be achieved. For example, if a ministry of health
wants to know which of two training and learner
support approaches will contribute the most to
improved provider performance, they can test the
two approaches and collect cost and results data.
Approach A costs $50,000 and results in 50
providers performing to standard one year later.
Approach B also costs $50,000, but results in only
40 providers performing to standard one year later.
The CEA for each approach would be as follows:
CEA for Approach A = program costs
= $50,000
units of results 50
= $1,000 per provider
performing to
standard
CEA for Approach B = program costs
= $50,000
units of results 40
= $1,250 per provider
performing to
standard
The CEA shows Approach A to be the more
effective, costing $250 less than Approach B for each
provider performing to standard. Approaches A and
B could also be assessed prospectively in the context
92
14
Cost and Results Analysis
Volume 1—Strategy
of applying the PI approach. To do this would re-
quire analyzing the estimated rather than actual costs
of each approach, along with the estimated results.
Cost-utility analysis (CUA) compares the cost of
alternatives with the subjectively determined ratings
(benefits or effectiveness) of those alternatives. CUA
may be used when effectiveness cannot be objectively
measured due to lack of data, lack of resources for
special studies, or other factors such as time con-
straints. An alternative form of CUA applies the
concepts of Disability Adjusted Life Years (DALYs)
or Quality Adjusted Life Years (QALYs), developed
by the World Bank and others in the 1990s, in
attempting to provide more objective denominators,
particularly for sector-level analysis and policy
support. DALYs and QALYs apply population-based
formulas to estimate years-of-life-saved, with weight-
ing for quality-of-life factors such as disability, in
measuring the impact of alternative health interven-
tions. (See the Glossary in Appendix 2 for definitions
of these terms.)
A recent publication, Development of WHO
Guidelines on Generalised Cost-Effectiveness Analysis,
discusses issues related to application of CEA as a
tool to aid in allocating a fixed health budget
between health interventions in such a way as to
maximize health in a society. This concept is referred
to as “sectoral cost-effectiveness analysis” (Murray,
Evans, Acharya, Baltussen, 1999). For example,
where “lives saved” is the desired result, an analysis
might show that more lives would be saved by
devoting most of a country’s health budget to
preventive measures such as expanding potable water
supplies and reducing incidence of malaria than by
applying those same resources to another mix of
activities. For a variety of reasons, these sorts of
analyses have not been performed in many countries;
when they have, applying the results is complicated.
For example, although more lives might be saved by
expanding potable water and reducing incidence of
malaria, there is a humanitarian imperative to
prevent childhood communicable diseases and treat
victims of illnesses and accidents. Thus, while the
results of a “sectoral cost-effectiveness analysis” may
provide valuable planning information, those results
must be weighed against other considerations in
making resource allocation decisions.
Return on investment (ROI) is similar to and based
upon a benefit-to-cost ratio in a CBA, where both
benefits and costs are shown as monetary values,
except that ROI is expressed as a percentage. Using
the example where program benefits are $60,000
and program costs are $20,000, ROI is calculated
as follows:
ROI (%) = net program benefits
x 100
program costs
= $60,000 - $20,000
x 100
$20,000
= 2
x 100 = 200%
1
Note: Net program benefits = program benefits –
program costs
The example shows that the activity being evaluated
has a 200% return on investment.
Figure 4
A cost-effectiveness analysis
of PI approaches in Indonesia
Operations research results from the Quality
Assurance Project demonstrate the cost-effectiveness
of alternative approaches for improving client-provider
interaction (CPI) among clinic-based FP service
providers in Indonesia. The study compares the cost-
effectiveness of combinations of training, training +
self-assessment, and training + self-assessment +
peer review to improve performance. It measures
direct and opportunity costs to providers for the
combinations of interventions and uses average
number of utterances per counseling session at
baseline and follow-up as the primary impact indicator,
with analysis also on the type and quality of utterance.
The cost and effectiveness data are combined to give
a cost-effectiveness ratio for the percent gain in utter-
ances per dollar of cost. This result is provided for two
categories of utterance: facilitative communication and
provision of medical and family planning information.
Kim, Putjuk, Kois and Basuki, 2000
100
Cost and Results Analysis
15
Volume 1—Strategy
Definition and data issues
As noted earlier, a key constraint in costing
alternative PI and other training and learning
approaches in international settings is the quantity
and quality of cost- and results-related information.
In the context of specific countries where PRIME II
works and in close collaboration with in-country
teams, PRIME II develops and applies instruments
and checklists to assess the availability of financial
information for conducting cost and cost-
effectiveness analyses, and for defining and
measuring results. PRIME II’s Performance
Monitoring Plan (PMP) provides a framework of
indicators and targets for measuring results, with
accompanying guidance on collecting and analyzing
data for reporting. Tools developed for costing and
Cost and Results Analysis help link costing elements
with the PMP and other PRIME II program
planning and results reporting at the country and
regional levels.
PRIME II’s Cost and Results Analysis strategy and
tools consider “life cycle” costs, including financial
and opportunity costs to providers and clients where
relevant, to ensure that PI interventions are sustain-
able. “Life cycle” costs refer to the costs likely to be
incurred over the full period of implementing an
activity. Omission of “life cycle” costs can lead to
underestimation of costs and distort the results of a
cost-effectiveness analysis.
Decisions must be made on a case-by-case basis as to
whether to include capital costs, such as the purchase
of equipment, and how to measure and allocate
personnel costs for a costing or cost-effectiveness
study (Dmytraczenko, Levin, et al, 1999). In
measuring personnel time and associated costs, direct
observation is generally preferable to provider recall
as a data collection method. In cases where there is
unused capacity of facilities, equipment and/or
personnel, a study may choose to focus only on
incremental costs of interventions being compared.
This approach is based on the rationale that the
unused capacity costs are already incurred or “sunk”
costs. Omitting the “sunk” costs of unused capacity
would not be appropriate in a situation where
elimination or conversion to alternative use of the
unused capacity resources is being considered.
From the perspective of our host-country counter-
parts/clients, costs of PRIME II technical support are
generally not considered in such analyses, as these
inputs are donor-supported, time-limited and not
sustainable by host countries. This is not to say that
other factors related to the need for and cost of
technical support should not be considered, simply
that considering the cost of international technical
support may not be useful to counterparts in
informing decisions, even though its availability is
valuable or necessary. An alternative might be to
make budgetary allowance for a transition from
international to local technical support, if required.
A critical step in any cost-effectiveness analysis is the
preparation of a checklist(s) of data to be collected
and the sources from which it should be obtained.
Definitions, instruments and methods need to be
feasible and compatible with counterpart institution
capacities and systems, and with PRIME II capacities
and resources. Considerable care must be taken in
adapting models and tools so that local conditions,
practices and data are taken into account.
An alternative term to cost-benefit analysis
or cost-effectiveness analysis
CBA and CEA are scientific methodologies requiring
a degree of rigor in data collection and comparison.
CEA encompasses data issues such as study design,
sample size, statistical significance and confidence
levels, and other adequacy measures in order to
establish causality between an intervention and
an outcome. This entails the use of pilot and
control groups and other measures that may
not always be feasible.
In certain instances it may be more realistic to
conduct a Cost and Results Analysis that serves the
same objective as CBA or CEA, but that takes into
account factors such as data limitations, overall
objectives and time and resource constraints. A Cost
and Results Analysis associated with a given
performance objective can serve the same decision
support purposes as more rigorous CEA. This is not
to say that CEA or CBA should not still be pursued
when possible. The point is that CEA may be seen at
one side of a decision support continuum where time
and resource investments are high, and that adequate
decision support results may be obtained at other
points along the continuum. It is useful to consider
the Pareto Rule, also known as the 80/20 rule, a rule
of thumb that postulates that 80 percent of resources
are utilized in activities that produce only 20 percent
of the output or results (see Appendices). As applied
27
16
Cost and Results Analysis
Volume 1—Strategy
to costing, the 80/20 rule suggests that effort should
focus on those components that have the greatest
impact on resource use and/or are programmatically
possible to change.
An analysis of the community pneumonia volunteer
program in Honduras by the PHR Project illustrates
a case in which a decision was made to conduct a
cost and effectiveness study (or Cost and Results
Analysis) instead of a CEA (Perdomo and Holley,
1998). Despite a rigorous CEA study design, the
authors encountered recording errors related to the
functional literacy of the volunteers, inadequacy of
cost accounting systems, and issues related to sample
size and incomplete reporting from health areas and
facilities. These factors caused the team to reclassify
the work from a CEA to a study of cost and
effectiveness. The study still produced valuable
results, documenting the costs and effectiveness of
the program, ways to improve it, and financial
requirements for expanding it. PRIME II seeks
similar information from its cost and results
analyses, whether they are prospective, as may
occur in a PNA, or retrospective, as in an end
of activity evaluation.
Documents you may be interested
Documents you may be interested
