Program Management in Sexual and Reproductive Health Services

 

Welcome to the programmatic area on program management in sexual and reproductive health (SRH) services within MEASURE Evaluation’s Family Planning and Reproductive Health Indicators Database. This is one of the subareas found in the service delivery section of the database. All indicators for this area include a definition, data requirements, data source(s), purpose, issues and—if relevant—gender implications. Good leadership and management are essential to organizational development, performance, and sustainability. An organization succeeds because of what it does (a shared commitment to accomplish something useful and important) and how it does it (the way it functions, decides, evaluates, adapts, and delegates). Good leadership and management are likewise essential to reproductive health (RH) programs and to the achievement of national RH goals. The indicators presented here describe attributes of a well-functioning system and generally measure progress in the use of a system, the effectiveness of individuals using the system, or the performance of the system.  Key indicators to monitor and evaluate program management can be found in the links at left.   Full Text Good leadership and management are essential to organizational development, performance, and sustainability. An organization succeeds because of what it does (a shared commitment to accomplish something useful and important) and how it does it (the way it functions, decides, evaluates, adapts, and delegates). Good leadership and management are likewise essential to reproductive health (RH) programs and to the achievement of national RH goals. Effective and efficient organizational performance is critical in the face of public health challenges (e.g., AIDS, tuberculosis), health-sector reform (e.g., decentralization, integration, financing), and the needs and desires of an ever-expanding number of clients of RH services from traditionally underserved groups (e.g., adolescents, rural and indigenous populations, men). The objective of management and leadership is to foster and sustain individual employee and organizational performance as well as overall program performance in delivering RH services. As depicted in MSH's Leading and Managing Framework, effective management and leadership consist of a series of essential functions, underlying dimensions, or elements. The functions of managing and leading are carried out simultaneously on a daily basis by multiple individuals at multiple levels of an organization, not just by senior staff.  In many instances, the "boundaries" between the management and leadership functional areas and other functional areas of RH service delivery overlap, because the management or leadership function spans each of the other areas. In other words, an organization's or program's success (or lack thereof) in performance in all of its areas of focus is at least in part attributable to a strength or weakness in its management and leadership. Factors determining how an organization does its work or accomplishes its objectives: effectiveness and functioning of individuals at all levels of the organization; management systems supporting their work; organizational culture; and adequacy of human and financial resources. Ideally, one should evaluate overall management and leadership capacity by taking a long-range view of the organization and the way it develops over time, evolving on the management development continuum characterized by four distinct developmental stages: emergent/initial, launch, consolidation, and mature. At the first stage, an organization begins to develop a particular management component. By the fourth stage, an organization operates extremely effectively with regard to the management component. Organizations (and the programs they support) pass through these stages at different rates, and evolve to the point that they have a clear mission, strong management structures and systems, and skilled leaders, managers, and staff who can effectively use these structures and systems. Management systems or capabilities within a single organization will often be in different stages of development because some managment systems will receive more attention than will others as the organization develops. For example, donors and the organization itself often focus on ensuring that a sound financial management system and practices are in place before they focus on developing an organization's human resource management system. In such a case, the financial management system may be in a more advanced stage of development than is the human resource management system. Organizational performance --which refers to what an organization does and how it does it -- always includes some element of customer satisfaction. One evaluates what an organization does in relation to the goals and objectives it has established. Evaluators should therefore define measures or indicators in relation to the specific long- and short-range objectives set by the organization, many of which are presented in this database. There are two distinct approaches to measuring management within RH programs. The first approach is to use a standard set of criteria based on national or international norms and standards. Some organizations in developing countries seek certification, for example, from the International Organization for Standardization (ISO), whereas others seek accreditation based on national standards. The second approach is to develop indicators for each management area and component in collaboration with the organization as part of an exercise to review and strengthen its management systems. This approach involves an assessment to determine the baseline stage of development of the organization. In this way, program managers and evaluators can tailor the standards against which organizational performance is being measured to the level of the organization's development as well as the specific context in which the organization functions and offers RH services. The indicators presented here are derived from actual experience in designing assessments of RH programs in developing countries rather than from rigid standards that are applied to health care organizations. Methodological Challenges of Evaluating Management and Leadership Self-assessment of performance generates ownership, but it may lack objectivity. Organizations may assess their own performance in a participatory fashion, involving staff from all levels, including board members, with the assistance of an external facilitator or a highly skilled internal facilitator. This is generally cheaper and less time-consuming than external assessments. It allows for organization-wide ownership of the findings and commitment to action because it builds upon existing strengths and addresses major weaknesses acknowledged by the staff. However, self-assessments can be subjective, unless the internal evaluator obtains hard data to verify the findings as part of the assessment process. Moreover, self-assessment requires careful and thorough consensus building among staff involved. External assessment, on the other hand, may be more objective, because it relies heavily on the review of organizational documents in addition to interviews with staff at all levels of the organizations. But it is likely to be time and resource intensive. Also, unless the assessment is carefully designed and implemented, the organization may not accept the findings or act on them.  Throughout this database, the term "evaluator" should be interpreted broadly to include both organizational staff responsible for internal assessments(also referred to as facilitators) as well as persons external to the organization. Many management indicators are not operationally precise. Just as self-assessments can be subjective, so can the actual indicators for good management.  Although efforts have been made to quantify and clearly define indicators, they are still subject to interpretation and varying points of reference.  Evidence of the causal link between management and outcomes remains elusive. Researchers have found it difficult to find conclusive evidence of a causal link between the indicators of program management/leadership and actual performance in developing country settings. For example, research from developing countries has yet to provide conclusive evidence that human resource interventions (e.g., leadership training) result in greater achievement of organizational objectives.  The many intervening variables between human development and end results (e.g., shifts in internal and external conditions, market shifts) make these links difficult to validate. Links between management interventions and organizational outcomes are not clear-cut in research in the public sector, particularly in health care.   Organization of Indicators The indicators describe attributes of a well-functioning system and generally measure progress in the use of a system, the effectiveness of individuals using the system, or the performance of the system. The indicators included in this section focus on four broad management elements found in all organizations: (MSH, 1999a). Mission:  A statement of purpose that frames the values guiding the organization or program and that provides consistency and meaning to its actions. The mission statement answers the question, "What does the organization do and why?"   Strategy:  The approach that organizations and programs select to define the activities enabling the fulfillment of the mission. Strategy answers the question, "How will the organization get to where it wants to go?" Organizational strategies should help prioritize and focus the organization's work, comply with its mission, and respond to the demands of the clients served and to the organization's potential market (clients it has the desire to reach). Structure: Addresses roles and responsibilities, lines of authority, and distribution of responsibility, in alignment with mission and strategy. The structure answers the question, "What is the framework and decision-making structure within which the organization operates?" A well-defined structure encourages individual and team/ group initiative and provides staff clarity in terms of the decision-making authority.   Systems: The interdependent management areas within an organization that allow it to do its work. Organizational systems answer the question, "How does the organization carry out its activities?" Key systems for health organizations and programs include human resource management, organizational planning, information systems, quality assurance, financial management, management of revenues, and management of supplies.     Key Systems for Health Organizations and Programs Human Resource Management The integrated use of systems, policies, and management practices to recruit, maintain, and develop employees in order to meet the desired goals of the organization. Effective human resource management supports employees in carrying out meaningful and satisfying work as well as help an organization to improve its level of performance and impact (FPMD, 1999). Planning A  systematic process to review, modify, and align key elements of the organization's mission, strategy, structure, systems, and program activities in light of changing internal or external conditions. Planning is an essential component in ensuring sustainability. Planning systems cover both annual and long-term planning that further the organization's mission, strategy, and goals/objectives. Management Information Systems (MIS) A set of components and procedures organized with the objective of generating information that will improve health care management decisions at all levels of the health system or organization. The MIS represents a key source of data for calculating indicators on service utilization, coverage, and overall performance; a comprehensive MIS includes distinct subsystems for management areas, such as human resources, logistics management, disease surveillance, finances, and workplan monitoring. Quality Assurance (QA) Improves service effectiveness and client satisfaction. Because QA emphasizes a process of constant improvement in operations, it requires long-term organizational commitment and teamwork (FPMD, 1993). Many elements of a high-functioning quality assurance system are embedded in other management systems. Financial Management Collects, records, and reports data on an organization's financial situation. It provides information that helps finance, program, and senior managers to make decisions about allocating resources. Management of Revenues Planning for revenue generation and diversification of funding sources through constituency building among clients and donors (current and potential). Organizations tend to have greater success in this area if they have multiple funding sources (e.g., donors, government, third party payments, local community, sale of services/products). Management of Supplies (Logistics) Without a functioning supply management system, an organization cannot deliver quality RH services to its clients. ____________ References: FPMD, 1993. "Using CQI to Strengthen Family Planning Programs." The Family Planning Manager 2, No. 1. Boston, MA: Management Sciences for Health.FPMD. 1999. "Human Resources: Managing and Developing your Most Important Asset." The Manager 8, No. 1. Boston: MSH.

Availability of a clear, strategic mission statement

Definition:

The existence of a mission statement, which is a written expression of purpose-- the overall reason an organization exists.

A mission statement is clear and strategic if it:

Data Requirements:

Evidence of a written mission statement; other information on the organization's mission.

Data Sources:

Organizational/program documents including plans, staff orientation materials, policy manuals and statements, and marketing materials.

Purpose:

This indicator provides a summary measure of the existence of a clear sense of direction in the form of a written mission statement. Having a clear mission statement that is well disseminated within an organization facilitates strategic planning and setting priorities based on the needs of current and potential clients.

Although programs and organizations without a formal mission statement may perform in a highly effective manner, a mission statement serves the important function of helping to keep staff focused on the accomplishment of long-term objectives.

Although the basic mission of an organization may remain the same for an extended period, the language of the mission statement must provide a clear focus and a priority for organizational strategies and activities.

This indicator is quantifiable on a scale of 0 to 5, as indicated below.

0 No written mission statement
1 Mission statement exists but fails to conform to the above criteria
2 Mission statement exists and conforms with only one of the above criteria
3 Mission statement exists and conforms with 2 of the above criteria
4 Mission statement exists and conforms with 3 of the above criteria
5 Mission statement exists and conforms with all 4 of the above criteria

Issue(s):

Even with a quantifiable scale for which to measure this indicator, this indicator is still somewhat subjective.  Evaluators should be mindful that although an organization may have a clear, strategic mission statement available, this is a benchmark indicator and a step toward addressing the more important issue, which is the use of the mission statement to guide the organization's objectives and strategic planning.  

A well-aligned strategic plan exists and is disseminated

Definition:

The fit between an organization's strategy and its mission and stakeholder needs

A strategy is the long-term plan according to which an organization aims to reach its goals and objectives through a series of activities, inputs, and results. "Well-aligned" means that the strategy fits the organization's mission and the needs of clients (existing and potential), within its policy and service-delivery environment.

Data Requirements:

Evidence of a written strategy; evidence of the quality of the strategy (coherence with mission and results of client-based needs assessment); evidence of analysis of market conditions and needs; evidence the strategy has been disseminated.

Data Sources:

Review of strategic plan or strategy documents, as well as needs assessment and competitor analysis; interviews with key staff (e.g., managers)

The box below indicates the criteria to be used in calculating a score on this indicator.

Purpose:

An organizational strategy should prioritize and focus the work, should comply with the mission, and should respond to the demands of the clients served and the organization's market. A clear strategy continuously and clearly defines the logic leading from activity to outcomes.

Moreover, the strategy must include evidence of linkage and responsiveness to its clients, community, funders, and potential new markets. The following 4-point scale from MSH's Management and Organizational Sustainability Tool has proven useful in measuring this indicator in reproductive health organizations and programs. 8

Descriptor Score
Organizational strategies are formulated with little concern for the perspectives of
clients and the demands of the market (the wider community).

 1

Client and community perspectives are discussed in formulating organizational
strategies, but there is no systematic assessment of these factors (e.g., no market
studies, no client interviews). There is no mechanism for involving community/
clients in formulating strategies. There is no analysis of competing services.

 2

Client needs and desires have been assessed, and markets for expanded and targeted services and products with the community have been defined. These single assessments are used repeatedly over time to guide the development of strategies. Community/clients are only sporadically involved in formulating organizational strategies. Analysis of competing services is carried out sporadically.

 3

The needs and desires of clients and the demands of the community are frequently reassessed to identify changes over time and to provide the basis for developing organizational strategies. Clients and community are systematically involved in formulating organizational strategies. A mechanism is in place for regularly analyzing competing services.

4

Issue(s):

Even with a quantifiable scale for which to measure this indicator, this indicator is still somewhat subjective.  Evaluators should be mindful that although an organization may have a strategic plan, like a mission statement, this is a benchmark indicator and a step toward addressing the more important issue, which is the implementation of the strategic plan.

References:

8 See Management and Organizational Sustainability Tool (MOST), MSH 1999a.

Percent of annual objectives achieved by the organization

Definition:

Programs or organizations set annual planning objectives at the beginning of the annual planning period and document them in an annual operational plan. Annual objectives should explicitly link to the broader strategic objectives and be achievable within the annual framework. They can include quantifiable outputs (e.g., number of HIV clients counseled) to higher-level outcomes involving behavioral or procedural change (e.g., new ways of tapping into funding sources or improved monitoring techniques).

This indicator is calculated as:

(Number of annual objectives achieved (or exceeded) by the organization / Total number of annual objectives set by the organization) x 100

An alternative indicator is, Capacity to reach annual objectives. The facilitator (evaluator) assigns a score from 0-3 for each objective, as follows.

Objective  Score

There are no stated objectives

0

Objectives partially met 1
Objectives met 2
Objectives exceeded 3


This alternative indicator is calculated as the mean score across all objectives.

Data Requirements:

List of all annual planning objectives; individual scores for each objective.

Data Sources:

Annual planning document; organizational strategy or other multi-annual planning document; semi-annual and/ or annual progress reports (e.g., to stakeholders); assessment by external evaluator or internal facilitator

Purpose:

This indicator provides a "results-oriented" measure of program or organizational planning performance. It is based on the premise that the overarching measure of good management and leadership is whether or not the organization achieves its stated objectives.

Issue(s):

The recommended measure for this indicator can only provide a general picture of overall functioning of the organization; the evaluator will require further in-depth analysis of the reasons for failing to obtain stated objectives.

Availability of logical and explicit organizational structure

Definition:

The structure of an organization refers to the staffing and decision-making framework that assigns personnel according to their authority and level of responsibility. The structure is clearly articulated when it provides clear lines of authority and accountability, distribution of responsibilities, and lines of communication.

Data Requirements:

Evidence that relationships, supervision, roles, and responsibilities have been formally defined; description of how decisions are made in the organization.

Data Sources:

Organizational chart; written job descriptions; policy manual(s); interviews with staff at all organizational levels.

Purpose:

This indicator measures whether an organization has a clearly defined structure - roles, responsibilities, and authority - both "on paper" and in practice. Evaluators can find evidence by reviewing whether the organization has:

Measuring these aspects requires developing a simple scale from 0 to 4. A program without any of the four documents listed above receives the lowest score (0), while one with all elements receives the highest score (4).

Organizational decision-making processes may or may not conform to those embodied in the formally defined organizational structure. To make this determination, one can assess the appropriateness of staff for the positions they fill and the extent to which they actually make the decisions called for in the "formal" document (e.g., by interviewing staff to determine how one or more recent important decisions were made).

Assessment for the indicator will tend to be more subjective than for the previous one, which involved simply measuring the existence of documents. However, evaluators can apply the same type of scale as in the previous indicator (ranging from no conformity to full conformity).

Issue(s):

The validity of the indicator is based on the assumption that a clear definition of roles, responsibilities, and decision-making in an organization promotes strategic and operational decision-making that optimizes the use of available resources. The organizational structure must be (at least partially) amenable to change as directed by the organization's management/leadership.

Gender Implications:

A logical and explicit organizational structure with clear lines of authority and accountability is an important indicator of management capacity. In many organizations, women are plentiful at lower levels of responsibility, but do not reach the top management positions. Many factors contribute to women's ability to rise to management positions in an organization, including their lower levels of school enrollment and literacy. Gender discrimination in the organization can be said to exist when women have the necessary educational qualifications and experience to compete for a particular job but are denied access on the basis of sex. Job descriptions and qualifications can also be written to exclude women by requiring skills or experience that women have no means to gain access to. Logical and explicit structure should include commitment to hiring and promotion on the basis of appropriately identified qualifications.

Percent of employees who have completed their annual performance reviews with their supervisors for the last performance period

Definition:

The organization's adherence to standards when it supervises and reviews employee performance.

Performance review, an assessment of the employee's performance by the supervisor and employee, is ideally based on jointly established work plans, performance objectives, and results related to expectations. The review is the cornerstone of the supervision process and an important element in overall performance management. Performance management is defined as the systems, policies, and procedures used by an organization to define and monitor the work that employees do and to ensure that the tasks and priorities of employees are consistent with the strategy of the organization. Performance reviews address the need of all staff for clear expectations of their work.

This indicator is calculated as:

(Number of employees who have completed their annual performance reviews with their supervisors for the last performance period / Total number of employees eligible for an annual performance review) x 100

Data Requirements:

Description of supervision and performance review standards; evidence that supervision and performance reviews adhere to standards and have been completed.

Data Sources:

Personnel files

Purpose:

Supervision and assessment of employee performance are critical functions of human resource management. Measuring adherence to standards assumes that the organization has written standards defining the requirements for a complete and high-quality review, as well as defining its frequency.

To measure the indicator, an evaluator (external to the organization or the Director of Human Resources for an organization) first reviews written human resources policy documents and establishes the standards for conducting supervision and the requirement for written documentation of the review. S/he then reviews a sample of personnel files (or all files if resources are available or the number of employees is small) and checks the documentation to see if supervision/performance review adheres to standards of timeliness, completeness, and accessibility.

Staff performance reviews should be applied system-wide, regardless of poverty status of clientele.

Issue(s):

Standards for completion vary greatly by organization/ program. For example, some organizations may require signatures by both the staff member and his/her supervisor in the instance of a joint performance review. In other instances, where a supervision checklist appears in "audit" form, only the supervisor may be required to sign the completed document. Some organizations have the additional requirement of documentation of salary changes/promotion; others require a written performance plan for the next period of review (e.g., 12 months).

The indicator assumes a well-documented supervision/ performance review system is in place.

Percent of key positions filled

Definition:

The effectiveness of the organization's human resource management system to fill key positions

The definition of "key" varies considerably by organization. For health service delivery organizations, clinical standards and organizational norms will dictate the definitions of key staff at the clinic level. The total number of required key staff is often determined as a ratio of physicians, nurses, auxiliary nurses, midwives, and other clinical staff to the catchment population. At the headquarters level of an organization, "key" may include department or unit heads, and critical technical support staff. In some countries, in large health organizations, more complex algorithms that include staff time available, type of patient load (inpatient, outpatient, deliveries, community visits, among others) and administration time are used to determine appropriate staffing levels.

Data Requirements:

Accurate count of key positions in the organization that are actually filled at the period under review and total number of key positions available in the organization.

Data Sources:

Personnel management information systems; personnel records; organization's organigram; facility survey; accurate payroll system; personnel policy manual.

Purpose:

The purpose of the indicator is to measure the effectiveness of the organization's human resource management system to fill key positions and thus to ensure organizational capacity to perform and achieve its objectives. The indicator measures actual performance against the ideal scenario in which all key positions are filled. Most reproductive health organizations spend the majority of their resources on staff; an incomplete complement of key staff can compromise quality of care.

Calculation of this indicator requires an organization to have a clearly defined and agreed-upon definition of "filled." Payroll systems may fail to provide accurate information if staff remain on a payroll (with or without pay) when they are on extended or permanent leave; hence the position is "filled," but no one is performing the functions of that staff member.

Because of inadequacies of personnel information systems, some organizations (especially public sector) rely on an annual count of employees as part of their management information system reports to provide the data for this indicator.

Where payroll and annual reporting systems are inaccurate, including questions about human resources on periodic facility assessments is common. A frequently used indicator is, Percent of facilities that have the full complement of staff on duty on the day of a site visit.

Issue(s):

An important limitation of this indicator is that it does not measure the competency of the staff filling key positions. Assessing competency requires a more timeconsuming, qualitative analysis of job descriptions, qualifications of staff in positions, and performance reviews.

Staff turnover rate

Definition:

The rate at which staff are leaving an organization or program as a proportion of the total staff employed during a reference period (e.g., 12 months).

The indicator is calculated as:

(# of staff who vacated their positions / # of staff employed by the organization or program) x 100

Data Requirements:

Accurate, up-to-date counts of staff who have left positions and of total number employed at the midpoint of the reference period (e.g., 12 months).

Data Sources:

Human resources information systems; personnel records; organization's payroll system (if accurate); "head count" survey (in the absence of routine personnel information system).

Purpose:

Staff turnover is an important way to measure both the effectiveness of the human resources management system and the overall management of an organization or program. It provides a complementary measure to the previous indicator on key positions filled. If turnover is high, the organization/program must incur additional costs of hiring new staff; these costs include interviewing, checking references, and start-up training, among others. Because human resources often consume greater than 70 percent of reproductive health program budgets, retention of qualified staff, or lack thereof, can have a very large impact on productivity and performance.

Issue(s):

Whereas this indicator can raise a "red flag" (signal possible personnel problems), human resource managers may lack the authority to solve the root causes of the problem (e.g., supervision, pay scales, promotion). Further understanding of the causes for turnover requires more in-depth analysis. Some organizations require exit interviews of all employees before departure; examination of these records should indicate if turnover relates to job satisfaction, pay issues, retirement, or other factors that the organization or program can address.

Generally, annual analysis is sufficient, although managers may want to examine this indicator more frequently in the case of a perceived increase in attrition. Managers will also want to review it over longer periods of time to facilitate long-term planning for hiring and staff development.

Availability and use of a coherent planning system

Definition:

The availability and utilization of a systematic process for planning.

Planning systems include systematic procedures for short-, medium-, and long-term plans. Their objective is to ensure support for and achievement of the organization's mission, goals, and strategies. A coherent planning system supports the efficient and effective development, implementation, and monitoring of plans.

This indicator measures both the availability and utilization of six key elements during the last planning cycle, as follows:

Available Utilized
Templates/formats for all planners to follow Templates used for most recent plan
Schedules for developing, monitoring, and updating the plan Schedules followed reasonably well for most recent plan
Manuals describing the planning process Manuals referred to during preparation of most recent plan
A mechanism to assure that activities in the plan are linked with budgets Activities in most recent plan are budgeted
A process for monitoring progress Systematic monitoring performed by managers for most recent plan
A system for generating progress reports Progress reports produced on regular basis for most recent plan

 

Evaluators can rank an organization on a 12-point scale, calculated by assigning one point for "available" and one point for "utilized" to each of the six items in the table presented here.

Data Requirements:

Evidence of a documented planning and budgeting process; evidence of existing strategic and operational plans; evidence of monitoring of plans and schedules.

Data Sources:

Organizational documents; interviews with key staff members.

Purpose:

Institutions that perform effective strategic planning can better understand and respond to changes in conditions affecting the organization, and they can more effectively apply available resources to client needs and generate demand for services (USAID, 1999). Managers should refer to such plans when they make management decisions and should monitor and adjust plans continuously to adapt to changing internal and external conditions. Hence, a coherent planning system requires not only the products (plans), but also the process (implementation) grounded in the strategic and budgetary realities of the organization.

Through observing organizational/programmatic documents and through interviewing key informants, an evaluator can consider whether key elements of a planning system exist and whether they have been used during the last planning cycle.

Issue(s):

Like many of the management indicators, one limitation of this indicator is that it is not operationally precise.  It also does not assess the quality and usefulness of the plans that are actually produced by the planning system. However, the next indicator, Number/proportion of organization/program units systematically using information to plan and monitor performance, addresses this issue.

References:

USAID. 1999. Health and Family Planning Indicators: Measuring Sustainability Volume II. Africa Bureau,
Office of Sustainable Development (AFR/SD). Washington D.C.: USAID.

Number/percent of organization/program units systematically using information to plan and monitor performance

Definition:

The capacity of managers to use routine information systems to monitor performance.

An effective management information system (MIS) processes raw data and produces information that allows decision-makers to understand how well the organization or program is performing.  This indicator can apply to information systems designed for producing information to monitor the overall or "bottom-line" performance of the organization/program or the performance of individual management units such as finance or human resources.

Organization/program units in the context of reproductive health refer to either service delivery points, branch offices, or management units/departments within headquarters.

As a percentage, this indicator is calculated as:

(Number of organization/program units systematically using information to plan and monitor performance / total number of organization/program units using information systems) x 100

Data Requirements:

Written (or computer generated) evidence of use of data (e.g. indicator charts, graphs); information system reports; planning objectives; number of units that show evidence of use of data; and total number of units.

Data Sources:

Assessment of the MIS, conducted during routine supervision or by an external evaluator; interviews with key staff.

Purpose:

When calculating this indicator, an evaluator examines the institutional documents specified above for the presence of the following three attributes of systematic monitoring:

Applying this indicator is common in reproductive health programs to measure the use of routine service indicators.  This set should include at least one indicator on each of the basic reproductive health services, such as family planning, antenatal and postnatal care, safe delivery services, STI/HIV treatment and prevention, as well as other programmatic indicators.

The indicator can also apply to a review of other information systems including disease surveillance systems, personnel or human resources, commodities and logistics, finance, and facilities and equipment.

Issue(s):

This indicator assumes that the information available to the evaluator is both timely and accurate.  The following two indicators in this section address these questions: Number/proportion of reporting units submitting a completed routine MIS report on time and Percent of data elements reported accurately in MIS reports.  This indicator does not measure the effectiveness of decision-making based on information that is monitored.  To do so would require a more complex, in-depth measurement process that involves interviews with key informants and intensive document review.

Number/percent of reporting units submitting a completed routine management information systems report on time

Definition:

The extent to which an organization makes information accessible in a timely fashion.

Management information system (MIS) refers to the mechanisms and procedures for the collection and use of routine data.

"On time" means the report is received within a specified time from the end of the reporting period. Common reporting periods are "within 7 days after the start of a new month," "within 14 days of the start of a new quarter," or other set period.

As a percentage, this indicator is calculated as:

(Number of reporting units submitting a completed routine MIS report on time / total number of reporting units submitting MIS reports) x 100

Data Requirements:

Records indicating exact dates when reports were sent from a collection point and received at a unit or office where they are aggregated.

Data Sources:

Log books; stamped and dated reports; computerized MIS databases.

Purpose:

This indicator provides a measure of the extent to which an organization makes information accessible in a timely fashion. The existence of complete and timely information is a pre-condition for measuring the previous indicator regarding the use of information for monitoring. The qualifier "on time" in the definition of the indicator highlights the need to monitor activities and outcomes at a pace consistent with the time frame that leaders and managers set for planning, monitoring, evaluating performance, and decision-making.

However, deadlines for submitting routine reports must be reasonable given the particular challenges to delivery. Experience in several countries has shown that pressure to submit monthly reports on time leads workers to omit some data. Thus, data are not only incomplete but also cannot be compared to data from institutions submitting figures for a full month.

Evaluators can apply this indicator to other types of reports critical to the management of reproductive health services. Such reports may be quarterly or annual performance reports (which draw on data from multiple sources), drug stock-out reports, notifiable disease or event reports, or periodic population-based reports. To be defined as "on time," a report must be received well before information is required for decision-making.

Issue(s):

The indicator does not measure the quality of the report submitted, nor does it measure the extent to which upper level management processes the reported data in a timely manner. A complementary indicator is: Percentage of reports processed (aggregated manually or entered into a database) according to deadline.

Percent of data elements reported accurately in MIS reports

Definition:

The accuracy of reporting and aggregation

A "data element" is a single datum input into an MIS. This term can refer to a single cell on a routine MIS report or in a data entry screen for automated MIS.

Data Requirements:

Primary data from health units/programs and aggregated data at all levels.

Data Sources:

Registers, patient records, and/or tally sheets; monthly/ quarterly reports from MIS.

Purpose:

This indicator is used in health programs to measure accuracy of reporting and aggregation. Inaccuracies can occur when data are recorded, tallied, transposed onto reporting forms, and aggregated.

Evaluators can select a number of sample data elements and can determine the extent of agreement between (1) data recorded in service registers, patient files, tally sheets or patient files, and (2) data reported on MIS forms.

Item  # Recorded   # Reported Consistent? 
Pill cycles  36 32  No 
 IUDs  5  5 Yes
 Inectables 18  18  Yes 

                               % accuracy 2/3 or 66%

 

To measure accuracy of aggregation, organizations frequently compare a sample of data elements from the raw data reported from multiple service delivery points (SDPs) to the aggregated total reported to higher levels.

For example:

 

SDP 1 

SDP 2 SDP 3  Actual Total  Reported Total  Consistent? 
Pill Cycles   32  84  14  130  130  Yes
IUDs    5 10  7  22  20  No
Injectables   18 21   23  62   49   No 

                                                           % accuracy 1/3 or 33%


Program or facility managers may inflate or under-report data for a variety of reasons. A more sophisticated indicator would therefore measure the relative difference between recorded and reported data to determine if the problem is over- or under-reporting of data. This approach is more time-consuming and tedious to calculate but better measures reporting accuracy.

If an evaluator wants to examine issues of data quality but lacks the time to do a detailed analysis of accuracy as required in this indicator, he/she may use a more basic measurement of an MIS that looks at completeness of data: Percentage of health facilities sending reports with no missing data.

Issue(s):

Field testing of this indicator shows that it provides a proxy indicator for the overall functioning of information systems. However, more in-depth analysis is necessary to explore whether the recorded data reflect reality. This assessment demands direct observation of those staff recording data, a technique that may be too costly to warrant routine use.

A system for quality assurance has been institutionalized

Definition:

Quality refers to offering a service or product in a way that consistently meets the clients' needs. Quality assurance (QA) is a generic term describing a number of management approaches (Continuous Quality Improvement (CQI), Total Quality Management (TQM)), all of which recognize that many organizational problems result from systems and processes, as well as from a lack of clear performance expectations, rather than negligence on the part of individuals. QA, as it applies to the management of reproductive health programs, generally involves the encouragement of staff members at all levels to analyze systems and processes, to use information to identify the nature and size of each problem, and to design and implement activities to improve services and client satisfaction. (For more detail on QA, see the Service Delivery - Quality of Care indicators.)

Data Requirements:

Evidence of the availability of quality standards and protocols; budget allocation for QA activities; performance/provider reviews of adherence to standards; client satisfaction feedback on quality; staff feedback on involvement in quality initiatives.

Data Sources:

Organizational documents including service delivery guidelines; interviews with managers, supervisors, and other staff at all levels; budget; staff performance reviews; training curricula; client satisfaction surveys; suggestion boxes.

Purpose:

This indicator measures organizational commitment to QA; more detailed indicators can be found in "Managing Quality and Clinical Services".10

From a management perspective, the following six items are essential to developing a composite score of commitment to QA:

By measuring this item, an evaluator will first understand if an environment or organizational culture of quality improvement exists.

Not all QA activities will require a separate line item budget; some of them are combined with other activities and are done only at marginal cost. Assessment of budget allocation can be difficult unless the budget is highly detailed/annotated.

This indicator is easy to measure but requires identification of protocols for service delivery (clinical protocols, counseling) as well as for management (e.g., supervision, storage of supplies, infection prevention, MIS reporting) for each major type of reproductive health service.

As with the previous item, this aspect requires measurement according to service delivery and management standards for each reproductive health area. It requires direct observation of staff (such as by a supervisor or mystery client); measurement can therefore be time consuming depending on the volume of services at a given facility.

Measuring client satisfaction is the principal means of knowing whether QA initiatives are reaping any benefits. Among the numerous methodologies for measurement, the most common is the client exit interview. One of the weaknesses of exit interviews is that clients sometimes forget details of a visit or do not know what practices are acceptable. Clients may fear impact of negative responses on availability of and access to services.

Measurement in this area can reveal gaps between client and provider understandings of quality. It can also help managers understand the extent to which providers feel encouraged or rewarded for taking initiative to address quality.

References:

Gutierrez, Maria. 2001. http://erc.msh.org/mainpage.cfm?file=2.2.6.htm&module=quality&language=English

The budget is linked to the annual operational plan for the current year

Definition:

The effectiveness of an organization or a program to match available resources with planned activities.

A budget is a document that projects the costs, and in many cases, the revenues of a defined activity, program, project, or organization. It is also a financial plan that quantifies programmatic goals and objectives by guiding the allocation of financial and human resources (MSH, 1999c).

Data Requirements:

Evidence that activities in the annual operational plan have been costed and that resources have been allocated to individual or sets of activities.

Data Sources:

Budgets; chart of accounts; operational plan(s).

Purpose:

This indicator measures the effectiveness of an organization or a program to match available resources with planned activities. Budgets may cover single activities (sub-budgets) or whole programs; a well-constructed budget allows for the "rolling up" of several sub-budgets into a total operating budget.

Issue(s):

Measurement of this indicator assumes a certain level of clarity of the budget. For this reason, those preparing budgets must provide information on the units of cost, that is, the number of participants, the specific number and type of materials, the number of days of per-diem, and the number of persons to receive per-diem, for example. This level of detail makes it easier to assign a value or cost to each unit. It also makes it easier to modify the budget because numbers in the plan shift over time. When budgets are broken down into unit costs, tracking changes in costs over time is easier. It is also easier to review quotes that come in from vendors, because historical data are available from prior budgets.

References:

MSH. 1999c. Understanding and Using Financial Management Systems to Make Decisions. Boston, MA:
Management Sciences for Health.

The financial management system produces accurate, timely information

Definition:

Financial management refers to managing an organization or program's resources to meet goals and objectives as effectively as possible by using those resources to carry out planned activities. A financial management system is composed of a series of tools and processes that permit the control, conservation, allocation and investment of an organization's or program's resources.

The system for scoring this indicator and assessing the effectiveness of a financial management system and its ability to generate accurate and useful information is as follows:

Descriptor  Score
Expenditures are tracked by budget-line items (e.g., inputs, salaries, utilities, materials) and are recorded as they occur. However, financial reports cannot be generated effectively. 1
Expenditures are not only tracked by inputs, but are also linked to services and materials purchased, and to the activities they support. Financial reports exist but are not used to analyze costs. 2
The financial system produces income/ revenue data and case flow analyses; costs are allocated by cost centers (e.g., products/outputs, service units, sets of services). Financial reports, which compare actual expenditures to budget, are sometimes used to analyze costs. 3
High-quality financial reports are linked to budgets and consistently used for management decisions, including allocation of resources. 4

 The underlying assumption in levels 2 through 4 is that reports are accurate and timely.

Data Requirements:

Information on planned and budgeted activities; expenditure information; evidence that financial reports are used for decision-making.

Data Sources:

Document review of budgets, case flow statements, income statements, balance sheets, and interviews with managers.

Purpose:

Managers must understand their current financial situation (liquidity) and their long-term position (solvency) if they are to lead towards effective performance. Evaluators can use a suggested scoring system to measure this indicator. The system combines elements of how expenditures are recorded and tracked and of how financial information is used to make decisions. These essential functions enable programs/organizations to understand their current and long-term liabilities.11

Issue(s):

Because measurement of this indicator requires performing a valid document review, evaluators must fully understand locally accepted accounting principles and reporting requirements. Ascertaining how management makes decisions will require not only review of documents such as operational plans, but also interviews with key decision-makers in the program or organization (MSH, 1999a).

References:

11 Embedded in levels 2 through 4 is the assumption that reports are accurate and timely.

Percent of annual revenue generated from diverse sources

Definition:

The ability to reduce dependency on single-source funding.

Revenue are monies or the equivalent received from sales, services, fees, donations, and grants. In the case of grants, only the portion actually spent is considered revenue; the balance may have to be returned to the donor.

Typically, the primary sources of funds available to reproductive health programs and organizations include: national governmental entities (such as Ministry of Finance/ Ministry of Health), local governmental entities (municipalities or districts); international donors or foundations; sales of services (through charging fees, contracting with other agencies and or insurance schemes)12; sales of supplies such as contraceptives; and donations from local donors, corporations or individuals.

Data Requirements:

Total resources (funding or in-kind donations) generated or received by the organization/program from each source.

Data Sources:

Income statements; revenue reports; audit reports or other financial records.

Purpose:

This indicator measures the ability of managers to use financial management systems to make decisions that will reduce dependency on single sources of funding. Diversity of funding sources spreads the organization's or program's risk of over-dependency on a single source of revenue, and allows the organization/program greater flexibility in determining future directions. Many NGOs find diversification critical to their sustainability over the long term. In addition, with decentralization and the subsequent requirement for local governments/health departments to generate some of their funds, public sector revenue diversification and cost sharing are now relatively common in developing country settings.

Issue(s):

This indicator assumes that organizations/programs have basic financial management procedures in place to provide revenue information. For newly decentralized public sector entities, this indicator may be difficult to measure because systems for aggregating information on the revenues generated from service delivery fees, for example, are nascent.

An important consideration in measuring this indicator relates to how funding is earmarked. An organization or program may have diversified sources of funding, but the donors may demand that money be used only for very specific programmatic activities, rather than for general operating or developmental costs that are crucial for sustainability. In addition, the earmarks may not align with the current strategic directions of the organization or program, and this situation may necessitate additional unforeseen expenditures. Hence, evaluators should use this indicator in conjunction with other indicators that permit a more in-depth financial analysis.

References:

12 These represent the principal forms of direct cost recovery although other mechanisms may exist.