With the approach of negotiations on the impending refurbishment of the IDA, [1] the debate resumes on methods of allocating aid based on “performance” and thus of indicators to measure the latter. The Notes du jeudi (Thursday Notes) deals with this subject for the first time, which is closely linked with questions of selectivity and effectiveness of aid.

The performance of the States in countries of the South in terms of governance is considered to be determinant for development and effectiveness of aid. Nevertheless, no consensus exists on the way to measure this performance. The present note offers an inventory of the principal groups of indicators of governance used to date, clarifies the logic on which they are based and examines their significance and their limits.

1) The Emergence and Development of Governance Indicators

The need for governance indicators is closely linked to the concept of selectivity of aid (cf. NdJ [Thursday Note] nº 25). A study published in 1997 by the economists Burnside and Dollar, clarified subsequently and strongly contested since, had maintained that there well and truly existed a correlation between aid and growth as soon as “good policies” exist. To support their assertion, Burnside and Dollar presented a model for 40 poor countries, 15 of which would have attained at that moment a “policy level” adequate for aid to be effective. According to this econometric model, by reallocating the aid received by these 40 countries only to the 15 countries that could make good use of it (over the course of the 5 preceding years), the average annual per capita growth rate of all 40 countries would have been higher. [2]

This result received considerable publicity. It was taken up right away by the World Bank (Assessing Aid, 1998) and the OECD (Development Cooperation Report, 1999), which served as justification for selective allocation of aid: “It is in countries that follow sound policies that aid will be the most useful”, adding that the others would not be affected by the reduction in an aid that had little effect and that they would be encouraged to improve their “policy level” in order to return to higher levels of aid.

Selectivity thus became a recurrent theme in ODA (Official Development Assistance). Gradually implemented by the majority of agencies, it replaced need (poverty level), the criterion more or less applied until then. The allocation formulas were modified, introducing indicators of “sound policies”, more widely called “indicators of governance”, already known since the beginning of the 1980s.

In the private sector, international investors also relied on such indicators, and all the more so since foreign direct investment in the developing countries grew considerably at this time (from 10 billion dollars at the beginning of the 1980s to more than 150 billion dollars since 1997).

These indicators, responding to the need of public and private donors to have tools to ensure the effectiveness of programs, corresponded to the essentially economic and managerial conception of governance during this period.

Today, this narrow view of governance has lost ground to a multidisciplinary approach, one that is endogenous above all, part of a long-term effort and conceived more as a process. Most of the aid agencies are using strategies of this type concerning governance. At the African level, NEPAD [3] places governance at the center of its peer review mechanism. From this perspective, the debate on the nature of the indicators and the method used to conceptualize and define them has been completely reopened.

2)The Most Well-Known Indicators

According to the World Bank, there are close to 140 sets of indicators available. Five sets are, however, most often used:

International Country Risk Guide (ICRG) is an evaluation system set up by a private American company. It is one of the indicators of governance most used by international investors since its creation in 1980. The ICRG evaluates risks on the financial and economic plane, based on “objective measurements” such as calculation of the national external debt, balance of funds, rates of growth and inflation, etc.

The ICRG also evaluates risks from a political angle, but the evaluation is then the result of subjective interpretations formulated by experts. The latter judge the apparent capacity of a government to remain in power, the socio-economic conditions that could lead to instability (unemployment, consumer confidence, poverty), internal and external political violence, corruption, religious and ethnic tensions, awareness of democratic responsibilities, quality of the bureaucracy, strength and impartiality of the legal system, respect for the law, etc.

Freedom House [4] is a special interest, not-for-profit group that likes to think of itself as independent (its avowed objective is to serve as a “clear voice for democracy and liberty throughout the world”), created in the United States in 1941 by personalities coming from all political perspectives.

Its major publication, Freedom in the World, evaluates political rights and civil liberties in different countries. The assessments come from the subjective opinions of the organization’s experts, on the basis of a questionnaire inspired by the Universal Declaration of Human Rights of 1948. This questionnaire brings together questions about political rights (the electoral process, plurality and political participation, operation of the government) and civil liberties (freedom of expression and belief, right to assemble, rule of law, individual rights). For each of the 192 countries evaluated (not counting certain “territories”, also assessed), Freedom House publishes two overall grades each year as well as a general assessment (“free” country, “partially free” country or “not free”), but not the performance on each specific subject.

Transparency International [5] (TI). No indicator of governance has attracted as much media interest as the Corruption Perceptions Index (CPI), published annually by TI since 1995. It is widely used by investors, donors, analysts and academics.

The CPI is essentially a synthesis of inquiries already completed by experts and national and foreign affairs circles on their perception of the level of corruption. In 2005, the CPI classified 159 countries on the basis of results from 16 different inquiries and evaluations carried out by 10 organizations between 2003 and 2005. A country’s score (between “10” for least corrupt and “0” for most corrupt) is then made public. A country must be subject to at least 3 inquiries or evaluations before it is integrated into the classification. That is why countries with a not very flattering reputation cannot be classified, for lack of sufficient data. Around fifty countries are not classified.

Variations in the classification of a country from one year to another do not result only from changes in perception about the level of corruption. They also come from changes in method on the part of Transparency International and from changes in the list of countries classified.

The World Bank’s [6] Country Policy and Institutions Assessment (CPIA), carried out each year, evaluates the quality of institutions and policies of client countries of the World Bank. This index is actually the principal criterion for allocating country credits (no-interest loans or gifts) from the IDA. [7]

The content of the CPIA has gradually evolved and today includes sixteen components, divided into four categories:

1. “economic management”, with three criteria: macroeconomic management, fiscal policy, debt policy;

2. “structural policies”, with three criteria: trade policies, policies linked to the financial sector, business environment;

3. “policies for social integration and equity”, with five criteria: gender equality, equity concerning public resources, strengthening of human capacities, social protection, policies and institutions linked to sustainable development;

4. “public sector management”, with five criteria: property rights, quality of budgetary and financial management, effectiveness of raising revenue, quality of public administration and transparency of the public sector.

The final score (“national performance score”) gives too much importance to the category “public sector management”. The complexity of the whole system is prejudicial to the transparency of the instrument: it is difficult for a country to know what it must do in order to improve its overall grade and thus increase the credits it will be allocated.

To standardize the grades between countries, the Bank constructs a detailed outline of the grading scales and carries out the grading of twelve countries beforehand, chosen at random, which will then serve as points of reference.

The “KKZ” of the World Bank Institute [8], launched in 1996 by Daniel Kaufmann and Aart Kraay, with the assistance of Pablo Zoido-Lobaton and Massim Mastruzzi (hence the designations “KK”, “KKZ” or “KKM”), measures the quality of governance in more than 200 countries on the basis of 31 sources of data produced by 25 different organizations, such as the ICRG, Freedom House, the CPIA and most of the sources used by Transparency International (see appendix 2 for a list of sources). First appearing biannually, it now is produced every year. In 2006, for the first time, almost all the data sources detailing the indicators were put online, with the indicators themselves.

The indicators of the World Bank Institute concern five aspects of governance: popular consultation and “accountability”, political stability, quality of government, quality of public policies, rule of law and control of corruption. They can be described as “composites” in the sense that they are constructed on the basis of hundreds of pre-existing indicators of perception.

Many people consider the “KKZ” indicators currently to be the most advanced. Along with the CPI of Transparency International, they have, moreover, played a major role in the governance debate on the international scene.

3) Questions in Connection with the Use of the Indicators [9]

Kaufmann and his team recognize that no indicator is perfect, a fortiori when it is a question of governance. While their work shows the greatest transparency about the inevitable margin of uncertainty in such an undertaking, the greatest flaw of all these indicators remains the lack of precision of the concepts on which they are based. There is not, in fact, any clear definition of complex concepts such as corruption, governmental effectiveness or investment climate. What is more, the aggregation of indicators from diverse origins, constructed to respond to different objectives, takes away much of the relevance from the indicators.

The choice of indicators. The indicators are focused. Some emphasize the administrative part of governance, for example, others the “private sector”; some emphasize the “supply” of governance (organizational capacity to deliver services), others on the “demand” (taking on responsibility). The KKZ indicator, which uses an obvious diversity of sources and criteria, in reality deals with economic governance.

“Perception” Indicators and “Objective” Indicators. There are two types of indicators: those that are based on perceptions and those that are constructed from objective facts. The second are no more reliable than the first. In fact, the choice of facts to use as much as their interpretation rests on subjective judgments. For example, the existence of specific laws against corruption does not automatically imply a smaller level of corruption than in a country without such laws: it depends, above all, on the application of these laws. Awarding better grades to countries that have regulations to make lay-offs easier (as the report “Doing Business” does, for example) corresponds to ideological prejudices. Indicators “based on facts” and those “based on perceptions” are two complementary and potentially useful sources of information.

The difficulty of comparisons over time. Governance, it must be emphasized, is a process. Its evaluation thus gains from comparisons made over time. There are at least two types of difficulty here: (i) the method selected and the composition of the indicators inevitably vary through the years, above all over the long term, and (ii) statistical imprecision makes comparisons uncertain, at least in the short term. On the second point, the creators of the WBI indicators decided to retain only statistically significant changes. On the first point, there is hardly any solution.

The question of transparency. In order to preserve the “proxy” role of indicators and not make them objectives in themselves, the World Bank has long kept secret the details of the CPIA grades for different countries. But conversely, the lack of transparency limits the capacity of countries to identify their specific weaknesses. Faced with growing criticism, the World Bank made the CPIA scores public for every country in 2006. The calculation of the allocation system based on performance has also been simplified. Most information concerning the indicators of the World Bank Institute is now also online (cf. supra).

The extreme simplification of the indicators. There is a temptation to want to reduce the complex reality of governance to a group of figures or a single grade. Due to the wide margins of imprecision, Kaufmann explicitly warns against the use of the KKZ indicators for the sole purpose of establishing classifications. He never merges his six distinct composite indicators into a single overall indicator; he always uses them separately. A synthesis of these indicators could, in fact, mask significant differences between countries. For example, in 2004, China and India would have had the same score with a single indicator. Whereas China is part of the upper half in terms of “governmental effectiveness” and only the lower quarter concerning “freedom of speech and responsibility”, India is located in the middle of these two areas.

The American Millennium Challenge Corporation (MCC) relies on composite governance indicators to allocate its aid. But, on the five KKZ criteria out of a total of sixteen used, the corruption index is decisive: if a country obtains a below average score, it cannot claim aid from the MCC. This is clearly contrary to the warnings of Kaufmann and his coauthors, who criticize the use of indicators taken separately.

Likewise, widely cited in the press, the CPI of Transparency International has played an essential role in the birth of the world movement against corruption. However, while Transparency International itself warns about the risks of interpreting the variations in country classifications (cf. supra), it is mainly these changes that are reported in the media.

The appropriateness of governance indicators. In most cases, the governance indicators assist foreign investors and donors in their decision-making and not national authorities. This drawback has provoked clear reservations on the part of a certain number of donors - which were expressed, in particular, at the annual assemblies of the World Bank and IMF in September 2006 at Singapore - such that they are less used by the World Bank, the reference remaining the CPIA.

4) New Initiatives

All these criticisms have led to the creation of new tools.

The governance profiles of the European Commission. Within the context of its new governance strategy, adopted in September 2006, the European Commission is developing “governance profiles”, inspired by the KKZ index, but which supplements it with a partnership approach. The method is currently based on the KKZ scores for 2004, but the governance profiles take into account changes since then and include three areas not covered by the KKZ grading system: social governance, regional and international context (role in regional integration) and character of the partnership with the European Union. Furthermore, sometimes Commission delegations reject evaluations of the World Bank Institute. Thus, the Commission noticeably raised Djibouti’s score, deemed to be too severe. [10] The consultation process between the European Commission and the member States in the elaboration of governance profiles, dialogue with local authorities and reference to the criteria of the African Peer Review Mechanism (APRM) has gradually provided a specific content to the European position.

The APRM, set up within the context of the NEPAD, also has its own indicators. It attempts to draw up a complete overview of a country’s performance in four areas: political and democratic governance, economic governance, corporate governance and socio-economic development. Within this framework, a large number of questions are analyzed, from those relating to landownership, regional issues, migration, ethnicity and culture to those related to the fragility of States.

A self-evaluation grid, drawn up in accordance with a standardized outline of key objectives, criteria and indicators, is distributed to the evaluated countries. For each key objective the norms to be respected are defined, that is, the treaties, conventions and constitutive declarations of international law and the African Union. The evaluation is supplemented by drawing up a National Action Plan for improving decision-making processes and respecting international standards concerning governance.

The APRM is winning over a growing number of African countries. It is neither a final exam nor a conditionality imposed by outside actors. It is based on the influence exercised by peers [11] and the participation of all parties to development (government authorities, unions, media, associations and NGOs, opposition parties, private sector, etc.). This opening of public space at the local and national levels also extends to regional and continental levels (Africa Governance Forum, discussion of evaluation reports in regional organizations, Pan-African Parliament and Regional Economic Communities).

DIAL (Développement, institutions analyses de long terme - Development, Institutions and Long-Term Analyses). An economic interest group founded on a partnership between the Institut de recherche pour le développement (IRD - Development Research Institute) and the AFD (Agence française de développement - French Development Agency) and supported by French Cooperation, has developed a statistical approach for perception inquiries by households. It is a question of the “perception” of the functioning of institutions, democracy at the national and local levels, performance of the administration, particularly in providing public services, incidence of corruption in everyday life and all of the “formal” freedoms: press, political parties, unions, etc.

This “participative” approach to governance is based on a rigorous methodology (sufficiently representative sample and precise statistical base). [12] On the other hand, DIAL’s statistical inquiries of African and Latin American households [13] involved the National Statistical Institutes of the countries (rather than the universities, policy research centers or public opinion companies. [14]) They are a module added to existing inquiries on formal and informal employment, living conditions and poverty (called “inquiries 1-2-3”), which makes correlation possible between the socio-economic situation of those surveyed and their evaluation of governance and democracy.

The DIAL researchers (see infra) have, moreover, demonstrated the significant margin of error entailed by the construction of overall indicators that are mainly based on the perceptions of experts by comparing them, using “mirror-inquiries”, to the perceptions of households.

The “Institutional Profiles” of the MINEFI (Ministére de l’économie, des finances et de l’emploi - Ministry of the Economy, Finances and Employment) and the AFD [15], the OECD and the UNDP. Since 2002, MINEFI and AFD have jointly developed a set of indicators concerning, to date, 85 developed and developing countries. The base is worked out from an inquiry undertaken by representatives in the covered countries. The range of institutional characteristics taken into account is broad: functioning of political institutions, public security, public governance, freedom of economic activity, opening and regulation of markets, security of transactions, corporate governance, social dialogue, social cohesion, etc. The base is then calibrated with the main existing indicators (in particular, CPI and KKZ), and then submitted to experts in the covered countries, who make corrections.

Several experiments of this type have also been undertaken in different parts of the world within the Metagora [16] framework of the OECD. The UNDP’s Oslo Governance Center is going to launch a governance indicators program to “evaluate, measure and monitor democratic governance” with which France and DIAL could be involved. [17]

All these initiatives have in common the close cooperation of local institutions in the definition of indicators and the elaboration of methods of producing results. With the necessary technical assistance, they can thus work out a system of indicators adapted to their specificities and take charge themselves of their regular production.

Governance indicators, which appeared with the recognition of the central place of governance in development policies, evolve together with the concept of governance itself. The expanded approach currently prevailing leads to reopening debate on the methods as well as the epistemological foundations of their elaboration. Since the annual assemblies of the World Bank and IMF in September 2006 at Singapore, several donors, including the European Commission and France, have openly demanded a reopening of discussion on the methods of allocating IDA credits. This debate has now been started within the context of negotiations for the fifteenth replenishment of the IDA, which must be completed in 2007.

The democratic governance approach developed by France [18] can assist in directing this debate towards the use of indicators constructed more in partnership with the recipient countries. The latter could, then, evaluate more effectively the quality of their policies. It is from this perspective that France and Canada, which share the same approach, are going to launch, within the context of the OECD, an initiative on governance indicators during 2007, in particular to compare the World Bank indicators with those of the APRM. This mechanism offers, in fact, the advantage of making their use easier and could facilitate a certain standardization of the existing systems.