Activites

Composite indicators contribute positively to the policy debate by informing and assisting policymakers in monitoring and evaluating processes. Besides assessing multidimensional socioeconomic phenomena, they enable cross-country comparison and are accessible to the general public. However, the construction of a composite index requires many steps, and researchers’ decisions are not trivial. If poorly constructed or misinterpreted they can convey misleading policy messages and hide relevant dimensions of wellbeing and sustainability.  The SPES Focus by London School of Economics and Political Sciences provides a statistical robustness analyses for five transition performance indicators and their findings reveal that these indicators are generally insensitive to components’ changes. Understanding the sensitivity of aggregated indicators to changes in their components is key to interpreting time variations and exploring further their economic and social implications.

The performance indicators

The five indicators – selected through a rigorous process – are:

1. Planetary Pressure Adjusted Human Development Index (PHDI),
2. Transition Performance Index (TPI),
3. Better Life Index (BLI),
4. Green Growth Index (GGI),
5. Sustainable Development Goals (SDG).

Focusing on Europe and taking data for the year 2019, authors simulate a 2% increase in each component and analyse the robustness of these indicators to changes in their inner components. Their findings reveal that these indicators are generally insensitive to components’ changes. This result is somewhat positive, because it suggests a broad stability against measurement errors.

However, it also highlights potential issues associated with value clustering. In many cases, the sensitivity of the indicator to changes on its components is not driven by the theoretical relevance, but the dispersion of the component values across countries. Authors argue that this limitation challenges an accurate assessment of sustainable development. For example, they found the Green Growth Index to show a low sensitivity to climate and resource use components, but a remarkable sensitivity to changes in the gender gaps in financial account ownership.

The indicators reliability and weighing

These results undermine the reliability of composite indicators as sole measures of economic and social transition towards sustainability. Countries improving in one or more dimensions or components may not find these changes reflected in the aggregate indicator if other economies also improve their situation by a similar rate. Besides, inappropriate thresholds, -whether sample based, or theoretically set too broadly or narrowly- can either smooth out or amplify the impact of changes in the indicator through the normalisation process.

One possible way to move forward could imply developing weights across multiple components. Despite weighting schemes have limitations, such as the difficulty of weighting different dimensions, recent advances in the literature propose techniques to improve their evaluation. Well-developed weighting schemes should highlight components with more theoretical relevance and compensate for the country-rank effect caused by the normalization, making indicators more sensitive to changes in transition-related components. These weights could also counterpart the perfect substitutability inherent to arithmetic aggregations. Avoiding in-sample thresholds, which exacerbate the effect of the analysis sample as well as the country’s relative position in the range of variation, and establishing theoretical limits, subject to continuous updates, would also contribute to reflecting changes in specific components.

Practitioners should always acknowledge the indicators’ limitations before compelling policymakers to interpret them. It is also mandatory to propose alternatives to enhance their ability to assess wellbeing. Well-defined target thresholds and values could serve to acknowledge the performance of countries beyond ranks, and get a deeper understanding of their performance. The big picture allows for a broad understanding of reality, but the devil – and the angels – are always in the details.

To read more on the methodology adopted fr the analysis and full discussion on results