The Think Tanks and Civil Societies Program at the University of Pennsylvania, led by James McGann, released the 2015 Global Go To Think Tank Index on Jan. 27.

The Think Tanks and Civil Societies Program at the University of Pennsylvania, led by James McGann, released the 2015 Global Go To Think Tank Index on Jan. 27. It is compiled from an international survey ranking the world’s leading think tanks based on a variety of criteria.

Think tank rankings are an increasingly useful tool for building public awareness about think tanks and promoting their development. However, we should not fetishize  the index. There are several limitations that should be noted.

For one thing, the sample is not complete. The 2015 index adopts a broad criteria for defining a think tank that includes autonomous and independent think tanks, quasi-independent and quasi-government think tanks as well as think tanks affiliated with governments, political parties, universities and enterprises. However, only 435 Chinese think tanks are included in the 2015 report, which is disproportionately small. A number of research institutions that operate under universities were not ranked. Also excluded were institutes affiliated with the government or the Party and ones with ties to the local Party schools.

Furthermore, the methodology is subjective since it relies on the judgments of experts. This year, 4,677 journalists, policymakers, think tanks, and public and private donors from 143 countries participated in the ranking process. The selection of people to take part in the evaluation was not balanced according to region and area of expertise. Moreover, policymakers were given questionnaires to evaluate think tanks and their experts. This created potential for respondents to be influenced by reputation. In addition, many evaluation indexes only measure  research achievements rather than the effect on policymaking, so they fail to accurately capture the practical impact of research.
 

The evaluation process is also lacking in scientific rigor. It is a three-step process. The aforementioned participants are   invited to make nominations, and people from this group volunteer to appraise and select finalists for the list, which is ultimately compiled by McGann’s research teams. Lastly, think tank experts examine the information and submit their own rankings. For one thing, most people are not aware of these expert’s credentials, areas of expertise and national origins. Without supervision, experts may favor think tanks they have relations with or the most popular ones instead of considering the influence on policymaking. The index system seems rigorous, but at its core, the ranking largely relies on the entirely subjective views of experts. The results are not reliable.
 

The competence of the research teams is suspect as well. Expertise  directly influences the reliability of research findings. Therefore, a professional and rigorous team is required to eliminate the subjective factors in evaluation. However, McGann is the only full-time staff member of his team and all others are interns. In the nomination process for the 2015 report, 50 interns carried out the duties of information collection for more than 7,000 think tanks.
 

Lastly, the results are not consistent. For example, in the list of the top 175 think tanks in the world, the Chinese Academy of Social Sciences (CASS) ranked 31st, ahead of the China Institutes of Contemporary International Relations which was in 39th place. But in the list of top 60 think tanks in Asia, CASS ranked tenth while the China Institutes of Contemporary International Relations ranked fourth place. In the same report, rankings of an institution differ in different lists.
 

McGann’s index is an attempt to evaluate the performance of think tanks, but we should realize its limitations and be skeptical about the rankings. In addition, China must build its own evaluation system and create scientific criteria for think tanks based on the national situation.

Liu Liang and Wu Sheng are from the Dongwu Business School (Finance and Economics School) at Soochow University.