GAO XIANG：Constructing a Chinese framework for social science evaluation
Gao Xiang
To build China’s discourse system in philosophy and the social sciences, fundamentally is to develop a system with Chinese characteristics, style and manner through presenting research achievements that demonstrate China's stance, spirit and accomplishments. We cannot achieve this goal without the power to set academic standards and evaluate academic research; without objective and impartial standards to determine what constitutes academic learning and what does not or distinguishing good academic research from bad, and without the discourse power to make these determinations, our academic research will lack necessary norms and guidance, and we will be unable to form a cohesive force and develop our own unique features. After 65 years of development since the founding of the P.R. China, and especially after 36 years of growth since reform and opening up, constructing a social science evaluation system with Chinese discourse is an unavoidable requirement for China’s academic research and basic premise for it to engage the world.
Problems on academic evaluation in China
Since reform and opening up and especially since the 1990s, China has consciously absorbed and learned from the academic evaluation systems of other countries through international exchange and as its own philosophy and social science research has developed. It has gradually established an academic evaluation system highlighting academic journals and institutions where both quantity and the level of academic publications are regarded as significant indicators. This system has played an important role in facilitating the supervision of research, encouraging scientific research and promoting philosophy and the social sciences, but its implementation has not been without difficulties.
The first problem is undue emphasis on quantitative indicators. China's current academic evaluation system tends to regard impact factor and citation rate as vital indicators for assessing the quality of social science research and the rating of academic journals. This can be justified to a certain extent. Quantitative indicators can provide necessary reference for academic evaluation; they are conducive to maintaining its objectivity and user-friendly. However, they are ultimately too limited to measure the quality and impact of academic research sufficiently. For example, a purely theoretical study might have a much lower impact factor and citation rate than an average study on applied policy, but these quantitative indicators do not capture the unique value the theoretical study may have as an article of civilization that gets passed down to later generations. In fact, a higher citation rate does not necessarily indicate that an article is particularly good, just as a lower citation rates does not mean an articles lacks academic merit—some masterpieces capable of resolving deficiencies in the field or expanding the academic scope may slip under the radar. Although they may seem objective and impartial, overemphasis of quantitative indicators is implicitly biased. We are currently seeing some of the drawbacks of this overemphasis in China. We put quantity before quality when evaluating institutions and researchers; we assess academic journals and presses foremost by citation rates and impact factors, while ignoring how well these journals or presses uphold a sense of academic responsibility and commitment.
The second problem is detachment from the reality. Under the current evaluation system, reviewers and researchers do not communicate enough in the process. This is a twofold problem. First, the system is too dependent on reviewers and does not have enough participation from scholars and journal editors. Reviewers alone can hardly be sure they represent an author's view on his or her discipline's development. Second, our current evaluation system is too general and does not make adjustments to meet different disciplines. In fact, humanities, social sciences, interdisciplinary studies, newly-emerging disciplines and traditional mainstream disciplines (e.g. historiography and economics) can differ enormously, beyond what any single system of indicators can capture. To improve academic evaluation, we have to start from the reality of each discipline’s development and represent the common will of the academic community.
The third problem is the corruption caused by individual interests. Academic research is an affair of timeless importance, and academic evaluation also has been integral to the success of research for many generations. Today's academic community is grappling with a rather serious tendency toward utilitarian behavior, as many of its members seek personal fame, individual interests and instant success. The inadequacy of the academic evaluation system, lack of transparency in evaluation standards, lack of openness in the evaluation process and insufficiency of academic supervision have enabled behaviors like rent-seeking by those in power and exchanging money for favors. For instance, in order to improve the impact factor of their journals, some editors seek out connections. Journals may make deals to cite articles from one another or even force authors to promise that their articles will be cited elsewhere. These behaviors have not only corrupted academic practice, but also damaged the impartiality and legitimacy of academic evaluation. Peer review should serve as a positive instrument to compensate for the inadequacy of quantitative evaluation, but it has also been adulterated in some cases. The process for selecting reviewers is often not transparent, and some reviewers may not evaluate independently and impartially, instead appraising an article by their relationship to the author, the authors' influence or their personal tastes. This damages the professionalism and authority of peer review.
Fourth, academic evaluation in China lacks value orientation. Academic evaluation should shed lights on researchers' orientations through assessing and rating the quality of academic research, publications and research institutions. Part of what makes an evaluation system sound and mature is its ability to maintain a distinguished standpoint and leading direction for academia. A system needs to provide impartial, objective and professional assessment to show what constitutes a success or failure in academic research, and thereby indicate its future direction. We can never make others enlightened if we are ourselves in darkness.
If we forget the mission of guiding academia and unequivocally pursue so-called independence and impartiality by solely relying on quantitative indicators while ignoring our own reflections on academia’s direction and applying a general criteria for all specific analysis, all the while turning a blind eye to interference in academic evaluation by non-academic sources, we will lose our ability to meaningfully and effectively evaluate academic work and become stuck in a limbo with no way out. Strengthening academic evaluation’s role as a guide is a very urgent problem for evaluation in Chinese social sciences.
Balance three relations
Chinese philosophy and social sciences have never before had as many opportunities as they do today, and an adequate evaluation system must benefit these fields in three ways. It must facilitate upholding the right academic orientation, strengthening the guiding role of the theoretical system of socialism with Chinese characteristics in research in philosophy and the social sciences. Second, it must help foster theoretical innovation and create an environment in which academia can thrive, building a Chinese discourse system in philosophy and the social sciences. Last, an academic evaluation system must enable Chinese academia to engage a global audience, conducting dialog on equal footing with international academia and contributing Chinese thought, experience and wisdom to world civilization.
In practice, we need to deal with three relationships.
The relationship between guiding ideological and being academic-centered. Compared with the natural sciences, philosophy and the social sciences have a clear ideological quality. To serve for class and state is essential nature that no school of thought can bypass. The present day global academic environment is defined by the interaction, integration and competition of various schools of thought and ideologies. China’s humanities and social sciences are the vehicles through which the excellent academic traditions of the Chinese people are carried down to later generations, and are the basis for China’s great process of modernization. With Marxism as their banner and soul, China’s humanities and social sciences serve the needs of socialism with Chinese characteristics. As we construct an academic evaluation system, we must follow these principles. Moreover, we need to realize that in Marxist academia, scientific nature and class nature do not contradict each other, nor do ideological guidance and academic-centered orientation. We cannot abdicate our historical responsibility of guiding academic direction, not can we reduce academic evaluation to political evaluation. We need to follow the principle of “letting one hundred flowers blossom and one hundred schools of thought contend,” facilitating exchange between various academic viewpoints as well as supporting and protecting the establishment and development of different academic schools. Through impartial and objective academic evaluation, we should encourage scholars to follow the scientific spirit and scientific principles. We should demonstrate the strong vitality and appeal of Sinicized Marxist academia through splendid academic achievements that can stand the test of practice and time.
The relationship between being based in China and engaging the world. The current evaluation system in China, especially evaluation of academic journals and institutions, has learned and borrowed a lot from Western experience. To a certain degree, this is necessary and justified, and it has played a positive role in establishing certain norms across social science research and publication in China. We have to keep in mind, however, that one of the most obvious problems we are facing is that China's voice is still relatively unheard in international academic discourse. Just looking at the evaluation of academic journals, the weight some people give to the Social Science Citation Index (SSCI) is particularly indicative. Without considering its original objectives, they unquestioningly use it as a tool for evaluating academic journals and articles. Some journals take great pride in being included in SSCI, and some institutions offer special awards to those who get published in an SSCI journal. In fact, this blind admiration of a foreign indicator system is actually because people’s understanding of it is quite shallow. This has a strongly detrimental effect on China’s academic development. Philosophy and the social sciences have a distinct national character; it should be the Chinese people themselves who have the final say in assessing the merit of Chinese academia. When constructing China’s academic evaluation system, we need to focus on our own methods while selectively employing international evaluation methods and learning from foreign experience. We must base our evaluation standards on the reality of Chinese academia and avoid narrowly pursuing a system that complies with mainstream international evaluation, let alone trying to mold our system to fit international norms. We have to be discerning and trust in our own wise judgment when we look at international academia. Chinese scholars need to articulate their own views in the global academic arena, making their voice on the future direction of academia heard. This is what we mean by “engaging the world”.
The relationship between quantitative and qualitative evaluation. Though it has much strength, we need to understand and bear in mind the limits of quantitative evaluation. Objective, accurate, and easy to use and understand, quantitative evaluation is an importance means for ensuring we remain impartial and avoid giving in to overly subjective or altogether arbitrary assessments. However, it alone is not enough. Solely evaluating articles or publication quantitatively cannot give us a rigorous, scientific appraisal of their actual value. It cannot encompass the real impact an article or journal has, which defies quantification. The contribution of a study cannot be determined merely by its citation rate; only the careful scrutiny and evaluation of an expert in the field, with reference to the field’s history and the current state, could accurately shed light on its true merit. Qualitative evaluation must therefore be included in our academic evaluation system. In short, qualitative evaluation without quantitative evaluation is subjective and arbitrary while quantitative evaluation without qualitative evaluation is incomplete. The evaluation system we are trying to construct should combine both quantitative and qualitative evaluation. Gathering sufficient statistical data and other research materials, we need to conduct multi-dimensional analysis, striving for a dynamic balance between quantitative and qualitative evaluation.
Continue to reform academic evaluation system
In reforming our academic evaluation system, we must uphold the guidance of Marxism and continue to serve socialism with Chinese characteristics. We must maintain our commitment to advancing theoretical innovation, creating a thriving academic environment, and strengthening China’s influence in international discourse. To accomplish these goals, we must observe four critical points.
Firstly, improving the peer review system. Peer review is now widely used both in China and the world as an approach to academic evaluation. Originally adopted for evaluating patent applications and later used by publications to assess submissions’ academic merit, the standard peer review procedure today involves a complicated process solidified over the course of two hundred years, which includes referee selection, anonymous review, collective vote and open discussion. The evolution of this process has been indispensible to the development of academic research and the formation of the academic community. Since it was introduced to China, peer review has played a significant role in enhancing research. However, in its current state, China’s peer review process is far from perfect. The system as a whole is not sound enough, procedures are not standardized and supervision is not adequate. In order to make the fullest use of peer review, we need to improve the expert selection process. When selecting reviewers, we need to consider the scope of their knowledge, their academic achievements, how capable they are at reviewing, and their academic integrity. Taking into account these criteria, we will avoid picking amateurs to review professionals, and academic evaluation will not be low-level and arbitrary. Academic journals and presses need to reinforce “double-blind” review and the “withdrawal system” to prevent the interference of personal connections or interests and ensure the peer review process remains impartial. They should also set up a system where authors can appeal their review if they believe they were in any way unfair or suspect misconduct, and reviewers are held accountable should the author’s appeal go through. We should have zero tolerance toward power-for-money deals, cliques of special interests and malpractice for selfish ends.
Secondly, strengthening the role of academic journals in the evaluation process. In the history of modern academia, academic journals have always been an important platform for releasing cutting-edge achievements. They have played an irreplaceable role in guiding academia’s development. Even in European and American academic circles, publication in a prestigious journal is a strong reflection of a scholar's ability. Improving the quality of academic journals is essential for strengthening their role in academic evaluation. Journals cannot select articles because of personal connections and pecuniary interests; ensuring impartiality will help journals establish and increase credibility. We need to establish a strict anonymous peer review system and multi-layer review system within an academic journals' editorial board, insulating the review process from interference by non-academic factors. The editorial board needs to stand at the vanguard of history and the times. The editors should follow the right political direction, and must also bring a profound knowledge of theory and academia to bear on submissions, so that they can discern and select particles of gold within the sand. If editors can fulfill this role, then journals can have a guiding influence in China's evaluation of academic publications.
Thirdly, enriching the content and tools of academic evaluation. Where there is academic activity, there is academic evaluation. Academic evaluation should not confine its focus to individual researchers, but instead expand its scope to include various disciplines, think tanks, universities, academia with particular regions and even international academia and online academic activities. When assessing an academic institution, we should consider its research accomplishments, the strength of its faculty and its overall atmosphere. We need to use a variety of indicators and approaches in our evaluation system—both quantitative and qualitative evaluation, outcome and process evaluation, and readers’ reviews and editors’ reviews, experts reviews and surveys. Concurrently, we should also listen to the authors' opinion. We should make better use of new media technology and innovate our evaluation approaches. For instance, within our system of indicators we should also include the impact an article, journal, institution, etc. has on the internet. We can also use online surveys, or invite experts from overseas to participate via remote video reviews and build a database of experts. To support the system, we should establish an online channel for reporting academic misconduct so that academic evaluation will be transparent and able to withstand public scrutiny and inspection.
Fourthly, establishing an independent, impartial and authoritative academic evaluation institution. Academic evaluation should pursue justice and impartiality. To archive this goal, it is necessary to establish an independent and authoritative institution to avoid interference from authors being reviewed and parties with individual interests. In contemporary China, an influential academic evaluation institution must have the following: adherence to the right political and theoretical direction; profound knowledge of academic development and clear knowledge of its missions and commitment; an adequate database and platform for investigation and analysis; sound evaluation standards, mechanisms and procedures; and a loyal, devoted and professional evaluation team with high credibility, appeal and organizational capability. Only an institution that meets all of these criteria can ensure academic evaluation remains objective and impartial.
Building an evaluation system for philosophy and the social sciences with a distinct Chinese character that is well-suited to China’s current situation is the basis for creating an environment in which Chinese philosophy and social sciences can thrive. It is also a precondition for establishing China’s own unique discourse and making Chinese scholarship more established in the world. This strenuous task will require the concerted efforts of all our colleagues in philosophy and the social sciences in China.
Gao Xiang is a professor, secretary-general of the Chinese Academy of Social Sciences and director of the Chinese Evaluation Center for Humanities and Social Sciences.
The Chinese version appeared in Chinese Social Sciences Today, No. 585, April 18, 2014
The Chinese link:
http://www.csstoday.net/xueshuzixun/guoneixinwen/89098.html
Translated by Jiang Hong
Revised by Charles Horne