Неэргодическая экономика

Авторский аналитический Интернет-журнал

Изучение широкого спектра проблем экономики

Opportunities for the Consolidation of Rating Products in the Internet Environment

The article reveals the experience of a project on consolidation of rating products within the single portal with related services for users. We describe the ideology of an open Internet platform, its structure and navigation, and specially designed analytical tools. We focus on the developed and tested trust index in relation to all the ratings on the website, which helps approach a possible solution to the problem of rating verification. We point out the fact that the proposed analytical verification tools are vulnerable to manipulation that is manifested in users’ hostile attacks on selective ratings by giving the ratings deliberately low trust scores. The paper shows the relevance of the portal designed, which is manifested not only in the number of views of its content, but also in the public reaction to various ratings of both individuals and organizations. In addition, we disclose the experience of using academic ratings over the period of several years to reveal undesirable trends and emerging threats in the market of economic research. We show an attempt to combine different ratings for the purpose of determining the extent of using educational and scientific potential of Russian regions. Video interviews allowed us to formulate ten principles in the development and use of ratings: neutrality (disinterest of the developer as a participant); pluralism (diversity of ratings); duration of the period of preparation; improvement of development methodology; openness of the object of rating and ranker; selectivity (optionality) of application, etc. These principles help raise the work of rankers to a higher level. We outline the ways of using the portal for communication between rankers and the expert community, for establishing a constructive dialogue between them and improving rating tools.

1. Introduction

 

Probably, it will be no exaggeration to say that the 21st century is a century of various ratings. Today, anything can be rated: countries, companies, universities, magazines, governors, athletes, etc. This situation is a consequence of the socio–economic life becoming more complex and the goods becoming more diverse, when people are no longer able to find their way in the huge mass of artifacts. They need clues – and they get them with the help of various ratings.

However, a rapid increase in the flow of rating information, in turn, raises serious concerns about the growing complexity of its selection and processing. For example, there are quite a few global rankings of universities, among which at least six are widely recognized: Academic Ranking of World Universities (ARWU), The Times Higher Education (THE), QS University Rankings, Webometrics, Performance Ranking of Scientific Papers for World Universities (PRSP), Center for World University Rankings (CWUR). Since the information presented in them is quite controversial, there emerge big disputes and disagreements both in the scientific and in the social environment. Not only the results obtained but also the methodologies used, the data sources, and their comparability with other rated products give rise to criticism.

In connection with the above, practical work on rating has led to the emergence of an independent direction in economic science, which is actively gaining momentum. For Example, Ph.Baty studies the accuracy of the sample used by rankers [1]; D.Jobbins examines the so–called “rating wars” between rankers [2]. L.Borghans and F.Corvers consider in detail the phenomenon of “Americanization” of European education under the influence of ratings [3]. Ph.Van Parijs discusses the validity of the weighting factors in the rating system and the “semantic” application of ratings [4]. D.Smith studies public opinion on issues of public awareness and trust in the existing ratings [5]. I.F.Aguillo et al., S.Van de Walle and R.Van Delft determine the consistency of different ratings [6; 7]. Interesting aspects of the problem of ratings manipulation were revealed by M.Scully and K.Watt [1], as well as by E.Gertler and his colleagues [8]. The use of bibliometric indices is considered in the works of D.Arnold and K.Fowler [9]; A.D.Alves, H.H.Yanasse and N.Y.Soma [10]; B.Hammarfelt and A.D.Rushforth [11]. Russian economists are also actively involved in the research on ratings. For example, E.D.Sverdlov, Yu.M.Arsky, V.A.Markusova and N.F.Chumakova study issues related to citation [12; 13]. In the works of S.S.Donetskaya we find the application of university ratings to the problem of assessing their competitiveness [14]. A lot of research work was carried out by A.M.Karminsky and A.A.Polozov; they study the rating movement, reveal its basic laws and assess ratings as a special type of expertise [15]. The problems of ranking Russian economic journals are considered in the works of A.Murav’ev, V.Polyakova, O.Tret’yakova, A.Rubinshtein, N.Burakov, O.Slavinskaya, F.Aleskerov and his colleagues [16; 17; 18; 19; 20; 21], etc.

The research presented in the article is a step toward streamlining the available information on ratings. To do this, a special section headlined “Ratings” was created on the Non-ergodic Economy website in 2016; the section was intended to become an open portal on rating. To date, this portal exists for over two years1. It should be noted that the practice of creating information and analytical databases of rating products already exists in Russia. In particular, there are several portals that “collect” all sorts of ratings on their pages. To date, the Internet-edition “Humanitarian technologies” is the most systematized and scientifically structured analytical website about the main directions and markets of humanitarian knowledge and technologies in Russia and abroad. Despite the existence of some experience in consolidation of rating products, some aspects of the present work are still poorly represented in analytical practice. So far, there have been no attempts to create and integrate a series of academic rankings, the need for which has been made obvious in recent years. Besides, there is no unified evaluation of the quality, reliability and popularity of rating products. These gaps need to be filled.

This situation was addressed in 2016 when the information-analytical portal “Ratings” was created; it specializes in the rating products of various phenomena in the sphere of economy; it focuses on the integration of academic ratings, consolidates the expert community on a single information platform by providing access to information for a wide range of users who can comment and discuss the proposed ratings; and it uses new online analytical support tools to assess the level of trust in a particular rating.

The purpose of the present article is to reveal the experience of creating the rating portal, to describe the problems that arose during the implementation of the project, and to determine the prospects of this project in improving the level of expert assessment of existing rating products.

 

Ideology and structure of the rating portal

 

The portal was created in order to achieve the following interrelated goals.

1. Creating an open interactive content of an information–analytical database of ratings. The fact is that the number of ratings in general and ratings that evaluate the same phenomenon is expanding at an increasing rate. Russia has accumulated a huge number of uniform rating products that are scattered on the Internet. This makes professional work with them extremely time–consuming. Thus, the task was to create an automated system for collecting and processing the entire array of rating products in Russia in the form of an open interactive information system to which any interested person can get free access. Our work resulted in the creation of the “Ratings” portal, which is an open archive of rating products of various phenomena at different levels (local, regional, national, global, etc.).

2. Subject structuring of rating products. To ensure more comfortable user navigation on the portal all the ratings were classified into five groups: country, regional, academic, educational and corporate. Although such a gradation of information materials is somewhat conditional, it helps navigate faster in the original array of ratings. By 2018, 31 rating products were placed on the portal, each of which accumulated rating results for several years (taking into account annual information, the portal contains about 90 ratings). This approach allows us to analyze the dynamics of ratings, and therefore the changes taking place in the respective markets.

We note that rating products were uploaded on the portal exclusively by the developer according to a simple rule: the core of the content consisted of exclusive ratings of the developers themselves; the core of “own” ratings gradually expanded due to “foreign” products in accordance with the structure of the portal. Thus, “foreign” ratings were uploaded in process of their detection in the Internet environment, and they went through preliminary information processing under the unified template for all rating products. The developer organizations whose ratings were posted on the portal include the Financial University under the Government of the Russian Federation, the Higher School of Economics, Expert RA Rating Agency, the New Economic Association, Internet title Nation’s Capital, Internet portal Career.ru, information and service portal “Indicator”. It should also be noted that due to the lack of a single portal of ratings it is currently not possible to estimate the size of not only the entire Internet market of these products, but also local (thematic) markets, which in turn does not allow us to establish the share of markets represented on the author’s portal.

3. Creating a “passport” of ratings. The spontaneity of the rating movement is also manifested in the fact that only the final results of ratings appear in the information field, while many aspects related to the product developers and the methodology of its preparation remain hidden from the general public. In this regard, the portal that we created differs from its predecessors in its specialization that goes hand in hand with the widest thematic field; besides, each rating product has a unified structure that requires additional processing of all the ratings presented on the website. Thus, each rating product is provided with a template table (actually, the “passport”), which includes information about the developer, date of creation and the frequency of rating preparation, official source of information (if the rating is borrowed from other portals and information resources), additional information on rating research, description of the rating technique, tabular data with the results of ranking by year. Such an approach corresponds to the global tradition of creating personalized scientific products in modern economic science. Thus, each rating has the addressee of the developer, who is responsible for possible inaccuracies in the data, irregularity of their provision, etc.

4. Availability of additional interactive options. Creating an open information and analytical database of ratings makes it possible not only to structure the available rating information, but also to present the opinion of users about the ratings in the form of a semantic link between the user profile and the corresponding rating publication. Semantically structured content provides easy access to the information on the website both to unregistered and registered users. However, compared to the former, the registered users have certain advantages associated with the two options available on the portal. The first one provides an opportunity to leave comments and to send messages and receive them from other registered users. The second option allows users to express the degree of their trust in a particular rating. For this purpose each table with the rating data has a special questionnaire in the form of a popup window that allows you to choose one of the answers to the question: “How much do you trust this rating?” The four answers provided are simple and qualitative: I fully trust (X4); I sooner trust (X3); I sooner don’t trust (X2); I don’t trust at all (X1). Each user can fill in the form only once. In the future, the questionnaire information is processed to calculate the trust index (I) for the corresponding rating by the following simple formula:

 

I = (0*Х1+0,4*Х2+0,6*Х3+Х4)*100%,

 

where X1, X2 X3 and X4 are the answer options that represent a Boolean variable; i.e., for example, X1=1, if the user specified this answer, or X1=0, if the user specified some other answer.

Thus, each answer option is assigned its own score. The overall trust index of the rating is the average of all scores of the users who participated in the voting. The presence of this procedure is reflected in the “passport” of the rating in the form of two parameters – the trust index (percentage) and the number of respondents (persons) involved in the formation of this trust index. The trust index itself lies in the range from 0 to 100% and has a transparent interpretation: if I≤50%, then the rating as a whole falls into the zone of low trust; otherwise it is in the zone of high trust.

5. A series of specialized video interviews. A database of video interviews of authoritative Russian experts4, which is a digest on the problems of compiling and using ratings, is one of the original options of the project. Experts who are directly involved in the preparation and use of ratings expressed non–trivial comments on the rating movement both in Russia and in the world. We chose the interviewees so that they could highlight comprehensively the most important aspects of rating, which in itself is of great importance for proper work with rating products.

 

Relevance of the project results

 

The functioning of the portal shows that the general public has an ambiguous attitude toward the project we have implemented. On the one hand, the statistics of views of the portal materials clearly indicates that a wide audience of users shows a certain interest in and demand for these materials (Tab.1). For example, the list of top 10 ratings of the portal shows that the potential of the demand for ratings ranges from several thousand to tens of thousands of readers. Strong fluctuations in the number of views for various materials can be partly explained by the fact that the information on the rating products was uploaded on the website unevenly – the difference in the life span of some materials reaches 1.5–2.0 years.

On the other hand, the recorded number of views of the materials clearly did not meet the expectations of the portal developers. In all likelihood, to increase the popularity of the site, it is necessary to take additional measures to promote it in the Internet environment and social media. This work was not carried out specifically, which is the direct reason for the modest statistics of visits to the portal. If the portal is properly managed, then in our opinion, we can count on attracting the attention of tens and hundreds of thousands of users, and in this sense we can talk about underutilization of the project capabilities and its incompleteness.

 

Table 1. Number of views of top 10 ratings as of February 11, 2018

 

 

However, the demand for the portal is not limited to its traffic statistics, even if it were very impressive. The fact is that the information posted on the portal causes a certain reaction of market participants and thus indicates their interest in the project. For example, two Financial University ratings – the rating of academic performance and popularity of economists of Russia and the “golden” rating of academic performance and popularity of economists of Russia – arrange the positions of domestic economists depending on their performance (publication rate in academic journals) and popularity (citation rate in academic journals). At the same time, the second rating clarifies the first one by considering the existing distortions in the information system and the facts of manipulating the original information.

Analysis of the Internet space has shown an increase in the popularity of the rating products under consideration; it is manifested primarily in the growing number of organizations that highlight in the media the fact that their employees are included in these ratings. Moreover, it is noteworthy that there emerges an interesting trend that the researchers now include the fact of their presence in these ratings in their biography. Thus, we can say that these ratings have received some recognition among the economists and are used as a kind of quality mark of scientists’ work. As of March 2017, there were 15 organizations that responded to the academic ratings of the portal. Some of them are as follows: Voronezh State University; ISEDT RAS [now – VolRC RAS], P.A.Stolypin Ulyanovsk State Agricultural Academy, Volgograd State University, Bashkir State University; Saratov Socio–Economic Institute (branch) of Plekhanov Russian University of Economics, Tver State University, Dagestan State University; Peter the Great St.Petersburg Polytechnic University5, etc.

Another information product of the portal – the rating of information openness of regional development corporations – was also in demand by the market of regional development corporations, and they immediately reacted to its appearance. Thus, the developers received a letter from the Tula Development Corporation with the request for clarification of their assessment and indication of additional sources of information on the activities of the corporation. In addition, general director of JSC Development Corporation of Bashkortostan Republic sent a letter of gratitude to the rating developers; he pointed out the importance of the rating and expressed willingness for further cooperation6. Of course, the regional mass media paid increased attention to the activities of corporations that were ranked as leaders in the new rating. Regional journalists interpreted these events as the success of local business structures under the patronage of the state. At the same time, the corporations that were placed at the end of the rating were not left without attention. Thus, regional journalists paid due attention to all of their “heroes”: to those who appeared at the top of the rating and to those who have been less successful in the process of informing the public about their activities. All this testifies to the fact that this information product was in demand by society. As of March 2017, there were nine regional development corporations that were mentioned in the news and media comments on their performance. They include the Development Corporation of Bashkortostan Republic, Development Corporation of the Omsk Oblast, Development Corporation of the Republic of Karelia, Development Corporation of the Smolensk Oblast, Development Corporation of the Kaliningrad Oblast, Development Corporation of the Vologda Oblast, Development Corporation of the Tula Oblast, Development Corporation of the Murmansk Oblast, Development Corporation of the Vladimir Oblast.

 

Analytical capabilities of the portal

 

Analytical work based on the portal materials is one more area in which the portal materials can be in-demand. In particular, based on the three waves of the regions’ academic performance rating (economy) for 2013–2015, we revealed alarming trends and hidden threats in the market of economic research. The analysis has led to the conclusion that the three components of this rating – the market of economists, economic journals, and higher economic schools – are subject to total concentration. An increasing number of regions are excluded from competitive processes in the Russian market of scientific research. Almost all subjects of economic science are grouped in a limited spatial area, while other regions remain without prospects for further development. Thus, we have found out that only 8–9 “active” regions form the leading regional core of Russia’s scientific space; in these constituent entities the best economic journals of the country are concentrated and, therefore, all the most valuable information in the field of economic science is collected in them [22]. Consequently, the rating products considered in dynamics allow us to carry out diagnostics of topical economic problems.

A similar analysis of the dynamics of the Rating of Russia’s leading economic journals for 2013–2015 has revealed the core of publications that consistently held the leading positions. In addition, it was possible to establish a relationship between the scientific level and reputation of journals, to give an economic interpretation of the relationship and to provide examples supporting it. We considered the important difference between the concepts of a journal’s reputation and the desire to publish an article in it. In addition, the analysis made it possible to build a typology of errors that impede the success of economic journals, to reveal their essence and authority with the help of specific examples. Among the typical mistakes the following were considered: unsatisfactory title of a journal, disregard for academic aesthetics, failure to meet the criteria of the national edition, flaws in the content policy, review failures, etc. [23]. Thus, in this case, the dynamics of the rating data allowed us to formulate proposals for the development of the market of economic journals and to help many of them adjust their development strategy.

The original analysis was carried out due to the substantial conjugation of two rating products – the regions’ academic performance rating (economy) and the educational rating of Russian regions. For this purpose, in particular, an assessment of the relation between the academic performance of regions in the field of economic science and their educational potential was carried out. The calculations have shown that only 12% of Russia’s regions produce research findings that correspond to their scientific and educational potential. At the same time, the core of effective regions is gradually shrinking, while the scientific and educational periphery is expanding [24]. Consequently, the educational potential of many regions is idle, which requires adjustment of the current scientific and educational strategy of regional development in Russia.

The academic community is discussing with increased interest the question of rating techniques, in particular, the legality of the use of standard indicators of the Russian Science Citation Index (RSCI), as well as the comparability of the results obtained. Different ratings available on one site facilitate the work in this direction by giving the researchers an opportunity to analyze their dynamics and consistency of the findings depending on the methodology of their construction. For example, researchers from Belgorod State University analyze two rating products posted on the site – the rating of academic performance and popularity of Russian economists and the golden rating of academic performance and popularity of economists [25]. Their poor consistency is shown and it is concluded that the calculation of scientists’ ratings on the basis of RSCI standard indicators without their reference to journal publications is erroneous, as it does not allow filtering out the citations of textbooks and monographs, dissertations and their abstracts. To be fair, we should note that the discrepancies in the results of the analyzed ratings are due to the very attempt of the developers to carry out such filtering. Nevertheless, the conclusions obtained by the authors led them to an idea of creating their own IF-coring Rule method that takes into account journal citations and impact factor of journals and makes it possible to identify significant errors in RSCI statistics on individual author profiles [26].

The rating movement on evaluating Russia’s leading economic journals generated a more powerful and effective discussion wave among the expert community. At the moment, we can say that there exist five most important ratings of Russian economic journals: A.Muraviev’s rating based on scientometric indicators of the RSCI system [16]; I.Sterligov’s rating based on expert assessments [27]; E.Balatsky and A.Ekimova’s hybrid rating; A.Rubinshtein’s “cluster” rating, based on expert survey data [28]; O.Tretyakova’s RSCI–based rating of the journals that are issued by RAS economic institutes [18]. The existence of alternative ratings did not go unnoticed, and it contributed to the emergence of works on their comparative analysis [29], their possible aggregation [30] and the construction of consensus ratings.

Thus, the concentration of rating products on the single portal is intended to facilitate analytical work on the search and processing of available information. However, underutilization of the potential of the site, which was mentioned above, brings forward the issue of mass involvement of analysts to the capabilities of this portal.

 

Experience in verification of rating products

 

Despite the universal expansion of the rating movement, there are no recognized techniques of rating verification. In fact, today every ranker independently develops methods to verify their ratings and prove that they work. In the vast majority of cases, the verification methods thus devised are extremely specific and cannot be applied to other rating products. However, the problem of ratings verification is on the agenda and needs to be addressed. This problem is especially evident in the presence of alternative ratings, each of which claims to reflect the reality correctly, but all of them give very different results.

Competitive ratings are also available in the framework of the consolidated rating portal we created. In particular, the market of the country’s academic economic journals is represented by five alternative rating products, which we have already mentioned above: the rating of journals in economics and related disciplines (A.A.Muraviev’s rating), the rating of Russian scientific journals of the Higher School of Economics (economics) (I.A.Sterligov’s rating), the “cluster” rating of Russian economic journals (Rubinshtein’s rating), the rating of Russia’s leading economic journals (Balatsky and Ekimova’s rating) and the united consensus rating of Russia’s leading academic journals. With regard to these ratings, it is quite legitimate to put a meta-scientific problem of assessing their adequacy and choosing among them the most acceptable and reliable one.

As mentioned before, in order to solving the task, the portal has been provided with an option to calculate the trust index. In theory, a rating that receives the highest value of this index could claim to be the “best” and most respected by users. Of course, this approach is only one of the possible methods of ratings verification, but in the absence of other methods its value increases and it allows us to solve the problem of evaluation of rating products at the expert level. However, the experiment on its practical implementation produced a negative result, showing that Russian society is unprepared for an adequate perception of the new analytical tool. Let us consider this issue in more detail.

The observation of five competing ratings of journals (Tab.2) has resulted in the following findings. According to the academic rankings of journals and economists developed by the Financial University under the Government of the Russian Federation, the average number of respondents did not exceed 100 people until November 2017. At that, the trust indices were about 70.0%. However, in the following period we observe a sharp increase in the number of respondents with a significant decline in the trust index. At the same time, analytical characteristics of the website register an almost simultaneous entry and voting of more than 100 respondents. This statistical “outlier” suggests a deliberate hostile action against specific rating products by means of manipulating the trust index estimates.

 

Table 2. Verification parameters for economic journal ratings

 

 

The absence of objectivity and logic in the estimates we have studied can be demonstrated as follows. The fact is that the consensus rating of Russia’s leading economic journals is an averaging of four competitive original rating products, three of which received a trust score from 62 to 86% (see Tab. 2). This means that the consensus rating of journals is by 3/4 a rating with a sufficiently high degree of trust, and therefore its trust index, according to our calculations, would have to be at least in the area of 58.5%, whereas in fact it was at 7.5%. This understating of trust in the integrated rating can only be explained by its affiliation with the developers of the rating of leading Russian economic journals, which were subjected to an artificial campaign of biased voting.

Thus, today the expert community of economists of Russia does not have the necessary objectivity and tolerance, which generates spontaneous impulses to manipulate the data on the portal by organizing pronounced “outbreaks” of increased activity of respondents. We recall that the manipulation of individual data within the RSCI became epidemic in its scale [31, 32]. And the situation worsens year by year. Thus, the index of academic ethics in Russia in 2016 amounted to 69.6%, and in 2017 – 64.9%. For the cities of the country, with the exception of Moscow and Saint Petersburg, the index in these two years amounted to 55.7 and 52.0%, respectively. The experiment carried out on the consolidated rating portal showed that scientometric manipulation by analogy with PR-campaigns can be “white”, i.e. aimed to improve their own image, and “black”, aimed to undermine unfriendly subjects – individuals and organizations.

However, it would be wrong to deny a new analytical approach based on the trust index. For example, if we do not take into account rating products, whose trust indices were tampered with, the average trust index for the rest of the ratings was 64.6%. This figure allows us to conclude that society tends to trust the rating products available at the information portal. In other words, when manipulating actions are neutralized, it can be expected that the existing ratings can become a working analytical material for many interested persons.

 

Attitude of the expert community toward the rating movement

 

The database of video interviews of Russian experts on the portal was designed on the premises that the interviewed specialists included the main participants of the process of compiling and using academic ratings. For this purpose, comments were taken from RAS academicians V.M.Polterovich (head of mathematical economics laboratory, Central Economics and Mathematics Institute of the Russian Academy of Sciences, deputy director of Moscow School of Economics, Lomonosov Moscow State University) and N.I.Ivanova (first deputy director of Primakov Institute of World Economy and International Relations). The opinion of the academic community was supplemented by an interview with E.B.Len’chuk (director of RAS Institute of Economics) and A.Ya.Rubinshtein (editorin–chief of the Journal of the New Economic Association, head of the scientific field “Theoretical Economics” at RAS Institute of Economics). The university community was represented by I.A.Bronnikov (deputy dean for academic affairs, political science department, Lomonosov Moscow State University), S.N.Sil’vestrov (director of the Institute of Economic Policy and Economic Security at the Financial University under the Government of the Russian Federation), and S.A.Tolkachev (first deputy director of the department of economic theory, Financial University under the Government of the Russian Federation); A.V.Savvateev (rector of Dmitry Pozharsky University) represented private higher education institutions. A.Gaganova (faculty of journalism at Lomonosov MSU) expressed the opinion of students. O.V.Tret’yakova (head of the department of editorial and publishing activity and scientific and information support, Vologda Research Center of RAS, deputy editor–in–chief of the journal Economic and Social Changes: Facts, Trends, Forecast) spoke as a representative of the publishing community. This whole range of views was supplemented by an opinion of a representative of the rankers community D.V.Petrosyants (an expert at the research group “National University Rankings” of Information Group “Interfax”). The domestic business, on whose behalf E.A.Gaganov (business development director of AO Italion) spoke, was not ignored either.

The uniqueness of the implemented project lies in the fact that each of the interviewed experts noted such a side of the rating process, which is not obvious at first glance. At the same time, some statements of experts were extremely categorical and radical, which helped “expose” the problems existing in this area. Let us consider the main points that were made by the experts in their interviews.

1. In today’s highly unstable environment and in the entire system of relations between market participants, ratings are necessary, as they allow us to record the situation, assess it and understand the changes taking place (S.N.Sil’vestrov). Moreover, many experts supported the idea of creating a single rating portal and supported its further maintenance and development (I.A.Bronnikov).

2. Ratings are necessary and very useful for research purposes, but they should not be used as the main source of information, but rather as a kind of auxiliary statistical array that helps organize and structure the analyzed subject area and denote certain landmarks (E.B.Len’chuk). Reassessment of ratings and their importance is extremely dangerous and counterproductive. We agree with D.V.Petrosyants, who pointed out: “A rating’s a lie, but it’s got a hint…” A rating is not an absolute truth, but it always contains important information about the object of evaluation. In other words, correct attitude toward ratings is the key to their effective use. For example, journal ratings are of interest to academics who publish their papers in them, but they are to no lesser extent needed by journal publishers, who can see their own place in the market and find tips on how they should adjust their development strategy (O.V.Tret’yakova).

3. Since ratings are influenced by many incoming circumstances, they are often onesided and should be used very cautiously to build systems of financial incentives (and sanctions!) (E.B.Len’chuk). Incentives and sanctions trigger the so-called Goodhart’s law with its inherent mechanisms of manipulating information and distorting the true picture. For example, attempts to evaluate researchers and scientific organizations only on the basis of bibliometric information give negative results. This thesis is a direct development of the previous one.

4. Ratings should not be turned into commercial projects. As a rule, business projects are extremely vulnerable both at the stage of collection and at the stages of their processing and use (N.I.Ivanova). The commercial interests of rankers can not only distort market information, but directly provoke its purposeful falsification.

5. Ratings should deal with not only and not so much (!) the best, but also the worst market participants (N.I.Ivanova). The fact is that the ratings are aimed at monitoring a limited number of market leaders, while outsiders who did not get into the ratings continue to work and make a negative contribution to the development of the economy. This state of affairs is the case for scientific journals, many of which do not claim to be in the top lists of the ratings, but are included in the VAK list and publish obviously poor and anti–scientific materials for a price. In other words, market participants who fall into the category of “dangerous” outsiders must not be overlooked. We can say that ratings should become a kind of full–fledged market research covering the entire market.

6. Rankers should comply with the principle of neutrality (disinterest as a participant). Otherwise, when rankers themselves are participants of the rating, it is likely that strong qualities of the rankers will be part of the rating technique, which will bring them to the forefront (E.B.Len’chuk).

7. Ratings should be based on three key principles: pluralism (diversity), duration of the period of preparation, and improvement of their formation technique (S.N.Sil’vestrov). Otherwise, the monopoly of the rating (absence of competing rating products), its point values (for one date) and one and the same (outdated over time) formation technique can cause serious harm due to errors and disorientation of the consumer.

8. The use of ratings should be based on the principle of openness of the object of rating and the ranker (D.V.Petrosyants). For example, when making a rating of journals, it is necessary that their texts are freely available; otherwise, a paradox arises: the more interesting an article, the less it is available. This effect prevents the formation of an adequate rating. The principle of ranker openness suggests that rankers have a good feedback from the community of rated entities. For example, when a rating is built, it should be discussed with the professional community to avoid resentment and discontent; if necessary, adjustments should be made to the rating technique and to the way the data are made public. Only in this case the rating itself acquires the necessary potential of “legitimacy” and trust.

9. For ratings to be used correctly, the principle of selectivity and optionality must be observed. This principle means that market participants voluntarily decide to use certain ratings in their activities. A striking example of this practice can be found in U.S. universities, which have two different systems of employment. In the most prestigious universities (Harvard, Stanford, Princeton, etc.), the decision on a particular candidate for the position is based on the assessment of their academic performance by university experts and the conclusion of the relevant commission; formal bibliometric data are not taken into account. In the weaker and peripheral universities, as a rule, there are no qualified experts able to assess an applicant adequately, this is why bibliometric data of an individual are actively used (V.M.Polterovich). This example illustrates the basic rule: the use of ratings is due to the lack of expertise (A.I.Rubinshtein). In other words, ratings and the expert community are mutually substituting mechanisms for evaluation. Meanwhile, there are other options for assessing the professional suitability of employees of U.S. universities. Thus, in the most prestigious universities, to obtain a permanent position, an employee must publish their papers in the most prestigious (top–rated) journals, whereas in the weaker universities there are no such requirements (V.M.Polterovich). In recent years, Russian universities are increasingly assessing employees according to their own rating systems. In other words, the rating and its user must correspond to each other in a certain sense.

10. Currently, it is necessary that quantitative ratings based on measurable indicators, and qualitative ratings based on expert surveys should coexist and, in some cases, combine. Some respondents speak in favor of quantitative ratings because of their clarity and transparency in contrast to anonymous experts whom we have no reason to trust (A.E.Gaganova). Other experts are equally categorical about the fact that it is not the ratings, but specific people who need to be trusted, and this approach forms the table of ranks directly in the mind of an individual on the basis of communication with other people (A.Ya.Rubinshtein). It was suggested that the rating of a university can be measured only by determining the level of intellectual debate between students and professors; therefore, the correct rating can only be in the mind of the person who goes to universities and participates in such discussions, while any quantitative indicators reflect completely different aspects of the work of an organization (A.V.Savvateev). The lack of consensus on what ratings are better means that the use of both is legitimate.

These principles and approaches to the development and use of ratings can prevent many errors associated with this assessment tool.

 

Conclusion

 

The experience of implementation of the project on consolidation of rating products described above has shown that there is currently a certain confrontation between the rating movement and the expert community, which extends to the confrontation between quantitative and qualitative ratings. In fact, we are faced with an ambivalent process: inadequate quantitative ratings are the result of a low qualification of experts (rankers), and similar inadequate and excessively subjective expert assessments are produced by a weak expert community and require the methods of evaluation to be more objective. Apparently, mutual adjustment of these two directions in Russia will go on for quite a while, until a reasonable equilibrium can be gradually reached. An important thing is that the growing practice of making ratings and the dialogue of rankers with the wider expert community will contribute to improving the quality of both the former and the latter. And the open Internet platform that we created for consolidation of various rating products can become a serious help in this mutual improvement.

Taking into account the thematic diversity of rating products, we should point out that their consolidation within the single portal is likely to lead to its “overload” and complicate the browsing considerably. In this regard, in the future it would be advisable to create a set of specialized rating portals, each of which would reflect the local (thematic) market as fully as possible. Moreover, within such portals it is necessary to upload not only domestic, but also foreign rating products, if possible.

 

References

 

1. Baty Ph. Measured, and found wanting more. Times Higher Education, 2010, July, 8. Available at: http://www.timeshighereducation.co.uk/story.asp?storycode=412341 (accessed: 21.02.2018).

2. Jobbins D. GLOBAL: First shots fired in ranking war. University World News, 2010, March 7. Available at: http://www.universityworldnews.com/article.php?story=20100305112613216 (accessed: 21.02.2018).

3. Borgans L., Covers F. The americanization of European higher education and research. Voprosy obrazovaniya=Educational Studies (Moscow), 2010, no. 2, pp. 5–44. (In Russian).

4. Parijs Ph. European higher education under the spell of university rankings. Ethical Perspectives, 2009, vol. 16, no. 2, pp. 189–206. DOI: 10.2143/EP.16.2.2041651. Available at: http://cdn.uclouvain.be/public/Exports reddot/etes/documents/2009u.Rankings.pdf (accessed: 21.02.2018).

5. Smith D.N. The history behind universities, league tables and the brand. Global Opinion Survey. New Outlooks on Institutional Profiles, 2010, February. Available at: http://science.thomsonreuters.com/m/pdfs/Global_Opinion_Survey.pdf

 (accessed: 21.02.2018).

6. Aguillo I.F., Bar-Ilan J., Levene M., Ortega J.L. Comparing university rankings. Scientometrics, 2010, vol. 85, no. 1, pp. 243–256. DOI: 10.1007/s11192-010-0190-z.

7. Van de Walle S. Publishing in Public administration: issues with defining, comparing and ranking the output of universities. International Public Management Journal, 2015, vol. 18, no. 1, pp. 87–107.

8. Gertler E., Mackin E., Magdon–Ismail M., Yi Y. Computing Manipulation of Ranking System. Istanbul, 2015. Available at: http://www.cs.rpi.edu/~xial/Files/GMMXYAAMAS15.pdf (accessed: 21.02.2018).

9. Arnold D.N., Fowler K.K. Nefarious numbers. Notices of the AMS, 2011, vol. 58, no. 3, pp. 434–437. Available at: http://www-users.math.umn.edu/~arnold/papers/impact-factors.pdf (accessed: 21.02.2018).

10. Alves A.D., Yanasse H.H., Soma N.Y. An analysis of bibliometric indicators to JCR according to Benford’s law. Scientometrics, 2016, vol. 107, no. 3, pp.1489–1499. DOI: 10.1007/s11192-016-1908-3.

11. Hammarfelt B., Rushforth A.D. Indicators as judgment devices: An empirical study of citizen bibliometrics in research evaluation. Research Evaluation, 2017, vol. 26, no. 3, pp.169-180. DOI: 10.1093/reseval/rvx018.

12. Sverdlov E.D. Citation mirages. Bibliometric assessment of the impact of scientific publications of individual researchers. Vestnik Rossiiskoi akademii nauk=Herald of the Russian Academy of Sciences, 2006. vol. 76, no. 12, pp. 1073–1085. (In Russian).

13. Arskii Yu.M., Markusova V.A., Chumakova N.F. Earth sciences: volumes and ratings of Russian publications. Vestnik Rossiiskoi akademii nauk=Herald of the Russian Academy of Sciences. 2007, vol. 77, no. 10, pp. 887–895. (In Russian).

14. Donetskaya S.S. Analyzing the competitiveness of Russian universities in global ratings. Vysshee obrazovanie v Rossii=Higher education in Russia, 2014, no. 1, pp. 20–31. (In Russian).

15. Karminskii A.M., Polozov A.A. Entsiklopediya reitingov. Ekonomika, obshchestvo, sport [Encyclopedia of ratings. Economy, society, sports]. Moscow: Forum, 2016. 448 p.

16. Murav’ev A.A. On scientific value of Russian journals in economics and related disciplines. Voprosy ekonomiki=Issues of economics, 2013, no. 4, pp. 130–151. (In Russian).

17. Pislyakov V.V. Russian journals on social sciences in foreign databases. Ekonomicheskii zhurnal Vysshei shkoly ekonomiki=Higher School of Economics Economic Journal, 2011, vol. 15, no. 2, pp. 268–269. (In Russian).

18. Tret’yakova O.V. Ranking of scholarly journals of economic institutes of the Russian Academy of Sciences. Ekonomicheskie i sotsial’nye peremeny: fakty, tendentsii, prognoz=Economic and Social Changes: Facts, Trends, Forecast, 2015, no. 5, pp. 159–172. DOI: 10.15838/esc/2015.5.41.11. (In Russian).

19. Rubinshtein A.Ya. Ranking of Russian economic journals: the scientific method or “numbers game”? Zhurnal Novoi ekonomicheskoi assotsiatsii=Journal of the New Economic Association, 2016, no. 2 (30), pp.162–175. (In Russian).

20. Burakov N.A., Slavinskaya O.A. Ranking of economic journals: sociological measurement. Vestnik Instituta ekonomiki Rossiiskoi akademii nauk=Bulletin of the Institute of Economics of the Russian Academy of Sciences, 2017, no. 1, pp.151–169. (In Russian).

21. Aleskerov F.T., Badgaeva D.N., Pislyakov V.V., Sterligov I.A., Shvydun S.V. An importance of Russian and international economic journals: a network approach. Zhurnal Novoi ekonomicheskoi assotsiatsii=Journal of the New Economic Association, 2016, no. 2 (30), pp.193–205. (In Russian).

22. Balatsky E.V., Ekimova N.A. The concentric model of Russian market for economic research. Mir novoi ekonomiki=The World of New Economy, 2017, vol. 11, no. 2, pp. 82–92. (In Russian).

23. Balatsky E.V., Ekimova N.A. Competition of economic journals in Russia: results of the three waves of ranking. Ekonomicheskaya politika=Economic Policy, 2017, vol. 12, no. 6, pp.178–201. (In Russian).

24. Balatsky E.V., Ekimova N.A. Russian model of pairing academic performance and educational potential of regions. Izvestiya UrGEU= Journal of the Ural State University of Economics, 2017, no. 5 (73), pp.109–127. (In Russian).

25. Moskovkin V.M., Sun Xingyuan. Economist rankings: modern state and prospects for future studies. Nauchnaya periodika: problemy i resheniya=Scholarly Communication Review, 2017, vol. 7, no. 1, pp. 20–45. DOI: 10.18334/nppir.7.1.38336. (in Russian).

26. Moskovkin V.M., Sun Xingyuan. Development of methods for determining the rankings of scientists based on the Russian Science Citation Index. Nauchno-tekhnicheskaya informatsiya. Seriya 1: Organizatsiya i metodika informatsionnoi raboty= Scientific and Technical Information Processing, 2017, no. 8, pp.23–28. (In Russian).

27. Sterligov I.A. Pilot project on evaluation of Russian scientific journals. Moscow: NIU VShE, 2014. Available at: http://slidegur.com/doc/225135/pilotnyj-proekt-vshe--po-e-kspertnoj-ocenke-rossijskih (accessed: 21.02.2018). (In Russian).

28. Rubinshtein A.Ya., Burakov N.A., Slavinskaya O.A. Soobshchestvo ekonomistov i ekonomicheskie zhurnaly (sotsiologicheskie izmereniya VS bibliometrii): nauchnyi doklad [Economists community and economic journals (sociological assessments vs bibliometrics): a scientific report]. Moscow: Institut ekonomiki RAN, 2017. 83 p.

29. Murav’ev A.A. Ratings of Russian journals in economics: comparative analysis. 2015. Available at: http://studydoc.ru/download/4796527 (accessed: 21.02.2018). (In Russian).

30. Subochev A.N. How different are the existing ratings of Russian economic journals and how to unify them? Zhurnal Novoi ekonomicheskoi assotsiatsii=Journal of the New Economic Association, 2016, no. 2 (30), pp. 181–192. (In Russian).

31. Balatsky E.V., Ekimova N.A. The problem of manipulation in the RSCI system. Vestnik UrFU. Seriya: Ekonomika i upravlenie=Bulletin of Ural Federal University. Series Economics and Management, 2015, no. 2, pp. 166–178. (In Russian).

32. Balatsky E.V., Yurevich M.A. The misalignment of Russian Economists’ scientometric indicators in RISC. Zhurnal Novoi ekonomicheskoi assotsiatsii=Journal of the New Economic Association, 2016, no. 2 (30), pp. 176–180. (In Russian).

 

 

 

 

Official link to the article:

 

Balatsky E.V., Ekimova N.A. Opportunities for the Consolidation of Rating Products in the Internet Environment// «Economic and Social Changes: Facts, Trends, Forecast», 2018, vol.11, no.2, pp.37–51.

226
3
Добавить комментарий:
Ваше имя:
Отправить комментарий
Публикации
В статье рассматриваются основные этапы становления малого бизнеса в России, дается оценка масштабов и эффективности его деятельности. На базе проведенных расчетов показано влияние развития малого предпринимательства на темпы экономического спада в 90-х годах. Критически оцениваются макроэкономические подходы к стимулированию деятельности малых предприятий. Намечены основные пути развития отечественного малого бизнеса.
В статье предлагается оригинальная методика оценки ущерба от законодательной деятельности федерального правительства в регионе. Методика представляет собой простую процедуру расчета, основанную на использовании принципа мультипликатора. Продолжением методики является схема расчета компенсационных выплат федерального центра региону, понесшему ущерб. На условном примере показано применение разработанной методики.
В статье рассматривается простая диффузионная модель роста нового рынка, в которую вводится дополнительный фактор – налоговая нагрузка. Такая модель позволяет понять долгосрочное влияние фискального фактора, который предопределяет асимпотические свойства объема рынка нового продукта и оказывает нелинейное воздействие на активность предприятия-инноватора. На условном примере проиллюстрированы возможности применения предложенной модели в прикладных расчетах.
Яндекс.Метрика



Loading...