Letter to CRUS regarding Bibliometric Evaluation
[ français | deutsch | english ]
Swiss Association for Research in Information Technology (SARIT)
The President
Prof Dr. Daniel Thalmann
EPFL-VRlab
1015 Lausanne
To the
President of the Conference of Chancellors
of the Swiss Universities (CRUS)
Professor Hans Weder
Post-office box 607
3000 Bern 9
Lausanne, 16.01.2008
CRUS — Project for the analysis of the services of research of Swiss universities based on bibliometric criteria
Mr. President,
We learned that CRUS started a project aiming at collecting bibliometric data in order to analyze the performance of research of the Swiss universities (Project "Bibliométrie 2007").
This project gives rise to the greatest reservations for the undersigned professors of information technologies of the Swiss universities, represented by SARIT [1], for several reasons:
1. The measurement of scientific performance through a bibliometric analysis causes behavior prejudicial to Information Technologies and science itself (by supporting quantity instead of quality, the publication of incremental results with little significance, increasingly frequent appearance of joint authors whose contributions are negligible, and the easiness of crossed citations, to quote only some well-known problems). We also refer in this respect to the article published recently, "Stop the numbers game" of David Parnas [2], one of the most famous researchers in Computer Science.
2. Bibliometric investigations are clearly not very reliable; see the article "For the evaluation of data processing by means of bibliometric analyses" of Friedemann Mattern [3]. In particular, the "frequency of citation" does not account for the quality of the researchers, because
- it depends more often on the social recognition of the researcher than excellence of his/her scientific work
- it favors researchers who work on fashionable topics
- it favors the fields of knowledge which traditionally publish shorter articles compared to those where publications are longer
- it cannot differentiate between the fashion and the substance of a paper
- it can favor the authors of "surveys", who are very frequently cited, compared to the authors of focused research papers
- a position article or even an erroneous article can be criticized and consequently well cited.
We can add to these criticisms the distortions by the absence of distinction between external and personal quotations, the "trusts or business of citation" and the large lists of authors. The same applies to "impact factor", which, in spite of relevant and repeated warnings (for example, Amin and Mabe 2000 [4]), is still used wrongly today as the key criterion for the evaluation of a person.
3. Measurements of impact using citation indexes of the ISI in the field of Information Technologies are often erroneous. This is because the culture of publication in Information Technologies is very different from that in other sciences. Whereas in other scientific fields, the publications in prestigious journals are more recognized than papers presented in conferences, this is not always true in Information Technologies: the rate of acceptance of contributions in top conferences, even in some workshops, is often lower than in top journals. As ISI quotation indexes refer only to contributions in scientific journals, and not in conference proceedings, their use as factors for evaluation lead to false conclusions.
In addition, some journals in information technologies, which have an excellent reputation, are unfortunately not covered by these indices because of their relatively young age.
The signatories do not oppose an evaluation of the quality of research for the academic staff in Information Technologies. We ask, however, that such be based on criteria of evaluation scientifically recognized and legitimate. Given the current state of knowledge, only an evaluation by experts (peer review) seems to us appropriate to satisfy these requirements ----- certainly not a purely bibliometric analysis. Such an analysis, even carefully carried out, could be at most used to refine the evaluation of an institution or possibly of a researcher in Information Technologies. The blind application of automated bibliometric measurement covering all scientific disciplines in an undifferentiated way would prove seriously harmful for science and information technologies.
To quote Albert Einstein: "Not everything that counts can be counted, and not everything that can be counted counts.".
Prof. Daniel Thalmann President SARIT As well as the professors of Information Technologies listed below.
1. All Professors of Computer Science at Swiss Universities are normally members of SARIT.
2. David Parnas (2007). "Stop the numbers game", Communications of the ACM 50, 11: 19-21.
3. Friedemann Mattern (2002). "Zur Evaluation der Informatik mittels bibliometrischer Analyse", Informatik-Spektrum 25, 1: 22-32. Friedemann Mattern (2002). "Bibliometric Evaluation of Computer Science — Problems and Pitfalls" http://www.vs.inf.ethz.ch/publ/slides/Mattern-Bibliometry-SARIT06.pdf
4. Mayur Amin, Michael Mabe (2000). Impact Factors: Use and Abuse. Perspectives in Publishing, No. 1 (October 2000). Elsevier Science. 1-6.
Signatures
| Name / Nom | Institution |
| Prof. Andrea Back | U. St. Gallen |
| Prof. François Bavaud | U. Lausanne |
| Prof. Abraham Bernstein | U. Zürich |
| Prof. Hanspeter Bieri | U. Bern |
| Prof. Torsten Braun | U. Bern |
| Prof. Didier Buchs | U. Genève |
| Prof. Joachim Buhmann | ETH Zürich |
| Prof. Horst Bunke | U. Bern |
| Prof. George Candea | EPF Lausanne |
| Prof. André Csillaghy | Fachhochschule Nordwestschweiz |
| Prof. Jacques Duparc | U. Lausanne |
| Prof. Boi Faltings | EPF Lausanne |
| Prof. Pascal Felber | U. Neuchâtel |
| Prof. Pascal Fua | EPF Lausanne |
| Dr. Patrick Furrer | Euresearch |
| Prof. Harald Gall | U. Zürich |
| Prof. Walter Gander | ETH Zürich |
| Prof. Benoît Garbinato | U. Lausanne |
| Prof. Solange Ghernaouti-Hélie | U. Lausanne |
| Prof. Eduard Glatz | Fachhochschule Ostschweiz |
| Prof. Martin Glinz | U. Zürich |
| Prof. Gaston Gonnet | ETH Zürich |
| Prof. François Grize | U. Lausanne |
| Prof. Markus Gross | ETH Zürich |
| Prof. Thomas R. Gross | ETH Zürich |
| Prof. Matthias Grossglauser | EPF Lausanne |
| Prof. Dominik Gruntz | Fachhochschule Nordwestschweiz |
| Prof. Jürg Gutknecht | ETH Zürich |
| Prof. Rolf Haenni | Berner Fachhochschule |
| Prof. Jürgen Harms | U. Genève |
| Prof. Pius Hättenschwiler | U. Fribourg |
| Prof. Matthias Hauswirth | U. Lugano |
| Prof. Michael Hess | U. Zürich |
| Prof. Lorenz Hilty | EMPA |
| Prof. Hans Hinterberger | ETH Zürich |
| Prof. Béat Hirsbrunner | U. Fribourg |
| Prof. Juraj Hromkovic | ETH Zürich |
| Prof. Jean-Pierre Hubaux | EPF Lausanne |
| Prof. Paolo Ienne | EPF Lausanne |
| Prof. Rolf Ingold | U. Fribourg |
| Prof. Gerhard Jäger | U. Bern |
| Prof. Mehdi Jazayeri | U. Lugano |
| Prof. Rolf Jufer | Berner Fachhochschule |
| Prof. Gerhard F. Knolmayer | U. Bern |
| Prof. Jürg Kohlas | U. Fribourg |
| Prof. Dimitri Konstantas | U. Genève |
| Prof. Peter Kropf | U. Neuchâtel |
| Prof. Martin Kropp | Fachhochschule Nordwestschweiz |
| Prof. Viktor Kuncak | EPF Lausanne |
| Prof. Friedemann Mattern | ETH Zürich |
| Dr. Jacques Menu | U. Genève |
| Prof. Hansjürg Mey | Prof. em. U. Bern |
| Prof. Bernard Moret | EPF Lausanne |
| Prof. Peter Müller | ETH Zürich |
| Prof. Thomas Myrach | U. Bern |
| Prof. Oscar Nierstrasz | U. Bern |
| Prof. Jürg Nievergelt | Prof. em. ETH Zürich |
| Prof. Martin Odersky | EPF Lausanne |
| Prof. Renato Pajarola | U. Zürich |
| Prof. Christine Parent | U. Lausanne |
| Prof. Cesare Pautasso | U. Lugano |
| Prof. Christian Pellegrini | U. Genève |
| Prof. Claude Petitpierre | EPF Lausanne |
| Prof. Rolf Pfeifer | U. Zürich |
| Prof. Yves Pigneur | U. Lausanne |
| Prof. Bernhard Plattner | ETH Zürich |
| Prof. Marc Pollefeys | ETH Zürich |
| Prof. Thierry Pun | U. Genève |
| Prof. Lutz Richter | Prof. em. U. Zürich |
| Prof. José D. P. Rolim | U. Genève |
| Prof. Timothy Roscoe | ETH Zürich |
| Prof. Ivo Sbalzarini | ETH Zürich |
| Prof. Helmut Schauer | U. Zürich |
| Prof. André Schiper | EPF Lausanne |
| Prof. Heiko Schuldt | U. Basel |
| Prof. Gerhard Schwabe | U. Zürich |
| Prof. Stefano Spaccapietra | EPF Lausanne |
| Prof. Burkhard Stiller | U. Zürich |
| Prof. Kilian Stoffel | U. Neuchâtel |
| Prof. Thomas Strahm | U. Bern |
| Prof. Peter Stucki | Prof. em. U. Zürich |
| Prof. Sabine Süsstrunk | EPF Lausanne |
| Prof. Nesime Tatbul | ETH Zürich |
| Prof. Stephanie Teufel | U. Fribourg |
| Prof. Daniel Thalmann | EPF Lausanne |
| Prof. Marco Tomassini | U. Lausanne |
| Prof. Christian Tschudin | U. Basel |
| Prof. Ulrich Ultes-Nitsche | U. Fribourg |
| Prof. Serge Vaudenay | EPF Lausanne |
| Prof. Thomas Vetter | U. Basel |
| Prof. Roger Wattenhofer | ETH Zürich |
| Prof. Alain Wegmann | EPF Lausanne |
| Prof. Eric Wehrli | U. Genève |
| Prof. Emo Welzl | ETH Zürich |
| Prof. Peter Widmayer | ETH Zürich |
| Prof. Carl August Zehnder | Prof. em. ETH Zürich |