Reviewers raise concerns about RAE gameplaying

The strategic games played by universities to inflate their ratings in the research assessment exercise have been laid bare by the peer-review panels that judged the submissions.

In a series of subject overview reports published by the Higher Education Funding Council for England this week, a number of the panels, which were responsible for assessing research quality in 67 disciplines across 159 institutions, raise concerns. They say that some universities excluded research-active staff from the exercise to artificially "exaggerate" their strengths, while others drafted in research stars on "unusual" contracts who were not fully integrated members of the research team.

David Otley, who chaired the panel that assessed economics, accounting and business and management studies, says in his report: "Some very strong units chose to submit only a proportion of their staff for assessment, despite the encouragement in the criteria for all research-active staff to be submitted."

Because "the panels had no information on the proportion of staff submitted", this meant that some departments "appeared stronger than others solely for this reason".

The business and management studies subpanel says in its report that it "remained concerned ... about the varying degree of selectivity that was apparent in the submissions".

"Although this knowledge was not used in making assessments, it is clear that some submissions included a very small proportion of academic staff from some institutions. This selectivity probably exaggerates the strengths of some institutions."

Unlike in previous RAEs, the Higher Education Statistics Agency was not able to release data showing the proportion of eligible academics left out of the 2008 RAE because of complaints that the guidance on eligibility for submission was unclear. This meant that there was no measure of "research intensity" in departments, prompting claims that the RAE results failed to show the true picture of research.

Times Higher Education understands that some RAE panels penalised perceived gamesmanship by giving a low ranking in the "esteem" and "research environment" assessment categories to departments they believed had submitted a low proportion of their staff. These categories will be published in RAE "subprofiles" in the spring.

The panels' reports also contain concerns about the recruitment of research stars. The subpanel on sociology is "struck by the fact that a few departments placed a great reliance on the presence of highly esteemed academics on fractional contracts or other unusual contractual arrangements".

It adds: "There was not always evidence that these academics ... made an effective contribution to the research culture."

The Asian studies panel's report notes that "the recent appointment of several short-term contract staff" in some institutions "puts a question mark over the sustainability of certain areas of research in the longer term".

Paul Marshall, executive director of the 1994 Group of small research-intensive universities, which submitted a higher proportion of their researchers than the larger research-intensive universities to previous RAEs, said: "Comments emerging from the panels are of great interest and will, we believe, be reflected in the subprofiles achieved for esteem and the research environment."

Les Ebdon, vice-chancellor of the University of Bedfordshire and head of the Million+ think-tank, said: "Some universities have selected their most active staff for submission to gain reputational advantage; whether this will also maximise funding remains to be seen."
---------
The games they play... and these are the ones that have come out. There are others...

From: http://www.timeshighereducation.co.uk