Dr Kate Charles, Head of Research & Commercialisation at the University of Portsmouth – my initial thoughts on Lord Stern’s REF Review recommendations. Comments and alternative perspectives most welcome.
Stern Review – An independent review of the REF
The Stern Review was published on 28th July, following a call for evidence early in 2016. This independent review makes recommendations on the future operation of the Research Excellence Framework (REF). The review examines the purpose and benefits of REF; problems and issues from REF2014, and sets out principles and recommendations for shaping future research assessment exercises.
The full report ‘Research Excellence Framework (REF) review: Building on success and learning from experience’ is available here:
Research Excellence Framework Review – Full Report
With further articles and comment available from a number of sources, including the Times Higher and The Conversation:
Stern review: submit all researchers to next REF
Stern review: response from the sector
Stress put on academics by the REF recognised by Stern Review
Recommendations outlined in the report, and of most interest to researchers, are set out in bold below, with my comments and further details from the report added as commentary. It is useful to bear in mind that the Stern Review was commissioned principally to look at efficiency and cost savings in future REF exercises, and it was recognised that much of this burden is currently borne by institutions – one would hope that this key principle is not lost in the detailed planning and implementation of the recommendations!
That the broad structure of the REF should not change.. And an interval of 5-7 years between exercises is reasonable.
Staff selection and outputs
1: All research active staff should be returned in the REF.
2: Outputs should be submitted at Unit of Assessment level with a set average number per FTE but with flexibility for some faculty members to submit more and others less than the average.
3: Outputs should not be portable.
The report states that ‘all academic staff who have any significant responsibility to undertake research’ are to be returned to REF and allocated to a Unit of Assessment. With ‘a common dataset’ to ‘support the accurate description of university research and teaching staff assessed in the REF and TEF’. This could perhaps mean that a proportion of academic staff time could be attributed to teaching vs research activities, made transparent through HESA reporting, or alternatively that anyone with [contractual] responsibility/allowance for research will need to be returned to the REF in future. Clearly, more details are needed on how ‘research active’ will be interpreted by Hefce.
A concern for non-research intensive universities will be whether the funding formula will then take further account of any volume measures, if selection is removed and all academic staff are to be submitted. Some institutions moved towards teaching-only contracts prior to REF2014, but how/whether this will affect the balance between REF and/or TEF performance needs to be carefully considered by institutions. This recommendation, together with the decoupling of outputs from individuals (see below) may in fact counter this trend towards teaching-only contracts, or it may accelerate it depending on the all important details of REF and TEF measures and how ‘research active’ is to be defined.
Another major change proposed, is to ‘decouple’ outputs from individuals. In REF2014, four outputs were required per FTE with reductions available for ‘special circumstances’. This put the onus on institutions and individuals to demonstrate specific individual circumstances, such as sickness absence, which may have resulted in a reduction of productivity during the REF period. This was clearly a very sensitive area for staff, so I believe that in general this move which would allow a unit to focus on the highest quality outputs across the unit as a whole will be welcomed by most. The suggested number of outputs is the equivalent of two per FTE submitted on average, but within a suggested range of zero to six per FTE.
To prevent the REF ‘transfer window’ the suggestion is that outputs should remain with the institution where they were produced. This would create a consistency with impact which remains with the institution in which the research was done. However, this suggestion may result in some further complications, where individuals/institutions are required to prove acceptance dates for research outputs, when a member of staff has moved institutions during the assessed period. The date of acceptance is already required by the REF Open Access requirements for journal articles and conference proceedings (articles must be deposited in an institutional repository within 3 months of acceptance), but how this will extend to other works such as monographs, books and book chapters is yet to be seen. Further, whilst the recommendation that ‘outputs should be submitted only by the institution where the output was demonstrably generated’ and the ’HEI where [the author] was based when the output was accepted’ is generally fair, the two may not be entirely congruent – it implies that outputs are prepared immediately as/after the work is produced and/or that there is no time lag between submitting an article and getting it accepted for publication, during which an author may transfer to another institution. Whilst I agree with the spirit of this recommendation, implementation would appear to add further complexity, rather than reduce the administrative burden of REF on institutions.
Assessment of outputs
4: Panels should continue to assess on the basis of peer review. However, metrics should be provided to support panel members in their assessment, and panels should be transparent about their use.
In line with the Metric Tide recommendations, Stern has recognised that metrics alone cannot replace robust peer review, but is supportive of the ‘appropriate use of bibliometric data’ by panels. The recommendation is that the weighting for outputs remains at 65%.
5: Institutions should be given more flexibility to showcase their interdisciplinary and collaborative impacts by submitting ‘institutional’ level impact case studies, part of a new institutional level assessment.
6: Impact must be based on research of demonstrable quality. However, case studies could be linked to a research activity and a body of work as well as to a broad range of research outputs.
7: Guidance on the REF should make it clear that impact case studies should not be narrowly interpreted, need not solely focus on socio-economic impacts but should also include impact on government policy, on public engagement and understanding, on cultural life, on academic impacts outside the field, and impacts on teaching.
On impact, there are suggestions for broadening the range of impacts, decoupling impacts from outputs, and reducing the number of impact case studies required with a ‘minimum of one impact case study in each Unit of Assessment (down from a minimum of two in REF2014)’. The main change here is for institutional level case studies, which could reflect the strategic cross-faculty approach to impact and showcase interdisciplinary work. At Portsmouth, the development of our interdisciplinary R&I themes could be key to developing these institutional level case studies. As regards broadening impact types, noting suggestions for greater inclusion of cultural impacts and public engagement, the devil will be in the detail – these types of impact are generally thought to be harder to robustly evidence. So, it remains to be seen whether the sector will fully embrace these suggestions for impact which were largely allowed in REF2014, and practically whether institutions will still favour impact case studies for which harder evidence can perhaps more easily be provided, e.g. economic impacts.
As anticipated, the recommendation is that the ‘Impact Template’ from REF2014 is absorbed into the Environment Template in future, with the overall weighting for impact comprising no less than 20%. So the weighting of individual impact case studies would be marginally increased, through the removal of the impact template and for smaller units submitting a single case study – this would be high currency indeed!
8: A new, institutional level Environment assessment should include an account of the institution’s future research environment strategy, a statement of how it supports high quality research and research-related activities, including its support for interdisciplinary and cross-institutional initiatives and impact. It should form part of the institutional assessment and should be assessed by a specialist, cross-disciplinary panel.
9: That individual Unit of Assessment environment statements are condensed, made complementary to the institutional level environment statement and include those key metrics on research intensity specific to the Unit of Assessment.
This is ‘institutional level Environment assessment’ is, I believe, generally a nod to research-intensives, which during the call for evidence proposed aggregated, volume-measures for research assessment. It is perhaps at odds with the recommendation for ‘supporting excellence wherever it is found’ which is retained as a key principle for REF and the RAND Europe/KCL report on high performing research units, which identified that they were largely autonomous from the institution in terms of culture and environment! In terms of reducing the burden, however, there should be economies for such institutional statements and retention of the UoA statements should hopefully allow for pockets of excellence to be sufficiently demonstrated. By a process of deduction (and simple mathematics) it is suggested that Environment weighting will remain at 15%.
Of course, it is important to note that these are recommendations at this stage and that a full consultation on implementation will need to take place. Indeed, Hefce have stated:
”The UK government and the devolved administrations will now need to consider the report and make their views known on next steps. The UK higher education funding bodies will also carefully consider the recommendations. Subject to the views of our respective ministers, we intend to launch a consultation as soon as possible before the end of 2016.”
In order to fulfil the Government’s ambition and complete the next REF by 2021, the following timetable is proposed in the Stern report:
- Through the summer and autumn, the UK Governments and funding councils should work together to translate the principles outlined in this report into the structures and formulae for submissions. Work will be required to test the impact of our proposals on scope for game playing and to mitigate against unintended consequences. Further work might also be required to model or pilot new ideas.
- By the end of the year a formal consultation should be issued so that the community can offer their views on the proposed process and the future REF formula. The decisions arising from this consultation should be published in the summer of 2017.
- As the shape of later years of the TEF exercise are revealed, Government should check the consistency of the two exercises and work to promote complementarity and mutual support and to alleviate tensions and burdens.
- The proposed process will allow sufficient time for universities to prepare for submissions to be collected in 2020 and the assessment to take place in 2021, with the final outcome of the next REF exercise to be published by the end of 2021 for implementation of the funding settlement in 2022.
Comments and discussion of this post and the Stern review are welcomed – please add them below. Alternatively, if you would like to contact anyone from Research and Innovation Services regarding this post please email email@example.com
This post is interesting. I can see the argument behind why they’ve suggested that research shouldn’t be portable, but it does present a slight logistical nightmare in terms of how to implement it.
Unless I’m missing something, this would mean that not only would all research publications need to be recorded in Pure, but we’d also need to record the university that each author was at when they were accepted for publication. It’s not that this isn’t (just about) doable, but I can see it being very time consuming. So it would have been useful if they’d asked for this at the start of the REF period, rather than what will be over 3 years late, by the time these recommendations get agreed!
There will be a consultation period from Hefce and also some piloting of the suggestions prior to implementation. Hopefully, this will iron out some of these practical issues.
One thing I didn’t mention is that ORCID (unique contributor identifiers) are recommended to be mandatory for the next REF, and recommendations for making better use of data that already exists. It would be a big job, but in theory Elsevier (assuming REF is going to use Scopus again) could add organisation identifiers and ORCIDs to its publication set (from 2014). Otherwise each output would have to be checked for the affiliation of contributors, as set out on the publication and further checked against employment dates. Although I guess, HEIs would only need to look at the outputs for those transferring between institutions during the REF period.
Anecdotally we have heard that one HEI, having anticipated this recommendation, has asked a new recruit to change their affiliation on articles in press… Game playing will not be eliminated, whatever the rules!
It does seem that it could be good, bad (or worse) or somewhere in between depending, as you note, on how things are finally defined (and on the decisions universities make).
There seem clear peculiarities with the non-portability and the decoupling of outputs and the invoking of flexibility. It is clearly sound to hold (as Stern does) that there should be a reduction in monitored output to reflect fractional staff (and is a continuance of the last REF). The review also suggests that there should be a reduction in no of papers to reflect those who retire/leave  which seems sensible. However, the move away from other special circumstances is problematic. While it may ‘allow a unit to focus on the highest quality outputs across the unit as a whole’, it will be subject to the set amount (which will depend on the definition of research active). Unless the definition is highly exclusionary (and possibly even then), the indisputable fact that the unit will be the sum of its parts means that failure to consider special circumstances may be discriminatory. It seems quite frankly surprisingly ill-thought through for the Stern Review to say that flexibility (i.e. others submitting more) is the way that members of staff who join mid-term will be accounted for (whereas there could be deduction for those who retire or leave) and that those who miss say half the REF period on maternity/paternity leave must be accounted for by others doing extra (whereas those who are on a 0.5 FTE contract will attract a deduction). Those figures are not particularly far-fetched but it is certainly not unforeseeable that a UoA with a young age profile may be adversely affected through the cumulative effect of such leaves against a UoA with an older age profile. The calculation of staff numbers may become a little more difficult but it would be fairer and would address the concern raised in the papers and online about ECRs being disadvantaged (as they would not be expected to contribute fully having left papers at the previous institution). I am not sure what lies behind this preference in Stern for ‘flexibility’.
It would be a worrying (and transformatively damaging) move if the definitions and details of the REF and TEF measures (or institutions’ reactions to them) were to lead to a division between teaching and research. The next steps will be interesting to say the least.
Many thanks for the post.
LikeLiked by 1 person
Many thanks, James. You make very good points here and it is valuable to get as many perspectives as possible on these recommendations. I would encourage all of our researchers to engage in the consultations that will undoubtedly follow – within the institution and with peers externally.
A good post from Chris Hewson at University of Salford: https://www.linkedin.com/pulse/impact-environment-reflections-stern-review-part-1-chris-hewson?trk=prof-post
Comments are closed.