(Bonus) Celebrating Research Diversity: The Hidden REF's Bid to Recognise Non-traditional Outputs

Intro: 00:00:00

Welcome to the Research Culture Uncovered Podcast, where in every episode we explore what is research culture and what should it be. You'll hear thoughts and opinions from a range of contributors to help you change research culture into what you want it to be.

Ged: 00:00:23

Hi, this is Ged Hall, and for those of you don't know me yet, I'm an Academic Development Consultant at the University of Leeds.

ellence Framework, or REF, in: 2014

But today I'm taking a wider view of research assessment by talking to Dr. Gemma Derrick. Gemma is an Associate Professor at the University of Bristol and works in the Centre of Higher Education Transformations.

Her research interests include researcher behaviour, academic practice, research evaluation, societal impact, the research workforce and its governance, particularly through review systems, and the politics and dynamics of knowledge production and translation. So because of those interests, she's investigated the effects of national audit frameworks, such as REF and lots of others, to demonstrate their strengths and weaknesses in related, in relation to their stated aims.

ewers operationalized the REF: 2014

Gemma, it's lovely to talk to you at last.

Gemma: 00:02:05

Hi, it's great to be here Ged, great to be talking to you.

Ged: 00:02:09

Fantastic. So when I was preparing for this interview, Gemma, I read your bio on the Bristol website. You're sniggering about it now, wishing I hadn't written this. Um, but I noticed you put, uh, put in a line about an unsuccessful career in musical theatre.

So before we dive into the, to the kind of like more, uh, staid stuff about research assessment, I'd love to hear a little bit about those, uh, those experiences, if you don't mind sharing them.

Gemma: 00:02:36

Well, you're over talking it a little bit, Ged. You said that it was a unsuccessful career in musical theatre. A career doesn't really happen until it starts.

So, it was more aspirations to be in musical theatre than anything else, and it was based on my love of singing and dancing, um, but although I was a decent dancer. I just can't sing. So perhaps it was more something that I had in my mind than something I pursued seriously. I really liked it. I still do put my earphones in, listen to music from musical theatre as well as all of the Disney classics and imagine myself singing them.

So it would never go away.

Ged: 00:03:13

Fantastic. So you've never, um, you've never thought about trying to sing or maybe even dance your research in a...

Gemma: 00:03:22

Well, I like my colleagues too much to sing in public. So I wouldn't do that to them, um, or anybody else that I don't know, um, to dance. No, that's the thing about a career that you envisaged many, many years ago.

I'm just not as flexible as I used to be.

[: 00:03:38

So there was a, there was a reason behind the singing of that, uh, so. Hence for asking, but, uh, I guess we better get on to what, uh, what your expertise is really all about. Um, so we're, we're talking today a week after the Hidden REF Festival and you're, you're on the, um, the organising committee for that.

So I wonder if you could tell us a little bit about the journey up to the. Up to the actual day of the festival last, uh, last week. Um, and the kind of things that you're hoping to achieve through it.

Gemma: 00:04:34

Oh, goodness. Leading you up to it. So, the Hidden REF started in 2019. And it was going to have the first exercise in 2020.

Um, and the whole, like, way, way that I got. on board is that our fearless leader, Professor Simon Hettrick, um, contacted me because he is, um, was his frame of, well, he's a software engineer and he's very, very concerned that, you know, what he was doing is a very, very, very high, you know, high calibre software engineer was not necessarily REF-able.

And there were many colleagues that felt the same. So he set up the Hidden REF and contacted me because he needed some evaluation experience of which I said, this is a brilliant idea. I want to be involved. You seem like a decent chap. Yeah, yeah. Let's, let's do this. Um, and then the committee grew, uh, with people who went, yeah, this is the right cause.

that point we started this in: 2019

in person until last Thursday at the Hidden REF Festival. So the fact that we have weathered the storm, um, we have produced a Hidden REF, uh, it was going to be 2020 but, you know, something happened in 2020 and, you know, it made us feel better that the real time REF, the real REF, got put back one year, so so did we, um, and we've, you know, Done the Hidden REF and carry through the momentum.

Um, and it's grown from more than just an exercise that we said we do because we're a whole lot of people who had interesting ideas about how the REF could be done better to reflect, you know, the research that is. being done, a broader range of research being done. And instead of saying, well, we don't like the REF, we decided to do it ourselves, basically.

So that's what the Hidden REF was. We weathered all the way through that. It's now grown to a movement, uh, that basically says that, yeah, there are really important outputs and aspects around research culture that are not REF-able and people who are essential to contributing to a vibrant, sustainable, excellent, fun, exciting research culture that we don't necessarily value as much as we can, because strategically on the organisation level, they're not REF-able.

And that has a follow on effect. So if something's not REF-able, it's not built into priorities at the organisation level. It's not built into, um, priorities at the organisation level, it's forgotten, it's overlooked, it's non prior, non valued, and it's not REF-ed. And so we went well. We're going to look at that.

And instead of thinking that the Hidden REF can be a big stress, we said, we didn't want our Hidden REF to be a big stress, which is why we focused on celebration rather than evaluation, because that's what evaluation is. It is a, um, it is, it is in a way of competition, but this is a way that we decided to run it where everybody wins.

Um, and it was the most fun I've had. and still continue to have to this day is to participate and work with this incredible group of people at the Hidden REF who are all very passionate about what they do and are just really nice people to, to work with and sit down and have a drink with too. So, um, wouldn't it be great if all our projects were the same?

Ged: 00:08:04

Absolutely. So looking at the Hidden REF website, it's, um, you know, you said it's a, it's a, it's a movement now. So I was just looking at the website and it said 'you will join a community that is fighting for a more effective and fairer system of evaluating success in research'. So, so from your, from your research, looking at the different systems that exist around the world, are there any that are, that are more effective and fairer in your, in your view?

Gemma: 00:08:33

It depends on what you mean by effective and fair. Okay, so all audit frameworks have choices made and parameters placed around criteria that dictate what it evaluates and what it doesn't. I think that you need to be very clear that these criterion are built upon priorities and these priorities are dictated mostly top down and are in line with government authorities or non-research community based, um, priorities. They sound good and no one's going to argue with them. We all want research to make, you know, society a better place. Uh, but an evaluation framework itself is going, it can't evaluate the entire aspect of, um, research. I mean, something is always going to be hidden.

But we wanted to make sure that just because it's hidden, it's not forgotten, and also because it's not forgotten, it's not non valued, because, um, participating in research is more than just producing four star case, uh, four star case studies for impact and, and, uh, monographs and publications that are also four star and participating in a research environment.

It requires a very complex interplay of things. people of products of communications of, um, strategies, all these sorts of things. And it's difficult. So evaluation framework, you know, by merit of having to maintain, be, be efficient because that's the big thing. Um, but also by, by. By spirit of main, you know, being able to, there's a certain amount of time, you've only got one year to do an entire evaluation of something like 191, 000 publications, and that's not including the case studies in the environment and the case impact templates, all the rest of it.

So there's, there's a certain amount of time. And that means pragmatically choices need to be made. These choices are reflectant in the criteria that is established. The criteria set on priorities and these priorities are not set by the research community themselves. So the Hidden REF was about giving voice to that research community.

and saying, Hey, this is what we value. And this is what we're going to celebrate by recognising and thanking them publicly for the, for the things that they enable us to do. And also for the outputs that we think are important, but are not necessarily REF-able. So this was what the Hidden REF was about.

The design of a lot of research frameworks are around the world. are, by merit of this being a very, very small field, are informed by the same voices over and over again, the same types of insights over again. And it can feel, as an ordinary researcher, at the bottom of the REF food chain, as very much a top down decision that's imposed on you, not something that you participate in and construct for your own, around your own values.

That's what the Hidden REF is about. The Hidden REF is, it was an exercise and now it's a grass-roots movement to inform. What we want to value and how we want to be evaluated. That's what the Hidden REF is about.

Ged: 00:11:22

So essentially. A mechanism of co-production to try and get a system that people can feel like they're buying into more.

Gemma: 00:11:31

Oh, that's putting it in very technical terms and I wouldn't say no. Um, but we, we, I think because I think because the landscape is changing so much. Um, and we are such a large group of dynamic people. We don't want to restrict what we are by any particular, you know. to criterion box that we are co-production on others.

I think, I think that good research and good research audit frameworks only happens by a two way interchange between, um, the people who run the exercise, the people who evaluate the exercise, and the people who are being evaluated. The main criticism against REF in the in the literature, and there's a lot, a lot of people hate the REF, is they feel not engaged with what they're being evaluated against.

uts. Um, but up until, right,: 2021

And what we're concerned about is the software engineer producing a very amazing four star piece of software that, for reasons that, uh, will become clear, uh, are pushed into, into producing an article. because the organisation sees that as a safe bet and less risky because these things are linked to funding outcomes.

Um, and then they're, they're, they're forced to do a publication. That publication might be two stars, so therefore not even at the fundable level at REF. And then they're undervalued, not because they're not doing good research, their software is fantastic, but because they're forced to compact, compacting their research into an output or a dissemination output that's not necessarily appropriate for their work in their field.

And this is what the Hidden REF wants to do.

Ged: 00:13:25

Just before we go on, I wonder if you could, you know, just for our international listeners that maybe haven't experienced REF or read much about it, say what. Two star, four star means?

Gemma: 00:13:36

Oh, okay. So there is a scale of zero to four for any of the outputs and for the impact case studies as well as the environment.

hree main criteria in the REF: 2021

Um, but that doesn't always, it's not always the case. Obviously the REF has a range of different, uh, uh, submissions along, uh, qualities along there. It's always going to be a portion that's one star, two star, three star, and four star. So university's like, I want it to win. I want it to be three and four star, which means they make competitive decisions about what.

There's a safe bet to get three and four star because we are inherently sometimes conservative when it comes to evaluation approaches because we do evaluate ourselves. We are expert on the REF panel, uh, where it's seen and perceived that these less risk, risky submissions are predominantly text traditional outputs such as publications and monographs to the exclusion of all the other things and all the other wonderful outputs that make research so incredible to work in and produce.

Ged: 00:15:26

Yeah. So, last week, the festival happened. You had lots of people there and lots of interesting discussions. Um, what did, um, what were the big talking points in terms of trying to, trying to move us towards that more fair, more effective system?

Gemma: 00:15:46

Well, uh, when, uh, a week after the REF 2028 initial decisions came out, I think that was like in July, June, July this year, um, the Hidden REF uh, established the uh, 5 percent manifesto, which was a challenge given to all universities um, around the country who would otherwise submit to the REF to, um, ensure that At that 5 percent of their output submissions contain, are these non-traditional outputs, are not publications and not books.

on traditional outputs in, in: 2014

We didn't think it'd be so difficult. to encourage, considering the vast amount of, uh, research out there, to ask institutes to double that, double that to 5 percent of their overall, that's 5 percent of about 200, 000 outputs submitted from all universities, to be something other than books. and publications.

Um, this we feel is in line with recognising the vast different ways in which people contribute to research and research culture. And by doing that, we strengthen our position, um, as a value to society by communicating the number of different ways that research results can be disseminated.

Ged: 00:17:13

Great. And, and what did people think were the ways that we can kind of move towards that aim of, uh, of, of a greater number of non-traditional outputs being submitted?

Gemma: 00:17:25

Well, I think the way to do that, and this was under discussion, our, our theory of change, if you want to call it that, Ged, is the fact that we know that institutions make, uh, calculated risk based decision making about what they're going to submit to the REF because the the consequences are so high and the cost is so high and the potential benefit is so high.

t Hidden REF will be in April: 2024

So we're hoping to run the Hidden REF every two to three years to make sure that we, we uh, Build this amount of sector confidence in the, in the transparent, fair, and perhaps competitive, uh, evaluation of non-traditional outputs, but at the same time teach the evaluators and feedback into that system that way.

nduct. together around the REF: 2028

So it's a combination of knowing when our universities are making decisions as well as giving them the confidence that these are going to be competitive outputs and they're not going to suffer the consequences of the fear associated with a non-traditional output being, um, being, being marked down.

Something I'd like to consider there from the research, we know that research evaluation is inherently conservative at times, especially when they know about something. If they feel confident in evaluating the object in front of them, they act conservatively. So this is why even though universities, you know, assess the vast majority of their submissions to the REF as three and four star, they only put their best forward.

When the REF outcomes come out, we see a pretty much an interesting distribution around. on the one through to four star, star stage, regardless of what the universities say. Um, however, when panels, uh, and research evaluation panels, and this is from my own research, as well as research in the peer reviewed literature about peer review, when researchers as panelists, uh, encounter something that they're unsure of, an ambiguous criteria, something they've never done before, Instead of acting conservatively, they act generously and they're actually far more likely to give these things higher scores because they don't want to be told they're wrong and no one's going to argue with a higher score.

So, um, which makes sense. And this is how groups work. Unfortunately, that kind of competitive advantage isn't being communicated. fed through to universities when they make the selections about what to put forward on their REF submissions. So we hope to clarify that by giving them more faith and security in the sector by providing more transparency on how to evaluate these non-traditional outcome outputs.

Ged: 00:20:41

I suppose in some ways the the act of doing the Hidden REF regularly would start to counterbalance that. Positive up correlation on, uh...

Gemma: 00:20:52

Well, we don't want anyone to get an unfair advantage. We want it to be fair. We just want it to reflect the vast way in which research influences and has output. So, I mean, I wouldn't...

It's funny right now. So, I do realise that by doing the Hidden REF, we may make it far more competitive and lower that score. But I don't think so. There is inherent value in these types of outputs. And I think it's far time we, um, we recognize them as well. Um, these outputs are excellent. They're not any less excellent because they're not published in a journal that no one can access.

In fact, they're probably even more excellent because they are more freely available, they are more accessible by people other than academics, and they do research culture more credit by communicating its value. So perhaps that's one way we can evaluate these things in the future.

Ged: 00:21:39

Okay. And, and in terms of, so I've asked about the next steps, but what specifically are the Hidden REF committee that you're part of?

You know, you've had a, I've given you a week to reflect on the festival. Um, so I want to, I want to really considered answer, obviously, uh, in terms of what your next steps are as a, as a group, do you, are you trying to grow the group? Are you Are you trying to diversify in terms of, uh, internationally maybe, or?

Gemma: 00:22:06

So we are all in therapy at the moment to recover from the Hidden REF Festival last week, um, but we're all in that together, so we're still a fantastic team. No, so the next, the next two priorities for the Hidden REF Festival is to run the Hidden REF 2024, which will be in April next year. So if anybody is listening and wants to put a submission forward, we welcome that.

, through, um, the Hidden REF: 2024

well, dictate is the wrong word, but we'll outline, but we'll outline all of the discussions and the, and the priorities and the ideas that were circulating around the Hidden REF Festival. There are amazing people there who wouldn't necessarily be central to the, to the, you know, mainstream REF decision making.

And so we will give them voice again, in line with our objectives to remain a grassroots organization. about how to evaluate these non-traditional outputs, both in the Hidden REF and then hopefully in the REF. Um, and that white paper will be authored by all of the people who attended the Hidden REF Festival last week, um, in Bristol.

And at the same time, I think that will mean that we'll get a whole lot of different people who want to take this further, both talking to their PVCs researchers, talking to their REF leads, talking to their VCs, if that's what it requires. To get them to sign the 5 percent manifesto and to, um, make sure that when submissions are going forward on the departmental level, they can say, what have you done?

. Uh, for the REF: 2028

Now it's people, culture and environment. Uh, and it's worth 25% of the overall. overall score, that puts it on par with an impact case study and that is going to be exceptionally important to consider all of these aspects that contribute to research culture, how we can, um, recognise them as well as mobilise them to make research culture within universities a nicer place to be.

Uh, research can be incredibly hostile, even in its best days, unfortunately, and we laugh about it and we develop dark sense of humour and, but, um, But that doesn't necessarily mean it should remain that way. So just because something's been the way that it's been done for 20 years doesn't mean it's the way that it should be in the future.

And we're hoping that Hidden REF can challenge this and contribute to that criterion as well by recognising a broader number of people, a broader number of roles, a broader number of aspects of research that are currently hidden by what we evaluate to be important and excellent in research culture.

Ged: 00:25:35

Brilliant. Um, just finally, as we kind of head towards the end of the, the interview, um, we've mentioned that the, the Hidden REF itself will happen next, um, calendar year. Um, so if people want to get involved either as a, I guess, on the review side or a submission side, how, how would they do that?

Gemma: 00:25:58

We'll get in contact with the organisation committee, um, the advisory committee.

I've forgotten what we call ourselves, Ged. It's been so long. It's just, now it's just, I know that everyone by name, I've got everyone's text messages. So, like, it's fine. Get involved, get in contact with us. Uh, go to the website, which is hiddenref- hidden-ref.org. Look us up, send us an email.

Um, get in contact and say, what can I do to help? Sign the 5 percent manifesto based both as an individual as well as actively, um, actively advocate within your institutions to get the institution to sign on to the 5 percent manifesto. Grab a badge, a 5 percent manifesto badge and, and push them and see what we can do.

We can, we can achieve this together and we are the most. Every single one of us the most important parts of research culture, so changing research culture is not about necessarily changing guidelines and policies, but how we work together as well as identifying what we value and pushing for what we value to be valued.

By those people who are not, um, everyday researchers. And when I say everyday researchers, that's a very broad range of people too, Ged. So people like you, for example, there's no way we could, we have a great research culture without people like you. But, um, this is what we're trying to do. So get in contact with us, um, have a go, uh, get people to five, uh, sign the 5 percent manifesto and get people in your institutions to sign the manifesto as well.

forgot, submit to Hidden REF: 2024

Ged 00:27:49

Fantastic. And we'll put all those, uh, links that you mentioned in the show notes so that people can, uh, can just click rather than try and rewind and write down what you, uh, what you wrote or what you said. Sorry. Um, Gemma. Many thanks for agreeing to be interviewed about the Hidden REF Festival. I know it was, um, it would have been nice for you to have another few days of relaxing from it, if you're actually managing to do that.

Um, so I'm sure life has been really hectic organising it, um, and I'm sure you really wanted to lie down. So thanks for, thanks for agreeing to be, um, to be interviewed. Um, so can I leave it to you to say goodbye to our listeners?

Gemma: 00:28:32

Sure. Thank you so much for listening. I'm actually chuffed to be asked and to talk to Ged here.

Thank you so much for listening and please do get in touch if you're interested in anything Hidden REF related or REF related. I'm here to help. Thank you for listening and goodbye.

Intro: 00:28:51

Thanks for listening to the Research Culture Uncovered podcast. Please subscribe so you never miss out on our brand new episodes. And if you're enjoying the discussions, give us some love by dropping a five star rating and written review as it helps other research culturists find us. And please share with a friend and show them how to subscribe. Thanks for listening, and here's to you and your research culture.