(Episode 123) Valuing Diverse Research Outputs: The Hidden REF and the 5% Manifesto with Simon Hettrick

Intro / outro [: 00:00:04

Welcome to the Research Culture Uncovered podcast, where in every episode we explore what is research culture and what should it be. You'll hear thoughts and opinions from a range of contributors to help you change research culture into what you want it to be.

Nick Sheppard [: 00:00:22

Hi, it's Nick, Open Research Advisor, based in the library here at the University of Leeds. For today's episode, I'm very happy to welcome Simon Hetrick, who is a professor at the University of Southampton. He's also director of Strategy at the Software Sustainability Institute and has led a campaign to gain recognition for research, software engineers, a new role in research, and he's now chair of the Hidden Ref, a national campaign that looks to recognize all research outputs and every role that makes research possible. So welcome to the podcast, Simon, and thank you for taking the time to talk to us. Hello. So, mainly today I wanted to talk to you about the Hidden REF and the 5% manifesto, I think, which is part of that as well. But I guess that initiative is bound up with some of your other work with the Software Sustainability Institute and the campaign to gain recognition for rses. Would that be fair to say?

Simon Hettrick [: 00:01:12

That's absolutely the case.

Nick Sheppard [: 00:01:14

So, yeah, perhaps we'll touch on that as well. But before. Sorry to interrupt beforehand when I realized you at Southampton, I just wondered if you. Are you a colleague of Les Carr?

Simon Hettrick [: 00:01:25

Yeah, no, I know Les Carr. So Les Carr was my line manager for a number of years, maybe eight, ten years, something like that. So he's somebody I've worked with very closely in the past.

Nick Sheppard [: 00:01:35

Well, I asked because. And he'll probably come around to eprints, so I think you'll be familiar obviously with the, the ePrints repository platform, which I think Les. It's a long time ago since I've seen Les, but he was a. One of the progenitors of that project, wasn't he, way back when?

Simon Hettrick [: 00:01:51

I think he was, yeah. Les Carris Gutteridge. I've met Chris as well, and I currently have an office next door to Justin Bradley, who currently leads all the development work on eprints.

Nick Sheppard [: 00:02:02

Right, right. So, yeah, we'll perhaps come back around to eprints, because in terms of different types of outputs, etc, it's something we're thinking about here, at least. We use eprints for the White Rose research online repository, as you, I'm sure you're aware. And I might actually do another sort of annex to this podcast, actually, with a colleague who looks after eprints, because we, after I spoke to you, we were talking about, you know, what the implications Might be for us. There's some of the hidden ref stuff, but we'll get onto that. So, yeah, just to come back to that question, that presumably it's all sort of bound up together, the different strands of your work, so perhaps you can just give us a bit of a background to the ssi, you know, its background and mission and how that relates to the hidden ref.

Simon Hettrick [: 00:02:45

So the Software Sustainability Institute, the SSI that we launched that in 2010, it was originally a research project funded by the EPSRC and basically it came about because the EPSRC, that's the engineering and Physical Sciences Research Council, they were concerned about the amount of software they were seeing in their proposals and the amount that researchers were talking about software, and they asked the question, who's helping researchers use good software? So there were various different groups that had a bit of a, sort of a side game on dealing with software, but there was no single institution for the UK that helped researchers use better software. That's where the software sustainability came up. We started in 2010. We're on our fourth phase of funding now running up to 2028, and we're now funded across UKRI. We're a national research facility. And the interesting thing about the SSI's job was we originally thought all we need to do is we'll go out there, we'll provide people who are software engineers with sort of good background knowledge with software, and that make the research world a better place. And obviously the big problem with that approach was there were eight of us and there were 240,000 researchers in the UK, so that's quite a big ask. So we quickly pivoted into looking for what can scale, what can we do to really change things.

Simon Hettrick [: 00:04:12

One of the things we did is we became the UK coordinators of Software Carpentry, which is a training course that gives researchers the right software engineering skills they need for research. We started building communities. We've got a brilliant fellowship with hundreds now, I think we're almost up to 300 fellows or maybe a little over, and they're all experts in different fields who use research software at universities and they go out and advocate and help us with our mission. And then one thing, the key thing for me that came out of it was in one of our collaborations workshops, which is a sort of event we run every year. In 2012, we came up with this concept of the research software engineer, and it was to address this problem where there were lots, there's lots of software being used in research. About 69% of all research is fundamentally reliant on software. But there wasn't a role for people who develop software in the research environment. So we looked into this and like, how is this happening? How is this software magically being written? And of course, as everybody knew at the time, it was being developed by people in postdoctoral positions.

Simon Hettrick [: 00:05:14

The problem with this was that people in postdoc positions, the success of their career is based on the number of publications they write and the amount of research funding which they're often not allowed to apply for, but they're spending their days writing software. So we'd locked people into this dead end position where they could no longer progress their careers because they were, they were writing software, not the publications and the other success metrics that a postdoctoral position is based on. We launched the Research Software Engineering campaign back in 2013 and it's been phenomenally successful. It's now a massive international community. There's many more than 10,000 RCs around the world that 14 different national associations. We've got our own society of RSE in the UK and it's embedded itself across UKRI and research funding policy. It's been a really important campaign.

Nick Sheppard [: 00:06:05

Well, I was just going to ask because RSE is research software engineers. We do have a team and built a developing team here at Leeds. I increasingly liaise with them because in my role, I guess I'm a research advisor, as I said, at the top, based in the library, but obviously we don't have the full overview of everything associated with open research. I'm a not coder. I don't, you know, I can't write code. I can't necessarily advise on licensing around code and that kind of thing. So that's one of the things I've been trying to sort of nail down from a strategic perspective here at the, at the University of Leeds around licensing for code. Maybe that's something you could give me some tips on.

Nick Sheppard [: 00:06:44

I'm not sure. It's something I've been working with my RSC colleagues on and I get the sense that that's maybe a bit of an issue across the sector. I know that it's been addressed hopefully by the UKRI policy, but there's not much there at the moment. Are you involved with that?

Simon Hettrick [: 00:06:58

Yes, I have been invited into a group which I haven't yet taken part in, so. So I'm not actually currently involved, but I will be in the future.

Nick Sheppard [: 00:07:08

I only asked because I, I was, because I've been working on this area myself. I was hopeful that there'd be something in there for me to maybe crib, but there isn't yet, so. But yeah, that's just an aside in terms of, you know, my sort of day to day practical concerns around this. But yeah, maybe just to take you back to how that work with the SSI relates to the hidden rare.

Simon Hettrick [: 00:07:27

So the work with rscs, the thing that was really interesting for me was the fact that this community grew like phenomenally quickly. And obviously we were pushing it and it was one of the things we wanted to happen at the ssi, but it had its own life and it grew very, very quickly. And the reason that happened was software is really important to research. The people who write software are really important to researchers. And so we were pushing it door. There was very, very quick adoption of this new role. And that's what really got me so worried at one stage back in 2018, where I thought, well, why did it take until the 2010s for us to realize this as a sector? And that led me to. It's about how we define what success means in research and how we attribute that success to different people who are involved.

Simon Hettrick [: 00:08:16

Software just wasn't a thing as far as research outputs are concerned. It was a means to get to a result and the result goes to a publication and you get your success based on that publication. What we needed to do if we really wanted to recognize people was to start caring about all of the outputs that are necessary to conduct research. Software is one of them, but there are many, many more. And that led me to look at the ref. And that's where I started to realize that maybe the way that the REF was being interpreted by universities wasn't quite right for the outcome we want. Which is better research.

Nick Sheppard [: 00:08:53

Yeah, because I mean, this is a conversation we've had quite a lot because strictly the REF has long, I think, recognized potentially other types of outputs. But in practice that's not really happened. Would that be fair to say?

Simon Hettrick [: 00:09:06

I think that would be pretty fair to say, yeah. So there are 21 different output categories and there have been since 2008, the research assessment exercise, which was a precursor to the reference. And I was really excited when I first came across this. I wasn't ref returnable at the time, so this was entirely new, brave new world for me. So I looked at the different app categories and I saw there under digital artifacts, category G was software. And this was like epoch defining for me as far as I was concerned. Because what it meant was I could look at all of the software that was being submitted to the ref. Obviously this was the software that universities were seeing as being good software.

Simon Hettrick [: 00:09:45

So I could tell a lot about how that being constructed. Would there be more software coming from universities with RSE groups or there's so much research that could be done. It was tremendously exciting. And then as the first stage, what I did was I plotted all of the outputs, all of those 21 different categories. I plotted them on a chart and I found that in the 2014 ref, which is the one I was looking at at the moment, at that time there were 186,000 outputs registered and in the software category there were 38. And then when you look at the 2021 data, around about the same number overall, around about 186,000 outputs and only 11 in software that time around. So how is this thing that 69% of all research relies on, how does it end up with only. I think it's 0.0002% of all the outputs submitted to the ref? There's obviously a problem there if we're not valuing all of the outputs that are fundamental to research.

Nick Sheppard [: 00:10:42

And that of course is software, which I guess is where you're coming from. But you mentioned there's a. What would even call them. I've heard them. There seems to be a lack of consensus there. I've heard them called non traditional research outputs, but perhaps that's not the best terminology either. I mean, what. What would you call them?

Simon Hettrick [: 00:11:00

What I'm calling them as changing. So. So, so it's a bit of a debate at the moment around what call these other outputs. So originally I was calling them non publication based outputs, but actually a lot of the outputs that you can submit to the REF but aren't are still publications, you know, everything from confidential reports to patent applications. So that name didn't work, so we went for non traditional outputs. And I never quite liked that name, but I was, I was in the committee that agreed to it, so I have to take some responsibility. And the problem was software is not traditionally submitted to the ref, but it's very much a traditional output as far as my community is concerned. So we've had a lot of pushback on that name.

Simon Hettrick [: 00:11:44

And then in the coming ref, in the 2029 one, they set up a new panel called the Research Diversity, Research Diversity Advisory Panel, rdap. And they didn't like the NTO name either. So they've gone for diverse outputs. But I don't like that name because it's not about being diverse, it's about representing what actually exists. So what We've done, I think what we've done is to be finalized, we're sticking with ntu, but we're calling them not traditionally submitted outputs, so we're keeping the same abbreviation but calling that. And that does work for me because they aren't traditionally submitted but they're very, very important and they are traditional outputs across lots of different fields.

Nick Sheppard [: 00:12:27

That's really interesting because in the course of my advocacy with academic colleagues around the University of Leeds, I often use the term now or encourage people to publish their outputs as first class research objects, which was a term I think I got from Hugh Shanahan. I don't know if you know Profess Shanahan from Royal Holloway professor of Open Science.

Simon Hettrick [: 00:12:45

Yes, yes, I've worked with him a.

Nick Sheppard [: 00:12:47

Lot and I think that was a term he used, you know, encouraging people to think throughout the research lifecycle, publish it with a persistent identifier, a doi, whatever, which again some of my colleagues I know do with software and code through GitHub and Zenodo etc. I don't know if you could talk us through that a little bit and how that should work and to what extent that is actually happening.

Simon Hettrick [: 00:13:07

So GitHub was launched in 2008 I believe and when we last looked at outputs that were being submitted, we quickly saw that it was really like becoming the place to store and it's a version control repository and it was becoming the place to store research software. So it's huge in research fish when you look at the outputs that are submitted, software outputs that are submitted to ResearchFish. GitHub is by far the main repository. We've worked with them a lot on code citation. So we have a citation file that they now have adopted so that you can understand how to cite code that's stored in GitHub. GitHub and you're absolutely right, they work directly with Zenodo. When you publish a release, you can snapshot it and get a doi, which gives you something that you can, you know, you can use to submit or write into publications or uses the representative like repository for that particular version of that software. So it's really, really handy way of storing your software and being able to snapshot it so that people understand exactly what was used in a publication.

Simon Hettrick [: 00:14:14

And it's certainly the advice that we, we give to people is to use GitHub.

Nick Sheppard [: 00:14:17

Because I wonder again, from my perspective, there's a bit of an infrastructure issue around this. So for example, here at Leeds we have the white rose research online eprints. We also have a separate data repository which is also eprints, but a separate instance of eprints and which is just for leads, not for the consortium of Sheffield, York and Leeds. So that's for data sets. We might link out to Zenodo or to GitHub or whatever. Different disciplines use different platforms, different repositories, et cetera. I mean, do you think there's an issue around that sort of disparate infrastructure and how that relates to different disciplines or that's just the way it is?

Simon Hettrick [: 00:14:54

I guess Disparate infrastructure and research seem to go hand in hand because you have a lot of. You rarely have an institutional policy on how to store NTO some non traditionally submitted outputs. So because they are quite new and they're. Although they're not new in themselves, the idea of storing them and using them as first class research outputs is quite new. So there's not been as much sort of agreement on the right way to do that yet. And universities have each taken their own tack and indeed disciplinary groups take a different tack and then other groups within universities do the same thing. So yeah, there's a lot of sort of disparate conversations going on about the right way to support these things. I think the issue of infrastructure has to be overcome so we do need to have a bit more sort of agreement on the right way to store different types of outputs.

Simon Hettrick [: 00:15:46

But I think that solution is going to be, at least in the software world will be related to a version control repository. GitHub is a useful one to use and it's a very popular one, a place to mint UIs. Zenodo I think is my preferred choice because it's very much a community supported resource and then things like eprints and to a certain extent a lot of organizations are now using pure. They are being opened up so that we can store data and other research outputs as well. Ultimately, if there's a DOI and it persists and that you can find that output for the long term, I think we can probably get around a lot of these infrastructure problems and it will just take some debate, some time and some coalescing of the community around which platforms to use.

Nick Sheppard [: 00:16:34

Yeah, I mean we've talked obviously about software and data. I mean what other examples of. What do you call them? Non traditionally submitted. Submitted.

Simon Hettrick [: 00:16:44

Submitted outputs. Yes.

Nick Sheppard [: 00:16:46

What might be some other examples of those?

Simon Hettrick [: 00:16:49

So it's very difficult to keep the list in your mind but I have it right here. I won't read my way through it but I mean there's a handful of different Publications, what we'd see as traditionally submitted outputs, everything from journal publications through to books. And then we have things like artifacts and exhibitions. Exhibitions are the second most most popular output category. After you get rid of all the publication, the traditional publication varieties, you've got things like confidential reports for external bodies, website content. I don't. And that one sort of mystifies me a little bit. I think it's maybe something that came about because it's a creation of its time.

Simon Hettrick [: 00:17:27

When these categories were first being talked about. The Internet was still quite a new and exciting thing. Maybe then digital and visual media translations, and there's a handful of other ones, but there's some interesting stories around them. And I find. I was recently talking to a professor in music and we were talking about the issue with getting recognition for the outputs that people who are working in musical research, how they should do it. And he was saying what they do currently is they. They write a composition and then they write a paper about that composition and then they submit the paper to the reference. And that just hit me as entirely bizarre and unnecessary step.

Simon Hettrick [: 00:18:06

If we're judging the composition and not the publication about the composition, then surely we should be submitting the composition itself. And the person I was talking to was very much of that opinion at the same time. And then you have things like. So again, data sets and databases, I mean, they can. To get those recognized by the ref. People are writing publications about them, but they. The data in the database itself is usually hugely more popular than the publication that comes out of it. So again, I don't see why we don't just focus on the data set and the database itself.

Simon Hettrick [: 00:18:43

I was talking to some people who work on producing documentaries and research documentaries and again, they were talking about. They have these wonderful documentaries that they submit to film festivals and get plaudits and awards for. But to get it into the ref again, they had to write a publication about it and then submit that publication. So I think we're adding a layer, an unnecessary abstraction layer to the whole process and a lot of unnecessary work. And really what we want to be doing is focusing on the outputs. And there's a second side to that. So it's not just the extra work it creates, but if we start caring about the outputs, then we will invest effort into those outputs to make sure that we're following best practices and when we are creating the outputs for the right kind of resources, rather than seeing them as some sort of necessary step towards getting a publication.

Nick Sheppard [: 00:19:32

Yeah, we've been having similar discussions here. With practice research, I think it's often called. So it's not something I've been directly involved with, but again, I've got a music, music musician, colleague who's very interested in this area and they've various projects looking at this type of thing and exactly what you're saying, you know, that it's mediated through the lens of traditional academia, I suppose to, to submit to the ref just on that website comment, that might be slightly tangential, but it's something that we do struggle with here, at least I'm sure elsewhere as well, is preservation of websites. Because people obviously, you know, they might have a website for a research project and they put loads of stuff on there. There might be reports, there might be data sets, they might be stuck because they're not coming to our repository, et cetera. And then of course that's not preserved in perpetuity. It might last five or 10 years and then suddenly it's kind of, well, what do we do this? We might have a database and that's difficult to preserve. So I don't know if that's, that's maybe part of that conversation around preserving websites.

Nick Sheppard [: 00:20:28

Is that something that you've been involved with?

Simon Hettrick [: 00:20:31

I've been involved in the conversations, but we're not directly aligned with preserving websites. I think the broader question is about sustainability of research outputs. And every field has its own problems. If you talk to some of the big data people, astronomers and the particle physicists and the bioinformaticians and those people, they're creating massive, massive data sets and they, they're simply too large to store everything. So then there's this whole question about, well, how do we, how do we boil that down, reduce it, abstract it, and then to something that we can conceivably store. And even then they've got massive problems with, you know, just the cost of storage.

Nick Sheppard [: 00:21:12

And then you've got the environmental impact.

Simon Hettrick [: 00:21:14

As well, I suppose, of storage, financial and environmental cost on all of these things. Absolutely. Then you've got things like website content and other like ephemeral sort of outputs that, you know that basically if you don't keep sustaining it, if you don't pay for that website to be hosted or whatever, then you lose the output. And we're obviously we must be all these questions, there's all these statements about how much written knowledge is created each year by our species and, you know, how it's just astronomically increasing and just think about how much we're losing constantly. That is a tremendously large issue for research and for just the society in general, I'm not sure what the solution is, but on the software side of things, we've been looking at that directly and sort of working out okay. So there has to be a set of decisions that are made about, you know, what needs to be stored, what needs to be maybe preserved and sort of stored somewhere, but not as directly accessible as some of it. And also, you know, maybe some things do need to be allowed to expire. But what doesn't happen with this sustainability argument is there usually isn't the resource to have the discussion about that and then to look at the different options and the benefits and disadvantages of each one.

Simon Hettrick [: 00:22:28

And it certainly isn't in our research culture we are very much a produce and then hope for the future. So we're focused on the production and not on the sustainability.

Nick Sheppard [: 00:22:39

Okay, We've had a quite wide ranging conversation around sort of related issues, but perhaps to bring it back to the hidden ref. And I think you mentioned the, the 5% manifesto. Can you tell us a bit more about that? And I guess the sector as a whole is moving to in the direction precipitated by the hidden ref, perhaps, but what's the 5% manifesto and what you're trying to do achieve with that?

Simon Hettrick [: 00:23:00

So the 5% manifesto is a. It's on our website, you can sign up to it. Hidden-ref.org is the website. Take a look. And basically what it is, it says that a university will say, will submit at least 5% of their outputs to the next ref in non traditionally submitted categories. Now, 5% is kind of, it's hardly the rallying call, is it? You know, what do we want? We want 5%. You know, it's really not particularly exciting and I think it's far, far too low for the importance of these outputs. But what 5% represents is it's almost just over double the number of non traditionally submitted outputs that were submitted to the last ref in 2021.

Simon Hettrick [: 00:23:40

That's 2.4%. So out of those 186,000 outputs, 2.4% represented absolutely everything else that exists in the research world that isn't a journal publication or a book. So the reason we chose that lower number was we thought, well, if we go for what we think it should be, where's that 50%, 60%, 70% more? I don't know, we'll never get a university to sign up. But 5%, a doubling of the average, that feels like something that we could persuade universities to do. You can sign up to it as an individual, so you can sign up to it, say, you know, I endorse this message. I believe that universities should submit more of these outputs and you can sign up to it as a university. Thus far we've only got one university signed up, that's Bath Spa. So a huge, huge round of applause for Bath Spa, as far as I'm concerned.

Simon Hettrick [: 00:24:30

But we are seeing other universities starting to talk about it and to work out whether they can implement the 5% manifesto themselves. I imagine by the time we get to ref 2029, we'll have a number of names on that list because right now a number of universities submit huge numbers of non traditionally submitted outputs, but they are still in the vast minority of universities overall. So I expect to see more names on it. And I think some of that will come about because of our lobbying and because it's obviously the right thing to do. We need to support and recognise all of the work that's being conducted in research.

Nick Sheppard [: 00:25:08

And did you mention, is there some sort of launch event or some event upcoming?

Simon Hettrick [: 00:25:12

Yes.

Nick Sheppard [: 00:25:12

So.

Simon Hettrick [: 00:25:13

Well, it came about in an odd way, the 5% manifesto. One of the main things we were going to be campaigning for was for the right for any role within research to be able to submit outputs to the next ref. And then Research England and the REF team just handed that on a plate to us back in 2023, I believe. So there was a change in the guidelines following the Future of Research Assessment Program that said, in the net ref, any staff member can submit their outputs. They don't have to be on academic contracts. Now this is a really important change. That means research software engineers and data stewards and research librarians and technicians and all these other overlooked roles can now get their work directly recognized through the ref. And that's obviously a good thing for research culture because that means universities will start to value them because there's a direct link between them, a good ref return and ultimately money.

Simon Hettrick [: 00:26:07

However, actually get universities to follow these guidelines is something different. Writing the guidelines is one thing, getting the universities to follow is another entirely. And that's not through any sort of bad action or lack of effort on the university side. It's just, it's a really significant change to the way that things have been done. Universities are slow moving and traditional organizations. So what we thought was, we'll put the 5% manifesto together and that'll be a way of having a rally, a campaigning vehicle that will, you know, can get people talking about, well, how many outputs are this? How many outputs will this university submit to the next ref and how many of those will be into the non traditionally submitted categories.

Nick Sheppard [: 00:26:50

Great, thanks. And just to say as well, when we publish this, I'll publish it with show notes as we call them. So I'll put some of the links into the website, a bit of information about this. So and I'll ask you, you know, you can send me any links that you do want added to that. Before I do let you go though I do, could I just ask you quickly about software licensing and where I can get sort of advice on that and policies around software licensing? Is that again, is that a challenge for other universities? Is it just.

Simon Hettrick [: 00:27:24

Oh yeah, absolutely. Because it's again, it's a new field so it's one of these things where no universities haven't set their agenda and stuff. I'm an ex patent attorney so I do have a bit of licensing background to me, but that's quite old now, 2007 I left that profession, but I have had an interest in some in discussions about software licensing. I'm not an expert, but I can talk about it.

Nick Sheppard [: 00:27:48

Okay, well thank you. I very well may take you up on that, but for now I'll let you go and thank you for your time. Nice to talk to you and speak again.

Outro [: 00:27:57

Thanks for listening to the Research Culture Uncovered podcast. Please subscribe so you never miss out on our brand new episodes. And if you're enjoying the discussion, give us some love by dropping a five star rating and written review as it helps other research culturists find us. And please share with a friend and show them how to subscribe. Thanks for listening and here's to you and your Research culture.