(S3 E2) Open research as an early career researcher and ReproducibiliTea with Kelly Lloyd

Intro: Welcome to the Research Culture Uncovered podcast, where in every episode we explore what is research culture and what should it be? You'll hear thoughts and opinions from a range of contributors to help you change research culture into what you want it to be.

Nick Sheppard: Hi, it's Nick, and for those who don't know me yet, I'm open research advisor based in the library here at the University of Leeds. You're joining us in season three of the Research Culture Uncovered podcast. We'll be speaking to colleagues from both the University of Leeds and from other universities and organizations about open research, what it is, how it's practiced in different disciplines, and how it relates to research culture.

s from the REDS Conference of: 2022

But now I'd like to introduce my guest for today. My colleague, Kelly Lloyd. Kelly is currently completing an ESRC funded PhD in the Leeds Institute of Health Sciences on the topic of investigating decision making in cancer preventative therapy. Kelly is also one of the organizers of the Reproducibilitea Journal Club here at Leeds which is how I first met you, I think?

Kelly Lloyd: Yes, it was

Nick Sheppard: Uh, so welcome to the podcast.

Kelly Lloyd: Thank you. Thank you for having me.

Nick Sheppard: And thank you for taking time out. I think you've been a bit busy lately?

Kelly Lloyd: Yes I have been a bit busy lately.

So the question is, have you finished a thesis yet? Yes. So I'm a final year PhD student and I'm happy to report that I submitted my PhD thesis last Thursday.

Nick Sheppard: Congratulations.

Kelly Lloyd: Thank you.

Nick Sheppard: So, um, what happens next? Not having a PhD myself. I'm not totally sure you get a...

And in many ways no . Cause [: 00:02:00

Nick Sheppard: Okay, well good luck with that.

Kelly Lloyd: Thank you.

Nick Sheppard: Um, are you nervous about the defense or,

Kelly Lloyd: yes. , but, um, I'm sure it'll be fine.

, you can explain that to us [: 00:03:00

Kelly Lloyd: Yes. So my origin story?

Nick Sheppard: Yeah, your origin story. Yeah. A superhero. You're casting yourself as a superhero!

ique it and discuss the book [: 00:04:00

I'd never heard open research or open science before, and this was like my first introduction to it, and I just thought it was so interesting. Have you read it?

Nick Sheppard: I've read parts. I haven't read the whole thing.

Kelly Lloyd: Oh, read it all. I loved it. And so yeah, from there I was really interested in open research. Just reading that book a learned a lot. So I will say about the book, it's called The Seven Deadly Sins of Psychology, but don't let the title put you off. I think it really can, a lot of it be applied to so many disciplines. It's just that Chris Chambers himself is from a psychology background, so it's definitely applicable to a lot of disciplines and it was from there that I got interested and then I got asked to help set up Leeds Reproducibilitea.

k and of Chris Chambers, but [: 00:05:00

Kelly Lloyd: I think the reason is so...Open science. A lot of the discussions came, at least the movement, how it sort of is at the moment. A lot of that came from psychology and I think a big part of that was just because a lot of psychology studies, really landmark study that really important in the discipline. People tried to replicate them and they couldn't be replicated. So it became this big thing that psychology can't be replicated and what are the issues? And that sort of bred into the discussions of open science, but., I mean, it's not just psychology. I think there is a perception, it's just psychology. It's not, and there are other disciplines that, science disciplines, I guess, cuz it's been replicated. So there's other scientific disciplines [00:06:00] that have tried to replicate landmark studies and also haven't been able to do it. So in cancer biology, there was a big replication project for cancer biology and they tried to replicate a bunch of landmark cancer biology studies, and they found firstly, it was really hard because there wasn't enough detail in the methods. Um, like there wasn't openly shared protocols. There's a lot of detail missing. A lot of data wasn't shared, and when they could replicate it more often, not, they did not find the same results. So it really is applicable to a lot of disciplines. I just think it needs to be sort of communicated from the key people in those disciplines. And it's just that the moment psychologists have been the ones mostly talking about it, so I think, researchers from other disciplines, but they think, oh, that's not to do with me because there's a psychologist talking about it.

it's important in all sorts [: 00:07:00

Kelly Lloyd: the treatment works.

Nick Sheppard: The treatment is based on studies that can be replicated.

Kelly Lloyd: It is much more scandalous in medical research when these things can't be replicated or conflicts of interest have resulted in, maybe selective reporting and, yeah.

Nick Sheppard: Well, we'll come touch onto some of those specific examples, but, you know, you've talked about reproducibility already, you know, and the, and the concept of replicating research results. But I mean, how would you define open research in your fields. I guess you've got more than one field with your

the whole cycle of research [: 00:08:00

Nick Sheppard: Well, yeah, but just, just on that, um, cuz we did actually work with you on a data set, didn't you? Yes. And, and you, in that case, as I recall, you didn't share all the transcripts, is that right?

Kelly Lloyd: So I, I did [: 00:09:00

Nick Sheppard: Even though you had permission, so they'd given, given that consent, but you would still an ethical consideration there I guess?

I phrased it cuz I said the [: 00:10:00

Nick Sheppard: Yeah, no, I'm just interested because, you know, we've, we've talked about this, it's, it is an issue I think, especially in qualitative research. You know, making sure that you're, the ethics is water tight and you actually sharing what you can, et cetera.

Kelly Lloyd: I wonder if I rejected so much data and identifiable information as I mentioned earlier, did I make my interview transcripts that I shared essentially kind of useless? Because there's just so much important data I took out potentially.

Nick Sheppard: So that's data sharing as part of open research or open science. Um, you mentioned methodology and the fact that that's not always sufficient. Is that an

Kelly Lloyd: yes

Nick Sheppard: is that a big issue in

quite a big issue in lots of [: 00:11:00

Nick Sheppard: Well, that puts me in mind, uh, of a recent conversation I had with Dr. Alex Freeman who you may know is uh, she's the inventor of a platform called Octopus, a new open research platform, and she talks a lot about narrative in research and how important narrative is, to the detriment of science sometimes, you know, you're trying to tell a story. Yeah. And that that's sort of the same sort of thing. It's like, well, okay, it might be boring, but this is crucial to actually, to science, you know, actually.

Kelly Lloyd: Yeah.

that part of the problem we [: 00:13:00

Kelly Lloyd: No. So, and it's just, as you mentioned, it's just not, it's a culture thing. It's not a priority. And this culture is also when you're getting papers like that, that say, oh, Don, Spend too much time on the methods, it doesn't matter. So that is the culture, and it is, as you mentioned, the. Important part. I dunno where I was going with that next point, but just that it is such an important part that we culturally, we don't teach people to spend enough time on describing.

somewhere. So there is more [: 00:14:00

And I don't wanna put you too much on the spot. Yeah. But I remember you talking, uh, I think recently about your, the, the challenges for yourself actually fully documenting that methodology. It's not easy, easy, I don't think to actually, yeah. I dunno. If you come back to your own methodology and how is it sufficient?

I realized I never specified [: 00:15:00

Nick Sheppard: So, I mean, this perhaps brings us around quite nicely to, well, you've mentioned two terms, which, uh, as I say, I've learned myself from you, but I still get a bit confused with registered reports versus pre-registration. Maybe you can give us a bit of a crash course in what pre-registration means and how that relates to registered reports. They're kinda the same thing. Are they?

tion in general, the general [: 00:16:00

Um, and also I mentioned that you put it onto a public repository, so it is a public repository, it's timestamped. I know that some people get quite worried about scooping, which is where you put your pre-registration online and people think that someone will then [00:18:00] do the study before you do and publish it, which I think is really unlikely, but if you are worried about that, you can embargo the pre-registration. So when the embargo lifts and when you've published the study, there's still the timestamp there when you originally timestamped it, but no one else could see it until the embargo was lifted.

and then the peer reviewers [: 00:19:00

Um, so at vital time you can make those changes. And another thing is the fact that um, if it's accepted, then you get in principle acceptance from the journal, where the journal has committed to publishing the full study, as long as you either adhere to the protocol or if there are any deviations, these are transparent and justified. And I think also mostly, in all cases that you should also get permission from the editors and peer reviewers to make those changes, rather than saying at the end, "we changed loads, so it's transparent" because they, you need [00:20:00] permission for it and like a justification. So for my study, I submitted it to a journal. They gave in principal acceptance, and then I went off and did the study. I recruited participants, I analyzed the data, everything I said I was gonna do worked, so I didn't make any changes. So I basically followed up my plan, like a recipe. I did it quite quickly actually. Once I had the data, I analyzed it quite quickly and I wrote up the rest of the paper. So the introduction methods couldn't change, apart from tense changes where I said, you know, we will do this, I changed it to, we did this. It was the only thing I could change. And then I basically wrote the rest of the paper. So I wrote the results and the discussion, um, and I gave it back to the journal and I said, look, I've done what I said I was gonna do that you already approved. And they went, yep, that's great. And I actually didn't even, you can get more feedback at this point. So there is stage two peer review and they can come back with more queries. I actually found that mine were happy. [00:21:00] Yeah. And so done.

Nick Sheppard: I mean that, I was gonna say, when you talking about pre registration, that sounds like quite a lot of work. Well it's effectively stuff that you'd write anyway. Just a different point in the process maybe?

and we are ready to go, once [: 00:22:00

Nick Sheppard: Yeah. That was something else that, [00:23:00] um, Alice Freeman talked about, in the context of Octopus and the fact that there's no...because everything's...again, culturally, we're so driven by positive, I say positive results by results that are finding an effect when actually science doesn't work like that often. No, you could, you could, you know, and that's perfectly valid to find.

Kelly Lloyd: I think it was a, oh, what, I'm gonna quote it badly, but I think there was a study where, I can't remember the exact statistics, but it was where they found that between five to 10% of study outcomes or publications in traditional publication format published null findings, but with the registered report format, it was much more around 60, 70% maybe are publishing null findings.

Nick Sheppard: Wow. That's a dramatic difference. Just so I'm clear then, so obviously you talk about pre registration and registered reports. Would a registered report be public also, or that's just for the journal article?

ly good point. Um, so when I [: 00:24:00

Nick Sheppard: In exactly the same way as...

Kelly Lloyd: Yeah, the steps are really similar, it's just that actually registered report, I had guaranteed publication and that was, yeah...any researchers out there will know that when you try and publish a study, sometimes it can take years. If one journal hangs onto it for six months and then says no, then it goes to another journal who hang onto it for six months and they say no, that's a year gone by. So for actually for my, this study, um, my final one that was a registered report, I got, um, the study up and running in March this year, and then the full study was accepted for publication in September. So it was really quick overall.

Nick Sheppard: And, but, and again, on registered reports, so you mentioned that not all journals do this though do they?

Kelly Lloyd: No. [: 00:25:00

Nick Sheppard: Is it quite few and far between? Is it, does it vary by discipline as well?

Kelly Lloyd: How many is it now? I think maybe 300 journals offer it in some formats, either they always offer it or one off specials and might be slightly up, um, offer that updates quite a lot, but the vast, vast majority are in psychology. So this was one of the few where I submitted is, uh, the British Journal of General Practice was one of the few journals that offers in Medicine, a registered report format. There weren't many others. Trying to think of what the others are called, but might be one or two others. Or maybe more than that, but for me, I felt like there wasn't many options, so if this journal said no, there's maybe one more journal I could try, and that was kind of it. So it's a bit limiting. When you work in medical and health research.

fact that, you know, there's [: 00:26:00

Kelly Lloyd: Mm-hmm. I think it's getting there very, very slowly. Some people argue it's so slow, it's not progressing enough. Um. Well the journal I mentioned, the British Journal General Practice only introduced registered reports in the last couple years. Um, I think the incentives are complicated. I think there is incentives from the journals that they...there are misunderstands of what registered reports are. So some people think it's a very positive, uh, positivist framework. Um, it doesn't account for exploratory research, but there are different formats and it can count for all types of research. So there's misunderstandings of what it is. I think there's an assumption there'll be a lot of work. And I do also know, I swear someone, some journals have mentioned that even when they did have it, people don't submit to it, because I [00:27:00] think there's also a cultural issue that there's only so many researchers who know about it and then so many researchers who are gonna submit their study like that. So it's a mixture.

Nick Sheppard: Can I just ask, um, with, I suppose there's one thing that I'm really not clear on as well, and not, not a concept we haven't discussed today, but I was talking with, uh, a colleague, Dorka, I think you might know my colleague Dorka, who was doing some of the case studies is Preprints so Preprinting. How does that interact with pre-registration or registered reports in particular? Can you still preprint or...

and that is fine as long as [: 00:28:00

Nick Sheppard: Yeah, I suppose I'm just trying to figure out the timeline, because if, if you, I mean, might you pre-register and then preprint and then a registered report and then the formal journal? I mean, how would, how would the timeline work I suppose?

done it at the stage where I [: 00:29:00

Nick Sheppard: I mean, there's a, yeah, I'm just thinking out loud. So, you know, and again, it's always interesting for me not being in academic, not being a researcher, just trying to figure out...

Kelly Lloyd: is more interesting because preprint is before peer review, but then registered reports have got peer review at the stage one stage so how's it work? It is more complicated when you think about it.

Nick Sheppard: Yeah, yeah, yeah. No, it just, uh, just occurred to me, but, uh, no, that's great. Thank you. I mean, we've already touched on some, I think, of the barriers to open research, but do you think there are any other particular barriers? I mean,there's the cultural aspect's perhaps that's the overriding barrier are there at the moment or...

isciplines. But I think it's [: 00:30:00

Nick Sheppard: Well, that, well that's the point isn't, I mean, why isn't it, I mean obviously people like yourself are very, um, passionate and enthusiastic about open science and for the good of science I think, but do we need to do more to incentivize it? I mean, people are [00:32:00] people. As a psychologist, you know...what, why, why should I, as a researcher, I've got to pay the bills, I've got to publish. It's not something that's maybe incentivized enough at the moment?

ts harder and the skills and [: 00:33:00

Nick Sheppard: Yeah. Well that, that's the subject of another podcast I'm hoping to do with a colleague, um, you know, Dr. Maddie Pownall.

Kelly Lloyd: Yes. Um, so really that's gonna be really interesting.

Nick Sheppard: So, yeah, we're yet to speak to Maddie, but, um, very interested in some of the work that she's doing about...

Kelly Lloyd: she's doing amazing work,

Nick Sheppard: trying to embed some of this stuff in the undergraduate curriculum, because they're of course the researchers of tomorrow.

Kelly Lloyd: Yes, a hundred percent, but then it comes to a complicated thing of...Maddie's doing amazing work in psychology, but we need to push it through in other disciplines, but who's gonna do in humanities, because you need each person who understands the language of that discipline to train in it because there's no point in me coming into humanities and talking about it cause I don't understand humanities.

alking to another colleague, [: 00:34:00

Kelly Lloyd: Yeah. That's never what you want. It needs to be, in an ideal world, and this is so much harder to actually do, it needs to be coming from the people like the leaders and the early career researchers in those disciplines who are gonna reimagine what open research is for their own disciplines in an ideal world. But then it's that culture as well...someone in humanities might feel like it's just STEM coming in trying to say how we, how to do our research and that's not pleasant. And those can be a lot of barriers. The culture barriers to open research is how people perceive it. So I don't know how much you've seen on Twitter, but there's also a big barrier that people don't like, well, tends to be more open science because there can be a bullying culture there.

Nick Sheppard: Yeah, yeah, yeah,

Kelly Lloyd: with a lot of

Nick Sheppard: so called "bropen" science.

e podcast to read the bropen [: 00:35:00

Yeah. I'll put the, I'll put the links in the show notes.

Kelly Lloyd: Yes, really, really good paper, um, I think it's blog post, but it's really, really good. And that's a really big issue, so I've seen it where, um,

Nick Sheppard: gatekeepers

Kelly Lloyd: gatekeepers,

Nick Sheppard: there's academic gatekeepers or now there's, um, and it can be gendered and...

cience because I don't wanna [: 00:36:00

Nick Sheppard: Can I ask if you've experienced that..?

Kelly Lloyd: No, I haven't actually...

Nick Sheppard: Or have you any colleagues that have experience?

Kelly Lloyd: I haven't experienced it. I'm not sure if colleagues have, but it is kind of my worst nightmare to put something out there and everyone on Twitter just rip into it and I think I'd cry. So when it happens to other people, I really feel for them, um, and I'm really scared of it happening to me. Which also means that you won't want to share your...I do want to share my data and everything, but the more you share, the more you open yourself up to criticism and that's really scary.

Nick Sheppard: Yeah um...

and whether Twitter is the best medium, especially now with...is another question perhaps

ectives of open science, um, [: 00:37:00

Nick Sheppard: and that cuts across culture as well, doesn't it? Yes. Not just about. Yes, it's culture. It's about the culture of the academy and, and, and, um, gender disparity, pay and all sorts of other things.

Kelly Lloyd: Yes science and research, it's just on the existing academic structure, which are already favouring, like, Western, white men and everything, it's already built on those existing power imbalances.

promote the adoption of open [: 00:38:00

Kelly Lloyd: So ReproducibiliTea is a journal club that originally started at Oxford, University of Oxford, by early career researchers who were quite interested in open science and they really felt that they needed this sort of community to discuss papers in this area. And it's expanded since then. So it's gone to, I think, I mean this is probably outdated statistics, but it's spread to over 140 institutes in over 27 different countries, and we have one in Leeds. So it started in 2018. I was one of the first groups as well that started it here and we basically, we meet...we have breaks, we're on a break at the moment cause I needed to finish my PhD, but when we're [00:39:00] up and running, we have a meeting every month, virtually, where we sort of, we assign a topic or a paper, but you don't have to read the paper. If you wanna read the paper, great, but we will summarize the paper as well. And sometimes we have guest speakers on an open research topic and then we have room for discussion. So we've had previous sessions on qualitative pre-registration and we had a guest speaker. And we've had them on preprints, the controversies in preprints. So there was a lot of controversies with, um, Covid and people putting preprints online that apparently people said they weren't very good studies, they wouldn't have passed peer review. It's hard to know because a lot of bad studies get through peer review, but that's kind of the discussion. Um. The origins of open science. Um, we've done quantitative data sharing. We've done stuff on more quantitative stuff like R programming and so many different things. We've tried to make it so it's inclusive to loads of different disciplines, um, and try to do different discipline [00:40:00] specific things as well. We had a geography session and we had a open science researcher from geography. I don't know anything about geography, but that's really interesting and try and have things over different disciplines and have multidisciplinary discussions and you know, as I said, it's informal. Maybe 20 minutes, 30 minutes, uh, presentations, and the rest is just informal discussions, questions in the chat, put your hand up or we can all just have a chat and see how we feel, what our opinions are. So you can learn a lot from it really, I think.

Nick Sheppard: Yeah. Well, I would, I would certainly vouch for that. I mean, I've been along, and I've, you know, a lot of what I've learned in my role is from the ReproducibiliTea sessions. Um, and you just encourage people to, to come along once...they'll see them advertised won't they on Twitter and...teams

Kelly Lloyd: and there's no, you don't have to...

an sort of have a nice time. [: 00:41:00

Nick Sheppard: Yeah, I'll put put contact details, maybe Twitter as well if you want, put links and

Kelly Lloyd: Yeah, we're always looking for volunteers, early career researchers. I think it's a great thing for your CV as well because open research is becoming more and more important, so if you wanna help out running sessions, please let me know.

Nick Sheppard: And we just emphasise, don't we, I think, the value of community, I mean, as I say, I met you through ReproducibiliTea. I've learned an awful lot from you on open research, maybe even you from the library as well?

Kelly Lloyd: Oh, I have. Yeah. 100 %

Nick Sheppard: You know, and that's the collaborative nature of open research

Kelly Lloyd: Yeah there were lots of things I didn't know...the licensing, like Creative Commons licensing, no idea about that. Really important. Data sharing. There's a lot I didn't know that I learned from the library.

Nick Sheppard: [: 00:42:00

Kelly Lloyd: Yeah. It is a great way to start with the community. I mean, there's Open Lunch as well, your sessions, which are really good. I guess ReproducibiliTea is quite a small and more informal discussion that'd be quite good if you want to really get to know people.

Yeah, yeah, and might we have to do it in person again? I mean, you said it's virtual because before the pandemic we...

I should probably set up a hybrid one. It is on my to-do list. I'm just not very good with technology. Mm-hmm. , so this could go very horribly wrong.

Nick Sheppard: And we do have a teapot of course, we've still got the teapot. It's just sat on the cupboard from...

Kelly Lloyd: I know I need to set the hybrid one, but I'm terrified of the tech side, of trying to do both.

le to join ReproducibiliTea. [: 00:43:00

Kelly Lloyd: thank you.

Nick Sheppard: ...just before I let you go, what's next for you now that you have finished your phD and, uh, and assuming you get a good viva and I've no doubt that you will, what's next? Are you staying at Leeds?

Kelly Lloyd: Yes, I have, um, a job interview on Thursday, so for a job in my department and I'm also writing an ESRC fellowship application for one year of funding they provide to, um, people who just finished their PhD, so hopefully if it all goes well at the interview, I'll still be around next year.

Nick Sheppard: Okay, and, uh, continue to work with you on open science and open research. So yeah, thank you very much and

Kelly Lloyd: thank you for having me.

Nick Sheppard: Thank you, see you again. Bye-bye.

Outro: Thanks for listening to the Research Culture Uncovered podcast.

pping a five star rating and [: 00:44:00

Thanks for listening, and here's to you and your research culture.