(S3 E3) Open research in the curriculum with Dr Madeleine Pownall

[: 00:00:00

[00:00:24] Nick: Hi, it's Nick, and for those who don't know me yet, I'm open research advisor based in the library here at the University of Leeds. You're joining us in season three of the Research Culture Uncovered Podcast where I'll be speaking to colleagues from both the University of Leeds and from other universities and organizations about open research, what it is, how it's practiced in different disciplines, and how it relates to research culture.

s from the REDS Conference of: 2022

But now I'd like to introduce my guest for today, Dr. Madeline Pownall, a lecturer here at Leeds in the School of Psychology. Dr. Pownall completed her PhD this year in 2022 in Social Psychology. She's a Leeds Institute for Teaching Excellence Fellow, that's LITE for short and has contributed extensively to the Framework for Open and Reproducible Research Training, or FORRT for short.

So welcome to the podcast, Maddy. Is it okay to call you Maddy?

[: 00:01:28

[00:01:30] Nick: And, and thank you for taking the time to talk to us.

[: 00:01:32

[00:01:34] Nick: Um, so perhaps we'll get onto your work with LITE and, and the FORRT communities. Uh, but firstly, congratulations on being HE Psychology teacher of the year 2022 which I spotted on on Twitter.

[: 00:01:45

[00:01:54] Nick: No, yeah. Congratulations. And I suppose, um, that is related to teaching, which we'll come on to...we want to talk to you about teaching and, you know, open research skills in the curriculum and that kind of thing. But perhaps to start with perhaps a little bit on, on your academic background, as I say, um, you are a lecturer here in the School of Psychology, but you only very quite recently finished your PhD?

[: 00:02:13

So it meant that when I first joined I did a lot of undergraduate teaching, mainly research skills, statistics, things like that. And my PhD was in, um, kind of like hardline social psychology, so sort of experimental social psychology, and then gradually more qualitative stuff. Um, but the, one of the main theories that I was using in my PhD was called Stereotype Threat Theory, which, um, kind of unbeknownst to me, or I quickly learned very quickly became the kind of like poster child for psychology's replication crisis.

So I started this PhD in: 2018

[00:03:39] Nick: Yeah, well certainly we've been working on that, uh, you and I a little bit and I'm really interested to talk about that in a bit more detail, but I suppose first to sort of address the, the whole psychology thing, I, I spoke to our colleague, uh, Kelly Lloyd recently, who also has a background in psychology, as I said to her, um, you know, it's come from psychology has it open research in, in some ways or a big part of it because of that replication issue?

[: 00:04:02

[00:05:42] Nick: I was just gonna say, I mean, I don't wanna put you, you know, on the spot for other disciplines perhaps, you know, ask you to talk to those disciplines, but is there perhaps a danger, do you think of people saying, you know, this is a, is a psychology problem, whereas actually there's evidence emerging that it does affect a much wider range of disciplines than, than just psychology?

[: 00:06:00

And I think that kind of set of questions you can ask for any discipline or any subdiscipline. So I've given like open research talks in like applied linguistics conferences and like physics conferences. And when I talk about things like what does robust research mean, that question kind of chimes with everyone, I think.

[: 00:07:27

[00:08:13] Maddi: Exactly.

[: 00:08:15

[00:08:57] Maddi: Yeah. So that, that whole paper kind of came out from...so all the authors on that paper are kind of self-defined early career feminist researchers. And what I was noticing is I was having a lot of coffees with a lot of my kind of PhD friends who all were really sympathetic to the values and kind of mission of open research or open science. They're like, yes, research should be transparent, it should be robust, it should be all of these things. We get it, we're on board. But what was...and there was kind of this kind of...what I was noticing is the sort of party line, if you like, of, oh, we think research should be robust and transparent that loads of people were subscribing to and endorsing. But then what was happening more kind of on the ground, particularly for early career people, and particularly for early career researchers who occupy research bases that are quite kind of contentious or marginalized, is that people were saying, yes, it's all well and good that this is good for research, but what does this mean for me as a researcher and I started having these conversations. Then we kind of brought together this dream team of, I think there's 10 of us on the paper, um, who were all kind of having these individual conversations around really trying to stress the importance that...of kind of bridging together the stuff that's good for research and what that actually means for the people doing the research.

Because if you have, um, all these efforts to, for example, like, make data more transparent and make things more open, which means that by definition it's more easy to find mistakes and it's more easy to kind of, um, question claims, then that's all well and good, but then you need to make sure that you've got a culture that can respond to things like questioning claims and finding mistakes in a way that's compassionate and in a way that's fair and kind of collegiate and what I was kind of noticing, from the more people I spoke to is that actually there wasn't, or there was a perception particularly in certain disciplines within, for example, like certain parts of social psychology or, um, psychology more broadly that people were saying, you know, I really, I really believe that data should be made open, but what then happens if I'm kind of out, what happens if I find a mistake? And there were a couple of notable examples of early career feminist researchers who, um, particularly on things like platforms like Twitter, were really experiencing like proper hostility, um, and aggression in the name of like open science. So we kind of wrote that paper to try and centre those experiences of basically that you're just trying to kind of remind the field, um, that when we're talking about research, it's not done in a vacuum like it's done by humans and the need to kind of humanise and make compassionate our, um, research culture above and beyond caring about the, the actual integrity of the research itself.

[: 00:11:56

[00:12:19] Maddi: yeah.

[: 00:12:20

[00:12:23] Maddi: Yeah. I think it's really important though that it's not a gender thing. This isn't a kind of like men versus women thing. Like I've, I'm, and I guess that my whole kind of feminist approach, um, makes that quite clear that I think that there's way more that kind of a feminist lens can offer above and beyond looking specifically at like gendered experiences.

I think for me it's about power, and it's about voice and about inequalities. Um, and Kirsty Whittaker and Olivia Guest make it really clear in their...um, where they first coined the term bropen science in a, um, article and The Psychologist that they're like, not all bros are men and not all men are bros or something. Um, and I think that that feels quite important because it is a little bit kind of tongue in cheek colloquial, this is the thing, but I think what has been really powerful is it has given, I guess, people like me and people like my early career colleagues and not early career colleagues, um, a kind of set of, or I guess a language to articulate that kind of feeling of, I don't really know if this culture is safe for me to be really...

[: 00:13:28

[00:13:29] Maddi: Of being exposed and being...so whenever I talk about like bropen science, I always have this slide that kind of has a scale and on, um, one axis is like researcher vulnerability and on the other there's research transparency and I think they are almost a direct correlation with, yes, it's great to be as transparent as we can be, but at least we need to have a bit of a conversation about what that means in terms of vulnerability of researchers, particularly researchers who might not have, kind of, good support that they can rely on or who might, um, occupy researcher topics that are, um...or that lend themselves more to kind of hostility and politics. You know?

[: 00:14:12

[00:14:50] Maddi: Yeah, so, so I guess I kind of went through my PhD and was interested in open research....I was kind of forced to be interested in open research because, um, nothing was replicating and then kind of turned my attention to...okay, well all of the stuff that I've been learning about open research has kind of come like on the grapevine and has been, you know, oh, I happened to be at events and happened to learn about this stuff. And now one of the things that I just think is really, really important that open research and these kind of conversations are integrated into training and like kinda undergraduate grassroots training. So FORRT is the Framework for Open and Reproduce Research Training. Um, and it's basically a community was started by, um, PhD students and now it's kind of grown and grown and grown.

It's basically a community of people all around the world who I guess believe that open and reproducible research should be integrated into, into training. And that includes everything from kind of, um, well everything actually from like kind of high school science all the way up to, um, sort of PhD training and beyond. Um, so we do things like...uh, there's different projects that are happening, so I led a project in FORRT that was all about articulating the impact of open research on student outcomes. Um, there's a whole load of work being done in FORRT around like EDI issues and neurodiversity, that we run systematic reviews, we write papers, we do studies, like it's a really kind of, um, productive community as well as being a space where people can chat. Um, so I think getting involved with FORRT was one of the best things that I ever did, um, because it is, it is a proper kind of big team science initiative, if you like. So we actually produce things as a, as a collective, which I think is quite powerful.

[: 00:16:45

[00:17:18] Maddi: Yeah, sure. So that whole project, so this came from...so most of the projects that happen with FORRT come from some kind of conversation on our Slack Channel and then someone says, right, let's do it, this will be our next thing. And a few people were saying in the community, that they really get the kind of moral case for why we should integrate open research into the curriculum or into training or whatever.

Um, so like, oh, it's, it's, you know, it's good for science and we need science to be transparent. All of this kind of stuff. Um, but what they were, or what some people were struggling with, is how to articulate the, kind of, impact on students. Um, to, to things like their, for example, university department. So what we wanted to create is basically a kind of synthesis of the evidence that looks at whether integrating open research actually does anything in terms of, um, student impact. So does it actually impact student outcomes in any way? Or is it more of a kind of, well, it's good for research, so it should be integrated into the curriculum. So we ran, um, a really big, I think there were 75 of us on the eventual paper, a really big systematic review that looked across those three different domains. So attitudes towards science. So does it impact how students trust science? Whether they think it's credible, whether they wanna have a career in science, does it impact their engagement? So do they like their studies more? Are they more motivated, are they more interested, um, and does it impact, um, statistical literacy? So are they able to do stats better if it's taught in an open research way? Um, and one of the big things from that review that we highlighted is the lack of good evidence. So now we are kind of working more on, I guess trying to like fill that gap in the evidence, um, and to really advocate for like good, empirical, robust investigations about what does integrating open research actually do for student outcomes?

[: 00:19:15

[00:19:49] Maddi: What, for researchers or for students?

[: 00:19:51

[00:20:26] Maddi: yeah, well I guess that...cause I have a, a very complicated thoughts and feelings about things like incentives, because, so I've, I've kind of over the past few years have made the switch from doing like kind of experimental, quantitative, statistical type psychology to doing now, like the 99% of the stuff that I do is, uh, qualitative or participatory and that kind of stuff. And actually what that's meant is that there's loads of open research practices that all of a sudden are just not compatible with the kind of research that I do.

And not in a way that, oh, it's because I need to think about it more, or because I need to frontload the work or need more time or more training, just purely because it's just not compatible. It's not, doesn't, it doesn't make sense to, for example, pre-register some of my qualitative work because it's really exploratory and it doesn't add anything. And so I've kind of come from, um, a place a few years ago where I was really advocating for like, there needs to be the kind of really formal top-down incentives for like engagement and open research to now thinking, Ooh, well actually there's some really legitimate reasons why people can kind of happily opt out of some open research practices, and I think the thing that we, the, like, the royal we, haven't yet figured out is how to differentiate people who are thinking about open research and have kind of made the, I dunno, epistological, methodological, ontological, whatever, decision that that open research practice isn't appropriate for them versus the people who don't engage with open research because they either kind of need more training or cause they haven't heard of it or because they don't want to, so, um, yeah, I think that that's...that feels like where we, the collective we, are kind of at with incentives. Um, cos it is like, there's a, there's a lot...and I've, I've written about this quite a lot recently, but there are a lot of open research practices that just don't make sense for some qualitative research. And some of my colleagues say this and they're like, oh yeah, but it, it kind of does make sense because it's flexible, but it doesn't, for me, most open research practices should be aiming to, um, kind of reduce questionable research practices, and if the questionable research practice isn't relevant for your method, then what is the point of engaging with that tool? Does that make sense?

[: 00:22:51

[00:23:54] Maddi: Exactly. And I think that there's a danger in the conversation around open research and qualitative research or creative research or participatory research that is, kind of, seems to be centering a bit around, oh, well these are all the ways that you can participate if you just think about it enough or if you build it into your ethics or things like that. And actually, what I would really welcome is a bit more of a kind nuanced, as you say, um, appreciation that yes, there's, there's all of these open research practices in a kind of buffet and actually not all of them will be appropriate and not all of them will be relevant. And that kind of goes beyond, oh, it's because you need time and training and more because you can make an informed decision about actually that that tool or practice isn't appropriate or isn't relevant.

Um, and yeah, so I think that that kind of move to...more like legitimising, opting out, which sounds counterintuitive., but I think it feels important in order to be, um, really inclusive, you know?

[: 00:25:02

[00:25:23] Maddi: yeah, that's true.

[: 00:25:23

[00:25:30] Maddi: Yeah, and it's like, we talk in the paper about a term, I, I don't even, we actually use this term, but about "slowpen" science, which I really, really like. And I think that this whole move to slow down research, um, feels really important. Like that feels like a really good progress. And I guess that kind of also by definition, um, like truly participatory research or citizen science or qualitative research is inherently slower. Like it's kind of, it's inherently more thoughtful and slower and your data's richer and it takes longer to analyse , and this kind of thing. So I do think that there's potential for, um, the kind of open research conversations and some of the existing, I guess, kind of like feminist research principles that they can be allies to one another. Um, and I think that this whole move to just really just slow, slow down, slow down, really think about kind of, what is your research question? How can this be done as open as possible, as closes as necessary, in a way that kind of is as close to the values of transparency and rigour in a way that makes sense for your approach to your research? And I, and I think that that to me is, is the real hallmark of really good research, when it's just slowed down a little bit.

[: 00:26:47

You know, we, we, we are not employed to support students, which will apply...so again, how do we sort of collaborate, I suppose, in, in terms of encouraging, um, better practice with students as well, as well as staff. Um, but perhaps just try and talk about that a little bit in terms of how we can perhaps embed it a bit in the curriculum? There's a program at Leeds called Curriculum Redefined isn't there, that we're hoping to maybe leverage a little bit in this direction?

[: 00:27:56

[00:29:38] Nick: unsettling?

[: 00:29:39

[00:30:12] Nick: And I guess you're just talking from a psychology perspective. Cause again, immediately I'm, I'm, I'm naturally trying to think broader than that, you know, and how can we bring that into other disciplines? I guess there's no easy answer to that is there? It's just, uh, we need to take a sort of longitudinal view and take it one step at a time, and hopefully the things that you learn from, from doing it in psychology, we can try to extend to other disciplines.

[: 00:30:39

So I think I would really kind of encourage, particularly Curriculum Redefined colleagues, also everyone, to think about, well it might mean that, you know, preregistration doesn't make sense for your discipline or you don't work with data so open data doesn't make sense, but those kind of fundamental principles....um, and I think things like inclusivity and compassion and that kind of thing, I think I see as being open research principles as well. So how can we, you know, design a curriculum that, um, champions those values?

[: 00:31:39

[00:32:18] Maddi: That's, I think that's above my pay grade. But the , that was a big question. Yeah. Cause I think things like, like all the, the things we looked about in the systematic review around like, um, engagement...attitudes towards science, scientific literacies, that really, that beyond kind of what does that look like for undergraduate students, that those kind of things, um, you'd hope that there's some kind of initiative to combat those just generally in the public. Um, and that's why I think that, cause I always try and talk about open research and psychology, not in terms of like, oh, we're having a crisis and nothing will replicate and everything's miserable. But more that this is such an exciting time to be, even just kind of thinking about science and knowledge and life because we're having this, like we're really taking seriously, kind of, evidence and methodology and how we communicate our science. And I think because there's things like, you know, um, it's all well and good...you could do a really great, really robust research study and it's all preregistered and the data is open and it's really robust and lovely. But then if the media reports it in a way that then kind of overstates the claims, for example, and then that's what gets communicated. And then there's this whole issue of like, credibility. Um, yeah. Isn't necessarily in the academy, but still is really important in terms of, um, how we communicate our research in a way that has kind of meaningful impact. So I think that yeah, the whole open research conversation is for everyone.

[: 00:33:49

[00:33:56] Maddi: Yeah, I preprint all the time.

[: 00:33:58

[00:34:09] Maddi: yeah, I think, I think preprints is, is really complicated. Cause I preprint everything kind of just by default, um, because there's, um, like there's some papers that are being cited all the time now as preprints, that are still stuck under review somewhere because,I don't know, couldn't find reviewers or anything. So that's really useful. And I think when I was kind of on the job market, it was really useful because I could show, oh, here's everything I've been working on, and it's all under review, but trust me, it exists. But I also think that, you know, peer review is there for a reason. Um, and trying to, kind of promote my work, but also caveat it gently with it hasn't yet been peer reviewed. But that is quite a nuanced thing to say and I get that like other kind of scientists and researchers may understand the nuances of that, but what does that look like, as you say, when it comes to, um, comms and media and kind of more public, uh, consumption of that science? So I think it is, it is complicated, but generally I'm a big fan of preprints. Cause I think that, It is one way of kind of, I guess like democratizing access to research about...

[: 00:35:20

[00:35:41] Maddi: Yeah, I've done, um...so it was a general rule whenever I peer review, I sign all of my reviews. So I think, in the spirit transparency, I sign my reviews, um, which feels a lot easier when you really like the , when you really like the paper. It, it felt very different, um, when, for example, you're a reviewer and you're like recommending rejection or when there's like, You know, things wrong with it. Um, I think what it does, um, or the kind of practice of open peer review is...it's, it's slightly kind of two sides to it, but I think it has the potential to combat some of those issues around compassion and hostility and that kind of thing, because more often than not, that stuff is done when there's a kind of veil of anonymity. Um, and I think kind of, having a culture where yes, we can like criticize, or not criticize, we can critique each other's work because that's important. But if you have to put your name to it, I think it could potentially be one mechanism to um, yeah, to kind of create more of like a compassionate culture.

But then we also see examples where people are just kind of actively hostile on Twitter, where their name's attached to the comments. So it doesn't work all the time, but I'm hoping that it, it should in theory be better for things like accountability. Um, or that's kind of my own experience that I feel it's a bit like, um...someone said to me when I was like, marking my first piece of student work, they were like, don't put anything on that piece of student work that you wouldn't be happy to sit across having a coffee with someone and say to them. And I think that having more of that kind of vibe with peer review, um, should help some of those things we talk about in the paper in terms of like hostility and unkindness and separating the research from the researcher and things like that.

[: 00:37:28

[00:37:54] Maddi: Um, so one of my big things is...so I'm trying to, um, I'm trying to design something that's, that's gonna be, um, useful to try and look at student outcomes more nationally, particularly in psychology. So it's very like early stages at the moment, cause I need to apply for funding, which...that whole process just scares me a bit. But I'm trying to design some kind of like randomized control trial thing that I'm sketching out at the moment with some colleagues in psychology that aims to look at...um, so for the institutions that really explicitly integrate open research into their curricula, do they show any difference in student outcomes compared to the institutions that don't? To try...everything that I'm doing at the moment is trying to kind of fill this gap in the literature of needing really good empirical evidence, to, I guess, kind of evidence or show any impact that open research has on student outcomes. Um, so that's, that's the biggie that I'm working on at the moment.

[: 00:38:54

Thanks very much for your time and, um, I'll let you get on. Thank you.

[: 00:39:16

[00:39:20] Outro: Thanks for listening to the Research Culture Uncovered podcast. Please subscribe so you never miss out on our brand new episodes. And if you're enjoying the discussions, give us some love by dropping a five star rating and written review as it helps other research cultures. Find us and please share with a friend and show them how to subscribe.

Email us@academicdevleads.ac.uk. Thanks for listening, and here's to you on your research culture.