(S3 E7) Octopus: a new way to open and transparent research with Dr Alexandra Freeman

[: 00:00:00

[00:00:25] Nick Sheppard: Hi, it's Nick, and for those who don't know me yet, I'm open research advisor based in the library here at the University of Leeds. You're joining us in season three of the Research Culture Uncovered podcast where we'll be speaking to colleagues from both the University of Leeds and from other universities and organizations about open research, what it is, how it's practiced in different disciplines, and how it relates to research culture.

s from the Reds Conference of: 2022

But now I'd like to introduce my guest for today, Dr. Alexandra Freeman. The creator of Octopus, a new open research platform and radical new approach to academic publishing designed to replace journals and papers as the place to establish priority and record your work in full detail. Open by design, the platform is being developed in partnership with Jisc and the UK Reproducibility Network with a grant from Research England.

Dr. Freeman is executive director for the Winton Center for Risk and Evidence Communication at the University of Cambridge. Previously, she had a 16 year career at the BBC working on series such as Walking With Beasts. Life in the Undergrowth, Bang Goes the Theory, Climate Change by Numbers and as series producer of Trust Me, I'm a Doctor, and, uh, you've won a number of awards, I believe Alex, including a BAFTA. What was the, what was the BAFTA for?

[: 00:01:51

[00:02:22] Nick Sheppard: I think I remember that. I remember the program, I'm not sure about the interactive side. Is that still out there? I might have to have a look at that with my kids myself. Is it still there?

[: 00:02:29

[00:02:34] Nick Sheppard: And uh, and I was listening to another podcast recently, that you did with Jisc and you were, you worked with David Attenborough. Is that right?

[: 00:02:41

[00:02:59] Nick Sheppard: And do you miss the BBC? Did you? Um.

[: 00:03:03

[00:03:19] Nick Sheppard: But it...so it does sound that, you know, with that sort of background, you were interested in communicating science, I guess?

[: 00:03:26

Um, and having started doing that, uh, I changed to doing that in 2016, and I found myself leading a research group with people, you know, doing postdocs in research. And that was when I really realized how much the sort of research assessment and the research culture relied on the way that researchers communicate their work. And that was when I started thinking, hmm, I think something needs to change here cos I'm not sure that the current journal system is really encouraging people to communicate in the way that would be best for science and research in general.

[: 00:05:10

[00:05:39] Alex Freeman: It is, I mean, it's not...so the research that we do in the Winton Centre is about how to communicate numbers and quantitative evidence and the quality of the underlying evidence to people. So those people might be, uh, patients trying to make decisions about healthcare, they might be a judge or a jury in court. They might be policy makers trying to develop policies. So all of these people are trying to make decisions based on evidence, and we work on how to communicate that evidence in such a way as to support that decision making. So how to help people weigh up the pros and cons and know about the quality of the underlying evidence. And so that quality of the underlying evidence can sometimes, you know, overlap with things like the reproducibility crisis. But it's not directly related to it.

[: 00:06:36

[00:06:47] Alex Freeman: it did, yeah.

[: 00:06:49

[00:06:59] Alex Freeman: Yeah, I mean, how long have you got? I think, uh, yeah, I mean, I knew about things like the reproducibility crisis before I came back to academia. I mean, you know, we probably all read about that kind of thing in the popular press, but really what struck me more than that, and more than the push for open access, were sort of broader issues with the research culture and the kind of publish or perish culture and also, working on the kind of communications that we did in the Winton Centre, it really struck me how much people needed to...well, there are kind of different ways of communicating. So when I was communicating in the media, it was quite right that my sort of incentives were built around how many people watched the programs and what people's reaction to the programs were. Did they find them engaging? Did they feel they learned something new? These were the sort of metrics that we got back, um, either immediately the next morning or in a week's time, and we were really judged on those.

It's quite, you know, we get quite a lot of feedback and we respond to that feedback constantly in the media and that's right, you know, we're basically an entertainment medium and when I started working on evidence communication, I really realized how different that needs to be, because your incentives now...you are not just trying to engage people with information, you are really trying to help people understand that information at a much deeper level and engaging emotions, and all of the kind of thing that, um, sort of normal science communication does is actually potentially a detriment for that. Um, you know, you're not just trying to sell a story and tell a good story and, you know, get people feeling kind of positive or angry or all of these emotions that you can kind of play with and engage in a narrative. You are trying to get people to really think about the numbers and make serious decisions and weigh things up and really have their critical faculties at their maximum.

And it really occurred to me that when we're writing for journal articles, if you look at the author guidelines, they kind of make sense from a media perspective. They're about, you know, keeping the word counts short, telling a good story, you know, putting the findings upfront and thinking about the impact.

But they don't make sense if your core objective is to help people understand the real detail. Uh, if you want people to be able to really assess the quality of that work. You know, all the sort of stuff that ends up being put in supplementary information is actually the really important stuff if you're trying to assess that work really for its intrinsic qualities and if you're trying to build on it.

So I started thinking, well, there's these two kinds of writing. There's writing to disseminate findings. If you're writing, to try and, uh, engage a policymaker or a practitioner and make them think, oh, this is something, you know, these are findings that I might put into practice. That's one thing, and that's all about the findings and the impact.

But if you're trying to communicate all the real depth of what's been done by whom and what happened, um, and all of the working, then that's a really different way of writing. Uh, it needs to be much fuller. It's probably not a linear narrative. It might use multiple different kinds of media. It's not just a kind of print thing. It might involve code and data repositories and videos, so, really the kind of thinking at the Winton Center that I was doing in the day-to-day research did sort of inform my thinking about the way that we communicate as scholars and seeing the research culture as well as somebody now working within a research group made me think, well, this is having a real impact on people's lives, people's careers, but also how they approach their work.

You know, I went to conferences where experienced, um, you know, really sort of, uh, high, high ranking professors were giving advice to early career researchers, and sometimes their advice was really, um, I felt was really bad advice because they were, you know, it was good advice in the environment that we're in, they were saying things like, if you want to get headlines, think about the headlines and then what research you need to do to get those headlines. And that really put my teeth on it, I thought, you know, that is not how we should be approaching research. It's not about the headlines, it's about the quality of the underlying work. So, you know, that may be how to succeed in the current environment, but if we want to help people do good research and incentivize good research, we need to change the current environment.

So that's why I thought we really need something quite radical that can help address things like the reproducibility crisis, open access issues...but also these bigger issues about what's incentivized and how research is assessed and what we think good means when it comes to research.

[: 00:12:53

[00:12:57] Alex Freeman: Well yes. And actually I was kind of also set...I mean, all of this was going on in my head, but it hadn't crystallized, it wasn't part of my day job. And then at the University of Cambridge, we have this organization that helps people come from outside academia and come and ask questions of academics. And so we occasionally get these emails saying, um, here's somebody who wants to have a chat. They're interested in X. Would you be able to spend an hour with them? Um, and somebody came and their question was, if you could start again with science, what would you do? And now that's a really big and challenging question, but it made me really think, well, the thing that I think would most change science for the better would be addressing this problem with the way that we communicate research, and hence the way that we assess research and hence the way that we approach research. And so it, it was the day before I was meeting him and I thought, gosh, I'd really better write down my thoughts so that I've got something cohesive to say.

So I wrote down all of this and emailed it to him to say, this is what I think we could chat about tomorrow. And he turned up to that meeting and said, yeah, that sounds great. Why aren't you doing it? So the ball was firmly in my court and I thought, well, fair point, I better do something.

[: 00:14:24

[00:14:33] Alex Freeman: Yes, and they've been working on it....it's had a very long heritage. So it was back in 2017 that I had this sort of night where I wrote everything down on about sort of, excuse me... so it was about 2017 when I wrote it all down and had this sort of night of writing about 10 pages of how I thought this new structure could work and since then I've gone out and talked about it with people. I then managed to raise some money to start building a really simple prototype so that I could show people how it might work. And then a more complex prototype, and then eventually UKRI, the UK's Government research funder, funded a full scale, uh, production of Octopus, and helped me go into collaboration with Jisc, who, um, really have the resources and technical capabilities to build something that can take the amount of traffic that Octopus is designed to take.

ry early product in, in June,: 2022

[00:17:07] Nick Sheppard: And Octopus...it, it refers to, well, eight legs I guess. Is it? There, there's eight , eight components of, of research that you've identified, I think. Is that right?

[: 00:17:17

So there are research problems, hypotheses, methods or protocols, uh, data or results, analyses, interpretations,and real world sort of implementations and then there are peer reviews. And so they form these kind of branching chains because everything in Octopus has to be linked to an existing publication because if you just, you know, allow people to put material into this kind of, like a repository, a soup of stuff, then it becomes...you know, people can publish data without publishing the method that it was, um, uh, collected according to, uh, or they can, you know, where do you put an analysis of a data set? You want all these things to be linked together. So in Octopus, if you publish an analysis, it has to be linked to the data. That data also has to be available, so you get these branching chains. Uh, each publication can have multiple peer reviews attached to it, and that forms this kind of, um, the, the skeleton, ironically, behind the invertebrate, Octopus structure. Um, and what we've done to start people off, because every publication needs to be attached to an existing publication, is that we've created, we've pre-seeded the data with existing research problems that we've extracted automatically using natural language processing from existing papers. So there are already something like 7,500 research problems or topics within octopus. And that means that there's a kind of...it's, it's almost like seeding a coral reef. There's already a structure there.

So if you come along with, uh, some research that you want to share. You should hopefully, fingers crossed, be able to find somewhere to attach your first research question to, and then you can publish your research question and then you can publish your hypothesis or theory behind that, uh, how you're going to approach it, and all of those publications can then attach to each other and, and flow from that.

[: 00:20:07

[00:20:58] Alex Freeman: Not yet so far. I mean, maybe I just always anticipate a lot of challenges anyway, but I...there's nothing that I haven't anticipated and the issues that people raise, you know, there are things that people worry about. Um, one of them, for instance, is the fact that peer reviews are post publication and they're not anonymous, so once a publication's out there in Octopus, then anybody can peer review it. But a peer review is treated like a public...any other publication type, uh, it is a publication in its own right.

So people worry about that because, uh, peer reviews in the current process, people are used to them being anonymous. And it's a slightly different role from how I see peer reviews going into the future. Um, and so people worry about, you know, can I feel like I can publish a critique of a more senior, in quotes, author's work? Um, there are concerns about...people worry that I wouldn't publish my ideas in Octopus because then I might get scooped. So these sorts of concerns, which I hope in the longer term the research culture will change because octopus is designed to change the way that we approach the way that we work. So you know, you can't get scooped if you are getting credit for the idea.

The idea is a piece of work in itself. You get it out there as quickly as possible. You can only be scooped until it's out there. Once it's out there. Octopus is like a patent office. It's out there with your name on, with the data on. There you go. Your idea, you get credit for that as an idea. You don't need to collect data for that idea.

So there are different...it's a different way of thinking, which I think will take, uh, quite a while to establish itself within the community. But everything about Octopus is designed with the principle of really helping people do the best quality research, uh, and really recognizing that and incentivizing it, and rewarding what we all as a community think is good practice.

So it is very, it is, it is ambitious in that it's trying to tackle pretty much every problem that I can see at the moment in one fell swoop. And some people kind of, you know, have a sharp intake of breath when they think, you know, well, can you really tackle everything at once? Well, I think we can, uh, because I think, you know, these are fundamental problems and if you tackle them at the root, and redesign the whole system, then yes, I think we can tackle them all at once.

[: 00:23:41

[00:24:36] Alex Freeman: Yes, I do. I do. Because everywhere I talk about Octopus, you generally get. An incredibly positive response to it. People understand what it's trying to do and why it's trying to do it, and that it is a model that can work. I mean, obviously in the dark hours of the early morning, sometimes I wake up with a slight cold sweat thinking, what if I'm wrong? But you know, it's been several years now of talking about this, and nobody has come up with a really fundamental flaw as to why this model isn't a much better model. You know, it's not perfect, but it's certainly a lot better than the current model.

Uh, so that's my first thing that gives me sort of positive vibes. But the other thing is that we're talking to all the different parts of the ecosystem. Som of course researchers feel that they can't change the world themselves just by the way that they publish. Institutions can't change the world because they also are contingent on what researchers do and what funders do, and so all of these different parts, the researchers, the institutions, the funders, the journals, we all have to kind of join hands and take a leap together. And actually I've talked to all of those four and everybody is happy to start taking that leap together. So I'm really hoping that in the next year we will all move together. You know, the funders are thinking about research assessment of the future, what the next REF's going to look like. Um, and, you know, other independent funders are all interested in, you know, making sure that they fund the best research that achieves the aims that they want.

They want high quality research, so they're really interested in research quality and research assessment and how it's done Institutions. They want to be able to see at a glance what all their different researchers are doing and how their research is being, um, uh, recognized by peers, um, and by funders. And octopus is designed to plug into, uh, institutional systems and provide them with the information that they want and that they need. and for researchers it's designed to make the publication process much easier for them. So, you know, I know what a pain it is to write a paper and get it through the review process and get it to publication. And actually so much of the time I've got a lot of research that it would be really nice to be able to get out there quickly, um, and get out there easily. And Octopus is designed to be able to do that and allow me to get credit for that. And journals who I initially wasn't sure how journals and octopus would work hand in hand, but actually had really positive conversations with several journals now about the fact that it...journals can still publish articles and in fact, increasingly journals are moving towards, especially the larger, more general journals, they're moving towards shorter articles, articles that are more to the point as far as their readers are concerned, and a lot of...the sort of the research work is therefore ending up in supplementary information, which can be a bit scattered and difficult to get hold of.

Well, Octopus can essentially form the basis for a research article. It can include all of the data and all of the analysis and all of the other work that goes on in a research project in a structured way. And then the journal article can kind of sit on top of that, and journals really appreciate that.

And in the future, you know, peer review is a difficult and expensive process for journals to organize. If there are peer reviews within Octopus....they're thinking about how they can use those to streamline their own systems and choosing what then gets published in their journals. So they're really positive about it as well. So I really think that, you know, we can all take this leap of faith with our, you know, holding hands in a row and stepping out into the, into the new future together.

[: 00:29:07

[00:30:04] Alex Freeman: Well, it is a challenge, yeah. I mean, I find it a challenge in my researcher job, uh, and I use several of those outlets. Um, but Octopus is designed to really bring all of those together. So if you are pre-registering already somewhere, if you are using protocols.io, if you are using Open Science Framework, any of these repositories or services, then Octopus can tie all of those together so you don't have to repeat work that you're already doing. If you're already using somewhere, then you can just use the URLs or the DOIs that you've created in these other services and put them in Octopus and kind of get them stamped in Octopus that, you know, this is how this fits in with all my other research. So if I've published a pre-registered hypothesis. In Octopus, I just need to say I have pre-registered the hypothesis, here it is, here's the DOI. And that then allows that hypothesis to be within my research chain to be research assessed. So I hope that by working with, um, you know, UKRI and people and institutions, we can really help researchers navigate the complex structure and bring it all into one place so that it's easier. It, it means that you don't have to give up doing what you, you are already doing. It allows you to tie together what you're already doing, and if you're not already using these services, then you can either purely use Octopus or you can use whichever service works best for you and tie it into your Octopus stuff, and Octopus will bring it all together for your institution.

[: 00:31:45

[00:31:47] Alex Freeman: Yes. Yeah, you log in with your ORCID, um, and it pulls in all the information that's attached to your ORCID so it already knows about all of your other publications. It will automatically pull those in from ORCID and present them to your institution. So when your institution wants to look at an author's page, they can see everything that that author's done, all the peer reviews that they've written, all of the publications outside of Octopus, all of their Octopus work. And increasingly we'll have little graphics to show, you know, how that...what sort of researcher that person is. You know, are they doing lots of collaborative work? Are they coming up with lots of ideas or are they a data collector? Are they an analyst? So it'll provide institutions with little miniature profiles of each researcher.

[: 00:32:36

[00:33:05] Alex Freeman: Yeah. So Octopus is all about encouraging and incentivizing intrinsic research quality. So what I mean by that is that so much journal articles end up being about the impact of the findings. But findings, although they're important and they might be our ultimate aim of a lot of research, they...they're not robust unless they're based on really high quality methods and ideas and hypotheses and data collection and all the other kinds of work. And I think that at the moment that means that we're driving people far too much towards it all being about the results and the findings and that we're putting less emphasis on the fact that we need specialists who are all about analysis, say, or data collection or hypothesis generation, and that these are important specializations and the quality of their work...it needs to be judged in its own right. Um, so that's why in Octopus we break up the unit of publication into these smaller pieces. because, you know, I think that each of those pieces represents a genuine sort of body of work that a person can produce.

And so that can increase, um, specialization because now if you are specialist in one part of the research process, you can be publishing just that part of the research process and collaborating sort of worldwide, you know. Um, but it also means that we can assess each of those units individually. So, as I said, peer review...you'd be peer reviewing, not a whole journal article, which might be, you know, seven or eight or more people's work, all kind of integrated together. Instead, you'd be peer reviewing a data set, or a hypothesis, or a protocol or an analysis. Um, so it means that the peer review is much more focussed, and we also have a rating system. And that now scares some people because they think, oh, you know, we don't wanna put a, a number on the quality of somebody's work. But actually it allows us to set criteria, predefined criteria, which define what we as a community think good means. So what does a good data set look like? What does a good analysis look like? And if we can come up with predefined criteria, and that will help people structure the way they're thinking about it, um, and rate those, which then once you've got a numerical rating, that does help a lot with the sort of digital world that we live in with sort of ordering search results and, you know, graphing results. But it will be also combined with the qualitative peer reviews. So I think together all of that helps us really, uh, refine research assessment from being something that's about quite a broad diffuse thing, like a journal article, and instead being much more focused on the real integrity and quality of a much smaller piece of work. And I think that's what we need to do.

[: 00:36:23

[00:36:50] Alex Freeman: Yeah, absolutely. So I think, um, first of all, go to Octopus - octopus.ac - and explore what the platform looks like. Have a read about how it works. But then what I would really encourage people to do is think about what your research questions are and publish your first research problem, which is quite a short, you know, it's about a paragraph or so, uh, publication that describes the research question that you are working on. And that then gives you a flavour of how easy it is to publish on Octopus. You shouldn't be worried...hopefully people won't be worried that their research questions are going to be scooped, so that's a kind of gateway entrance to it. And then have a think about. What, um, you might not be able to publish in the journal article at the moment.

Have you got small data sets or ideas that you are really, genuinely never going to carry on and work on further? Why not publish those in Octopus? Have a go see how it feels. You can then get credit for publishing those. So those will now appear on your record for your institution to see. Maybe somebody will peer review them for you and rate them.

d then, as I said, next year,: 2023

So, you know, I think really think ahead now about work that you are doing. For instance, I've got work in my team where we've got quite...multiple analyses and complex work that's gone on. When we publish it as a paper, it's only going to be a small aspect of that work. So we've talked to the journal about publishing all the stuff that we've done on Octopus, and then relating that to a sort of narrative thread that we're going to submit to the journal, and they're very happy with that concept. So have a think about whether you've got work like that that might fit the Octopus structure. So yes, have a look, have a think, and then I...

[: 00:39:25

[00:39:46] Alex Freeman: Yeah, I mean, it was originally a STEM proposition because that's my background, but actually we've had really good conversations with people in arts and humanities and social sciences, uh, and many of those feel that this way of working will actually suit their disciplines as well. So out of the, um, 7,500 or so problems in there already, uh, there are some topics that cover all of hopefully, uh, humanities, um, history. We've got lots of that in there. So there are places for arts and humanities to start branching their problems out as well, I hope. But we're really keen over the next year to work closely with people in areas where they don't feel that Octopus currently necessarily sounds like it's suiting them because I think a lot of it is about we need to learn what language people use in different disciplines, and make sure that that helps people then see how, for instance, you know, a protocol or a method might be better described to fit the way that people work in other disciplines that we're less familiar with.

[: 00:41:03

[00:41:10] Alex Freeman: Absolutely

[: 00:41:11

[00:41:14] Alex Freeman: Yeah, absolutely.

[: 00:41:16

[00:41:22] Alex Freeman: Oh it's been a pleasure. Thank you. I can always talk about Octopus for a long time.

[: 00:41:26

[00:41:35] Alex Freeman: Yeah. And do drop us a line...dpo drop us line if anyone wants to chat.

[: 00:41:41

[00:41:43] Alex Freeman: Perfect. Thank you.

[: 00:41:44

[00:41:50] Outro: Thanks for listening to the Research Culture Uncovered podcast. Please subscribe so you never miss out on our brand new episodes. And if you are enjoying the discussions, give us some love by dropping a five star rating and written review as it helps other research culturists find us. Please share with a friend and show them how to subscribe.

Email us at academicdev@leeds.uk. Thanks for listening, and here's to you and your research culture.