Peer review is critical for the acceptance of open textbooks by the wider world. Perceptions of uncertain quality of open textbooks is often cited as a barrier to adoption. Good, transparent review processes will help solve this problem.
While many open textbooks are comprehensively reviewed, there is currently no accepted standard or process for review.
We’ve put together a Peer Review Working Group to help the Open Textbook community come together to develop a clear approach to review. (Would you like to join us? Go here!)
We’ve currently got around 20 members in this group, from as many institutions, including: the Open Textbook Network, OpenStax, SUNY, Oregon State, and Ryerson University. You can find a recap of our first meeting (including the full video) here.
During this meeting, we identified a number of challenges for reviewing Open Textbooks, and our group hopes to provide some solutions, with subgroups focusing on the following issues.
Please join us if you think you can help us figure out smart approaches to reviewing Open Textbooks.
Group 1: Open Textbook Review Board The Rebus Community has proposed to create a review board that can been leveraged to help find reviewers for open textbooks. Members of the board will be contacted when a book becomes available for review and make efforts within and possibly beyond their institutions to find volunteers. This group will define and help to launch the board. Join Group 1 here.
Group 2: Defining Categories of Review This group will work to define the purpose and structure of the different kinds of review that could be valuable to an open textbook. They will also help guide the development of dedicated tools and resources for each kind of review. Join Group 2 here.
Group 3: Environmental Scan of Peer Review Tools, Processes & Research This group will conduct a scan of the existing peer review market in order to inform the development of new review processes. Join Group 3 here.
Group 4: Creating Standard Markers for OT Review This group will discuss creating a standardised system to clearly identify the kinds of review that any given textbook has undergone. This will link to the work of group 2, who are defining different categories of review. Join Group 4 here.
Group 5: Recognition for Reviewers This group will discuss ideas of how to recognise and reward reviewer contributions. This may eventually feed into the creation of something similar to the Mozilla Contributorship Badges project, recognising all kinds of participation in open textbook projects, including peer review. Join Group 5 here.
Signup for the groups will stay open, but we encourage you to join by May 20 so we can start moving forward on each of them.
In this month’s Office Hours, organized by Rebus Community and the Open Textbook Network, special guests and participants discussed their experiences with OER workflows.
Watch the video or read a recap of the session below.
Guests included Allison Brown, SUNY Geneseo; Rebel Cummings-Sauls, Kansas State University; Billy Meinke, University of Hawaii; and Anthony Palmiotto, OpenStax.
Karen Lauritsen of the Open Textbook Network introduced this month’s guests, asking them to briefly describe the processes they use when making open textbooks.
Billy Meinke, an OER technologist at UHawaii, works with faculty at his institution to shepherd them through the process of creating OERs. He shared an OER workflow diagramthathecreated, which provides a high-level view of the major steps involved in adopting and adapting an OER: priming, pre-production, design, development, and publishing. In order to make for more efficient workflows, Billy said, OER training at all UHawaii campuses also now includes Pressbooks training, to make individuals comfortable with the software they use for book production.
Rebel Cummings-Sauls is the director for the Centre for Advancement of Digital Scholarship at Kansas State University and specializes in copyright and Open Access. KSU provides grant funding for faculty to create open access textbooks and resources. Rebel said KSU prefers faculty to complete the resources commissioned within a year, though projects can take longer. Payment is one incentive she uses to keep projects on track. Rebel mentioned that one factor that can cause delays is that faculty members often want more rounds of private feedback on their textbook before it goes public.
Anthony Palmiotto is the editorial director at OpenStax, which creates open textbooks that are competitive with market-leading texts for specific college courses and makes them available free on the OpenStax website, generally under CC BY licenses. The OpenStax workflow begins with preparation: collecting market research, competitive benchmarking, educational research, length requirements, and so on. OpenStax involves faculty in this process, reaching out to them via surveys and at conferences. Often those who get involved in this way will continue to work on the project in later stages. OpenStax selects a team of faculty to work on the book as authors and reviewers. Later stages of production include revisions, originality checking, art rendering, fact checking and accessibility checks, and XML production. Anthony said OpenStax books typically take 18 months to 2 years to reach completion.
Allison Brown joined us from OpenSUNY, and described the production process for SUNY’s open textbooks. After a manuscript has been received, the workflow includes peer review, author revisions, copy editing, typesetting, proofreading, and finally publication. She said pain points included transitions (when a manuscript moves between collaborators such as writers and editors; or across platforms, such as from Word to Pressbooks) and copy editing. She said it’s important to forewarn authors of the expectations of them post-copy editing. To make the workflow more manageable, OpenSUNY conducts a thorough needs assessment of each manuscript prior to production. Allison says one common thread that ensures successful book projects is healthy communication with authors. She is transparent with authors, staff and freelancers, and clearly outlines expectations at various stages of the production process. She also mentioned the importance of someone acting as project manager to ensure projects stay on track.
The ensuing discussion touched upon a number of topics. Among these, the group discussed how to involve students in beta use and testing of books post-publication. Rebel said her institution obtains feedback on the books through several methods–evaluations, surveys, quizzes and forms from students whose faculty use the text in their class–and that they receive the bulk of student feedback and input in the first semester of a book’s use. Participants asked how OpenStax operationalizes originality checks and incorporates ancillary resources. Anthony said they use iThenticate and conduct spot checks of sections of copy with Google. Participants asked whether proofreading, copy editing, and design is assigned to freelancers or done in-house. Allison and Rebel both said that it varies with project, with Allison adding that she sometimes outsources cover design to students. One of the major themes that came out of this discussion was the importance of educating faculty and students about CC licenses and their implications, as well as copyright and fair use guidelines. New Prairie Press’ Permission to Publish was shared as a resource.
Peer review is an integral component of traditional academic publishing, but in the world of open textbooks, there is currently no common standard for reviewing in open textbooks. To address this need, Rebus Community has formed a working group to explore peer review for open textbooks. The group held its first meeting May 3 to discuss how the community of open textbook practitioners could build a coherent strategy and best practices around peer review for open textbooks.
Below is a recap of the meeting. You can read next steps and how to get involved here.
Deb Quentel from CALI started the session by describing CALI’s editorial review process for the lessons and textbooks they produce. Deb said CALI has an editorial review board consisting of potential reviewers in specific subject areas that she reaches out to first when looking for reviewers. If no one has the relevant expertise, she asks board members to leverage their networks to find appropriate reviewers. If that fails, the author may suggest potential peers as reviewers. Finally, Deb scouts reviewers online from relevant websites and publications.
In CALI’s peer review process, reviewers are asked to look at subject matter accuracy and whether the content covers the terrain to be expected. Reviewers also evaluate the sophistication of the end-of-chapter questions that are included for students. Reviewers provide a no-more-than-two-page memo of feedback. At CALI, peer reviewers are anonymous and usually review one chapter of a text. About 75% of the chapters for any given book get reviewed. Deb outlines her process in further detail here.
Next, Hugh McGuire of Rebus Community outlined various types of pre- and post-publication review that he hypothesized may need to be done to an open textbook:
Subject matter expert review (a formal, pre-publication review, similar to traditional peer review)
Open review, in which anyone can make comments (less formal)
Student and faculty feedback on a published, beta version of the textbook
Ongoing feedback (how does feedback get back to the authors following publication?)
Revision and updates (especially for subjects that change frequently)
Karen Lauritsen from the Open Textbook Network gave an overview of OTN’s review process. All new OTN member organizations are able to send faculty to workshops as part of their professional development in open textbooks. In return, they are required to provide a light review for one book in the Open Textbook Library and are given a rubric from which to do so. Karen said that roughly 40% of these reviewers end up adopting a textbook from the library for classroom use.
The discussion then moved on to standardizing types of peer review processes for open textbooks, perhaps by using markers or badges to transparently delineate which processes were employed on a given text. Billy Meinke from the University of Hawaii shared the Mozilla Science Contributorship Badges and noted that ideally peer review badges would exist both for content and for contributors. This would serve the dual purpose of giving recognition to reviewers along with credit they can use in their tenure and promotion process.
Billy said that Mozilla Science Badges were linked to ORCID ids. Hugh said Rebus will be looking for ways to do the same. Karen added that the markers ought to be non-hierarchical, but still provide adequate recognition.
Nicholas Persa from UW-Madison acknowledged the need for one portal of access that enables and encourages community review – a model that Rebus is hoping to build.
Participants agreed on many topics, such as the need to incentivize reviewers either with payment or recognition, the need for post-publication review in classrooms and continual updating of open textbooks, and badges or markers to validate both content and reviewers. However, a number of questions remained unanswered: what specific value can we offer reviewers, particularly when monetary payment is not an option? Should pre-publication review be done before copyediting? How do we begin making these markers and defining levels of completion for peer review?
Universities such as The University of British Columbia seem to be making strides in recognizing the importance of OERs on tenure committees, and as this spreads to other institutions, there will be a growing need to develop best practices and standards for processes like reviewing.
The Rebus Working Group will continue to work on these and other issues pertaining to peer review. Next steps will be posted on the forum. Join the conversation on the Rebus Community forum thread as we try to figure it all out!
At Rebus, we think one of the best ways to produce open textbooks is by involving students in the process.
That’s why we’re really excited about the handbook we have in development, A Guide to Making Open Textbooks With Students. The book is designed to help faculty find or create projects involving students in the creation of open textbooks or open resources.
The guide will contain:
an overview of open pedagogy – what it is, why it’s valuable, learning objectives and outcomes
resources and assignments for faculty wanting to adopt open pedagogy in their classrooms
case studies of projects in which faculty and students collaborated to create open textbooks and similar open educational resources
student perspectives on their experiences working on such projects
tools you can use for open textbook projects, such as licensing guides, student agreements and more.
Currently our focus is on finding students who have worked on open textbook projects to write 300- to 500-word narratives about their experiences.
If you know of someone who has worked on such a project and might be willing to contribute such a sidebar, please sign up to the forum and reply to this post, or encourage them to do so.
We also still welcome contributions from faculty, so if you have an idea for a resource that you can contribute, we’re all ears–reply here.