Introduction
Katherine Skinner, Educopia Institute
Through the Next Generation Library Publishing project (2019–2022, generously funded by Arcadia Fund), Educopia Institute, California Digital Library, and Stratos, in close collaboration with COAR, LYRASIS, and Longleaf Services, seek to improve the publishing pathways and choices available to authors, editors, and readers through strengthening, integrating, and scaling up scholarly publishing infrastructures to support library publishers. In addition to building publishing tools and workflows, our team is exploring how to create community hosting models that align explicitly and demonstratively with academic values.
In this Values and Principles Framework and Assessment Checklist, we are piloting a tool that can help scholarly publishing service providers demonstrate their adherence to and support for agreed-upon academic values and principles.
We invite feedback and engagement with these ideas and with this work, which is generously supported by a grant from Arcadia, a charitable fund of Lisbet Rausing and Peter Baldwin. This Framework and Assessment Checklist and its affiliated documentation have been prepared by Sarah Lippincott and Katherine Skinner and edited by Nancy Adams, Hannah Ballard, Terra Graziani, Jennifer Kemp, Brandon Locke, Catherine Mitchell, Kristen Ratan, Oya Reiger, Kathleen Shearer, John Sherer, Nick Shockey, Eric van Rijnand Sarah Wipperman.
Background
Over the last 30 years, well over 100 manifestos, open letters, and documents have been prepared by scholars, librarians, publishers, and other scholarly communications players in explicit hopes of reclaiming or recalibrating the scholarly publishing landscape. These statements have pointed to a set of relatively consistent ideas regarding what values and principles the statements’ authors believe that scholarly publishing providers should seek to uphold in their work, including representative governance; sustainability; equity, diversity, and inclusion; transparency; openness; and reproducibility/reusability.
Thus far, most of these values and principles statements have been issued by attendees of a hosted event or a set of participants in a project, community, or association to guide behavior. With few exceptions, these documents have not been structured to enable users to assess or demonstrate their compliance. Instead, their users often signal their support via signing or invoking a statement, and signatories to a statement may work in direct conflict with its terms with no fear of repercussions.
After studying these statements in 2019–2020, our team recommended reworking values and principles statements to include measurable actions, so that adherence can be readily assessed and audited. In a white paper published in March 2020, we proposed a methodology for assessment that mission-driven scholarly publishing service providers can use to demonstrate their adhesion to agreed-upon academic values and principles. Following the paper’s positive reception, we have drafted a pilot framework to further test this concept.
The Next Generation Library Publishing project now seeks feedback on this framework. With this assessment framework, we hope to differentiate those with close adherence (perhaps with a designation, e.g., Trusted Community Partner) and to incentivize better alignment between publishing tools, services, and platforms and the scholarly communities and publics they ultimately serve.
How to Use the Framework and Assessment Checklist
Please note that for the purposes of this public review period, the Assessment Checklist is intentionally separated from the Values and Principles Framework in this first iteration for ease of review. We recommend bringing up this Framework on one half of your screen and the Assessment Checklist on the other half so that you can see them simultaneously (or use two monitors if you are lucky enough to have them).
Our proposed model of use for this framework is as follows:
Self Assessment: Entities would begin by conducting a self assessment, gathering independent evidence to substantiate each relevant value and principle, as based on the indicators and the assessment checklist. Gathering and organizing these materials will signal that an entity is ready to undertake a second-stage review.
Review by Trained Peers: A second stage review of that evidence would then be conducted by trained peers (process TBD). This review would define and recognize the entity’s position within a spectrum of practice, possibly by assigning a designation(s) for entities that meet or surpass a threshold (e.g., “Trusted Community Partners”). The second stage review might also include recommendations for growth towards values-aligned practices. The review process will be intentionally designed to avoid cost quandaries that so often beset assessment and audit frameworks. We recommend that assessment results be made publicly available.
Ongoing Reviews: Finally, any assessment would necessarily be repeated on a regular basis in order to account for changes over time within each entity. There will need to be a process for retracting designations for organizations and communities. There will also need to be specified conditions under which additional reviews may be requested, e.g. if a Trusted Community Partner is acquired by a new hosting organization.
The framework consists of six top-level values: diversity, equity, and inclusion; transparency; openness and interoperability; access to knowledge; financial and organizational stability; and representative governance. Each value includes several principles or ways that the value might manifest. Each principle includes several indicators, or statements that lend themselves to measurement.
Each of these indicators is elaborated upon in the assessment checklist, with a set of concrete, verifiable queries and suggestions for possible evidence that an entity might provide to show its adherence to that Indicator.
For example, under the value “Transparency,” there are three principles (see Figure 1 below). The first Principle is that “The community/organization gives stakeholders visibility into its internal operations and practices.” This principle contains two Indicators, that the community/organization publicly documents 1) its internal operations, and 2) its finances.
Figure 1: Snapshot of Transparency Value, Principles, and Indicators
Principle | Indicators |
|---|
The community/organization gives stakeholders visibility into its internal operations and practices. | The community/organization publicly documents its internal operations. The community/organization publicly documents its finances |
The community/organization engages in equitable contracting and pricing practices. | The community/organization publicly documents its contract terms and pricing for all products and services. |
The community/organization provides visibility into its collection, storage, and use of user data. | The community/organization has publicly accessible policies and practices for protecting the confidentiality and privacy of personal information. |
For the first Principle, the checklist then specifies questions for each of the two indicators:
|
|---|
Are your mission, scope, and activities clearly and publicly stated?
Possible Evidence: Website or other publicly available environment includes the community/organization’s mission, scope, and activities
Is your incorporation and tax status and/or your relationship to an incorporated host or fiscal/legal sponsor clearly and publicly described and documented?
Possible Evidence: Website or other publicly available environment includes concrete information about the community/organization’s official business identity, disclosing the legal incorporated entity name, its tax status, its national jurisdiction, and any “Doing Business As” names it is associated with.
Is your governance process publicly documented?
Possible Evidence: Website or other publicly available environment includes a description of the ownership and control structure, and how decisions are made, e.g., bylaws, charter, or other official internal management structure rules
Do you publicly distribute your governance meeting agendas, minutes, and official decisions?
Possible Evidence: Website, public email list, or other publicly available environment with active and ongoing records of the governance body’s meetings and decisions.
If you are a publisher, do you provide public documentation regarding editorial, peer review, or other quality control practices for published materials?
Possible Evidence: Policies posted on website or other publicly available environment
If you are creating or managing software or code, do you provide public documentation regarding your code review process, release schedule, and quality control practices?
Possible Evidence: Policies posted on website or other publicly available environment
Do you have a publicly available privacy statement?
Possible Evidence: Privacy statement posted on website or other publicly available environment
Are your Terms and Conditions (if applicable) publicly available?
Possible Evidence: Terms and Conditions posted on website or other publicly available environment
Do you publicly share the following financial information: budget, budget-to-actuals, statement of financial position, end-of year financial statements, operating reserves, and net worth?
Possible Evidence: Annual report, project-based disclosures, Website or other publicly available environment
With which of the following groups do you share detailed financial information, including your community/organization’s net gains and losses, at least quarterly? Leadership, Full-time employees, All employees, Members, Users, Partners, General public
Possible Evidence: Mailings, messages, meeting presentations, minutes, Website or other publicly available environment
Do you publicly document and/or disclose all of your sources of funding?
Possible Evidence: Website or other publicly available environment, Annual Report, mailings
|
An entity using this assessment framework would answer each of these questions, identifying the evidence that can be used to substantiate a more formal external review. Once the external review is complete, a designation(s) would be granted, and then the entity would be reassessed on a regular basis.
Use Cases
We believe this framework may be applicable to at least three core use cases:
For scholarly publishing infrastructure providers, this framework provides a way to:
Measure adherence to academic values and principles
Demonstrate that adherence to clients and prospective clients (perhaps via spectrum-scoring and/or a designation, e.g., “Trusted Community Partner”)
Compare their adherence back to their own baseline over time to show progressive engagement
Compare their adherence to that of peers
Identify concrete ways to improve adherence over time
For libraries, presses, and societies, this framework provides a way to:
Inform decisions about which scholarly publishing infrastructure providers to work with
Demonstrate their own commitment to working with academic mission-driven scholarly publishing infrastructure providers
Monitor how their scholarly publishing infrastructure providers align with academic mission over time
Incentivize them to invest in working relationships with “Trusted Community Partners”
For funders and investors in academic publishing, this framework provides a way to:
Inform decisions about which scholarly publishing infrastructure providers to fund
Demonstrate their commitment to funding academic mission-driven scholarly publishing infrastructure providers
Monitor how their scholarly publishing infrastructure providers align with academic mission over time
Incentivize grantees/recipients and prospective grantees/recipients to align their activities with the academic mission they support
We are building upon the existing base of academic values and principles, as documented by more than a hundred initiatives and communities over the last three decades. We seek to provide a framework that offers concrete ways to gauge and measure stakeholder behaviors against these values and principles. Ultimately, institutions that demonstrate higher adherence via the framework should, at least in theory, be acting as “Trusted Community Partners” or some similar designation. Establishing such “trustedness” will depend upon a scholarly communications provider’s transparency and provision of verifiable evidence of its adherence to the values and principles. We will be testing this framework in a series of pilot assessments later this year, and we will share our findings publicly.
In this work, we have striven to account for differences in organizational structures, recognizing that these may influence the implementation of values and principles. The categorization into a set of interconnected-but-distinct values, each with its own set of assessment questions, enables organizations and communities to see their adherence to each value, not just to the set as a whole. A broad range of players—emergent and established, nonprofit and for profit, small and large—have roles to play in a healthy scholarly communication environment, and we are currently studying and trying to account for where fundamental differences in their organizational objectives and abilities may necessarily impact how they can respond to and enact certain values and principles. For example, a program hosted by an academic institution is often disallowed its own board and freestanding governance; instead, it must formally adhere to the Board of Trustees or other official governing body of the academic institution itself. Such a program may build in ways to engage an extended community of stakeholders in governance roles, but these will have to manifest differently than that of a nonprofit (501c3, for example) organization.
We also try to account for and address our own national biases. This framework is being produced in the U.S., and our project team includes partners largely based in the U.S. Being inclusive of a wide variety of geographical and national contexts is complicated, even in more globally representative project teams than our own, and it is a challenge we commit to taking on in this work. This pilot framework currently does not claim any extensibility beyond North America and parts of Europe; however, we are actively seeking to engage a much broader range of locations and fiscal and cultural environments in the next phase of our work (with funding contracts supporting this involvement). We acknowledge this not to excuse the biases we know we have introduced but to recognize them and to commit to addressing them as we continue to build the framework.
We are mindful that indicators or accountability processes/assessments have to strike a tough balance. They need to provide enough structure and information to engender trust, guide investments, and incentivize alignment with shared principles and standards but not so much structure and information that it creates artificial barriers to entry for these marketplaces. The lead authors of this framework have grave concerns about how assessment frameworks have been used in other fields and environments and also about the costs associated with implementing assessments and the selection, credentialing, and monitoring of assessors/auditors. We are also concerned that often, assessment tools lend legitimacy to players that produce documentation and procedures that appear on the surface to meet a requirement but that actually fail to accomplish the spirit of the value or principle that the assessment tool is trying to measure.
We welcome engagement with this framework in any form (direct comment, outreach to the authors, outreach to the team). We also trust the extended community to help us assess whether a framework like this is worthwhile within our field, and, if so, what protections need to be put into place to help maximize its positive impact and guard against its misuse.
Models Consulted
For a full list of the values and principles documents and assessment frameworks we have studied, please also refer to the white paper published in March 2020 by Katherine Skinner and Sarah Wipperman: Encouraging Adherence to Values and Principles in Scholarly Publishing: A case for assessment strategies.
The assessment models that we especially leaned upon and learned from as we prepared this framework include the following:
Values, Principles, and Indicators
VALUE: Diversity, Equity, and Inclusion
Diversity, Equity, and Inclusion Assessment Checklist
Principle | Indicator(s) |
|---|
The community/organization advances diversity, equity, and inclusion | The community/organization attracts, welcomes, and retains stakeholders (staff, leadership, volunteers, and users/clients) from diverse lived experiences with a range of skill sets and viewpoints The community/organization ensures that its own practices and policies promote equitable treatment and opportunity |
The community/organization creates inclusive spaces, encouraging diverse participation in organizational activities | The community/organization has processes designed to maintain and enforce as safe an environment as possible for all people that engage with the community/organization or its activities The community/organization’s activities and policies reflect and are responsive to the distinct needs and contexts of diverse stakeholder groups |
The community/organization provides equitable access to its tools, services, and outputs | The community/organization ensures its outputs (including products, tools, and services) adhere to industry standards for accessibility |
The community/organization works with others that commit to advancing diversity, equity, and inclusion | The community/organization evaluates and selects its suppliers and partners based in part on their explicit adherence to equitable standards and inclusive practices with its staff and its clients |
VALUE: Transparency
Transparency Assessment Checklist
Principle | Indicator(s) |
|---|
The community/organization gives stakeholders visibility into its internal operations and practices | The community/organization publicly documents its internal operations The community/organization publicly documents its finances |
The community/organization engages in equitable contracting and pricing practices | The community/organization publicly documents its contract terms and pricing for all products and services |
The community/organization provides visibility into its collection, storage, and use of user data | The community/organization has publicly accessible policies and practices for protecting the confidentiality and privacy of personal information |
VALUE: Openness and Interoperability
Openness and Interoperability Checklist
Principle | Indicator(s) |
|---|
The community/organization strives to create open, interoperable products and services | The community/organization actively chooses open technologies, standards, and protocols The community/organization seeks to make its tools/services/infrastructure interoperable with other tools/services/infrastructure |
The community/organization facilitates use and reuse by humans and machines | The community/organization enables and encourages use and reuse of content and code |
The community/organization facilitates broad understanding of its systems, interfaces, and infrastructure | The community/organization provides its members and users with appropriate agency and control over content and code The community/organization ensures the portability and durability of content that it hosts on behalf of members and users |
VALUE: Access to Knowledge
Access to Knowledge Assessment Checklist
Principle | Indicator(s) |
|---|
The community/organization contributes to the development of a vibrant knowledge commons | The community/organization participates in, collaborates with, and supports a broader network of other organizations/communities that help to reduce exclusions from knowledge production, sharing, and/or use The community/organization commits to the development of a robust knowledge commons |
The community/organization engages in activities that advance access to knowledge | The community/organization prioritizes access to knowledge |
VALUE: Representative Governance
Representative Governance Assessment Checklist
Principle | Indicators |
|---|
Governance systems should center and empower community stakeholder voices | The community/organization’s governance provides community stakeholders with decision-making roles and responsibilities The community/organization’s governance encourages participation and input from community stakeholders The community/organization’s governance processes and lines of authority are transparent and documented openly |
Governance systems should distribute control and decision-making power among community stakeholders | The community/organization officially engages representatives from active stakeholder communities in fiduciary oversight and management accountability The community/organization ensures that representative voices change over time |
VALUE: Financial and Organizational Stability
Financial and Organizational Stability Assessment Checklist
Principle | Indicator(s) |
|---|
Community/organization should pursue a stable and secure future | The community/organization thinks about and actively plans for the future |
Community/organization should engage appropriate control over its fiscal and legal status | The community/organization maintains control over and monitors its own finances The community/organization understands, maintains, and communicates its legal and organizational identit(ies) |
Community/organization should document and share its succession planning and sunsetting options | The community/organization clearly documents its succession plans The community/organization clearly documents its sunsetting and dissolution plans |
Creative Commons Attribution 4.0 International License (CC-BY 4.0)