Skip to main content
SearchLoginLogin or Signup

Assessment Checklist

June 2020
Published onJul 27, 2020
Assessment Checklist
·
key-enterThis Pub is a Supplement to

This Checklist is intentionally separated from the Values and Principles Framework in this first iteration for ease of review. We recommend bringing up the Framework on one half of your screen and this Assessment Checklist on the other half so that you can see them simultaneously (or use two monitors if you are lucky enough to have them).

This assessment checklist is designed to help scholarly communications funders, organizations, and communities to first identify how well their operations align with academic values and principles, and then pinpoint concrete tasks they can undertake to improve that alignment over time. Those that demonstrate high alignment with academic values and principles tend to seek out a close engagement with a community of practice to whom they intentionally offer both control and responsibility.

As described in more detail in the Values and Principles Framework’s Introduction, we are mindful that assessment instruments and processes have to strike a tough balance. They need to provide enough structure and information to engender trust, guide investments, and incentivize alignment with shared principles and standards but not so much structure and information that it creates artificial barriers to entry for these marketplaces. The lead authors of this framework have grave concerns about how assessment frameworks have been used in other fields and environments and also about the costs associated with implementing assessments and the selection, credentialing, and monitoring of assessors/auditors. We welcome advice on how to structure this work so that it is neither one-size-fits-all, nor punitive in nature, but instead engages and incentivizes communities and organizations to move steadily towards stronger alignment with values and principles over time.

Diversity, Equity, and Inclusion

The community/organization attracts, welcomes, and retains stakeholders (staff, leadership, volunteers, and users/clients) from diverse lived experiences1 with a range of skill sets and viewpoints

  1. Do you have a plan or policy that helps you to achieve specific goals regarding your diversity (e.g., in staff recruitment, member/user recruitment, and leadership recruitment activities)?

Possible evidence: plan, policy, or statement

  1. Do you actively use guidance developed for hiring practices to help you recruit and hire candidates from diverse lived experiences?

Possible evidence: guidance sources and where they have been implemented in hiring and recruitment processes

  1. Do you know how staff members/leadership team members/volunteers of diverse lived experiences sense their belonging as staff members?

Possible evidence: documentation from reviews, team calls, HR survey and follow-up activities

  1. Does your leadership team fairly represent the diversity of your stakeholders?

Possible evidence: documentation, policies, evaluation practices

The community/organization ensures that its own practices and policies promote equitable treatment and opportunity

  1. Is advancement in the community/organization based on fair and transparent criteria?

Possible evidence: criteria and how it is applied in advancement

  1. Are salaries fair and equitable?

Possible evidence: current salaries or pay scale and job descriptions

  1. Do you offer or require training for all employees on topics related to diversity, equity, and inclusion?

Possible evidence: trainings offered/required in the last 12 months

  1. Do you offer or require training for all employees on topics related to accessibility for persons with disabilities?

Possible evidence: trainings offered/required in the last 12 months

  1. Do you have specific, measurable goals related to improving diversity, equity, and inclusion in your community/organization, in your field, or in society?

Possible evidence: goals list and measures

The community/organization maintains and enforces a safe environment for all people who engage with the community/organization or its activities

  1. Do you have a code of conduct, standards of conduct, or other policy documentation regarding acceptable behavior in community spaces?

Possible evidence: documentation specifically addressing expectations and rules regarding how people interact within the community/organization’s forums (e.g., listservs, calls, events, presentations, booths, working groups, projects, etc.)

  1. Which of the following does it explicitly cover: Staff; Community members; Leadership; Speakers, presenters, attendees, sponsors, and vendors for events; Verbal comments; Written comments; Images; Physical interaction; Intimidation; Disruption; Harassment; Stalking; Physical spaces; Virtual environments; Other

Possible evidence: documentation

  1. Is there a clearly documented pathway for reporting discrimination, harassment, or other harmful behavior and is it openly and easily available to those that come into contact with your organization?

Possible evidence: documentation and its location

  1. Is there a clearly documented pathway for responding to reports of violation and for enforcing your code of conduct, and is it openly and easily available to those that come into contact with your organization?

Possible evidence: documentation and its location

The community/organization’s activities and policies reflect and are responsive to the distinct needs and contexts of diverse stakeholder groups

  1. Do your products/services specifically impact or target underserved populations?

Possible Evidence: research, outreach, communications, or education plan or mission/vision/values

  1. Do your pricing or membership models seek to make participation attainable for stakeholders around the world and within diverse institutional contexts?

Possible Evidence: pricing model and/or membership model, (e.g., sliding scale or open source/open access)

  1. Do you engage diverse stakeholders in the design, co-creation, and evaluation of products, activities, or services?

Possible Evidence: research, market research, UX work, or other activities demonstrating involvement and intentionality of involvement

  1. Do your products/services support language diversity?

Possible Evidence: publications or outputs in languages beyond English

The community/organization ensures its outputs (including products, tools, and services) adhere to industry standards for accessibility

  1. Do your outputs explicitly prevent disparate access through compliance with W3C standards and guidelines?

Possible Evidence: conformance to Web Content Accessibility Guidelines (WCAG 2.0 or WCAG 2.1); Authoring Tool Accessibility Guidelines (ATAG 2.0); User Agent Accessibility Guidelines (UAAG 2.0); Accessible Rich Internet Applications (WAI-ARIA 1.1);

  1. Do you adhere to relevant national or international accessibility standards?

Possible Evidence: conformance to the European Union’s Web Accessibility Directive2 or Section 508 of the Rehabilitation Act3 in the United States

  1. Are the digital information and services you provide available in formats appropriate for users with disabilities, in a timely manner, and at no additional cost to those users?

Possible Evidence: captioning on audio-visual media, transcripts for audio-visual media; compliance with The Web Video Text Tracks Format (WebVTT) and Timed Text Markup Language (TTML)

The community/organization evaluates and selects its suppliers and partners based in part on their explicit adherence to equitable standards and inclusive practices with their staff and their clients/users

  1. Do you evaluate the hiring, staff support, and leave policies and practices of your suppliers and partners for equitability and attentiveness to diversity and inclusivity?

Possible Evidence: Checklist or contract language that ensures each supplier and/or partner meets specified goals; Procurement policy that includes a stated intent to use small businesses, women-owned business enterprises, and minority-owned business enterprises wherever possible or feasible

  1. Do you ensure your suppliers and partners are clear and transparent in their own pricing and negotiation strategies?

Possible Evidence: Checklist or contract language specifying that the supplier and/or partner does not engage in nondisclosure agreements or use exploitative business practices, such as bundling, unreasonable price increases, platform lock-in, or deceptive marketing

  1. Do you evaluate the policies and practices of your funders for equitability and attentiveness to diversity and inclusivity?
    Possible Evidence: Checklist or contract language specifying that funders
    consider supporting diversity and inclusivity among their funding priorities

Transparency

The community/organization publicly documents its internal operations

  1. Is your incorporation and tax status and/or your relationship to an incorporated host or fiscal/legal sponsor clearly and publicly described and documented?

Possible Evidence: Website or other publicly available environment includes concrete information about the community/organization’s official business identity, disclosing the legal incorporated entity name, its tax status, its national jurisdiction, and any “Doing Business As” names it is associated with.

  1. Is your governance process publicly documented?

Possible Evidence: Website or other publicly available environment includes a description of the ownership and control structure, and how decisions are made, e.g., bylaws, charter, or other official internal management structure rules

  1. If you are a publisher, do you provide public documentation regarding editorial, peer review, or other quality control practices for published materials?

Possible Evidence: Policies posted on website or other publicly available environment

  1. If you are creating or managing software or code, do you provide public documentation regarding your code review process, release schedule, and quality control practices?

Possible Evidence: Policies posted on website or other publicly available environment

  1. Are your mission, scope, and activities clearly and publicly stated?

Possible Evidence: Website or other publicly available environment includes the community/organization’s mission, scope, and activities

  1. Do you publicly distribute your governance meeting agendas, minutes, and official decisions?

Possible Evidence: Website, public email list, or other publicly available environment with active and ongoing records of the governance body’s meetings and decisions.

  1. Do you have a publicly available privacy policy?

Possible Evidence: Privacy policy posted on website or other publicly available environment

  1. Are your Terms and Conditions (if applicable) publicly available?

Possible Evidence: Terms and Conditions posted on website or other publicly available environment

The community/organization publicly documents its finances

  1. With which of the following groups do you share detailed financial information, including your community/organization’s net gains and losses, at least quarterly? Leadership, Full-time employees, All employees, Members, Users, Partners, General public

Possible Evidence: Mailings, messages, meeting presentations, minutes, Website or other publicly available environment

  1. Do you publicly document and/or disclose all of your sources of funding?

Possible Evidence: Website or other publicly available environment, Annual Report, mailings

  1. Do you publicly share the following financial information: budget, budget-to-actuals, statement of financial position, end-of year financial statements, operating reserves, and net worth?

Possible Evidence: Annual report, project-based disclosures, Website or other publicly available environment

The community/organization publicly documents its contract terms and pricing for all products and services

  1. Do you publicly disclose contract conditions for all of your products and services?

Possible Evidence: Website or other publicly available environment, sales and marketing materials

  1. Do you publicly disclose pricing for all of your products and services?

Possible Evidence: Website or other publicly available environment, sales and marketing materials

  1. Do you publicly disclose your terms of use, both for any software, platform, or infrastructure you host and for any content and data that you host or collect?

Possible Evidence: Terms of Use or License posted on Website or other publicly available environment

  1. Do you employ non-disclosure agreements?

Possible Evidence: self-assessment

  1. Do you have a publicly available policy against utilizing non-disclosure agreements?

Possible Evidence: Policy statement posted on Website or other publicly available environment

The community/organization has publicly accessible policies and practices for protecting the confidentiality and privacy of personal information

  1. Do you make public your policies regarding the collection and use of user data or personal information?

Possible Evidence: Policy posted to Website or other publicly available environment

  1. Do you comply with the General Data Protection Regulation (GDPR)?

    Do you allow users to “access their personal data and information about how this personal data is being processed”?

    Possible Evidence: Policy documentation

    Do you follow industry-standard protocols for data anonymization?

    Possible Evidence: Policy documentation

  2. Do you limit collection of user data to the minimum required for service provision?

Possible Evidence: Policy and process documentation

  1. Do you prohibit the selling of user data or insights derived from your user data?

Possible Evidence: Policy documentation

  1. Do you allow users to easily opt out of data collection?

Possible Evidence: Policy and process documentation, Website statement or clear opt-out option presented to users

Openness and Interoperability

The organization/community favors open technologies, standards, and protocols

  1. Do you make all software required to run your infrastructure available under an open source license?

Possible Evidence: Policy and process documentation, Website statement

  1. Are all of the software components in your stack available under open source licenses, including those that you did not develop locally?

Possible Evidence: Policy and process documentation, Website statement

  1. Is all documentation required for stakeholders to successfully implement or run your software/infrastructure available publicly and freely?

Possible Evidence: Policy and process documentation, Website statement

  1. Do you work within your industry to develop or set open standards?

Possible Evidence: Engagement in communities of practice, records of work undertaken with working groups, projects, standards-setting bodies, etc.

  1. Are your own publications available as Open Access documents with licensing that clearly specifies how they may be shared?

Possible Evidence: Open Access policy, percentage of community/organization’s publications available Open Access, CC licensing

  1. Do you use industry standard and/or open file formats wherever possible?

Possible Evidence: Platforms and tools use (or easy export) open file formats; content is stored in open file formats; infrastructure adheres to open standards wherever feasible

  1. Do you work with other major adopters of relevant standards to achieve consistent implementation?

Possible Evidence: Records or examples of collaborative work undertaken with other adopters within working groups, projects, standards-setting bodies, etc.

The community/organization seeks to make its tools/services/infrastructure interoperable with other tools/services/infrastructure

  1. Do you provide an API that facilitates interaction between your tool/service/infrastructure and others and/or that enables users to extend your tool/service/infrastructure?

Possible Evidence: open documentation, active engagement with other tools, Key Performance Indicators for the API(s)

  1. Do you maintain a publicly available product development roadmap?

Possible Evidence: Product development roadmap on website or other publicly available environment

  1. Does your platform/service/tool (if applicable) capture COUNTER-compliant metrics for publications?

Possible Evidence: COUNTER-compliant reports

  1. Do you invite your users to submit ideas for integrations between your product/service and other products/services?

Possible Evidence: User feedback forums, discussion notes, survey

  1. Do you evaluate what other tools/services/infrastructure your user base regularly relies upon?

Possible Evidence: Survey, market research

The community/organization seeks to enable and encourage use and reuse of content and code

  1. Do you make the content, metadata, and code you produce immediately, openly, and freely available, findable, and reusable?

Possible Evidence: Release via a standardized communications protocol; use of open licenses (e.g., CC-BY for content, and MIT or GNU GPLv3 or equivalent for code)

  1. Do you avoid placing restrictions on how others can implement your stack?

Possible Evidence: Policy and process documentation favors licenses like MIT and GNU GPLv3, Website statement, evaluation of practices

  1. Do you avoid placing restrictions on how others can share the outputs your entity produces?

Possible Evidence: Policy and process documentation favors licenses like CC-BY, CC-BY-SA, and CC-BY-NC; Website statement; evaluation of practices

  1. Do you employ persistent identifiers where appropriate?

Possible Evidence: Use of persistent identifiers (e.g., DOIs) or permalinks

  1. Do you ensure that any metadata you produce or provide meets domain-relevant standards to promote reproducibility/reusability?

Possible Evidence: Metadata documentation, Policy and process documentation

  1. Do you make your outputs machine-actionable through the utilization of general-purpose, open technologies?

Possible Evidence: Policy and process documentation

  1. Do you engage in Web archiving and/or digital preservation to ensure longevity for your content and code?

Possible Evidence: Use of web archiving (PermaCC, ArchiveIt, WebCite, Archive.is) or digital preservation repository (e.g. Chronopolis, APTrust, LOCKSS) for content and code

The community/organization provides its members and users with appropriate agency and control over content and code

  1. Do you have mechanisms in place to assist users in migrating off of your infrastructure?

    1. If yes, what costs are associated with these?

    2. If yes, what events could trigger a request for such assistance?

    3. If yes, in what format(s) can the user receive the content back?

Possible Evidence: APIs, export functions, exit strategies, migration to an agreed-upon format specified in initial contract, testimonials

  1. Do you disclose the use of algorithms used in your products or services and document their general functions?

Possible Evidence: Documentation (Actionable? Can a user understand and assess and potentially reproduce the algorithm?)

  1. Do you indicate how your metadata was derived, especially metadata that has been collected algorithmically?

Possible Evidence: Documentation, Policies and procedures

  1. Do you involve your users and members in creating and maintaining a product development roadmap for code and features?

Possible Evidence: Documentation, Policies and procedures. (Pay special attention to how it is developed and maintained, with whom it is shared, and how new features are prioritized)

  1. Do you implement relevant usage data metrics and make these available to users?

Possible Evidence: Documentation, procedures4

Access to Knowledge

The community/organization participates in, collaborates with, and supports a broader network of other organizations/communities that contribute to reducing exclusions from knowledge production, sharing, and/or use

  1. Do you invest in and/or financially support community initiatives and/or non-profit organizations that advance access to and reduce exclusions from the production, sharing, and/or use of knowledge?

Possible Evidence: Significant time, effort, and/or financial support provided to such community initiatives

  1. Do you actively and freely share your expertise, content, and/or infrastructure in order to promote access to and reduce exclusions from knowledge production, sharing, and use?

Possible Evidence: Open resources provided with adequate documentation to make them useful to broad, unrestricted audiences and stakeholders

The community/organization commits to the development of a robust knowledge commons

  1. Does your content, data, and code adhere to FAIR Guiding Principles?

Possible Evidence: Documented adherence to FAIR Guidance Principles (Findable, Accessible, Interoperable, Reusable)

  1. Do you do community outreach and/or advocate on behalf of individuals, communities, and organizations that experience exclusions from knowledge production, sharing, and use?

Possible Evidence: Documented evidence of work, publications, events, activities, statements, or other advocacy work.

The community/organization prioritizes access to knowledge as core to its mission and services

  1. Does your mission or strategic plan include a commitment to reducing exclusions from knowledge production, sharing, and usage?

Possible Evidence: Strategic plan, mission statement

  1. Do you have a leadership or senior position with clear accountability for delivering on and evaluating your community/organization’s work to reduce exclusions from the production, sharing, and/or use of knowledge?

Possible Evidence: Job or role description

  1. Do you internally evaluate your impact (using specific metrics or key performance indicators) in reducing exclusion from knowledge production, sharing, and usage?

Possible Evidence: Logic model, outcomes assessments shared with staff, leadership, and/or community members

  1. Have you participated in external evaluations to assess your impact in reducing exclusion from knowledge production, sharing, and usage?

Possible Evidence: Report or audit results

Representative Governance

The community/organization’s governance provides community stakeholders with decision-making roles and responsibilities

  1. Do you engage elected/appointed representatives from community stakeholder groups in the official, decision-making governing body?

Possible Evidence: Bylaws or other formal, legal governance documentation names and describes the role of this governing body

  1. Do you require that major decisions about the community/organization that affect stakeholders (especially decisions about its organizational form, ownership, and pricing) be approved by this official, decision-making governing body?

    Possible Evidence: Bylaws or other formal, legal governance documentation that describe decision-making protocols

The community/organization’s governance encourages participation and input from community stakeholders

  1. Do you make governance meeting agendas, minutes, and official decisions openly available to community stakeholders?

Possible Evidence: Documentation accessible by community stakeholders

  1. Do you have an openly documented process for soliciting/resolving community stakeholder concerns and/or objections about governance decisions?

    Possible Evidence: Process clearly described in the Website or another publicly available location

  2. Do you officially involve stakeholder community representatives in reviewing and amending the governance system?

Possible Evidence: Bylaws, evidence of participation in a review/amendment process

  1. Do community stakeholders give input into governing body appointments/elections?

Possible Evidence: Solicitation process, election process documentation

The community/organization’s governance officially engages representatives from active stakeholder communities in fiduciary oversight and management accountability

  1. Does a governance body that includes stakeholder representatives protect the organization’s assets and members’ investments?

Possible Evidence: Documented role in overseeing the creation and approval of the annual budget, financial quarterly close reports, annual fiscal reporting, annual audit/review

  1. Does a governance body that includes stakeholder representatives recruit, hire, and oversee the director/manager/CEO?

Possible Evidence: Bylaws, evidence of annual reviews, evidence of participation in recruitment and hiring of the director

  1. Does a governance body that includes stakeholder representatives provide direction for the organization?

Possible Evidence: Bylaws, Policy setting evidence, Strategic planning evidence

The community/organization ensures that representative voices change over time

  1. Do you set and enforce term limits for representatives?

Possible Evidence: Bylaws, Minutes from governance meetings, elections, number of representatives that change over time

  1. Do you have a process for stakeholders to express concerns or objections regarding who is serving as their representative?

Possible Evidence: Process clearly documented in the Bylaws, on the Website or in another publicly available location

Financial and Organizational Stability

The community/organization thinks about and actively plans for the future

  1. Do you document your mission-driven goals and milestones on a regular basis?

Possible Evidence: Strategic plan, logic model, documented goals

  1. Do you measure progress towards these goals and milestones at least annually?

    Possible Evidence: Documented assessment of goals

The community/organization maintains control over and monitors its own finances

  1. Do you have access to and control over your bank account(s), budget, ledger/log of all receivables and expenditures, and any accounting software and/or bookkeeping environment?

Possible Evidence: Review of each element

  1. Do you have a chart of accounts that details your categories of expenditures and receivables?

Possible Evidence: Chart of accounts, accounting report demonstrating the use of the chart of accounts

  1. Do you create an officially approved annual operating budget for the overall community/organization each year?

Possible Evidence: Annual budget developed by the community/organization that is approved by its governing body

  1. Do you rectify your financial books at least quarterly with review and sign-off from a designated governing body representative(s)?

Possible Evidence: Signed quarterly reports or other evidence of successful review

  1. Do you have written financial policies that are adequate for the size and complexity of your community/organization and that include explicit internal controls?

Possible Evidence: Accounting manual, policy documentation

  1. Do you have an annual independent audit or review conducted of your financial statements?

Possible Evidence: Audit or Review documentation

  1. Are there at least two, non-interdependent funding streams for the entity (e.g., memberships, subscriptions, services, grants, contracts, donations)?

Possible Evidence: Documented revenue streams

  1. Do you know how to describe your legal and organizational identity (e.g., incorporation, taxable status, charity status, etc)?

Possible Evidence: Official documentation (e.g., incorporation documents, tax exemption letter, B-Corp certificate, host organization documents) as compared to self description

  1. Do you clearly document your community/organization’s legal and organizational identit(ies) such that community members and customers/clients understand what the legal entity can do and who has authority to speak for it?

Possible Evidence: Disclosure via Website or other public forum, contractual language

  1. Do you engage a legal advisor and/or lawyer to review all contracts (both those you prepare and those you sign)?

Possible Evidence: Legal representation

  1. If you are not a free-standing organization but instead are hosted by another legal entity, do you transparently and publicly acknowledge this hosting relationship?

Possible Evidence: Disclosure via Website or other public forum, contractual language

  1. If you are hosted by or housed within another entity, do you have a documented pathway for separation that can be enacted by your governance group?

Possible Evidence: Documented separation pathway and trigger events

The community/organization clearly documents its succession plans

  1. Do you have a policy that describes how organizational ownership or responsibility may be transferred, and does this policy explicitly prohibit contracts/agreements to be reassigned without express permission from community stakeholders?

Possible Evidence: Documentation review

  1. Do your governance documents include a legal commitment to maintaining your organization’s mission regardless of company ownership?

Possible Evidence: Documentation review

  1. Do you have a succession strategy that identifies entities that will steward products and services if or when your company ceases to operate?

Possible Evidence: Succession documentation

The community/organization clearly documents its sunsetting and dissolution plans

  1. Do you have a financial reserve set aside that is adequate to cover sunsetting costs?

Possible Evidence: Documented financial reserve and sunset budget

  1. Do you have a plan documenting what happens when your company ceases to operate?

Possible Evidence: Documented sunset/dissolution plans

  1. Do you document, test, and regularly update specific pathways for any user-owned or generated content to be migrated to another platform or service in the case of dissolution?

Possible Evidence: Agreements with similar platforms/services, migration strategies and pathways, contracts with users/members, testing documentation

Comments
4
?
Gerald Beasley:

This is valuable work thank you, but Sustainable Development appears to be absent from the Values, Principles and Assessment Checklist, and I’m not sure why, since a commitment to SD would seem to be important.

?
Ana Enriquez:

Could there be a pre-quiz of the activities in which the organization engages, to determine which questions should be asked? For example, this one really only applies to organizations that develop products/software.

?
Ana Enriquez:

It would help to define this more rigorously. Is “stack” intended to include use of Adobe Reader (which is not open source)? What about layout software?

Paige Mann:

Another way to expand participation is to offer consortially-managed solutions. I know that smaller libraries interested in IRs may need to rely on their consortia if they’re ever to have an IR or journal publishing service.