
Description
June 2020
This Checklist is intentionally separated from the Values and Principles Framework in this first iteration for ease of review. We recommend bringing up the Framework on one half of your screen and this Assessment Checklist on the other half so that you can see them simultaneously (or use two monitors if you are lucky enough to have them).
This assessment checklist is designed to help scholarly communications funders, organizations, and communities to first identify how well their operations align with academic values and principles, and then pinpoint concrete tasks they can undertake to improve that alignment over time. Those that demonstrate high alignment with academic values and principles tend to seek out a close engagement with a community of practice to whom they intentionally offer both control and responsibility.
As described in more detail in the Values and Principles Framework’s Introduction, we are mindful that assessment instruments and processes have to strike a tough balance. They need to provide enough structure and information to engender trust, guide investments, and incentivize alignment with shared principles and standards but not so much structure and information that it creates artificial barriers to entry for these marketplaces. The lead authors of this framework have grave concerns about how assessment frameworks have been used in other fields and environments and also about the costs associated with implementing assessments and the selection, credentialing, and monitoring of assessors/auditors. We welcome advice on how to structure this work so that it is neither one-size-fits-all, nor punitive in nature, but instead engages and incentivizes communities and organizations to move steadily towards stronger alignment with values and principles over time.
Do you have a plan or policy that helps you to achieve specific goals regarding your diversity (e.g., in staff recruitment, member/user recruitment, and leadership recruitment activities)?
Possible evidence: plan, policy, or statement
Do you actively use guidance developed for hiring practices to help you recruit and hire candidates from diverse lived experiences?
Possible evidence: guidance sources and where they have been implemented in hiring and recruitment processes
Do you know how staff members/leadership team members/volunteers of diverse lived experiences sense their belonging as staff members?
Possible evidence: documentation from reviews, team calls, HR survey and follow-up activities
Does your leadership team fairly represent the diversity of your stakeholders?
Possible evidence: documentation, policies, evaluation practices
Is advancement in the community/organization based on fair and transparent criteria?
Possible evidence: criteria and how it is applied in advancement
Are salaries fair and equitable?
Possible evidence: current salaries or pay scale and job descriptions
Do you offer or require training for all employees on topics related to diversity, equity, and inclusion?
Possible evidence: trainings offered/required in the last 12 months
Do you offer or require training for all employees on topics related to accessibility for persons with disabilities?
Possible evidence: trainings offered/required in the last 12 months
Do you have specific, measurable goals related to improving diversity, equity, and inclusion in your community/organization, in your field, or in society?
Possible evidence: goals list and measures
Do you have a code of conduct, standards of conduct, or other policy documentation regarding acceptable behavior in community spaces?
Possible evidence: documentation specifically addressing expectations and rules regarding how people interact within the community/organization’s forums (e.g., listservs, calls, events, presentations, booths, working groups, projects, etc.)
Which of the following does it explicitly cover: Staff; Community members; Leadership; Speakers, presenters, attendees, sponsors, and vendors for events; Verbal comments; Written comments; Images; Physical interaction; Intimidation; Disruption; Harassment; Stalking; Physical spaces; Virtual environments; Other
Possible evidence: documentation
Is there a clearly documented pathway for reporting discrimination, harassment, or other harmful behavior and is it openly and easily available to those that come into contact with your organization?
Possible evidence: documentation and its location
Is there a clearly documented pathway for responding to reports of violation and for enforcing your code of conduct, and is it openly and easily available to those that come into contact with your organization?
Possible evidence: documentation and its location
Do your products/services specifically impact or target underserved populations?
Possible Evidence: research, outreach, communications, or education plan or mission/vision/values
Do your pricing or membership models seek to make participation attainable for stakeholders around the world and within diverse institutional contexts?
Possible Evidence: pricing model and/or membership model, (e.g., sliding scale or open source/open access)
Do you engage diverse stakeholders in the design, co-creation, and evaluation of products, activities, or services?
Possible Evidence: research, market research, UX work, or other activities demonstrating involvement and intentionality of involvement
Do your products/services support language diversity?
Possible Evidence: publications or outputs in languages beyond English
Do your outputs explicitly prevent disparate access through compliance with W3C standards and guidelines?
Possible Evidence: conformance to Web Content Accessibility Guidelines (WCAG 2.0 or WCAG 2.1); Authoring Tool Accessibility Guidelines (ATAG 2.0); User Agent Accessibility Guidelines (UAAG 2.0); Accessible Rich Internet Applications (WAI-ARIA 1.1);
Do you adhere to relevant national or international accessibility standards?
Possible Evidence: conformance to the European Union’s Web Accessibility Directive2 or Section 508 of the Rehabilitation Act3 in the United States
Are the digital information and services you provide available in formats appropriate for users with disabilities, in a timely manner, and at no additional cost to those users?
Possible Evidence: captioning on audio-visual media, transcripts for audio-visual media; compliance with The Web Video Text Tracks Format (WebVTT) and Timed Text Markup Language (TTML)
Do you evaluate the hiring, staff support, and leave policies and practices of your suppliers and partners for equitability and attentiveness to diversity and inclusivity?
Possible Evidence: Checklist or contract language that ensures each supplier and/or partner meets specified goals; Procurement policy that includes a stated intent to use small businesses, women-owned business enterprises, and minority-owned business enterprises wherever possible or feasible
Do you ensure your suppliers and partners are clear and transparent in their own pricing and negotiation strategies?
Possible Evidence: Checklist or contract language specifying that the supplier and/or partner does not engage in nondisclosure agreements or use exploitative business practices, such as bundling, unreasonable price increases, platform lock-in, or deceptive marketing
Do you evaluate the policies and practices of your funders for equitability and attentiveness to diversity and inclusivity?
Possible Evidence: Checklist or contract language specifying that funders
consider supporting diversity and inclusivity among their funding priorities
Is your incorporation and tax status and/or your relationship to an incorporated host or fiscal/legal sponsor clearly and publicly described and documented?
Possible Evidence: Website or other publicly available environment includes concrete information about the community/organization’s official business identity, disclosing the legal incorporated entity name, its tax status, its national jurisdiction, and any “Doing Business As” names it is associated with.
Is your governance process publicly documented?
Possible Evidence: Website or other publicly available environment includes a description of the ownership and control structure, and how decisions are made, e.g., bylaws, charter, or other official internal management structure rules
If you are a publisher, do you provide public documentation regarding editorial, peer review, or other quality control practices for published materials?
Possible Evidence: Policies posted on website or other publicly available environment
If you are creating or managing software or code, do you provide public documentation regarding your code review process, release schedule, and quality control practices?
Possible Evidence: Policies posted on website or other publicly available environment
Are your mission, scope, and activities clearly and publicly stated?
Possible Evidence: Website or other publicly available environment includes the community/organization’s mission, scope, and activities
Do you publicly distribute your governance meeting agendas, minutes, and official decisions?
Possible Evidence: Website, public email list, or other publicly available environment with active and ongoing records of the governance body’s meetings and decisions.
Do you have a publicly available privacy policy?
Possible Evidence: Privacy policy posted on website or other publicly available environment
Are your Terms and Conditions (if applicable) publicly available?
Possible Evidence: Terms and Conditions posted on website or other publicly available environment
With which of the following groups do you share detailed financial information, including your community/organization’s net gains and losses, at least quarterly? Leadership, Full-time employees, All employees, Members, Users, Partners, General public
Possible Evidence: Mailings, messages, meeting presentations, minutes, Website or other publicly available environment
Do you publicly document and/or disclose all of your sources of funding?
Possible Evidence: Website or other publicly available environment, Annual Report, mailings
Do you publicly share the following financial information: budget, budget-to-actuals, statement of financial position, end-of year financial statements, operating reserves, and net worth?
Possible Evidence: Annual report, project-based disclosures, Website or other publicly available environment
Do you publicly disclose contract conditions for all of your products and services?
Possible Evidence: Website or other publicly available environment, sales and marketing materials
Do you publicly disclose pricing for all of your products and services?
Possible Evidence: Website or other publicly available environment, sales and marketing materials
Do you publicly disclose your terms of use, both for any software, platform, or infrastructure you host and for any content and data that you host or collect?
Possible Evidence: Terms of Use or License posted on Website or other publicly available environment
Do you employ non-disclosure agreements?
Possible Evidence: self-assessment
Do you have a publicly available policy against utilizing non-disclosure agreements?
Possible Evidence: Policy statement posted on Website or other publicly available environment
Do you make public your policies regarding the collection and use of user data or personal information?
Possible Evidence: Policy posted to Website or other publicly available environment
Do you comply with the General Data Protection Regulation (GDPR)?
Do you allow users to “access their personal data and information about how this personal data is being processed”?
Possible Evidence: Policy documentation
Do you follow industry-standard protocols for data anonymization?
Possible Evidence: Policy documentation
Do you limit collection of user data to the minimum required for service provision?
Possible Evidence: Policy and process documentation
Do you prohibit the selling of user data or insights derived from your user data?
Possible Evidence: Policy documentation
Do you allow users to easily opt out of data collection?
Possible Evidence: Policy and process documentation, Website statement or clear opt-out option presented to users
Do you make all software required to run your infrastructure available under an open source license?
Possible Evidence: Policy and process documentation, Website statement
Are all of the software components in your stack available under open source licenses, including those that you did not develop locally?
Possible Evidence: Policy and process documentation, Website statement
Is all documentation required for stakeholders to successfully implement or run your software/infrastructure available publicly and freely?
Possible Evidence: Policy and process documentation, Website statement
Do you work within your industry to develop or set open standards?
Possible Evidence: Engagement in communities of practice, records of work undertaken with working groups, projects, standards-setting bodies, etc.
Are your own publications available as Open Access documents with licensing that clearly specifies how they may be shared?
Possible Evidence: Open Access policy, percentage of community/organization’s publications available Open Access, CC licensing
Do you use industry standard and/or open file formats wherever possible?
Possible Evidence: Platforms and tools use (or easy export) open file formats; content is stored in open file formats; infrastructure adheres to open standards wherever feasible
Do you work with other major adopters of relevant standards to achieve consistent implementation?
Possible Evidence: Records or examples of collaborative work undertaken with other adopters within working groups, projects, standards-setting bodies, etc.
Do you provide an API that facilitates interaction between your tool/service/infrastructure and others and/or that enables users to extend your tool/service/infrastructure?
Possible Evidence: open documentation, active engagement with other tools, Key Performance Indicators for the API(s)
Do you maintain a publicly available product development roadmap?
Possible Evidence: Product development roadmap on website or other publicly available environment
Does your platform/service/tool (if applicable) capture COUNTER-compliant metrics for publications?
Possible Evidence: COUNTER-compliant reports
Do you invite your users to submit ideas for integrations between your product/service and other products/services?
Possible Evidence: User feedback forums, discussion notes, survey
Do you evaluate what other tools/services/infrastructure your user base regularly relies upon?
Possible Evidence: Survey, market research
Do you make the content, metadata, and code you produce immediately, openly, and freely available, findable, and reusable?
Possible Evidence: Release via a standardized communications protocol; use of open licenses (e.g., CC-BY for content, and MIT or GNU GPLv3 or equivalent for code)
Do you avoid placing restrictions on how others can implement your stack?
Possible Evidence: Policy and process documentation favors licenses like MIT and GNU GPLv3, Website statement, evaluation of practices
Do you avoid placing restrictions on how others can share the outputs your entity produces?
Possible Evidence: Policy and process documentation favors licenses like CC-BY, CC-BY-SA, and CC-BY-NC; Website statement; evaluation of practices
Do you employ persistent identifiers where appropriate?
Possible Evidence: Use of persistent identifiers (e.g., DOIs) or permalinks
Do you ensure that any metadata you produce or provide meets domain-relevant standards to promote reproducibility/reusability?
Possible Evidence: Metadata documentation, Policy and process documentation
Do you make your outputs machine-actionable through the utilization of general-purpose, open technologies?
Possible Evidence: Policy and process documentation
Do you engage in Web archiving and/or digital preservation to ensure longevity for your content and code?
Possible Evidence: Use of web archiving (PermaCC, ArchiveIt, WebCite, Archive.is) or digital preservation repository (e.g. Chronopolis, APTrust, LOCKSS) for content and code
Do you have mechanisms in place to assist users in migrating off of your infrastructure?
If yes, what costs are associated with these?
If yes, what events could trigger a request for such assistance?
If yes, in what format(s) can the user receive the content back?
Possible Evidence: APIs, export functions, exit strategies, migration to an agreed-upon format specified in initial contract, testimonials
Do you disclose the use of algorithms used in your products or services and document their general functions?
Possible Evidence: Documentation (Actionable? Can a user understand and assess and potentially reproduce the algorithm?)
Do you indicate how your metadata was derived, especially metadata that has been collected algorithmically?
Possible Evidence: Documentation, Policies and procedures
Do you involve your users and members in creating and maintaining a product development roadmap for code and features?
Possible Evidence: Documentation, Policies and procedures. (Pay special attention to how it is developed and maintained, with whom it is shared, and how new features are prioritized)
Do you implement relevant usage data metrics and make these available to users?
Possible Evidence: Documentation, procedures4
Do you invest in and/or financially support community initiatives and/or non-profit organizations that advance access to and reduce exclusions from the production, sharing, and/or use of knowledge?
Possible Evidence: Significant time, effort, and/or financial support provided to such community initiatives
Do you actively and freely share your expertise, content, and/or infrastructure in order to promote access to and reduce exclusions from knowledge production, sharing, and use?
Possible Evidence: Open resources provided with adequate documentation to make them useful to broad, unrestricted audiences and stakeholders
Does your content, data, and code adhere to FAIR Guiding Principles?
Possible Evidence: Documented adherence to FAIR Guidance Principles (Findable, Accessible, Interoperable, Reusable)
Do you do community outreach and/or advocate on behalf of individuals, communities, and organizations that experience exclusions from knowledge production, sharing, and use?
Possible Evidence: Documented evidence of work, publications, events, activities, statements, or other advocacy work.
Does your mission or strategic plan include a commitment to reducing exclusions from knowledge production, sharing, and usage?
Possible Evidence: Strategic plan, mission statement
Do you have a leadership or senior position with clear accountability for delivering on and evaluating your community/organization’s work to reduce exclusions from the production, sharing, and/or use of knowledge?
Possible Evidence: Job or role description
Do you internally evaluate your impact (using specific metrics or key performance indicators) in reducing exclusion from knowledge production, sharing, and usage?
Possible Evidence: Logic model, outcomes assessments shared with staff, leadership, and/or community members
Have you participated in external evaluations to assess your impact in reducing exclusion from knowledge production, sharing, and usage?
Possible Evidence: Report or audit results
Do you engage elected/appointed representatives from community stakeholder groups in the official, decision-making governing body?
Possible Evidence: Bylaws or other formal, legal governance documentation names and describes the role of this governing body
Do you require that major decisions about the community/organization that affect stakeholders (especially decisions about its organizational form, ownership, and pricing) be approved by this official, decision-making governing body?
Possible Evidence: Bylaws or other formal, legal governance documentation that describe decision-making protocols
Do you make governance meeting agendas, minutes, and official decisions openly available to community stakeholders?
Possible Evidence: Documentation accessible by community stakeholders
Do you have an openly documented process for soliciting/resolving community stakeholder concerns and/or objections about governance decisions?
Possible Evidence: Process clearly described in the Website or another publicly available location
Do you officially involve stakeholder community representatives in reviewing and amending the governance system?
Possible Evidence: Bylaws, evidence of participation in a review/amendment process
Do community stakeholders give input into governing body appointments/elections?
Possible Evidence: Solicitation process, election process documentation
Does a governance body that includes stakeholder representatives protect the organization’s assets and members’ investments?
Possible Evidence: Documented role in overseeing the creation and approval of the annual budget, financial quarterly close reports, annual fiscal reporting, annual audit/review
Does a governance body that includes stakeholder representatives recruit, hire, and oversee the director/manager/CEO?
Possible Evidence: Bylaws, evidence of annual reviews, evidence of participation in recruitment and hiring of the director
Does a governance body that includes stakeholder representatives provide direction for the organization?
Possible Evidence: Bylaws, Policy setting evidence, Strategic planning evidence
Do you set and enforce term limits for representatives?
Possible Evidence: Bylaws, Minutes from governance meetings, elections, number of representatives that change over time
Do you have a process for stakeholders to express concerns or objections regarding who is serving as their representative?
Possible Evidence: Process clearly documented in the Bylaws, on the Website or in another publicly available location
Do you document your mission-driven goals and milestones on a regular basis?
Possible Evidence: Strategic plan, logic model, documented goals
Do you measure progress towards these goals and milestones at least annually?
Possible Evidence: Documented assessment of goals
Do you have access to and control over your bank account(s), budget, ledger/log of all receivables and expenditures, and any accounting software and/or bookkeeping environment?
Possible Evidence: Review of each element
Do you have a chart of accounts that details your categories of expenditures and receivables?
Possible Evidence: Chart of accounts, accounting report demonstrating the use of the chart of accounts
Do you create an officially approved annual operating budget for the overall community/organization each year?
Possible Evidence: Annual budget developed by the community/organization that is approved by its governing body
Do you rectify your financial books at least quarterly with review and sign-off from a designated governing body representative(s)?
Possible Evidence: Signed quarterly reports or other evidence of successful review
Do you have written financial policies that are adequate for the size and complexity of your community/organization and that include explicit internal controls?
Possible Evidence: Accounting manual, policy documentation
Do you have an annual independent audit or review conducted of your financial statements?
Possible Evidence: Audit or Review documentation
Are there at least two, non-interdependent funding streams for the entity (e.g., memberships, subscriptions, services, grants, contracts, donations)?
Possible Evidence: Documented revenue streams
Do you know how to describe your legal and organizational identity (e.g., incorporation, taxable status, charity status, etc)?
Possible Evidence: Official documentation (e.g., incorporation documents, tax exemption letter, B-Corp certificate, host organization documents) as compared to self description
Do you clearly document your community/organization’s legal and organizational identit(ies) such that community members and customers/clients understand what the legal entity can do and who has authority to speak for it?
Possible Evidence: Disclosure via Website or other public forum, contractual language
Do you engage a legal advisor and/or lawyer to review all contracts (both those you prepare and those you sign)?
Possible Evidence: Legal representation
If you are not a free-standing organization but instead are hosted by another legal entity, do you transparently and publicly acknowledge this hosting relationship?
Possible Evidence: Disclosure via Website or other public forum, contractual language
If you are hosted by or housed within another entity, do you have a documented pathway for separation that can be enacted by your governance group?
Possible Evidence: Documented separation pathway and trigger events
Do you have a policy that describes how organizational ownership or responsibility may be transferred, and does this policy explicitly prohibit contracts/agreements to be reassigned without express permission from community stakeholders?
Possible Evidence: Documentation review
Do your governance documents include a legal commitment to maintaining your organization’s mission regardless of company ownership?
Possible Evidence: Documentation review
Do you have a succession strategy that identifies entities that will steward products and services if or when your company ceases to operate?
Possible Evidence: Succession documentation
Do you have a financial reserve set aside that is adequate to cover sunsetting costs?
Possible Evidence: Documented financial reserve and sunset budget
Do you have a plan documenting what happens when your company ceases to operate?
Possible Evidence: Documented sunset/dissolution plans
Do you document, test, and regularly update specific pathways for any user-owned or generated content to be migrated to another platform or service in the case of dissolution?
Possible Evidence: Agreements with similar platforms/services, migration strategies and pathways, contracts with users/members, testing documentation