Abstract

Our library has encountered a variety of challenges when supporting faculty through the tenure and promotion process. The Indiana University Purdue University Indianapolis (IUPUI) campus standards and processes, like that of many others, preferentially reward outcomes and impact over the process of inquiry. Additionally, a narrow range of peer-reviewed products is preferred—journal articles, books, conference proceedings, etc. Given the emphasis on reputation and impact in our campus standards, candidates are expected to demonstrate that their research has had an effect on the world beyond campus. Thus, the range of evidence used in dossiers often centers on funding and citation-based metrics, with other metrics considered as secondary. Over the past decade, the research metrics services provided by our library to faculty candidates has evolved significantly. We began by retrieving traditional bibliometrics and teaching others how to do so. As we repeatedly encountered gaps in the data at the level of individual faculty members, we adopted a more proactive stance in our support. We continually advocate for broader consideration of the types of products that are valued, the range of evidence used in dossiers, and the types of impact discussed in statements. In many cases, we use evidence from the publishing and informetric literature to corroborate individual experiences and advocate for change. As our campus implements a new Diversity, Equity, and Inclusion (DEI) pathway for promotion and tenure, we are challenged to adapt so that we can effectively support faculty who choose this pathway. In this commentary, we will discuss the points of intervention to proactively engage with scholars.

This essay describes the approach at IUPUI University Library for supporting faculty through the tenure and promotion process. Our work in this area developed out of traditional reference services and existing engagement with the IUPUI Office of Academic Affairs (OAA). When Heather Coates joined the library in 2011 as the Digital Scholarship & Data Management Librarian, there was a standing invitation from the OAA to provide a workshop on retrieving citation counts and associated information from citation databases such as Scopus and Web of Science. This workshop was part of an annual series designed for faculty seeking promotion and/or tenure (P&T). As a conversation about altmetrics emerged, librarians at IUPUI began to explore the world of metrics beyond citation databases and the literature of bibliometrics and research evaluation.

Over the past decade, we have observed how the scholarly publishing ecosystem and associated metrics fail to capture some of the most significant outcomes and impacts of research on our campus—changing professional practice, influencing local and state policy, and developing relationships with communities and increasing their access to resources. Despite increasing availability of informetrics, coverage of some types of scholarship such as highly-interdisciplinary and community-engaged scholarship remains uneven.

In conversation with scholars and administrators, we have gathered local evidence for the variety of forms and products of scholarship that are valued and celebrated across campus. For example, the evidence summaries produced for faculty candidates describe gaps in particular databases or aggregators for that individual. We have aggregated this information over the years to document which metrics sources are likely to have more complete coverage for an individual or department. However, interdisciplinary research frequently challenges these patterns. We use this knowledge and our position in the library to help them recognize that practices or products which might be unusual for their field or department are not unusual at the campus level. This awareness sometimes enables scholars to claim a broader range of products as significant in their case. In other scenarios, faculty whose work has a strong community- or public-facing component are empowered to claim credit for that work and its associated impacts. In many cases, our advocacy builds on existing work to update and/or reform the P&T processes, such as the IUPUI Faculty Learning Community for Public Scholarship. We strive to cultivate change in ways that respect the unique case and situation of each scholar, often by introducing or connecting them with others who have successfully navigated a similar path.

Most of our advocacy is centered around expanding the understanding of the available metrics and empowering scholars to take control of their own scholarly record through the promotion of open and equitable practices within the scholarly ecosystem. In particular, we encourage scholars to consider more open methods for distributing both their scholarship and expertise to relevant audiences, which can then generate valuable documentation of the types of impact they seek to demonstrate. By supporting them in using Open Access (OA), open-source, and community-controlled infrastructure, we hope to empower scholars to tell more diverse and authentic stories of impact.

Outreach

We prioritize reaching out to early-career scholars and pretenure faculty so that we can introduce proactive strategies to help them create and use a broader range of evidence in demonstrating a more comprehensive range of impact (Figure 1). In reaching out early, we are able to raise awareness of the limitations of bibliometric and altmetric evidence, break down the challenging process of generating evidence and documentation into feasible tasks, and encourage them to integrate documentation strategies into their scholarly processes.

We also promote existing open infrastructure for each of these tasks. For example, helping a new faculty member to create an ORCID in their first year can save them time when submitting a manuscript or applying for funding since data associated with an ORCID is interoperable. Combined with the deposit of publications and other scholarly products into our institutional repository, IUPUI ScholarWorks, they are effectively distributing their work and getting credit for it with little effort on their part. IUPUI ScholarWorks, an open access repository which is available to the public, provides metrics related to views, downloads, and the location of users. These metrics are frequently not available for conference presentations, posters, white papers, and other types of grey literature. While a faculty member could manage their own website to share their scholarship and generate similar metrics, many faculty choose not to spend their time in this way or do not have the technical skills to do so. Additionally, the metadata and preservation capacity of IUPUI ScholarWorks tends to make scholarly products more visible and durable than content hosted on a faculty-managed website. Given the 5–6 year tenure clock that is common at many institutions, these elements are crucial for early-career faculty to generate as much evidence as possible about the access and use of their scholarship. 

Impact metrics

Inside the boundaries of the traditional promotion and tenure process, value is typically placed on the journal impact factors (JIFs) and certain high-profile journals from commercial publishers. Certain disciplines, such as the humanities, might value smaller specialized journals. However, smaller publishers and specialized venues might not have the infrastructure to generate the same metrics as larger and well-resourced publishers.

Within the P&T process, value continues to be attributed to journals and presses (i.e., dissemination outlets). This becomes a challenge for librarians when consulting with faculty because what works for one scholar might not work for another. Placing value on journal metrics (e.g., JIF or CiteScore) over item-level metrics (e.g., downloads or citation counts) is a disservice to scholars; their process for selecting a venue for publication can stem from a variety of reasons driven by personal experience. Knowing this, our approach to promoting open and alternative ways for early-career scholars and pretenure faculty ensures that they have access to more item-level evidence (e.g., citations, downloads, etc.) that they can include in their P&T dossier.

While scholars are typically familiar with the idea of scholarly or academic impact, other types of impact are less familiar. In our experience supporting faculty, most enter their first academic position with little understanding of the evidence used for P&T. The clinical translational sciences movement introduced many academic health sciences researchers to the notion of “bench to bedside” or translational impact. That movement has been expanded and sustained on our campus through the work of the Center for Translating Research to Practice (TRiP). We have also found the Becker Library Model for Research Assessment and the Commonwealth Scientific and Industrial Research Organisation (CSIRO) Impact Evaluation Framework, tremendously useful for expanding the conversation. The Becker Model describes five areas of impact, along with a detailed list of indicators for each category of impact. We have used the Becker model as a prompt to ask researchers and administrators to deconstruct and challenge their notions about impact and evidence to demonstrate it. For example, during sessions with the IUPUI LIFT Mentoring Circles, we provide examples in which we map out the possible effects of a local scholar’s work across different types of impact - advancing knowledge, community benefit, translating research into practice, etc. Many faculty on our campus are both practicing professionals and researchers. We discuss the limitations of bibliometric and altmetric evidence derived from publications and point out that such metrics cannot demonstrate impact upon communities, much less the economy or policy. Other evidence is necessary to tell the full story.

Our Process

Each faculty case for P&T is idiosyncratic, so our support focuses on individual consultations. Consultations often begin with questions about the desired outcomes and impact of their work. Starting there helps librarians to gain valuable context about the bigger picture and the “why” underlying the scholar’s work as well as the connections they are already making between their scholarship and real-world effects. By introducing faculty to more formal approaches to research evaluation, such as the CSIRO Impact Evaluation Framework, we aim to empower scholars to tell a more complete and authentic story of their scholarship that is supported by robust documentation. In many cases, the act of discussing different types of impact prompts them to recall existing documentation which could be strong evidence for their case. During workshops and consultations, we use examples (Table 1) to prompt their consideration of these possibilities.

Table 1: Example mapping of research goal to different types of impact and the supporting evidence

Ensuring that the impacts and evidence presented in dossiers are aligned with local priorities and understood by reviewers requires a shared understanding of the limitations of current approaches and metrics. To that end, we have used evidence from the literature and tools such as the INORMS SCOPE model to engage administrators and reviewers in parallel conversations about bibliometrics, altmetrics, and types of impact. Often, these conversations develop from a consultation or workshop. A common example centers on the Journal Impact Factor (JIF) and the many ways in which it is not fit for use in evaluating individual scholars or publications. Motivated by common misconceptions which arise during consultations, we gathered evidence from the literature to write a brief about how use of the JIF can be detrimental to a scholar’s case. From the perspective of a reviewer, the JIF is not a good predictor for the number of citations any given article published in that journal will receive. While an “evaluation of dissemination outlets” is a required element in our campus P&T process, we have advocated for consideration of item-level metrics in addition to the evaluation of journals, books, presses, and other venues to encourage recognition of the broader context. Discussing the limitations and strengths of metrics as well as the assumptions built into institutional evaluation processes raises awareness of the gap between the local priorities and available evidence. In some cases, these conversations have resulted in changes to school or departmental guidelines, but changes in process and attitudes are more difficult to measure.

Scholars who publish in specialized or highly interdisciplinary journals cannot be expected to accrue the same number of citations as those who publish in more general journals or fields. Many people recognize that comparing citations across disciplines can lead to inequity but may not be aware of the options for interpreting metrics in the appropriate context. We encourage use of field normalized citation metrics (e.g., Field Weight Citation Index, Field Citation Ratio) when available and encourage reviewers to consider differences in authorship and publication rates when completing an evaluation. Field normalized metrics allow for comparison across subject areas, which is helpful for administrators and campus reviewers. Over time, we have seen increasing comfort with and acceptance of a broader array of metrics and types of impact.

Public & Community-engaged Scholarship

Scholarship that engages the public and community partners often generates varying forms of knowledge making and dissemination practices, not centered around journal articles, books, book chapters, of conference proceedings. This work tends to produce intangible outcomes, such as developing community partnerships to validate a research question or involvement in a local or regional dialogue related to the work. With some planning and support, these outcomes can be captured and counted towards cases of promotion and tenure. Campus leadership has demonstrated willingness to recognize this type of work through the creation of the Integrative DEI case. This creates an opportunity for us to better serve scholars whose work is not “binned” into research, teaching, and service. Thus, this case allows the interrelated nature of DEI work to be recognized. Despite these changes, the campus P&T process still heavily emphasizes and values products over process; conveying the importance of integrative scholarship demands more of faculty pursuing this case since project outcomes are not automatically captured by existing digital infrastructure or tangible products may not be a key effort.

Faculty who largely focus their time on cultivating relationships and lasting partnerships are faced with the tension between doing the work and recording the impact of their work so it can be visible and valued. Our current methods for consulting with scholars whose work is more interdisciplinary and/or DEI focused tends to require manual methods of capturing evidence to demonstrate the value of their work outside of peer-reviewed dissemination methods. For example, publicly engaged research often relates to the work of interdisciplinary and/or DEI scholars. This type of research is subject to public audiences and peer review and requires a large investment in time to cultivate relationship with community. Trying to quantify the labor associated with engaging with communities is a task that the university and its departments and schools have been actively trying to capture. By establishing Resources to support publicly engaged scholar-practitioners, Public & Engaged Scholarship Review Committee (PESRC), ENGAGE! Journal, and The Community Engaged Research Group (CERG) the university has taken steps to support scholarship that falls outside the traditional publishing ecosystem. Recently implemented systems such as The Collaboratory may also provide a low-barrier resource for faculty to document integrative work and track their engagement with community partners.

The Integrative DEI case is designed to provide the scholar with more flexibility with respect to the products and types of impact that are recognized. This case type adds value to the work that is already happening and it establishes a pathway for scholars to get credit for all the work they do. Generating and capturing effective evidence of interdisciplinary and/or DEI scholarship still remains an area of challenge for library support since current methods remain largely manual. One of the most effective methods we use when working with interdisciplinary and/or DEI scholars is advanced web searching. Oftentimes this results in some media mentions or discovery that a result of a project created change within a profession or informed policy, though the responsibility largely remains on the scholar to keep track of evidence related to their work. This could range from tracking number of participants that attend a program, soliciting evaluations or input from community partners, adoption of course elements or syllabi by other professionals, and recording direct engagement hours with a community. This is no easy task and is an activity that detracts from what scholars value most—doing the work.

Next Steps

As we move forward, we are prioritizing scholars who continue to operate in a system that does not fully capture or reward their efforts. By prioritizing scholars who are underserved and underrepresented in the current publishing system, we are helping to ensure that value is given to the process of inquiry as well as products that are less visible. By increasing our support for these scholars, our aim is to contribute to a campus culture which values this work. Such a culture will ultimately benefit all IUPUI scholars, including students who are interested in these types of scholarship. Of course, we recognize that we are a small part of the larger community of support. Recently, as an extension of our outreach, librarians within the Center for Digital Scholarship have begun cross-training colleagues in Educational Services to expand our service capacity and leverage the expertise of subject liaisons. Our colleagues who serve as subject librarians have existing expertise and relationships with their departments, schools, areas, and units across campus. This provides great capacity to expand the conversations around research evaluation processes and supporting evidence used in P&T. Despite this extension of outreach and changes to campus P&T guidelines and standards, we anticipate many opportunities to support schools and departments in navigating new types and sources of scholarship and evidence. Conversations continue with administrators, schools, and campus leaders to understand the opportunities provided in this new pathway. Ultimately, we seek to contribute to a campus culture in which faculty can tell an authentic story of their scholarship while successfully obtaining promotion and/or tenure.