Abstract

Criteria and guidelines for review, promotion, and tenure (RPT) processes form the bedrock of institutional and departmental policies, and are a major driver of faculty behavior, influencing the time faculty spend on different activities like outreach, publishing practices, and more. However, research shows that many RPT guidelines emphasize quantity over quality when evaluating research and teaching, and favor bibliometrics over qualitative measures of broader impact. RPT processes rarely explicitly recognize or reward the various public dimensions of faculty work (e.g., outreach, research sharing, science communication), or, when they do, relegate them to the service category, which is undervalued and often falls heavily on women and underrepresented groups. There is a need to correct this mismatch between institutional missions or values—often focused on aspects like community engagement, equity, diversity, and inclusion, or public good—and the behaviors that are rewarded in academic assessments. We describe recent efforts to promote RPT reform and realign institutional incentives using a values-based approach, including an overview of workshops we ran at the 2023 Council of Graduate Departments of Psychology (COGDOP) Annual Meeting, the Association for Psychological Science (APS) Annual Convention, and the American Anthropological Association (AAA) Department Leaders Summer Institute. These workshops were designed to guide participants through the process of brainstorming what values are important to them as departments, institutions, or more broadly as disciplines, and which faculty behaviors might embody these values and could be considered in RPT evaluations. We discuss how similar activities could promote broader culture change.

Corresponding author: Erin C. McKiernan, [email protected] or [email protected]

Introduction

The past decade has seen growth in national and international efforts to reform academic assessment [1], including but not limited to the San Francisco Declaration on Research Assessment (DORA), the Leiden Manifesto for Research Metrics [2], the Hong Kong Principles for assessing researchers [3], the National Academies of Sciences, Engineering, and Medicine (NASEM) Roundtable on Aligning Incentives for Open Scholarship, and the Coalition for Advancing Research Assessment (CoARA). This movement has been driven by increasing recognition that our current evaluation systems are ineffective at best, not properly incentivizing practices such as responsible [4] or reproducible research [5], and discriminatory at worst, disadvantaging young scholars, women [6], and underrepresented groups like faculty of color [7].

Review, promotion, and tenure (RPT) expectations or requirements, both at institutional and departmental levels, can be strong drivers of faculty behavior. Surveys show that faculty express concerns about achieving tenure and promotion, and change their behaviors accordingly by adjusting time spent on teaching, research, or service activities [8] and by making publishing decisions based on both perceived and formal RPT criteria [9]. Research also shows that there is a mismatch between the values institutions purport to uphold, as expressed in their mission statements, and the behaviors they explicitly recognize and reward in RPT evaluations. For example, many universities tout the importance of ‘disseminating knowledge’ [10], and ‘serving the local area’ or ‘civic duty/service’ [11]. However, while the words ‘community’ and ‘public’ are used frequently within RPT guidelines, the public dimensions of faculty work (e.g. outreach, research sharing, science communication) are either not recognized or are relegated to the service category [12]. Service is often both undervalued in academic evaluations [8], and falls disproportionately on the shoulders of women [13] and faculty of color [14]. This is first and foremost an equity problem, but it is also a mission problem. Through misaligned incentives, universities may be losing—or worse, actively driving out—the very people who serve their communities and help institutions meet their societal missions [15]. 

Interventions and strategies

Driven by a motivation to help spur change in academic assessment, this year we (the current authors) ran a series of workshops, primarily for department chairs. Our rationale for targeting this audience was two-fold. First, as academics, chairs possess a deep understanding of the norms and standards of their disciplines, as well as the goals and culture of their departments. Additionally, their administrative roles make them familiar with institutional policies and procedures. This creates fertile conditions for the socialization of potential policy revisions, both upward (to university leadership) and outward (to fellow faculty in the department). We note that there are also complementary efforts to drive RPT reform at the presidential level (e.g., the Higher Education Leadership Initiative for Open Scholarship, HELIOS), the faculty senate level (e.g., University of Vermont [16]), and the early-career researcher level (e.g. Young SiT [17]). Our hope is that future coordination across these levels will encourage scale.

We held four workshops over the course of 2023: two in-person workshops at the 2023 Council of Graduate Departments of Psychology (COGDOP) Annual Meeting in February; one in-person workshop at the Association for Psychological Science (APS) Annual Convention in May; and one online workshop as part of the American Anthropological Association (AAA) Department Leaders Summer Institute in June. The overall perspective of these workshops was inspired by prior work by members of our team on studying [18] and re-aligning incentives [19], as well as the work of the Humane Metrics Initiative (HuMetricsHSS), all of which seek to reframe evaluation processes and promote a values-based approach [20]. The overall strategy of such an approach is to first identify the values important to the institution, department, discipline, or other relevant communities, then identify key faculty behaviors that align with those values, and finally to center those behaviors in developing an evaluation framework. 

For all workshops, we implemented a mixture of short talks, guided discussions, and breakout groups (slide decks are available at https://doi.org/10.17605/OSF.IO/G92U4). We began with brief presentations from our team to set the stage by highlighting research that shows the mismatch between institutional or disciplinary values and what is currently rewarded, problems with metrics such as Journal Impact Factor, and existing inequities in academic evaluation (i.e. ‘the system is broken’). Next, we placed this reform work in a broader context by talking briefly about ongoing U.S. national and international reform efforts (i.e. ‘you are not alone’). Then, we showed an example of success from one of the current authors (MRD) who, as Chair of the Department of Psychology at the University of Maryland, was able to push forward a broad set of reforms to their promotion and tenure guidelines in 2022 [21] (i.e. ‘you can do this’). Finally, we talked about disciplinary considerations, and the idea that departments can customize both the values they want to focus on and the behaviors or indicators to fit their needs (i.e. ‘enabling multiple pathways to success’). This led us into the breakouts, where participants discussed in small groups what each value meant to them, and how they might measure these values via behaviors (i.e. ‘the embodiment of that value’) in academic assessments. These discussions were designed to flip perspective from focusing on the traditional categories of research, teaching, and service, to thinking more broadly about how a certain value might be represented in different ways across the full range of faculty activities.

For guided discussions and breakouts, we took a subset of the proposed values from HuMetricsHSS, focusing on the ones we deemed most relevant for both the discipline and, where applicable, the underlying theme of the conference or session. For example, our two COGDOP workshops were embedded into the conference program such that they followed after different plenary speakers and were meant to continue on a theme. The first day centered on faculty recruitment and retention through an equity lens. Therefore, for this first workshop, we chose to focus the breakout discussions on values such as accessibility, diversity, equity, and inclusivity. The second day, there was an emphasis on U.S. federal policies, especially new requirements emerging under the 2022 OSTP memo on “Ensuring Free, Immediate, and Equitable Access to Federally Funded Research” [22]. Since a key aspect of the memo was research integrity, we focused the breakout discussions in this second session on values such as quality, rigor, and reproducibility. Though we were not involved in organizing the overall agenda of the annual COGDOP meeting, in retrospect, we believe that embedding RPT reform discussions into larger ongoing conversations of interest to the discipline, is a good strategy to emphasize their timeliness and relevance.

During the AAA and APS workshops, we were again intentional about asking participants what the values of accessibilty, equity, and community engagement meant to their specific disciplines. For example, anthropologists highlighted the importance of recognizing disparate knowledge systems, and establishing non-exploitative and truly collaborative relationships with the communities they study. In this context, we talked about behaviors such as involving tribal authorities in research design, co-authoring scholarship with local communities, and making sure the outputs of scholarship were publicly available but shared with community rights and autonomy in mind (e.g., according to the CARE Principles for Indigenous Data Governance). We believe it was key to the success of these events that they were convened by leading professional societies in their respective disciplines, who are also members of the Alliance for Open Scholarship (All4OS) and have committed to developing discipline-specific norms for sharing scholarship and improving incentives (e.g. see recent statements from the Academic Data Science Alliance and the American Society for Microbiology). The signaling that their societies are engaged in this work surely reinforced the timeliness and viability of RPT reform for workshop participants.

Findings

Overall, the feedback from participants of all four workshops was positive, and we were pleased at the depth of discussions. In general, department chairs recognized the many problems with the current evaluation systems, including the widespread misuse of bibliometrics, and agreed there is a need for better incentives that more accurately capture and assess faculty activities in line with disciplinary and institutional values, and also produce more equitable outcomes. Department chairs seemed eager to collectively brainstorm how this could be done, and some mentioned that they would take these ideas back to their departments for further consideration. In addition, following the meeting, about a dozen department chairs from COGDOP expressed interest in engaging in collective action to enact broader change.  

We did notice that some values represented more of a challenge than others. For example, while breakout groups generated many ideas for how to define and measure faculty behaviors associated with accessibility or rigor, they struggled to do the same with values such as inclusivity. In addition, several values were highly discipline specific. Whereas psychology chairs had several ideas on how to measure reproducibility, for example, anthropology chairs discussed how this value might be differently defined or not necessarily applicable in certain types of research. There was also overwhelming consensus that the definitions for certain values, like quality, are not always clear and may depend strongly on disciplinary standards or research culture. Again, we suggest that this is one area where professional societies (e.g., All4OS members), as leaders in their respective academic fields, could help provide disciplinary norms and guidance. 

Next steps

As we move forward with these efforts, our goal is to triangulate among engaged professional societies, institutions, and departments via a HELIOS-All4OS partnership. We want to test the idea that buy-in from university leadership (as observed by HELIOS membership), societal guidance on norms and good practices (through All4OS member statements), and mission-driven department chairs/faculty will make the work of reforming RPT both easier in which to engage and more rapidly actionable.

However, we recognize that comprehensive cultural change requires thinking not just about tenure and promotion, but the full embodiment of the university mission and the incentive system that underlies that mission. This will mean reforming how we conduct faculty searches and hiring, the basis for annual or merit reviews, criteria for institutional funding opportunities or awards, and other forms of recognition. In other words, culture change will require continuity and consistency throughout the evaluative system (i.e. mutually reinforcing vectors). Cross-sector initiatives such as the NASEM Roundtable on Aligning Incentives for Open Scholarship, which the ORFG helps to coordinate, are an example of efforts to catalyze, harmonize, and scale culture change efforts.

While such change may seem daunting, the benefits to be realized from improving academic incentives are many: increased public trust in research through greater transparency, more rapid progress in solving societal problems through better reporting and sharing, democratization of knowledge through both greater access to information and increased participation of traditionally underrepresented researchers, and a more equitable system where everyone has opportunities to both learn and contribute. 

References

1. Pain, E. (2023). How academia is exploring new approaches for evaluating researchers. Science Careers  https://doi.org/10.1126/science.caredit.adj4869. 

2. Hicks, D., Wouters, P., Waltman, L., de Rijcke, S. & Rafols, I. (2015). Bibliometrics: The Leiden Manifesto for research metrics. Nature 520, 429–431 https://doi.org/10.1038/520429a

3. Moher, D., Bouter, L., Kleinert, S., Glasziou, P., Sham, M.H., Barbour, V., Coriat A-M., Foeger, N., & Dirnagl, U. (2020). The Hong Kong Principles for assessing researchers: Fostering research integrity. PLoS Biology 18(7): e3000737. https://doi.org/10.1371/journal.pbio.3000737

4. Pontika, N., Klebel, T., Correia, A., Metzler, H., Knoth, P., Ross-Hellauer, T. (2022). Indicators of research quality, quantity, openness, and responsibility in institutional review, promotion, and tenure policies across seven countries. Quantitative Science Studies 3 (4): 888–911. https://doi.org/10.1162/qss_a_00224 

5. Edwards, M.A. & Roy, S. (2017). Academic Research in the 21st Century: Maintaining Scientific Integrity in a Climate of Perverse Incentives and Hypercompetition. Environmental Engineering Science, 34(1) https://doi.org/10.1089/ees.2016.0223

6. Murphy, M., Callander, J.K., Dohan, D., Grandis, J.R. (2021). Women’s Experiences of Promotion and Tenure in Academic Medicine and Potential Implications for Gender Disparities in Career Advancement: A Qualitative Analysis. JAMA Network Open 4(9):e2125843. https://doi.org/10.1001/jamanetworkopen.2021.25843  

7. American Psychological Association, APA Task Force on Inequities in Academic Tenure and Promotion. (2023). APA Task Force Report on Promotion, Tenure and Retention of Faculty of Color in Psychology. Retrieved from https://www.apa.org/pubs/reports/inequities-academic-tenure-promotion.pdf

8. Harley, D., Acord, S. K, Earl-Novell, S., Lawrence, S., & King, C. (2010). Assessing the Future Landscape of Scholarly Communication: An Exploration of Faculty Values and Needs in Seven Disciplines. eScholarship Publishing. UC Berkeley: Center for Studies in Higher Education. Retrieved from https://escholarship.org/uc/item/15x7385g

9. Niles, M.T., Schimanski, L.A., McKiernan, E.C., Alperin, J.P. (2020). Why we publish where we do: Faculty publishing values and their relationship to review, promotion and tenure expectations. PLoS ONE 15(3): e0228914. https://doi.org/10.1371/journal.pone.0228914 

10. Bayrak, T. (2020). A content analysis of top-ranked universities’ mission statements from five global regions. International Journal of Educational Development. 72: 102130. https://doi.org/10.1016/j.ijedudev.2019.102130 

11. Morphew, C.C. & Hartley, M. (2006). Mission Statements: A Thematic Analysis of Rhetoric across Institutional Type, The Journal of Higher Education, 77:3, 456-471, https://doi.org/10.1080/00221546.2006.11778934 

12.  Alperin, J.P., Muñoz Nieves, C. Schimanski, L.A., Fischman, G.E., Niles, M.T. & McKiernan, E.C. (2019). How significant are the public dimensions of faculty work in review, promotion and tenure documents? eLife 8:e42254.https://doi.org/10.7554/eLife.42254 

13. Guarino, C.M., Borden, V.M.H. Faculty Service Loads and Gender: Are Women Taking Care of the Academic Family?. Research in Higher Education 58, 672–694 (2017). https://doi.org/10.1007/s11162-017-9454-2 

14. Domingo, C. R., Gerber, N. C., Harris, D., Mamo, L., Pasion, S. G., Rebanal, R. D., & Rosser, S. V. (2022). More service or more advancement: Institutional barriers to academic success for women and women of color faculty at a large public comprehensive minority-serving state university. Journal of Diversity in Higher Education, 15(3), 365–379. https://doi.org/10.1037/dhe0000292 

15. Jimenez, M.F., Laverty, T.M., Bombaci, S.P., Wilkins, K., Bennett, D.E. & Pejchar, L. (2019). Underrepresented faculty play a disproportionate role in advancing diversity and inclusion. Nature Ecology & Evolution 3, 1030–1033 https://doi.org/10.1038/s41559-019-0911-5 

16. University of Vermont Faculty Senate Research, Scholarship, and the Creative Arts Committee (2023). Resolution on Open Access and Open Science. Retrieved from: https://drive.google.com/file/d/1MqyjVGRRMnTCsVS48dESYCxfc0DEbcA3/view 

17. Algra, A. Koopman, I., & Snoek, R. (2020). How young researchers can re-shape the evaluation of their work. Nature News. https://www.nature.com/nature-index/news/how-young-researchers-can-re-shape-research-evaluation-universities 

18. Aguiar, O. Rethinking Research Assessment for the Greater Good: Findings from the RPT Project. ScholCommLab Blog https://www.scholcommlab.ca/2022/05/04/findings-from-the-rpt-project/ 

19. Dougherty, M. R., Slevc, L. R., & Grand, J. A. (2019). Making Research Evaluation More Transparent: Aligning Research Philosophy, Institutional Values, and Reporting. Perspectives on Psychological Science, 14(3), 361-375. https://doi.org/10.1177/1745691618810693

20. Agate N., Long, C.P., Russell, B., Kennison, R., Weber, P., Sacchi, S., Rhody, J., & Thornton Dill, B. (2022). Walking the Talk: Toward a Values-Aligned Academy. Humanities Commons. https://doi.org/10.17613/06sf-ad45 

21. University of Maryland, Department of Psychology. Department Policies and Initiatives. Retrieved from https://psyc.umd.edu/about-us/department-policies-and-initiatives 

22. Nelson, A. via U.S. Office of Science and Technology Policy (2022). Ensuring Free, Immediate, and Equitable Access to Federally Funded Research. Retrieved from https://www.whitehouse.gov/wp-content/uploads/2022/08/08-2022-OSTP-Public-access-Memo.pdf 

    
    

This work is licensed under a Creative Commons Attribution 4.0 International License.

Please attribute the authors and the publication venue when citing this work.