Holistic Review Toolkit

II: Why AMIGA?

Across the U.S. higher education administrators and faculty are leading initiatives that focus on expanding faculty ranks with individuals from a wider range of different perspectives and backgrounds. Although graduate study is a significant part of the pipeline to becoming a faculty member, graduate education and admissions specifically, have received little attention.

In 2020 social inequality and access to resources manifested nationally with the realization that people of color are more likely to catch COVID-19. Moreover, racial injustice exposed the extent that black and brown people experience policing and other forms of institutional racism. Students, staff and faculty alike have long pondered, "What can one do to foster social justice in academia?"

Demographic forecasts detail a significantly changing US population by 2050. In California, public colleges and universities are already enrolling substantial numbers of college students who have been underrepresented in higher education. The Alliance for Multi-campus, Inclusive Graduate Admissions (AMIGA) project focuses on supporting holistic graduate admissions to increase access to graduate education and to develop more opportunities to becoming faculty.

To that end, the AMIGA project supports equitable holistic review methods for graduate admissions that potentially open access to graduate school for students from a wider variety of backgrounds, such as those who are first generation college or have experienced significant economic hardship. It is only with attention to equitable methods that institutions can select faculty from a pool of candidates with increasingly different backgrounds, perspectives and experiences while preparing the brightest minds to tackle society’s most intractable problems.

III. The University of California Commitment to Diversity

The mission of the University of California (UC) is to serve society as a center of higher learning, providing long-term societal benefits through transmitting advanced knowledge, discovering new knowledge, and functioning as an active working repository of organized knowledge. The UC Regents Policy 4400 recognizes that, “The diversity of the people of California has been the source of innovative ideas and creative accomplishments throughout the state’s history into the present,” and that UC’s core mission, “is to serve the interests of the State of California, it must seek to achieve diversity, inclusion, and accessibility among its student bodies and its employees.” The UC Statement on Diversity defines diversity as referring, “To the variety of personal experiences, values, and worldviews that arise from differences of culture and circumstance." Employing equitable holistic review practices serves to remove barriers to the recruitment, retention, and advancement of talented students and to support cohorts that are more likely to reflect the diversity of the state of California.

The UC campuses must comply with the California Constitution which prohibits consideration of race, ethnicity and gender in the public university, however it is permissible to consider background and life experiences that shaped academic or professional paths and/or that influenced an applicant’s decision to pursue a graduate degree. The UC Guidelines For Addressing Race and Gender Equity in Academic Programs in Compliance with Proposition 209 provide a roadmap to consider. Departments can take into account a wide variety of indicators in the evaluation of candidates, including potential for leadership and significant life experiences, rather than admitting students solely on a narrow range of traditional indicators." A greater emphasis on applicant background and experiences will lead to equitable selection practices contributing not only to student success, but the academic excellence of student cohorts.

IV. Developing a Rubric

Equitable graduate admissions review requires fair, clear and consistent evaluation across committee members. It can be accomplished broadly through faculty admission training and the use of a scoring template (rubric) that admissions committee members develop and norm prior to application review. Discussing and finalizing selection criteria and scoring method/s among colleagues is critical to achieving consensus that supports validity and reliability. Consider the following issues to ensure the development of an equitable and valid rubric for your graduate program.

• Discipline specific skills – In context, weigh whether an applicant's ability to take specific courses may be affected by other factors other than choice (e.g., a particular public institution may not offer Latin or Greek language study).

• GPA – Contextualize the GPA by supporting evaluation with factors that may have been influential, such as a change of majors, personal circumstances or limited access to particular coursework, e.g., a non-research intensive college or university may not offer the depth of course offerings as a research intensive institution.

• Recommendation letters – Consider the questions that faculty respond to for recommendation letters. Be cognizant that unfamiliar faculty, programs or institutions do not signal a less talented or prepared applicant.

• Research experience – Consider the type and depth of research experience in the context of the undergraduate institution, availability of graduate preparation programs, and necessity to work extensively to pay for college expenses.

• Personal background – Consider the background and life experiences that have shaped an applicant’s academic and/or professional path and influenced their decision to pursue a graduate degree.

• Standardized test scores – Consider the equitable use of scores, such as the GRE, GMAT and others in context of other information. If temporarily or permanently suspended, how might corresponding skills be evaluated from the current application information?

V: Promising Practices for Holistic Review

1. Add comprehensive graduate application information to the department website.

   A successful holistic admission cycle begins with in-depth graduate program and application information on program websites that do not assume every prospective graduate student is knowledgeable about applying or preparing for graduate school. Prior to the admissions cycle, review the graduate program website for equitable and comprehensive practices (e.g., clear expectations for writing sample, requirements for specific courses or particular level of math skills) and make changes accordingly.

2. Review online application content.

   Review the electronic application to identify the information required from each applicant. Work with the Graduate Division to customize additional fields for a particular graduate program.

3. Document existing practices.

   Consider the graduate admissions process that occurred in the preceding admissions cycle. Ask the previous year's graduate admissions chair to record and share the admissions process with the new admissions chair so that faculty will have a starting point for the present cycle.

4. Frame faculty discussions.

   Among program faculty and admissions committee members discuss the connection between graduate admission decisions, the future of the field and the academy as a whole. Research on the so-called “pipeline problem” may be generative. However, these discussions must follow governing guidelines and policies for a particular institution, e.g., in California, these discussions cannot include race/ethnicity, gender and national origin.

1. Assemble admissions committees early.

   In order for admissions committee members to participate in pre-admissions discussions and take advantage of faculty development opportunities, graduate program leadership should identify the new or continuing graduate admissions chairperson and admissions committee members by the start of the fall quarter or semester.

2. EXPAND VARIETY of admissions committee.

   Depending on the number of applications, consider involving more faculty, and possibly advanced graduate students, in application review. Advanced graduate students will acquire essential professional development skills through participation. Also, include faculty and graduate students from a wide variety of backgrounds and experiences on the committee.

3. Identify a vice chair of admissions.

   If the continuing admission chair is in their final year of graduate admissions service, securing a vice chair to support and learn from the current admissions chair will facilitate a smooth transition of established holistic review practices in the next admissions cycle.

4. Review admissions data.

   Admissions committees should review graduate program data for applications, enrollments and completions for previous years to understand past outcomes and trends and to set goals for the current admission cycle (e.g., updating admissions criteria, developing a rubric).

5. Review admissions resources.

   Review websites to identify broad and specific tools and information that support equity in graduate admissions.

6. Offer holistic review training.

   Faculty may not know what holistic review entails. Strongly encourage admissions committee faculty to participate in holistic review training provided by the university or led by knowledgeable colleagues.

7. Utilize a SCORING TEMPLATE.

   With faculty colleagues, develop a rubric (scoring template) for the graduate admissions process. Identify key applicant preparation, interests, and experiences and determine how they will be evaluated (e.g., high, medium, low). They could include discipline specific skills, research experience, coursework preparation, or personal background and experiences. The admissions committee will apply the rubric to evaluate applicants' materials. Consider using more than one rubric for different parts of the admission process, e.g., a separate rubric for interviews.

1. Review commonly used terms.

   Discuss terms that are commonly used to understand an applicant’s background (e.g., first generation college or economic disadvantage), to establish common understandings across faculty who may otherwise start with distinctly different reference bases.

2. Review application fields.

   Locating and triangulating applicant information (using multiple sources of application data to potentially support an applicant's background) both reduces review time and supports identifying key skills and backgrounds. Reviewers familiar with the application can, therefore, move quickly from one page of the application to another. Also, knowing where to find specific information supports equitable practices by reviewing the full file (e.g., taking into consideration an applicant's first generation status or by understanding why an applicant had limited research experiences by reviewing the résumé that demonstrated extensive work experience to pay for college).

3. Focus on skills, not test scores.

   Given the temporary or permanent waiver of standardized test scores, identify the representative skills formerly associated with test scores and determine alternative information to evaluate those skills.

4. norm the review process.

   Prior to reviewing current applications, meet to confirm criteria among admissions and broader program faculty. Norm the review process by individually employing the developed rubric, guidelines for its use and using sample applications from previous years for practice. Gather to discuss individual scores, negotiate differences and finalize methods. When norming is used in admissions review, it results in a more consistent and transparent review process across multiple reviewers.

5. Meet after initial review.

   Plan for the committee to meet following individual scoring to discuss top applicants and to reach consensus. Committee members can share impressions, scores of individual applicants and practice the give and take that accompanies mutual decision making.

VI. Taking Stock

The following items assist both the admissions chair and committee members to form shared knowledge and procedures with which to implement holistic review. By taking stock of current admissions practices, the committee can then assess how holistic review methods can be tailored for implementing a successful and equitable admissions process.

• In the previous cycle, were there written procedures and instructions that guided the admissions process?

• Were there any changes recommended to the admissions process that support equitable practices? Outline them and think about the process of implementing those changes.

• What are the official and unofficial policies regarding the use of standardized test scores?

• Identify the level of admissions experience of each committee member. Is a committee member experienced, a newcomer or a mix of both? What admissions training has each committee member participated in?

• How many faculty members participated in last year’s application review?

• How is the workload distributed among members of the committee?

• Outline the process of how the committee decides who to admit. Are there specific predefined criteria the committee considers when admitting graduate students?

• What were the instructions given to the admissions committee previously? How were the admissions instructions disseminated?

• If you previously participated in the admission process, are there changes you would recommend be implemented?

• What are the criteria that are used to review applicant files? Do these address the competencies needed to prepare and succeed in your graduate program?

• Based upon the established selection criteria where will you look for this information within an application?

• Are you up to speed on current knowledge about graduate admissions, such as how to norm a scoring template/rubric?

• Review the admissions instructions and procedures shared by the chair. Are you comfortable with them or would you like additional information, such as implicit bias modules or background on equitable graduate admission methods?

• Do the admission instructions sufficiently prepare committee members to review applications equitably? Consider how they may and may not prepare you. What else is needed?

VII. Faculty Development

Although it may be challenging to know where to begin when implementing an equitable and inclusive admissions process, extending one's basic knowledge is a good starting place.

UC REGENTS POLICY

• UC Regents Policy 4400

Holistic Review

• Holistic Review in Graduate Admissions

Implicit Bias

• The Influence of Unconscious Bias

  Noncognitive Measures

• Noncognitive Measures for Higher Education Admissions

• The Role of Noncognitive Constructs and Other Background Variables in Graduate Education

Rubrics

• An Effective Rubric Norming Process

• IGEN and C-CIDE Rubric Design & Use

VIII. Recommended Reading

• Guterl, F. (2014, October 1). Diversity in Science: Why It Is Essential for Excellence. Retrieved October 23, 2019, from https://www.scientificamerican.com/article/diversity-in-science-why-it-is-essential-for-excellence/.

• Michel, R. S., Belur, V., Naemi, B. and Kell, H. J. (December 2019). Graduate Admissions Practices: A targeted review of the literature. ETS Research Report-19-33.

• Posselt, J. R. (2014). Toward inclusive excellence in graduate education: Constructing merit and diversity in PhD admissions. American Journal of Education, 120(4), 481-514.

• Posselt, J. R. (2016). Inside graduate admissions: Merit, diversity, and faculty gatekeeping. Harvard University Press.

• University of California. (2016). Guidelines For Addressing Race and Gender Equity in Academic Programs in Compliance with Proposition 209. Retrieved from https://www.ucop.edu/general-counsel/_files/guidelines-equity.pdf

• Fine, E., & Handelsman, J. (2012). Reviewing Applicants: Research on Bias and Assumptions. Women in Science & Engineering Leadership Institute University of Wisconsin-Madison.

• Milkman, K. L., Akinola, M., & Chugh, D. (2015). What happens before? A field experiment exploring how pay and representation differentially shape bias on the pathway into organizations. Journal of Applied Psychology, 100(6), 1678–1712. doi: 10.1037/apl0000022

• Moule, J. (2009). Understanding Unconscious Bias and Unintentional Racism. The Phi Delta Kappan, 90(5), 320-326. Retrieved from, http://www.jstor.org/stable/20446107

• Woo, S. E., LeBreton, J. M., Keith, M. G., and Tay, L. (2023). Bias, fairness, and validity in graduate-school admissions: A psychometric perspective. Perspectives on Psychological Science, 18(1), 3-31.

• Gardner, S. K. (2013). The Challenges of First-Generation Doctoral Students. New Directions for Higher Education, 2013(163), 43–54. doi: 10.1002/he.20064

• Miller, C., & Stassun, K. (2014, June 11). A Test that Fails. Retrieved from https://www.nature.com/naturejobs/science/articles/10.1038/nj7504-303a.

• Moneta-Koehler, L., Brown, A. M., Petrie, K. A., Evans, B. J., & Chalkley, R. (2017). The Limitations of the GRE in Predicting Success in Biomedical Graduate School. Plos One, 12(1). doi: 10.1371/journal.pone.0166742

• Newman, D. A., Tang, C., Song, Q. C., & Wee, S. (2022). Dropping the GRE, keeping the GRE, or GRE-optional admissions? Considering tradeoffs and fairness. International Journal of Testing, 22(1), 43-71.

• Sedlacek, W. E. (2004) Why we should use noncognitive variables with graduate and professional students. The Advisor: The Journal of the National Association of Advisors for the Health Professions. 24 (2), 32-39.

• Kent, J. D., & McCarthy, M. T. (2016). Holistic Review in Graduate Admissions: A Report from the Council of Graduate Schools. Council of Graduate Schools. Retrieved from, https://cgsnet.org/ckfinder/userfiles/files/CGS_HolisticReview_final_web.pdf

• Michel, R. S., Belur, V., Naemi, B. and Kell, H. J. (December 2019). Graduate Admissions Practices: A targeted review of the literature. ETS Research Report-19-33.

IX. Project Contacts

The AMIGA Project was generously funded by the Andrew W. Mellon Foundation. The success of the Project was based on graduate faculty and Graduate Division collaborations that include humanities and humanistic social science fields at both UC Davis and UCLA. Dr. Julie Posselt (USC), project assessment lead and advisor, also contributed invaluable advice and input. Moreover, staff at both UC Davis and UCLA provided critical support and vision. Finally, the AMIGA Project supported multiple graduate students that offered their talent, insights and time to all aspects of the Project. ~ Josephine Moreno