Getting To Know Your Applicants and Learners By Using Situational Judgment Tests (SJTs)
An SJT can provide some insights that can't be easily measured by other methods
by Lauren Day-Page, PharmD, CCP, Clinical Pharmacy Practitioner, G. V. (Sonny) Montgomery VAMC
Have you ever reviewed applicants for admission to a program or job and wondered who they were beyond the CV? What if I told you there was a method of testing that would allow you to obtain some insight into their social skills, such as critical thinking, teamwork, conflict management, interpersonal skills, cultural awareness, and leadership competencies? All of these attributes can be assessed using situational judgment tests (SJTs).1
Situational judgment tests (SJTs) have been used in employment testing and military intelligence for almost a century. They are sometimes used to assess personnel selection and to make promotion decisions.2 In the healthcare setting, this method has been used to evaluate attributes such as empathy, adaptability, integrity, and collaboration.3 The examinee is given a scenario and instructed to consider several potential responses and then rate or rank them based on the appropriateness of the responses. The score on each test item is determined based on how closely the examinee’s selections correspond to a rubric established by subject matter experts.2 This evaluation tool has advantages and disadvantages depending on how it will be used.
The basic components of SJT include a realistic scenario with several potential responses to the scenario (See Figure 1). Responses are typically where the variability in scores or a person’s behaviors may become evident. Examinees are evaluated based on what they “would do” (rather than what they “should do”), and the responses typically show a strong correlation with personality constructs. Should-do instructions assess examinees’ ability to apply knowledge, whereas would-do instructions assess examinees’ behavioral tendencies.2 Respondents can rank, rate, or select the most/least/best/worst responses. The rank format requires examinees to rank-order response options from most to least effective.2 Rate format instructs test takers to rate each response option, typically with a number corresponding to the appropriateness of the response (e.g., 1=very appropriate, 2=somewhat appropriate, 3=somewhat inappropriate, or 4=very inappropriate).2 The most/least/best/worst selection response format allows examinees to identify the most and least effective options.2
SJTs have been used to assist with determining eligibility for admissions or acceptance into post-graduate training programs.3 Some are now using it to evaluate social behaviors, professional attributes, and non-cognitive skills related to interprofessional collaboration, team building, and establishing rapport with patients. It has also been used to assess students’ preparedness for advanced practice experiences or capstone courses.3
SJTs are growing in popularity in education programs for the health professionals. These unique tests can be used to identify non-cognitive skills that require development and can prompt reflective learning.4 For example, an SJT has been used to measure attributes such as empathy, integrity, and critical thinking among pharmacy students in the United States and Australia.4 This method could be useful in a team-based learning (TBL) setting, where students discuss various situations with their assigned team. It has the potential for students to learn how to reasonably assert themselves when others do not agree. Another advantage of using SJTs mentioned by multiple authors is the capacity to assess a large group of learners simultaneously.3,4
Scoring SJT can be seen as a significant disadvantage because there is no “objective” right or wrong answer. The scores truly rely on the test developer’s expertise and the subject matter experts (SMEs) who set the “key” for each item. This leaves an SJT open to ambiguity and variability, raising concerns about reliability and bias. Having a process to create SJTs that reduces the potential for legal challenges and improves diversity is an additional challenge. SMEs have their own behavioral tendencies, cultural backgrounds, religious beliefs, and interpretations of scenarios. The experts should be a diverse (e.g., gender, race, professional discipline) group that represents the prospective test takers.4 Validity research on SJT should be conducted to determine whether the SJT items elicit the intended constructs (e.g., empathy, collaboration) to be measured.3 Research has been conducted to assist with improving the validity of the response and scoring process. Additionally, a valid and reliable SJT requires a substantial time commitment to develop.
Lastly, another potential problem is that professionalism and other social traits evolve over time. Thus, using SJTs may complicate the learning process if the ideals and values of the school or faculty differ from the assessment. SJTs may reveal expected behaviors but may not promote professional identity formation.4
This testing method is a potentially useful assessment in academia if it can accurately predict future performance and behaviors. Based on student responses, an SJT might provide educators with information to customize lesson plans and provide students with feedback. I envision this assessment could be used to assist with creating rotation schedules, selecting leaders among a cohort, or developing activities to foster professionalism in post-graduate programs. If appropriate efforts are taken to create an SJT that is valid, reliable, and unbiased, this method could be an effective way to select applicants with high-fidelity and strong interpersonal skills.
References
1. U.S. Office of Personnel Management Policy, Assessment, and Selection. Situational Judgement Tests [Internet]. Washington, D.C.: Office of Personnel Management [accessed 8/13/2024].
2. Whetzel DL, Sullivan TS, and McCloy RA. (2020) Situational Judgment Tests: An Overview of Development Practices and Psychometric Characteristics Personnel Assessment and Decisions 2020; 6(1): Article 1.
3. Wolcott MD, Lobczowski NG, Zeeman JM, McLaughlin JE. Exploring the Role of Item Scenario Features on Situational Judgment Test Response Selections. Am J Pharm Educ 2021 Jun;85(6):8546.
Reed BN, Smith KJ, Robinson JD, Haines ST, Farland MZ.Situational Judgment Tests: An Introduction for Clinician Educators. J Am Coll Clin Pharm2022;5: 67-74.