In this post we describe the basics of how to run an Angoff standard setting workshop.
The purpose of a certification exam is to determine which candidates are qualified and which are not. There needs to be a score below which the candidate fails to qualify for the certification. This score is called the "standard" or "cutscore" for the exam. One way to think about an exam standard is to consider lining up all of the candidates in order of their score on the exam. The standard is that point in the line below which the candidates fail and above which the candidates pass.
There are many methods for setting the standard for an exam. These methods are divided into two broad categories: examinee-centered methods and test-centered methods. In the examinee-centered methods candidates rate themselves by answering the questions on a survey. These survey responses are used in conjunction with the candidates' scores on the exam items to generate the standard (cutscore) for the exam. This requires a sufficient quantity of candidate responses to be effective. In the test-centered methods, subject matter experts (SMEs) review the questions on the test and make predictions of how the minimally qualified candidate (MQC, more on this later) would perform on each item. The Angoff methology falls into the test-centered category.
The first step in the Angoff process is to define and describe very clearly the skills, knowledge, and experience of the MQC. The MQC is the candidate who scores just barely above the standard. In our rank-ordered line of candidates the MQC is the last candidate to pass the exam. The MQC had just enough skill, knowledge, and experience to barely pass the exam. The MQC is the context of the entire Angoff standard setting process. Everything must be done in the context of the MQC description. It is vitally important that the SMEs connect with the MQC description. As the Angoff process goes on the SMEs will rate how the MQC will perform on each item and it is important that differences in the ratings be based on a difference in opinion of how the MQC will perform and NOT based on a difference in understanding of the MQC description.
After establishing the minimally qualified candidate description and cementing it as the context for the Angoff standard setting workshop, the Angoff workshop is generally conducted in multiple rounds. Essentially, each SME is asked to predict how an MQC would perform on each item. The predictions must be independent and based on each SME's individual experience working with candidates who have the skills, knowledge, and experience described in the MQC description. Multiple rounds involve some discussion between the rounds. More on that later.
The SMEs are asked to predict how many MQCs out of 100 would get each item correct. This is sometimes a hard concept for SMEs to wrap their heads around. It is possible that MQCs pass but get different items correct. Where does that variation come from? MQCs are not minimally qualified in the same way. The MQCs have at lease the minimal level of skills, knowledge, and experience in each area but the MQCs vary in where they have skills, knowledge, and experience beyond that minimal level. Because of this varying "more than minimal qualifications" and where all questions are designed to be harder than all MQCs can complete successfully, there is a resulting variation in how the MQCs perform.
After SMEs have completed their predictions for a round they come together as a group and talk about each item that has a range between the low and high predictions that exceeds a certain threshold. The purpose of the discussion is NOT for the SMEs to sway the other SMEs to their way of thinking. The purpose of the discussion is for the SMEs to share insights and perspective that might be helpful to each other. After hearing the discussion the SMEs can update their predictions if they want to.
After the SMEs have made their predictions and the rounds are all complete, the workshop is complete and the standard can be determined from their predictions. The output of the Angoff standard setting workshop is a number that represents the lowest number that a candidate can score on the exam and still pass.
Major Standard Setting Methods
Verifying the Location of the Standard