Unfair Treatment?: The Case of Freedle, the SAT, and the Standardization Approach to Differential Item Functioning; The College Board Responds
Maria Veronica Santelices and Mark Wilson:
In 2003, the Harvard Educational Review published a controversial article by Roy Freedle that claimed bias against African American students in the SAT college admissions test. Freedle’s work stimulated national media attention and faced an onslaught of criticism from experts at the Educational Testing Service (ETS), the agency responsible for the development of the SAT. In this article, Maria Veronica Santelices and Mark Wilson take the debate one step further with new research exploring differential item functioning in the SAT. By replicating Freedle’s methodology with a more recent SAT dataset and by addressing some of the technical criticisms from ETS, Santelices and Wilson confirm that SAT items do function differently for the African American and White subgroups in the verbal test and argue that the testing industry has an obligation to study this phenomenon.
The Harvard Educational Review has published a research article by Maria Veronica Santelices (Pontificia Universidad Católica de Chile) and Mark Wilson (University of California, Berkeley) that is critical of the Differential Item Functioning (DIF) analyses used in the construction of the SAT®. Unfortunately, this work is deeply flawed. It utilizes only partial data sets, focuses on a student sample that lacks representation and diversity, and draws conclusions that do not match the data. Simply stated, this research does not withstand scrutiny.
The SAT is a fair assessment, and many years of independent research support this. It is the most rigorously researched and designed test in the world and is a proven, reliable measure of a student’s likelihood for college success regardless of student race, ethnicity or socioeconomic status. There is no credible research to suggest otherwise. While a few critics have promoted the notion that the test results indicate bias in the tests themselves, this theory has been by and large debunked and rejected by the psychometric community.
In reviewing this article, our researchers identified a number of fundamental flaws in the data analysis, and they also expressed serious concerns about the conclusions reached by the authors. Key concerns with this study include the following:
