Researchers were trying to tackle AI bias…
Society is increasingly investing in AI in many spaces, from education to health, to justice. Given their scale, quick delivery, and accuracy, AI-powered technologies offer considerable promise in increasing access and improving services. Even so, over the last few years, we have quickly learned that AI biases can severely undermine these advances for certain groups, often historically marginalized groups. Technologies implemented in the justice system, for instance, have been found to be biased against people of color; health technologies have been found to work less effectively on Black patients.
These biases – and the unfair outcomes they are associated with – have led to a flurry of efforts to redress these biases. Phew, we all thought. We’re finally ensuring that these technologies are fair! In fact, as research led by our team members has found, algorithms developed to ensure fairness may actually lead to worse outcomes.
Indeed, in some cases, this work has led to “unfair fairness…”
How is that possible? Well, many of these algorithms adopt very simplistic definitions of fairness, for instance, where fairness means reducing gaps in performance or outcomes between demographic groups. The result is that these algorithms attempt to realize “fairness” in two key ways: 1) improving performance for worse-performing groups or 2) degrading performance for better-performing groups. Let’s take an example…say that an algorithm implemented in the justice system systematically misclassifies Black defendants who in fact have a “low risk” of reoffending (that is, committing a crime again) as having a “high risk” of reoffending. (This could be for many reasons, e.g., biased data underlying the algorithm.) That’s a big problem, because judges often take the likelihood of reoffending into account when determining whether to release a defendant on bail (that is, whether to let them await trial at home vs. in jail). One way we might try to address this fairness issue is to improve performance for Black defendants; we might, for instance, tell the algorithm to flip the “high risk” classifications for Black defendants that it is the least confident about. In doing so, the performance gap between Black, and say, white defendants, would shrink.
But there’s a cost associated with all of this, of course: the system’s accuracy for Black defendants! Over time, the number of Black defendants mistakenly flipped to “low risk” of reoffending (when they are, in fact, at a “high risk” of reoffending) would grow. At some tipping point (when the number of mistakes the system is making is just too many!), we might decide that we simply cannot afford for the system to be any more inaccurate for Black defendants. At that point, if gaps between demographic groups persist, we have only one option to achieve “fairness,” as defined: reduce performance for white defendants. In other words, we’d now be telling the algorithm to flip the “low risk” of reoffending classifications for white defendants that it is the least confident about.
But a lot of folks would argue that marking someone that was previously at a “low risk” of reoffending as instead at a “high risk” of reoffending has serious ethical implications (and indeed, is quite different from marking someone that was previously “high risk” as “low risk”)! Indeed, it really seems to offend the “innocent until proven guilty” principle…and it means some defendants that may have had the chance to await trial at home (and more effectively prepare a legal strategy for court) won’t have the chance to do that. Moreover, making this change does nothing for the disadvantaged group… it simply reduces the gap in “fairness” from a mathematical standpoint.
Our team was deeply concerned about this phenomenon, which we have labeled “leveling down.” Leveling down runs counter to the objectives of algorithmic fairness and broader equality goals in society: to improve outcomes for historically disadvantaged or marginalized groups. Despite this, it receives very little attention…something that we hope this website can change.