Practical Considerations for AI & Integrated Data Systems: Implicit Bias

November 6, 2023
Morgan Sexton, Katherine Kalpos, and Amelia Vance
9

There has been lots of hype recently around artificial intelligence (AI), including its magical promises and potential risks. While it may be tempting to get swept up in the potential around AI and incorporate it into Integrated Data Systems (IDSs) right away, it is crucial to understand that AI is only as good as the information fed into it and the soundness of the algorithms that it relies on. Additionally, the implementation of AI into IDSs can create significant ethical and legal challenges. In this emerging issues update, we explore (1) what AI is, (2) how AI is being used by government agencies, (3) what incorporating AI into IDSs may look like in practice, and (4) practical considerations for IDS stakeholders who are considering incorporating AI into their IDSs.

Agencies are already increasingly using automated tools to make decisions about enforcement, caseload management, benefits, and the application of rules. Agencies as varied as the IRS, BLS, SSA, FDA, EPA, SEC, FCC, and USPTO employ automated systems for governmental tasks once done by humans. For example, the Department of Veterans Affairs uses AI in administering veteran benefits while the Department of Education uses an automated chatbot to help navigate student loan applications. The Department of Health and Human Services has sponsored the creation of an AI-based tool to detect illegal opioid sellers. The Food and Drug Administration has similarly begun using AI in criminal investigations and to mine through online reports on unsafe food.

While there are many ways that governmental entities are adopting and using AI (see this inventory, parts 1 and 2 of this report, and pages 12-14 of this report for more), our focus is on a particular use case: how governmental agencies are incorporating AI into their IDSs and systems utilizing the data housed within an IDS.

An arm holding a sign that says "HELP!" extends out from behind multiple stacks of paperFor example, imagine a hypothetical state-run, public-facing, informational website about state-level government benefits (similar to the federal government’s Benefits.gov). The state wants to personalize the website’s content for users to improve their experience by presenting users with information about the benefits they are most likely to be eligible for at the top of their screen. To accomplish this, the state wants to use individual-level data within a multi-agency IDS to pre-screen a user’s eligibility for various benefits and prioritize content based on which benefits are most likely to be relevant to that individual. The state would most likely do this by implementing AI to enable this functionality to operate in an accurate, efficient manner at scale.

There are many ways that incorporating AI in IDSs and systems utilizing the data housed within an IDS can benefit government agencies. Lobel explains that “[a]lgorithms reveal patterns often unseen by humans'' and that “[a]utomation and digitization can also alleviate the burdens of administrative paperwork, which a 2021 Presidential Executive Order describes as a burden that exceeds nine billion hours annually with regard to federal agencies.” Utilizing AI can empower agencies to help more people, more efficiently without having to increase staffing. AI can also be used to quickly take care of repetitive, procedural, and standardized tasks that can take humans a long time to complete (i.e. filling out basic questionnaires when applying for unemployment compensation) and free up time for people to devote to tasks that require more thoughtful input (i.e. actual human conversations).

That being said, there is significant disagreement about AI’s benefits–especially in terms of its impact on equity. On one hand, Lobel argues that “the upsides of AI are immense. Automated decision-making is often fairer, more efficient, less expensive, and more consistent than human decision-making,” a view that, according to experts like former FTC Commissioner Noah Phillips, should be more closely examined. On the other hand, Commissioner Phillips also supported the conclusion of a 2022 FTC report, which stated that “government and companies should ‘“exercise great caution in either mandating the use of, or over-relying on, [AI] tools’” and that “humans should continue to be involved.” In a January 2023 brief, the Center for Democracy and Technology stated that an “important lesson for the government is that AI is not necessarily objective or fair compared to alternatives. One reason for this is that many uses of AI involve data, but data is inherently biased. This is especially true for government agencies that want to change historical trends in data like student achievement gaps or unhoused rates.”

A report from Upturn and Omidyar Network finds that “[t]oday’s automated decisions are socio-technical in nature: They emerge from a mix of human judgment, conventional software, and statistical models. The non-technical properties of these systems — for example, their purpose and constraining policies — are just as important, and often more important, than their technical particulars.” This is especially true in cases where AI is used to generate predictive risk scores to aid human decision-making in governmental agencies (as seen in the screening tool discussed in depth in the next section).

On a more theoretical level, Virginia Eubanks, author of Automating Inequality, explained how governmental agencies are using AI to ration access to benefits:

Often [government automated decision-making] tools are created in [a] context where the assumption is that there are an inevitable shortage of resources, that there will never be enough resources, and that what we need these systems for is to make more neutral objective decisions about who gets access to their basic human rights. So I think of that as like these tools are sort of addicted to doing digital triage often, but triage is a really bad metaphor for what these tools are supposed to do. Triage is really only useful when there are more resources coming in the future. And if we’re using these tools to make decisions that acknowledge and rationalize and justify some people getting access to their human rights while others do not, then what we are doing is actually digital rationing, not digital triage.” (starting at 42:57)

It's likely that governmental entities will continue to explore how–and under what circumstances–AI can be incorporated into their technologies (including multi-agency IDSs). As noted in an article from the Brookings Institute, “Government organizations more rapidly introduce new [information and communication technology]; more importantly, they develop standards and rules for its use.” There has been a lot of action at the federal level focused on understanding the emergence of AI and exploring how it could be regulated (see Appendix A for a high-level overview of some recent activity). It’s clear that AI is a top priority right now for federal and state regulators, likely including those responsible for overseeing the operations of state-level IDSs.

“In many ways, the AFST is the best-case scenario for predictive risk modeling in child welfare. The design of the tool was open, participatory, and transparent. Elsewhere, child welfare prediction systems have been designed and implemented by private companies with very little input or discussion from the public. Implementation in Allegheny County has been thoughtful and slow. The goals of the AFST are intentionally limited and modest. The tool is meant to support human decision-making, not replace it.” (Automating Inequality, p. 171)

But despite its well-intentioned design and implementation, critics like Eubanks have argued that the AFST still perpetuates inequalities, especially in the form of “poverty profiling”:

“Like racial profiling, poverty profiling targets individuals for extra scrutiny based not on their behavior but rather on a personal characteristic: living in poverty. Because the model confuses parenting while poor with poor parenting, the AFST views parents who reach out to public programs as risks to their children.” (Automating Inequality, p. 158)

Following this criticism, Allegheny County released a statement claiming that “[Automated Inequality] has numerous inaccuracies and several key points require correction,” noting that, in fact, a family’s receipt of “public benefits (e.g. SNAP and TANF)” actually lowers that family’s AFST score, and that, “[n]ationally, the disproportionate involvement in child welfare of those impacted by poverty is well documented. Eubanks explained that part of her critique was focused on the fact that Allegheny County’s system included data only about “families using public services not on those that access private resources for parenting support. Because this will result in higher risk scores and more scrutiny of poor and working-class families, I believe the system is unjust and discriminatory.”

Questions remain regarding the AFST’s variable weights, error rate, and potential limitations on human discretion in the screening process, but the more important questions are at a higher level: should AI ever be used as part of a determination about which families should be investigated? An article from the New York Times, “Can an Algorithm Tell When Kids Are in Danger?,” features viewpoints from both angles. While pediatrician, Rachel Berger, claimed that predictive analytics can bring objectivity to extremely subjective child protective decisions, former commissioner of NYC’s Administration of Children’s Services, Gladys Carrión, expressed concerns about infringing on individual’s civil liberties “under the guise that we are going to help them.”

APPENDIX: Policymaker Actions Related to AI

This report provides general information, not legal advice, and following the recommendations or tips within does not guarantee compliance with any particular law.