(PS4-36) Race, Treatment Cost, and Justice: LLMs in Opioid Sentencing and Treatment Decisions

Abstract Text: As Artificial Intelligence (AI) gains a foothold in our world, researchers are concerned about how this technology will become increasingly entrenched in the criminal justice system and play a role in sentencing decisions (Ryberg, 2024). Already, predictive algorithms have been incorporated into our justice system to predict recidivism risk (Linhart & Schubert, 2024). Given the well-documented racial bias of AI in various fields (Heinz et al., 2023; Glantz, 2024) and ethical concerns of adopting AI algorithms in sentencing decisions (Rybert & Roberts, 2023), this study aims to further explore how AI considers cost and race when making sentencing decisions.
Possession of illicit opioids is heavily sanctioned; however, incarceration does not stop opioid use. Treatment is more effective and presents less of a financial burden to taxpayers. This study utilizes large language models (LLM) to get a better understanding of how race and cost information of incarceration versus treatments for opioid use impact sentencing attitudes toward people who use opioids. These results will provide information about important public health topics and will help identify notable moderators for the treatment of people with opioid convictions.
Vignettes about a black or white man who was arrested for opioid possession after being stopped for a traffic violation were presented. For each trial, the LLM was presented with one of the vignettes, and half of them were also presented with information regarding the cost to taxpayers of putting the defendant in prison or a drug counseling program. To assess its sentencing decisions, the LLM was asked to rate its support for sentencing the defendant to mandatory drug counseling or prison. The LLM was also asked to complete the following modified public stigma subscales: Personal Responsibility Beliefs Subscale from the AQ-27 (Corrigan et al., 2003), items from the Day’s Mental Illness Stigma Scale measuring belief about lack of recovery and lack of treatability (Day et al., 2007), a dangerousness scale (Penn et al., 1999), and the Social Distance Scale (Lebowitz et al., 2016) assessing social acceptance, a Bayesian racism scale (Uhlmann et al., 2010), a modified nine question moral responsibility scale (Nadelhoffer & Brewer, 2008). A Python program was used to complete 200 independent trials with two LLMs (ChatGPT -4o & Gemini 2 flash).
Data analysis is ongoing, but preliminary results have been calculated. A 2 (LLM) X 2 (Cost) X 2 (Race) ANOVA had a statistically significant main effect for LLM by cost interaction. When no cost information was provided, ChatGPT-4o had a higher endorsement of prison than Gemini. The same ANOVA was calculated for drug counseling and had a statistically significant interaction for LLM, race, and cost. Gemini reported lower mean values for drug counseling across cost and race. Hierarchical regression was used to assess predictors for prison and drug counseling. LLM and cost information were entered on the first step, followed by the public stigma and racism scales. For prison, both perceived danger and racism were found to be unique predictors (R2adj = .12, p < .05). For drug counseling, the LLM, cost information, social acceptance, and danger were unique predictors (R2adj = .54, p < .001).