Recent stories in the media are raising important questions about whether risk and needs assessments are biased against people of color or women. As with all such tools, accuracy depends greatly on design and proper implementation.
If your agency is using a risk and needs assessment, you should know the tool is performing and develop a plan to remediate any issues (e.g., scoring inconsistencies or low predictive accuracy) you may discover.
1. Validate your assessment tool.
- Do a validation study that is conducted by an independent party. Most risk and needs assessment tools have been validated by the developer as part of the initial construction of the tool, but a validation study by an objective third party should be conducted by each agency. Validating is fairly straightforward, although results may vary depending on the statistical test used. Ideally, predictive validity should be measured using multiple statistical tests.
- Determine the tool’s predictive accuracy by race and gender, as well as for your overall population. A properly validated tool may result in different average scores by race or gender, which may reflect true disparities in criminal justice involvement or enforcement in a jurisdiction. To be valid, risk and needs assessments must predict reoffending (e.g., new arrests and/or convictions) and the degree of accuracy in making those predictions should be close to equal by race and gender groupings. The validation study should examine each item on the tool by race and gender to isolate any items that may be contributing to bias.
2. Assess quality of implementation of the risk and needs assessment tool.
- Pinpoint the cause of any racial or gender bias that the validation study identified. Determine if the problem is attributable to the tool itself (e.g., certain items are weighted in a manner that is contributing to unintended bias) or if there is a problem in the way it’s being administered (e.g., assessors are not scoring the tool according to established guidelines).
- The Council of State Governments (CSG) Justice Center developed a multi-point inspection protocol for assessing and improving the quality of your risk and needs assessment implementation (called the Risk Assessment Quality Improvement checklist). To learn more about this tool, contact Jen Kisela (jkisela@csg.org).
- Conduct inter-rater reliability activities and focus groups to review how scoring protocols are being used in practice. Often there is a difference between the policies and procedures that guide use of risk and needs assessment and the actual way the tool is being implemented. Inter-rater reliability exercises will highlight scoring differences among assessors and focus groups may provide additional information around why practices no longer align with written policies and procedures.
3. Develop a plan to address any bias with the risk and needs assessment tool itself (e.g., current weighting of items is causing unintended bias) or how it is being implemented and used (e.g., it is not being scored correctly).
- Establish protocols to continually monitor the administration of risk and needs assessment tools. Agencies should create protocols to adequately train new assessors and then to regularly review, via direct observation or recording, how assessors conduct the interview and calculate the score.
- Provide regular booster trainings to keep staff skills current and accurate. In Hawaii, for example, scoring inconsistencies highlighted a need to update the assessment’s scoring guide and retrain assessors. Hawaii’s action plan led to significant updating of scoring rules and a massive statewide retraining effort for assessors.
- If you decide that changing risk and needs assessment tools is necessary, you may want to consult this guide on risk and needs assessment instruments that have been validated and implemented in correctional settings. But, recognize that adopting any new tool will require several years of implementation before a new validation test can be conducted to confirm improvements have been made.
Additional resources
For examples of sound methodology for validation studies see:
- Predictive validity performance indicators in violence risk assessment: a methodological primer
- Reporting guidance for violence risk assessment predictive validity studies: the RAGEE Statement
- Performance of Recidivism Risk Assessment Instruments in U.S. Correctional Settings
- Thirty Years of Research on the Level of Service Scales: A Meta-Analytic Examination of Predictive Accuracy and Sources of Variability
For examples of best practices for implementing risk and needs assessment see:
For primers on risk and needs assessment for policymakers, judges, prosecutors, public defenders, and behavioral health providers see:
- Risk/Needs Assessment 101: Science Reveals New Tools to Manage Offenders
- The Role of Offender Risk Assessment: A Policy Maker Guide
- Offender Risk & Needs Assessment Instruments: A Primer for Courts
- Adults with Behavioral Health Needs Under Correctional Supervision: A Shared Framework for Reducing Recidivism and Promoting Recovery
You might also be interested in