Choosing the Right Data Strategy for Behavioral Health and Criminal Justice Initiatives

Read pdf Online Download pdf

Collecting data and evaluating programs should be a critical element of all criminal justice and behavioral health initiatives. However, the data collection process can be overwhelming and unclear. Programs need to determine what data to collect, how often, and how to use them to assess impacts, set goals, and make improvements. This brief will help project coordinators and research partners working with government entities or nonprofit agencies identify, collect, and prioritize the appropriate data as well as provide recommended metrics for behavioral health and criminal justice-related programs. Photo credit: ConnectVector via Shutterstock.

Amelia Vorpahl, Sarah Wurzburg, Sheila Tillman, Faye Taxman | October 2021 | Choosing the Right Data Strategy for Behavioral Health and Criminal Justice Initiatives

Collecting data and evaluating programs should be a critical element of all criminal justice and behavioral health initiatives. Data help illustrate what is working, what needs to change, and which stated outcomes are achieved. In the short term, data can be used to set goals and assess needs, and in the long term, data can help answer research questions, evaluate program outcomes, and inform continuous program improvement.

Despite the benefits of data collection, however, the process can be overwhelming and unclear. Programs need to determine what data to collect, how often, and how to use them to assess impacts, set goals, and make improvements. There are often data collection requirements from funders, whether they are federal (such as the Justice and Mental Health Collaboration Program), state, local, or philanthropic. This can lead to programs needing to quickly develop data collection and analysis processes. Often, programs also work to develop research partnerships through local agencies or universities to support development of data collection plans, logic models, and analyses and to conduct evaluations.

This brief will help project coordinators and research partners working with government entities or nonprofit agencies identify, collect, and prioritize the appropriate data as well as provide recommended metrics for behavioral health and criminal justice-related programs.

Selecting Key Metrics

Project coordinators, researchers, implementation teams, and advisory groups need to identify what data are appropriate to collect and track for their program goals, which can be overwhelming to narrow down. Many of the decisions about what metrics to track will depend on a specific jurisdiction’s program and its priorities. It is paramount to ensure that the behavioral health and criminal justice partners agree on the definitions for each of the key metrics. Developing a shared understanding of and language about the information can be an important step in the process.

When selecting key metrics, elements to consider include collecting data at both the individual and systems levels; using quantitative and qualitative data; and prioritizing equity by

	Key Benefits of Data Collection
Increases program accountability	Streamlines decision-making	Enhances understanding of what contributes to program success or failure	Optimizes service delivery	Communicates the program’s value to key stakeholders

disaggregating data by race, gender, ethnicity, and other characteristics to address disparities (see the Appendix for types of data collection). There are many pre-established behavioral health and criminal justice metrics to draw on when determining what to collect, including the following:

Interface Process Measures

Gauge to what extent behavioral health and criminal justice systems interact and collaborate to identify and treat people at both the system and individual levels. Key metrics include rates of identification and referral, levels of program engagement and completion, insurance enrollment rates, and capacity of systems and individuals to provide or access programming.

Housing Instability

Measures how many people are having trouble finding and keeping stable housing within the criminal justice system. Measurements include number of people booked, screened, or connected to housing services, as well as staff capacity for homelessness support. Housing tenancy metrics also identify the number of people who remained housed for specific time periods.

Police-Mental Health Collaboration (PMHC)

Assesses how effective collaboration programs between law enforcement and behavioral health partners are at safely and appropriately responding to people with mental health needs. Key data points include the level of community need (number of calls or arrests) as well as administrative outcomes, such as the number of officers receiving specialized mental health and stabilization training.

Stepping Up Initiative

Helps counties create programs to reduce the number of people with mental illnesses in local jails. There are four key metrics for people with mental health needs: how many are booked into jail, their average length of stay, the percentage connected to treatment, and recidivism rates.

Data Sources

There are different sources from which to collect data, which can impact how the data can be used. There should be a structured method of collection while using any source to ensure good data quality.

Data Type	Source	Usually Contains	What it Tells You
Quantitative	• Management information systems/administrative data • Case management software • Special data collection	These types of quantitative data sources contain objective information and statistics, like demographic characteristics of participants; dates; and types of events, such as timing of participation in a program or service, results from screenings or assessments, case management goals, and other behavioral health metrics.	These sources can inform researchers about participant characteristics and outcomes, the number of processes people participate in; the length of time between events; and participation rates of screening, assessment, and treatment.
Qualitative	• Organizational surveys • Participant surveys • Interviews • Focus groups	Descriptive sources like surveys and interviews can include directed questions, subject matter interviews, and other information that explores program context and focuses on the subjective opinions and perspectives of individuals doing the work or participating in a program. There are some standardized survey instruments that can be used for some topics.	Qualitative data sources aim to capture human experience by assessing the opinions and perspectives of program staff, administrators, partners, and participants. They can identify common themes and examine support for program changes or innovation.

Determining How to Use Data

Beyond just collecting data, researchers also need to analyze the data to support evaluations and continued funding and assess program impacts, processes, or outcomes. These types of evaluations are important for helping to determine whether a program is functioning as intended and meeting its goals as well as to inform continuous improvement. The specific collection and evaluation option chosen by a researcher also needs to connect to their objectives, as different types of evaluations assess different program elements.

Types of Data Evaluation	What It Is	What It Tells You	Considerations
Implementation study	Data are used to address and improve health care implementation and delivery, including how to use research in practical application	Identifies factors influencing implementation processes and outcomes, including how to introduce potential solutions into a health system or scale them	Resources: Budget, staff, expertise level, existing secondary data or need to collect primary data Stakeholder engagement to capture all points in implementation process
Process evaluation	Data are used to report on the progress of a program or improve future program procedures	To what extent programs and policies have been implemented as intended and what the results have been	Scope: Evaluating one aspect of a program’s activities or the program as a whole? Timing: Starting at the beginning of implementation or after it has already started? Resources (see above)
Outcome evaluation	Data are used to examine the effects or results of a program	How the program has impacted the target population and whether it has been effective in meeting objectives	Timing: Program must already be fully implemented Scope: Assessing long- or short-term impacts? Resources (see above)
Continuous quality improvement	A multi-stakeholder quality management process uses data to focus on activities to improve community health needs Involves strategies to ensure that the innovation is being implemented and to address barriers to change	Measures consistency, success, and effectiveness of behavioral health services	Look at systems, not just people. Identify and engage multisystem stakeholders Sustainability: Ensure analysis and improvement is ongoing, not a one-off project
Cost benefit/ cost savings	Data provide a way to compare cost of program with its effects	The direct and indirect costs of program resources compared with outcomes	Ensuring accuracy of costs and benefit estimates. Some benefits may not be fully measurable.
Impact evaluation	Data assess how the program affected outcomes, including to what extent any effects were intentional and if effects would have happened in the program’s absence	Helps determine whether the program is meeting objectives, including how and why it is working or not	Identifying a comparison group to determine the counterfactual (would effects have otherwise happened)
Developmental evaluation	Real-time data are used to inform ongoing decision making incorporated into program development and implementation	What initial results say about expected progress and how data on changing circumstances can lead to program adaptations in real time	Often requires extensive field work as well as stakeholder collaboration
Effectiveness evaluation	Data help assess whether a program met its intended goals and objectives	How successful a program was in meeting its objectives, including assessing results vs. expectations and why a program may have failed or exceeded timelines. Focused on long-term systematic and aggregated impact.	Timing: Program must be fully implemented for a period of time Readiness: Jurisdiction must be willing and able to look at systems level Resources (see above)

Six Key Tips for Collecting Data and Selecting Evaluation Methods

1. Develop a research partnership

Partner with researchers in local agencies or universities to build data capacity, set up data collection processes, create data-sharing or security protocols, and support the development of evaluations.

2. Develop a logic model and determine key metrics

After the program has identified its vision, mission, and goals, develop a logic model for the program to help select metrics. This process will help ensure that the goals of the program are matched to the data that are prioritized for collection.

3. Establish a clear data collection plan

Review and determine the different sources (e.g., case management system; management information system, such as jail management system; Excel spreadsheet; organizational survey; participant survey; focus groups; interviews, etc.) of data available for the program. Develop a data collection plan based on the available data sources and the data use agreements that have been set up between criminal justice and behavioral health partners. A clear collection process should also include standardized procedures, quality assurance, and privacy and security protocols. Partners should work with researchers to outline who is responsible for each data source; where the data are stored; how they are used; and how they are reported back to program staff, community stakeholders, and agency leaders.

4. Ensure organizational buy-in

Educate staff on how the data are used and how to give feedback. The logic model can be used to ensure a common understanding of how the program works. The model should be updated as changes to the program occur. Everyone in the program should view data collection as a core procedure that is built into routine operations, not just a one-time activity. Staff need to be continually involved in conversations about how the data are used to support improved outcomes for the participants.

5. Develop an evaluation plan

Collaborate with a research partner to develop an evaluation plan, which will help ensure that the data collection, logic model, and eventual evaluation are all coordinated.

6. Craft a compelling story

The program should determine what audience it should reach and how this audience wants to receive information. Compelling messaging that helps contextualize data for different audiences is essential. When data are shared on an ongoing basis with stakeholders through an advisory group (such as a criminal justice behavioral health council or criminal justice coordination council) that includes the heads of agencies, county commissioners, and community members, it can help support the initiative’s long-term success, funding, and ability to innovate.

Appendix: Criminal Justice and Behavioral Health Metrics

Objective: Identify and refer people in the justice system with behavioral health needs to the appropriate care and services.
Interface Process Measures: Set 1	Metric	Description
	Screening rate	Percentage of people who screened positive for behavioral health issues using a validated screening tool
	Clinical assessment rate	Percentage of people clinically assessed as needing behavioral health services using a validated assessment procedure
	Referral rate	Percentage of people screened or assessed for behavioral health conditions who are referred to a treatment program or linked to services
	Initiation	Percentage of people who are referred to behavioral health services who start services within 14 days of the screening or clinical assessment

Objective: Ensure that people in the justice system participate in and complete the appropriate behavioral health treatment.
Interface Process Measures: Set 2	Engagement	Individual participates in at least two treatment sessions within a 30-day period of initiation
	Retention	Length of stay in treatment for different levels of care, including outpatient treatment, intensive outpatient treatment, therapeutic community, and counseling services
	Successful completion rate	Percentage of individuals (people who are arrested, defendants, incarcerated people, people on probation/parole) who successfully complete treatment
	Medication-assisted treatment rate	Percentage of individuals (people who are arrested, defendants, incarcerated people, people on probation/parole) screened or assessed for behavioral health conditions who require medication and who are provided medication in a justice setting
	Compliance with treatment plan	Percentage of individuals who comply with their treatment plans developed by the behavioral health specialist and the individual

Objective: Offer access to a seamless system of care that provides people in the justice system who have behavioral health needs with the appropriate services.
Interface Process Measures: Set 3	Continuum of care	Percentage of individuals who transitioned from one program to another (step up or step down) within 30 days
	Continuity of care	Percentage of individuals who transitioned from one phase of programming to another within 30 days
	Transitioning of care	Percentage of individuals who moved from one type of programming (i.e., substance use disorders or mental illness) to another, such as vocational or educational services

Objective: Ensure that a range of behavioral health programming is available in both correctional and community settings to effectively address people’s needs.
Interface Process Measures: Set 4	Uniform screening protocols	Core set of screening processes and criteria accepted by justice and treatment agencies to identify individuals with substance use disorders and mental illnesses
	Insurance enrollment	Percentage of individuals in the justice system who are enrolled in health insurance
	Enrolled in insurance at time of release	Percentage of individuals released from prison or jail who are enrolled in health insurance
	Responsivity rate	Percentage of individuals in the justice system who can access appropriate services for their specific needs
	Availability of programming	Capacity of the system to provide appropriate levels of care for individuals (in prisons, probation/parole, jail, or community settings) who have specific needs
	Access rate	Percentage of individuals who can access a given program
	Participation rate	Percentage of individuals who partake in programs during their period of incarceration or supervision

Objective: Expand access to safe and stable housing supports for people in the justice system with behavioral health needs.
Metrics for People Experiencing Housing Instability or Homelessness	Homelessness rate	Percentage of people who are experiencing homelessness and booked into jails or prisons
	Housing instability rate	Percentage of people screened for housing instability
	Service connection rate	Percentage of people connected to housing or homeless services (e.g., permanent housing, transitional housing, use of emergency shelters)
	Staff capacity for homelessness support	• Percentage of dedicated or partially funded staff for homelessness outreach or case management • Percentage of staff who receive education on homelessness and homelessness support services
	Housing tenancy	Percentage of individuals who remained housed at specific intervals (e.g., 90 days, 6 months, 12 months, 24 months after release)

Objective: Effectively respond to people with behavioral health needs who come into contact with law enforcement.
Police- Mental Health Collaboration (PMHC): Level of Need	Level of need	Percentage of calls for service involving people who have mental health needs
PMHC: Minimized Arrests	Arrest rate	Percentage of arrests involving people who have mental health needs
	Arrest rate	Percentage of people who have mental health needs who have more than 1 arrest in last 12 months
	Disposition/ resolution of call	Disposition/resolution of call (e.g., arrest, resolved at scene, transported for voluntary evaluation, detained for involuntary evaluation, referral to mental health treatment)
PMHC: Reduced Repeat Encounters	Repeat call rate	Percentage of repeat calls to the same location
PMHC: Reduced Use of Force	Use of force rate	Percentage of encounters with people who have mental health needs where force was used
	Use of force rate	Type of force used by officers during encounters with people who have mental health needs
	Officer injury rate	Percentage of police encounters with people who have mental health needs that result in injury to officers

Objective: Effectively respond to people with behavioral health needs who come into contact with law enforcement.
PMHC: Administrative and Process Outcomes	Officer training rate	Percentage of officers receiving mental health and stabilization training
	Officer training rate	Percentage of officers trained in select PMHC response models
	Officer training rate	Percentage of shifts covered by trained officers
	Officer training rate	Percentage of dispatchers trained on PMHC response models
	Officer training rate	Percentage of mental health-related calls receiving a response by a trained officer

Objective: Reduce the prevalence of people with behavioral health needs in local jails.
Stepping Up Initiative: 4 Key Metrics	Jail bookings	Percentage of people with mental illnesses booked into jail
	Length of stay	Average length of stay for people with mental illnesses in jails
	Connection to treatment	Percentage of people with mental illnesses connected to treatment
	Recidivism rate	Recidivism rates for people with mental illnesses

Project Credits

Writing: Amelia Vorpahl, Sarah Wurzburg, Sheila Tillman, CSG Justice Center; Faye Taxman, Center for Advancing Correctional Excellence at George Mason University

Research: Sheila Tillman, CSG Justice Center

Advising: Faye Taxman, Center for Advancing Correctional Excellence at George Mason University

Editing: Leslie Griffin, Emily Morgan, CSG Justice Center

Design: Michael Bierman

Public Affairs: Ruvi Lopez, CSG Justice Center

Web Development: Eleventy Group

This project was produced in collaboration with the Center for Advancing Correctional Excellence at George Mason University and was supported by Grant No. 2019-NT-BX-K001 awarded by the Bureau of Justice Assistance. The Bureau of Justice Assistance is a component of the Department of Justice’s Office of Justice Programs, which also includes the Bureau of Justice Statistics, the National Institute of Justice, the Office of Juvenile Justice and Delinquency Prevention, the Office for Victims of Crime, and the SMART Office. Points of view or opinions in this document are those of the author and do not necessarily represent the official position or policies of the U.S. Department of Justice.

ABOUT THE AUTHORS

Amelia Vorpahl

Former Content Development Specialist, Communications and External Affairs

Amelia Vorpahl led the writing and strategic development of an array of content that advances the CSG Justice Center's goals, including policy reports, fact sheets, interactive tools, and web content. Before joining the CSG Justice Center, Amelia worked on the

Sarah Wurzburg

Deputy Division Director, Behavioral Health

Sarah Wurzburg oversees technical assistance focused on behavioral health, diversion, and reentry and serves as the lead for projects related to substance use, mental illnesses, and housing. She leads the work on the development of community responder programs, including a

...

toolkit that supports sites in development of non-police responses to people in crisis. Previously, Sarah was a research analyst at the National Association of State Alcohol and Drug Abuse Directors, Inc., where she was the team lead for Youth and Women’s Services and was the primary author of research reports on youth substance use disorder treatment, driving under the influence, and Medicaid. Sarah has also worked as a juvenile court advocate and in community substance use disorder prevention. She received her BA from DePauw University in English (writing) and her MA in social services administration with a focus on policy analysis from the University of Chicago.

Sheila Tillman

Senior Policy Analyst, Behavioral Health

Sheila Tillman provides technical assistance to Justice and Mental Health Collaboration Program grantees, focusing on the intersection of behavioral health and criminal justice. She also provides support on a variety of behavioral health and reentry projects. Before joining the CSG