Choosing the Right Data Strategy for Behavioral Health and Criminal Justice Initiatives

Collecting data and evaluating programs should be a critical element of all criminal justice and behavioral health initiatives. However, the data collection process can be overwhelming and unclear. Programs need to determine what data to collect, how often, and how to use them to assess impacts, set goals, and make improvements. This brief will help project coordinators and research partners working with government entities or nonprofit agencies identify, collect, and prioritize the appropriate data as well as provide recommended metrics for behavioral health and criminal justice-related programs. Photo credit: ConnectVector via Shutterstock.

Amelia Vorpahl, Sarah Wurzburg, Sheila Tillman, Faye Taxman | October 2021 | Choosing the Right Data Strategy for Behavioral Health and Criminal Justice Initiatives

Choosing the Right Data Strategy for Behavioral Health and Criminal Justice Initiatives

Collecting data and evaluating programs should be a critical element of all criminal justice and behavioral health initiatives. Data help illustrate what is working, what needs to change, and which stated outcomes are achieved. In the short term, data can be used to set goals and assess needs, and in the long term, data can help answer research questions, evaluate program outcomes, and inform continuous program improvement.

Despite the benefits of data collection, however, the process can be overwhelming and unclear. Programs need to determine what data to collect, how often, and how to use them to assess impacts, set goals, and make improvements. There are often data collection requirements from funders, whether they are federal (such as the Justice and Mental Health Collaboration Program), state, local, or philanthropic. This can lead to programs needing to quickly develop data collection and analysis processes. Often, programs also work to develop research partnerships through local agencies or universities to support development of data collection plans, logic models, and analyses and to conduct evaluations.

This brief will help project coordinators and research partners working with government entities or nonprofit agencies identify, collect, and prioritize the appropriate data as well as provide recommended metrics for behavioral health and criminal justice-related programs.

Selecting Key Metrics

Project coordinators, researchers, implementation teams, and advisory groups need to identify what data are appropriate to collect and track for their program goals, which can be overwhelming to narrow down. Many of the decisions about what metrics to track will depend on a specific jurisdiction’s program and its priorities. It is paramount to ensure that the behavioral health and criminal justice partners agree on the definitions for each of the key metrics. Developing a shared understanding of and language about the information can be an important step in the process.

When selecting key metrics, elements to consider include collecting data at both the individual and systems levels; using quantitative and qualitative data; and prioritizing equity by

Key Benefits of Data Collection
Increases
program
accountability
Streamlines
decision-making
Enhances
understanding of
what contributes to
program success
or failure
Optimizes
service
delivery
Communicates
the program’s
value to key
stakeholders

disaggregating data by race, gender, ethnicity, and other characteristics to address disparities (see the Appendix for types of data collection). There are many pre-established behavioral health and criminal justice metrics to draw on when determining what to collect, including the following:

Interface Process Measures

Gauge to what extent behavioral health and criminal justice systems interact and collaborate to identify and treat people at both the system and individual levels. Key metrics include rates of identification and referral, levels of program engagement and completion, insurance enrollment rates, and capacity of systems and individuals to provide or access programming.

Housing Instability

Measures how many people are having trouble finding and keeping stable housing within the criminal justice system. Measurements include number of people booked, screened, or connected to housing services, as well as staff capacity for homelessness support. Housing tenancy metrics also identify the number of people who remained housed for specific time periods.

Police-Mental Health Collaboration (PMHC)

Assesses how effective collaboration programs between law enforcement and behavioral health partners are at safely and appropriately responding to people with mental health needs. Key data points include the level of community need (number of calls or arrests) as well as administrative outcomes, such as the number of officers receiving specialized mental health and stabilization training.

Stepping Up Initiative

Helps counties create programs to reduce the number of people with mental illnesses in local jails. There are four key metrics for people with mental health needs: how many are booked into jail, their average length of stay, the percentage connected to treatment, and recidivism rates.

Data Sources

There are different sources from which to collect data, which can impact how the data can be used. There should be a structured method of collection while using any source to ensure good data quality.

Data TypeSourceUsually ContainsWhat it Tells You
Quantitative• Management information systems/administrative data
• Case management software
• Special data collection
These types of quantitative data sources contain objective information and statistics, like demographic characteristics of participants; dates; and types of events, such as timing of participation in a program or service, results from screenings or assessments, case management goals, and other behavioral health metrics.These sources can inform researchers about participant
characteristics and outcomes, the number of processes people participate in; the length of time
between events; and participation rates of screening, assessment, and treatment.
Qualitative• Organizational surveys
• Participant surveys
• Interviews
• Focus groups
Descriptive sources like surveys and interviews can include directed questions, subject matter interviews, and other information that explores program context and focuses on the subjective opinions and perspectives of individuals doing the work or participating in a program. There are some standardized survey instruments that can be used for some topics.Qualitative data sources aim to capture human experience by assessing the opinions and
perspectives of program staff,
administrators, partners, and
participants. They can identify common themes and examine support for program changes or innovation.

Determining How to Use Data

Beyond just collecting data, researchers also need to analyze the data to support evaluations and continued funding and assess program impacts, processes, or outcomes. These types of evaluations are important for helping to determine whether a program is functioning as intended and meeting its goals as well as to inform continuous improvement. The specific collection and evaluation option chosen by a researcher also needs to connect to their objectives, as different types of evaluations assess different program elements.

Types of Data
Evaluation
What It IsWhat It Tells YouConsiderations
Implementation studyData are used to address
and improve health care
implementation and
delivery, including how to
use research in practical
application
Identifies factors
influencing implementation processes and outcomes, including how to introduce potential solutions into a health system or scale them
Resources: Budget, staff, expertise level, existing secondary data or need to
collect primary data

Stakeholder engagement to capture all points in implementation process
Process evaluationData are used to report on the progress of a program
or improve future program procedures
To what extent programs and policies have been implemented as intended and what the results have beenScope: Evaluating one aspect of a program’s activities or the program as a whole?

Timing: Starting at
the beginning of
implementation or after it has already started? Resources (see above)
Outcome evaluationData are used to examine
the effects or results of a
program
How the program has impacted the target population and whether it has been effective in meeting objectivesTiming: Program
must already be fully
implemented

Scope: Assessing long- or short-term impacts?

Resources (see above)
Continuous quality
improvement
A multi-stakeholder quality management process uses data to focus on activities to improve community health needs Involves strategies to ensure that the innovation is being implemented and to address barriers to changeMeasures consistency, success, and effectiveness of behavioral health servicesLook at systems, not just people.

Identify and engage multisystem stakeholders

Sustainability: Ensure analysis and improvement is ongoing, not a one-off project
Cost benefit/
cost savings
Data provide a way to
compare cost of program with its effects
The direct and indirect costs of program resources compared with outcomesEnsuring accuracy of costs and benefit estimates. Some benefits may not be fully measurable.
Impact evaluationData assess how the
program affected
outcomes, including to what extent any effects were intentional and if effects would have happened in the program’s absence
Helps determine whether the program is meeting objectives, including how and why it is working or notIdentifying a comparison group to determine the counterfactual (would effects have otherwise happened)
Developmental
evaluation
Real-time data are used to inform ongoing decision making incorporated into program development and implementationWhat initial results say about expected progress and how data on changing circumstances can lead to program adaptations in real timeOften requires extensive field work as well as stakeholder collaboration
Effectiveness
evaluation
Data help assess whether a program met its intended goals and objectivesHow successful a
program was in meeting its objectives, including assessing results vs. expectations and why a program may have failed or exceeded timelines. Focused on long-term systematic and aggregated impact.
Timing: Program must be fully implemented for a period of time

Readiness: Jurisdiction must be willing and able to look at systems level

Resources (see above)

Six Key Tips for Collecting Data and Selecting Evaluation Methods

1. Develop a research partnership

Partner with researchers in local agencies or universities to build data capacity, set up data collection processes, create data-sharing or security protocols, and support the development of evaluations.

2. Develop a logic model and determine key metrics

After the program has identified its vision, mission, and goals, develop a logic model for the program to help select metrics. This process will help ensure that the goals of the program are matched to the data that are prioritized for collection.

3. Establish a clear data collection plan

Review and determine the different sources (e.g., case management system; management information system, such as jail management system; Excel spreadsheet; organizational survey; participant survey; focus groups; interviews, etc.) of data available for the program. Develop a data collection plan based on the available data sources and the data use agreements that have been set up between criminal justice and behavioral health partners. A clear collection process should also include standardized procedures, quality assurance, and privacy and security protocols. Partners should work with researchers to outline who is responsible for each data source; where the data are stored; how they are used; and how they are reported back to program staff, community stakeholders, and agency leaders.

4. Ensure organizational buy-in

Educate staff on how the data are used and how to give feedback. The logic model can be used to ensure a common understanding of how the program works. The model should be updated as changes to the program occur. Everyone in the program should view data collection as a core procedure that is built into routine operations, not just a one-time activity. Staff need to be continually involved in conversations about how the data are used to support improved outcomes for the participants.

5. Develop an evaluation plan

Collaborate with a research partner to develop an evaluation plan, which will help ensure that the data collection, logic model, and eventual evaluation are all coordinated.

6. Craft a compelling story

The program should determine what audience it should reach and how this audience wants to receive information. Compelling messaging that helps contextualize data for different audiences is essential. When data are shared on an ongoing basis with stakeholders through an advisory group (such as a criminal justice behavioral health council or criminal justice coordination council) that includes the heads of agencies, county commissioners, and community members, it can help support the initiative’s long-term success, funding, and ability to innovate.

Appendix: Criminal Justice and Behavioral Health Metrics

Objective: Identify and refer people in the justice system with behavioral health needs to the appropriate care and services.
Interface Process
Measures: Set 1
MetricDescription
Screening ratePercentage of people who screened positive for behavioral health issues using a validated screening tool
Clinical
assessment rate
Percentage of people clinically assessed as needing behavioral health services using a validated assessment procedure
Referral ratePercentage of people screened or assessed for behavioral health conditions who are referred to a treatment program or linked to services
InitiationPercentage of people who are referred to behavioral health services who start services within 14 days of the screening or clinical assessment
Objective: Ensure that people in the justice system participate in and complete the appropriate
behavioral health treatment.
Interface Process Measures: Set 2EngagementIndividual participates in at least two treatment sessions within a 30-day period of initiation
RetentionLength of stay in treatment for different levels of care, including outpatient treatment, intensive outpatient treatment, therapeutic community, and counseling services
Successful
completion rate
Percentage of individuals (people who are arrested, defendants, incarcerated people, people on probation/parole) who successfully complete treatment
Medication-assisted
treatment rate
Percentage of individuals (people who are arrested, defendants, incarcerated people, people on probation/parole) screened or assessed for behavioral health conditions who require medication and who are provided medication in a justice setting
Compliance with
treatment plan
Percentage of individuals who comply with their treatment plans developed by the behavioral health specialist and the individual
Objective: Offer access to a seamless system of care that provides people in the justice system
who have behavioral health needs with the appropriate services.
Interface Process
Measures: Set 3
Continuum of carePercentage of individuals who transitioned from one program to another
(step up or step down) within 30 days
Continuity of carePercentage of individuals who transitioned from one phase of
programming to another within 30 days
Transitioning of
care
Percentage of individuals who moved from one type of programming
(i.e., substance use disorders or mental illness) to another, such as
vocational or educational services
Objective: Ensure that a range of behavioral health programming is available in both
correctional and community settings to effectively address people’s needs.
Interface Process
Measures: Set 4
Uniform screening
protocols
Core set of screening processes and criteria accepted by justice and
treatment agencies to identify individuals with substance use disorders
and mental illnesses
Insurance
enrollment
Percentage of individuals in the justice system who are enrolled in health
insurance
Enrolled in
insurance at time
of release
Percentage of individuals released from prison or jail who are enrolled in
health insurance
Responsivity ratePercentage of individuals in the justice system who can access
appropriate services for their specific needs
Availability of
programming
Capacity of the system to provide appropriate levels of care for
individuals (in prisons, probation/parole, jail, or community settings) who
have specific needs
Access ratePercentage of individuals who can access a given program
Participation ratePercentage of individuals who partake in programs during their period of
incarceration or supervision
Objective: Expand access to safe and stable housing supports for people in the justice system
with behavioral health needs.
Metrics for People
Experiencing
Housing
Instability or
Homelessness
Homelessness
rate
Percentage of people who are experiencing homelessness and booked
into jails or prisons
Housing instability
rate
Percentage of people screened for housing instability
Service
connection rate
Percentage of people connected to housing or homeless services (e.g.,
permanent housing, transitional housing, use of emergency shelters)
Staff capacity for
homelessness
support
• Percentage of dedicated or partially funded staff for homelessness
outreach or case management
• Percentage of staff who receive education on homelessness and
homelessness support services
Housing tenancyPercentage of individuals who remained housed at specific intervals
(e.g., 90 days, 6 months, 12 months, 24 months after release)
Objective: Effectively respond to people with behavioral health needs who come into contact
with law enforcement.
Police-
Mental Health
Collaboration
(PMHC): Level of
Need
Level of needPercentage of calls for service involving people who have mental
health needs
PMHC: Minimized
Arrests
Arrest ratePercentage of arrests involving people who have mental health needs
Arrest ratePercentage of people who have mental health needs who have more than
1 arrest in last 12 months
Disposition/
resolution of call
Disposition/resolution of call (e.g., arrest, resolved at scene, transported
for voluntary evaluation, detained for involuntary evaluation, referral to
mental health treatment)
PMHC: Reduced
Repeat
Encounters
Repeat call ratePercentage of repeat calls to the same location
PMHC: Reduced
Use of Force
Use of force ratePercentage of encounters with people who have mental health needs
where force was used
Use of force rateType of force used by officers during encounters with people who have
mental health needs
Officer injury ratePercentage of police encounters with people who have mental health
needs that result in injury to officers
Objective: Effectively respond to people with behavioral health needs who come into contact with law enforcement.
PMHC:
Administrative
and Process
Outcomes
Officer
training rate
Percentage of officers receiving mental health and stabilization training
Officer
training rate
Percentage of officers trained in select PMHC response models
Officer
training rate
Percentage of shifts covered by trained officers
Officer
training rate
Percentage of dispatchers trained on PMHC response models
Officer
training rate
Percentage of mental health-related calls receiving a response
by a trained officer
Objective: Reduce the prevalence of people with behavioral health needs in local jails.
Stepping Up
Initiative: 4 Key
Metrics
Jail bookingsPercentage of people with mental illnesses booked into jail
Length of stayAverage length of stay for people with mental illnesses in jails
Connection to treatmentPercentage of people with mental illnesses connected to treatment
Recidivism rateRecidivism rates for people with mental illnesses

Project Credits

Writing: Amelia Vorpahl, Sarah Wurzburg, Sheila Tillman, CSG Justice Center; Faye Taxman, Center for Advancing Correctional Excellence at George Mason University

Research: Sheila Tillman, CSG Justice Center

Advising: Faye Taxman, Center for Advancing Correctional Excellence at George Mason University

Editing: Leslie Griffin, Emily Morgan, CSG Justice Center

Design: Michael Bierman

Public Affairs: Ruvi Lopez, CSG Justice Center

Web Development: Eleventy Group

This project was produced in collaboration with the Center for Advancing Correctional Excellence at George Mason University and was supported by Grant No. 2019-NT-BX-K001 awarded by the Bureau of Justice Assistance. The Bureau of Justice Assistance is a component of the Department of Justice’s Office of Justice Programs, which also includes the Bureau of Justice Statistics, the National Institute of Justice, the Office of Juvenile Justice and Delinquency Prevention, the Office for Victims of Crime, and the SMART Office. Points of view or opinions in this document are those of the author and do not necessarily represent the official position or policies of the U.S. Department of Justice.

ABOUT THE AUTHORS


Image for:
Amelia Vorpahl
Content Development Specialist, Communications and External Affairs
Amelia Vorpahl leads the writing and strategic development of an array of content that advances the CSG Justice Center's goals, including policy reports, fact sheets, interactive tools, and web content. Before joining the CSG Justice Center, Amelia worked on the
...
policy and communications teams of several major advocacy organizations. In her previous role as senior communications manager at Oceana, Amelia led media outreach, content creation, and messaging strategy for various regional and federal campaigns. Amelia holds a BA in journalism from the University of Wisconsin and an MPA from the University of Texas at Austin.
Read More
  • Image for:
    Sarah Wurzburg
    Program Director, Behavioral Health
    Sarah Wurzburg oversees technical assistance focused on behavioral health, diversion, and reentry and serves as the lead for projects related to substance use, mental illnesses, and housing. She leads the work on the development of community responder programs, including a
    ...
    toolkit that supports sites in development of non-police responses to people in crisis. Previously, Sarah was a research analyst at the National Association of State Alcohol and Drug Abuse Directors, Inc., where she was the team lead for Youth and Women’s Services and was the primary author of research reports on youth substance use disorder treatment, driving under the influence, and Medicaid. Sarah has also worked as a juvenile court advocate and in community substance use disorder prevention. She received her BA from DePauw University in English (writing) and her MA in social services administration with a focus on policy analysis from the University of Chicago.
    Read More
  • Image for:
    Sheila Tillman
    Senior Policy Analyst, Behavioral Health
    Sheila Tillman provides technical assistance to Justice and Mental Health Collaboration Program grantees, focusing on the intersection of behavioral health and criminal justice. She also provides support on a variety of behavioral health and reentry projects. Before joining the CSG
    ...
    Justice Center, Sheila was a project assistant at Policy Research Associates in Delmar, New York, where she assisted on a variety of research projects. Sheila has a BA in criminal justice from Russell Sage College and received her MA in criminal justice from the University of New Haven.
    Read More
  • You might also be interested in

    New Support Centers Offer Behavioral Health Resources and Assistance

    New support centers offer resources, training, and assistance to improve responses to people with behavioral health needs.

    Read More

    Apply Now for Short-Term, Free Support in Winter/Spring 2021

    Apply for short-term, free support to better understand how people with mental illnesses move through your local criminal…

    Read More

    Federal Spending and Pandemic Relief Package Signed into Law

    Congress recently approved a $1.4 trillion omnibus spending package for 2021 and a $900 billion COVID-19 relief package.

    Read More