Choosing the Right Data Strategy for Behavioral Health and Criminal Justice Initiatives
Collecting data and evaluating programs should be a critical element of all criminal justice and behavioral health initiatives. However, the data collection process can be overwhelming and unclear. Programs need to determine what data to collect, how often, and how to use them to assess impacts, set goals, and make improvements. This brief will help project coordinators and research partners working with government entities or nonprofit agencies identify, collect, and prioritize the appropriate data as well as provide recommended metrics for behavioral health and criminal justice-related programs. Photo credit: ConnectVector via Shutterstock.
Choosing the Right Data Strategy for Behavioral Health and Criminal Justice Initiatives
Collecting data and evaluating programs should be a critical element of all criminal justice and behavioral health initiatives. Data help illustrate what is working, what needs to change, and which stated outcomes are achieved. In the short term, data can be used to set goals and assess needs, and in the long term, data can help answer research questions, evaluate program outcomes, and inform continuous program improvement.
Despite the benefits of data collection, however, the process can be overwhelming and unclear. Programs need to determine what data to collect, how often, and how to use them to assess impacts, set goals, and make improvements. There are often data collection requirements from funders, whether they are federal (such as the Justice and Mental Health Collaboration Program), state, local, or philanthropic. This can lead to programs needing to quickly develop data collection and analysis processes. Often, programs also work to develop research partnerships through local agencies or universities to support development of data collection plans, logic models, and analyses and to conduct evaluations.
This brief will help project coordinators and research partners working with government entities or nonprofit agencies identify, collect, and prioritize the appropriate data as well as provide recommended metrics for behavioral health and criminal justice-related programs.
Selecting Key Metrics
Project coordinators, researchers, implementation teams, and advisory groups need to identify what data are appropriate to collect and track for their program goals, which can be overwhelming to narrow down. Many of the decisions about what metrics to track will depend on a specific jurisdiction’s program and its priorities. It is paramount to ensure that the behavioral health and criminal justice partners agree on the definitions for each of the key metrics. Developing a shared understanding of and language about the information can be an important step in the process.
When selecting key metrics, elements to consider include collecting data at both the individual and systems levels; using quantitative and qualitative data; and prioritizing equity by
| Key Benefits of Data Collection | ||||
|---|---|---|---|---|
| Increases program accountability | Streamlines decision-making | Enhances understanding of what contributes to program success or failure | Optimizes service delivery | Communicates the program’s value to key stakeholders |
disaggregating data by race, gender, ethnicity, and other characteristics to address disparities (see the Appendix for types of data collection). There are many pre-established behavioral health and criminal justice metrics to draw on when determining what to collect, including the following:
Interface Process Measures
Gauge to what extent behavioral health and criminal justice systems interact and collaborate to identify and treat people at both the system and individual levels. Key metrics include rates of identification and referral, levels of program engagement and completion, insurance enrollment rates, and capacity of systems and individuals to provide or access programming.
Housing Instability
Measures how many people are having trouble finding and keeping stable housing within the criminal justice system. Measurements include number of people booked, screened, or connected to housing services, as well as staff capacity for homelessness support. Housing tenancy metrics also identify the number of people who remained housed for specific time periods.
Police-Mental Health Collaboration (PMHC)
Assesses how effective collaboration programs between law enforcement and behavioral health partners are at safely and appropriately responding to people with mental health needs. Key data points include the level of community need (number of calls or arrests) as well as administrative outcomes, such as the number of officers receiving specialized mental health and stabilization training.
Stepping Up Initiative
Helps counties create programs to reduce the number of people with mental illnesses in local jails. There are four key metrics for people with mental health needs: how many are booked into jail, their average length of stay, the percentage connected to treatment, and recidivism rates.
Data Sources
There are different sources from which to collect data, which can impact how the data can be used. There should be a structured method of collection while using any source to ensure good data quality.
| Data Type | Source | Usually Contains | What it Tells You |
|---|---|---|---|
| Quantitative | • Management information systems/administrative data • Case management software • Special data collection | These types of quantitative data sources contain objective information and statistics, like demographic characteristics of participants; dates; and types of events, such as timing of participation in a program or service, results from screenings or assessments, case management goals, and other behavioral health metrics. | These sources can inform researchers about participant characteristics and outcomes, the number of processes people participate in; the length of time between events; and participation rates of screening, assessment, and treatment. |
| Qualitative | • Organizational surveys • Participant surveys • Interviews • Focus groups | Descriptive sources like surveys and interviews can include directed questions, subject matter interviews, and other information that explores program context and focuses on the subjective opinions and perspectives of individuals doing the work or participating in a program. There are some standardized survey instruments that can be used for some topics. | Qualitative data sources aim to capture human experience by assessing the opinions and perspectives of program staff, administrators, partners, and participants. They can identify common themes and examine support for program changes or innovation. |
Determining How to Use Data
Beyond just collecting data, researchers also need to analyze the data to support evaluations and continued funding and assess program impacts, processes, or outcomes. These types of evaluations are important for helping to determine whether a program is functioning as intended and meeting its goals as well as to inform continuous improvement. The specific collection and evaluation option chosen by a researcher also needs to connect to their objectives, as different types of evaluations assess different program elements.
| Types of Data Evaluation | What It Is | What It Tells You | Considerations |
|---|---|---|---|
| Implementation study | Data are used to address and improve health care implementation and delivery, including how to use research in practical application | Identifies factors influencing implementation processes and outcomes, including how to introduce potential solutions into a health system or scale them | Resources: Budget, staff, expertise level, existing secondary data or need to collect primary data Stakeholder engagement to capture all points in implementation process |
| Process evaluation | Data are used to report on the progress of a program or improve future program procedures | To what extent programs and policies have been implemented as intended and what the results have been | Scope: Evaluating one aspect of a program’s activities or the program as a whole? Timing: Starting at the beginning of implementation or after it has already started? Resources (see above) |
| Outcome evaluation | Data are used to examine the effects or results of a program | How the program has impacted the target population and whether it has been effective in meeting objectives | Timing: Program must already be fully implemented Scope: Assessing long- or short-term impacts? Resources (see above) |
| Continuous quality improvement | A multi-stakeholder quality management process uses data to focus on activities to improve community health needs Involves strategies to ensure that the innovation is being implemented and to address barriers to change | Measures consistency, success, and effectiveness of behavioral health services | Look at systems, not just people. Identify and engage multisystem stakeholders Sustainability: Ensure analysis and improvement is ongoing, not a one-off project |
| Cost benefit/ cost savings | Data provide a way to compare cost of program with its effects | The direct and indirect costs of program resources compared with outcomes | Ensuring accuracy of costs and benefit estimates. Some benefits may not be fully measurable. |
| Impact evaluation | Data assess how the program affected outcomes, including to what extent any effects were intentional and if effects would have happened in the program’s absence | Helps determine whether the program is meeting objectives, including how and why it is working or not | Identifying a comparison group to determine the counterfactual (would effects have otherwise happened) |
| Developmental evaluation | Real-time data are used to inform ongoing decision making incorporated into program development and implementation | What initial results say about expected progress and how data on changing circumstances can lead to program adaptations in real time | Often requires extensive field work as well as stakeholder collaboration |
| Effectiveness evaluation | Data help assess whether a program met its intended goals and objectives | How successful a program was in meeting its objectives, including assessing results vs. expectations and why a program may have failed or exceeded timelines. Focused on long-term systematic and aggregated impact. | Timing: Program must be fully implemented for a period of time Readiness: Jurisdiction must be willing and able to look at systems level Resources (see above) |
Six Key Tips for Collecting Data and Selecting Evaluation Methods
1. Develop a research partnership
Partner with researchers in local agencies or universities to build data capacity, set up data collection processes, create data-sharing or security protocols, and support the development of evaluations.
2. Develop a logic model and determine key metrics
After the program has identified its vision, mission, and goals, develop a logic model for the program to help select metrics. This process will help ensure that the goals of the program are matched to the data that are prioritized for collection.
3. Establish a clear data collection plan
Review and determine the different sources (e.g., case management system; management information system, such as jail management system; Excel spreadsheet; organizational survey; participant survey; focus groups; interviews, etc.) of data available for the program. Develop a data collection plan based on the available data sources and the data use agreements that have been set up between criminal justice and behavioral health partners. A clear collection process should also include standardized procedures, quality assurance, and privacy and security protocols. Partners should work with researchers to outline who is responsible for each data source; where the data are stored; how they are used; and how they are reported back to program staff, community stakeholders, and agency leaders.
4. Ensure organizational buy-in
Educate staff on how the data are used and how to give feedback. The logic model can be used to ensure a common understanding of how the program works. The model should be updated as changes to the program occur. Everyone in the program should view data collection as a core procedure that is built into routine operations, not just a one-time activity. Staff need to be continually involved in conversations about how the data are used to support improved outcomes for the participants.
5. Develop an evaluation plan
Collaborate with a research partner to develop an evaluation plan, which will help ensure that the data collection, logic model, and eventual evaluation are all coordinated.
6. Craft a compelling story
The program should determine what audience it should reach and how this audience wants to receive information. Compelling messaging that helps contextualize data for different audiences is essential. When data are shared on an ongoing basis with stakeholders through an advisory group (such as a criminal justice behavioral health council or criminal justice coordination council) that includes the heads of agencies, county commissioners, and community members, it can help support the initiative’s long-term success, funding, and ability to innovate.
Appendix: Criminal Justice and Behavioral Health Metrics
| Objective: Identify and refer people in the justice system with behavioral health needs to the appropriate care and services. | ||
|---|---|---|
| Interface Process Measures: Set 1 | Metric | Description |
| Screening rate | Percentage of people who screened positive for behavioral health issues using a validated screening tool | |
| Clinical assessment rate | Percentage of people clinically assessed as needing behavioral health services using a validated assessment procedure | |
| Referral rate | Percentage of people screened or assessed for behavioral health conditions who are referred to a treatment program or linked to services | |
| Initiation | Percentage of people who are referred to behavioral health services who start services within 14 days of the screening or clinical assessment | |
| Objective: Ensure that people in the justice system participate in and complete the appropriate behavioral health treatment. |
||
|---|---|---|
| Interface Process Measures: Set 2 | Engagement | Individual participates in at least two treatment sessions within a 30-day period of initiation |
| Retention | Length of stay in treatment for different levels of care, including outpatient treatment, intensive outpatient treatment, therapeutic community, and counseling services | |
| Successful completion rate | Percentage of individuals (people who are arrested, defendants, incarcerated people, people on probation/parole) who successfully complete treatment | |
| Medication-assisted treatment rate | Percentage of individuals (people who are arrested, defendants, incarcerated people, people on probation/parole) screened or assessed for behavioral health conditions who require medication and who are provided medication in a justice setting | |
| Compliance with treatment plan | Percentage of individuals who comply with their treatment plans developed by the behavioral health specialist and the individual | |
| Objective: Offer access to a seamless system of care that provides people in the justice system who have behavioral health needs with the appropriate services. |
||
|---|---|---|
| Interface Process Measures: Set 3 | Continuum of care | Percentage of individuals who transitioned from one program to another (step up or step down) within 30 days |
| Continuity of care | Percentage of individuals who transitioned from one phase of programming to another within 30 days |
|
| Transitioning of care | Percentage of individuals who moved from one type of programming (i.e., substance use disorders or mental illness) to another, such as vocational or educational services |
|
| Objective: Ensure that a range of behavioral health programming is available in both correctional and community settings to effectively address people’s needs. |
||
|---|---|---|
| Interface Process Measures: Set 4 | Uniform screening protocols | Core set of screening processes and criteria accepted by justice and treatment agencies to identify individuals with substance use disorders and mental illnesses |
| Insurance enrollment | Percentage of individuals in the justice system who are enrolled in health insurance |
|
| Enrolled in insurance at time of release | Percentage of individuals released from prison or jail who are enrolled in health insurance |
|
| Responsivity rate | Percentage of individuals in the justice system who can access appropriate services for their specific needs |
|
| Availability of programming | Capacity of the system to provide appropriate levels of care for individuals (in prisons, probation/parole, jail, or community settings) who have specific needs |
|
| Access rate | Percentage of individuals who can access a given program | |
| Participation rate | Percentage of individuals who partake in programs during their period of incarceration or supervision |
|
| Objective: Expand access to safe and stable housing supports for people in the justice system with behavioral health needs. |
||
|---|---|---|
| Metrics for People Experiencing Housing Instability or Homelessness | Homelessness rate | Percentage of people who are experiencing homelessness and booked into jails or prisons |
| Housing instability rate | Percentage of people screened for housing instability | |
| Service connection rate | Percentage of people connected to housing or homeless services (e.g., permanent housing, transitional housing, use of emergency shelters) |
|
| Staff capacity for homelessness support | • Percentage of dedicated or partially funded staff for homelessness outreach or case management • Percentage of staff who receive education on homelessness and homelessness support services |
|
| Housing tenancy | Percentage of individuals who remained housed at specific intervals (e.g., 90 days, 6 months, 12 months, 24 months after release) |
|
| Objective: Effectively respond to people with behavioral health needs who come into contact with law enforcement. |
||
|---|---|---|
| Police- Mental Health Collaboration (PMHC): Level of Need | Level of need | Percentage of calls for service involving people who have mental health needs |
| PMHC: Minimized Arrests | Arrest rate | Percentage of arrests involving people who have mental health needs |
| Arrest rate | Percentage of people who have mental health needs who have more than 1 arrest in last 12 months |
|
| Disposition/ resolution of call | Disposition/resolution of call (e.g., arrest, resolved at scene, transported for voluntary evaluation, detained for involuntary evaluation, referral to mental health treatment) |
|
| PMHC: Reduced Repeat Encounters | Repeat call rate | Percentage of repeat calls to the same location |
| PMHC: Reduced Use of Force | Use of force rate | Percentage of encounters with people who have mental health needs where force was used |
| Use of force rate | Type of force used by officers during encounters with people who have mental health needs |
|
| Officer injury rate | Percentage of police encounters with people who have mental health needs that result in injury to officers |
|
| Objective: Effectively respond to people with behavioral health needs who come into contact with law enforcement. | ||
|---|---|---|
| PMHC: Administrative and Process Outcomes | Officer training rate | Percentage of officers receiving mental health and stabilization training |
| Officer training rate | Percentage of officers trained in select PMHC response models | |
| Officer training rate | Percentage of shifts covered by trained officers | |
| Officer training rate | Percentage of dispatchers trained on PMHC response models | |
| Officer training rate | Percentage of mental health-related calls receiving a response by a trained officer |
|
| Objective: Reduce the prevalence of people with behavioral health needs in local jails. | ||
|---|---|---|
| Stepping Up Initiative: 4 Key Metrics | Jail bookings | Percentage of people with mental illnesses booked into jail |
| Length of stay | Average length of stay for people with mental illnesses in jails | |
| Connection to treatment | Percentage of people with mental illnesses connected to treatment | |
| Recidivism rate | Recidivism rates for people with mental illnesses | |
Project Credits
Writing: Amelia Vorpahl, Sarah Wurzburg, Sheila Tillman, CSG Justice Center; Faye Taxman, Center for Advancing Correctional Excellence at George Mason University
Research: Sheila Tillman, CSG Justice Center
Advising: Faye Taxman, Center for Advancing Correctional Excellence at George Mason University
Editing: Leslie Griffin, Emily Morgan, CSG Justice Center
Design: Michael Bierman
Public Affairs: Ruvi Lopez, CSG Justice Center
Web Development: Eleventy Group
This project was produced in collaboration with the Center for Advancing Correctional Excellence at George Mason University and was supported by Grant No. 2019-NT-BX-K001 awarded by the Bureau of Justice Assistance. The Bureau of Justice Assistance is a component of the Department of Justice’s Office of Justice Programs, which also includes the Bureau of Justice Statistics, the National Institute of Justice, the Office of Juvenile Justice and Delinquency Prevention, the Office for Victims of Crime, and the SMART Office. Points of view or opinions in this document are those of the author and do not necessarily represent the official position or policies of the U.S. Department of Justice.
ABOUT THE AUTHORS
On March 9, 2024, President Joe Biden signed a $460 billion spending package for Fiscal Year 2024, allocating…
Read More
Empathetic Policing: Mason City Police Department Launches Virtual Reality Training Program to Help Officers Better Understand Behavioral Health Crises
Read More
Biden Signs Six-Bill Spending Package Funding Key Criminal Justice Programs
On March 9, 2024, President Joe Biden signed a $460 billion spending…
Read More
Taking the HEAT Out of Campus Crises: A Proactive Approach to College Safety
The sharp rise in school shootings over the past 25 years has…
Read More