PDF version

GATHERING USER PERSPECTIVES TO SUPPORT EFFECTIVE COURT OPERATIONS

A Statement from the Action Committee

Our Committee supports Canada’s courts in their modernization efforts. It provides guidance for addressing challenges, and highlights opportunities and innovative practices to modernize court operations and improve access to justice for court users.

1. Context and Background

As noted in the companion to this publication, entitled Understanding User-Centred Justice, improving access to justice requires incorporating the needs and perspectives of the people who use the courts – including litigants, victims, and witnesses – when making decisions about court operations. There are many ways to learn about the perspectives of court users, and using different approaches will help leaders in the court system to both gather evidence to help them understand what is working in their courts and collect ideas for how to reform areas that need to improve.

2. Engaging with Users

A court may choose to seek information about user perspectives in a range of ways, depending on the nature and scope of the initiative or reform, the resources available, the population it wishes to reach, and the types of information it is seeking to gather. Gathering user perspectives does not always require a proactive engagement process. Spontaneous letters and complaints or queries to registry staff about user experience, as well as observing how court users interact with the system, can all provide valuable information. Understanding how people experience the courts can also involve reviewing data that has already been gathered by the court, or by academics, civil society, and government.

These approaches can go a long way to increasing a court’s understanding of user needs and wants, but there are times when a targeted approach will be necessary to gather the information needed to make appropriate decisions. At its most basic, an engagement strategy could involve placing a short survey on the court’s website to ask users about their experience. If a court wants feedback on a wide range of topics or is looking for input to guide reforms, though, it may be necessary to host a townhall meeting or conduct an in-depth written survey, one-on-one interviews, or a focus group of affected individuals.

All approaches for gathering information work best when used in combination with others. For example, starting with a few open-ended interviews can help to identify trends or themes that could then be explored with more people through a survey. Focus groups or interviews with select respondents could be used to add qualitative data to the quantitative information gathered through an initial survey or to validate assumptions contained in reports about challenges in the justice sector. Pairing user feedback, no matter how it is obtained, with self-assessment or direct observation can also make it easier to evaluate the success of an initiative.

2.1 General Considerations

Regardless of the chosen method(s) of engagement, some common design considerations apply to developing an engagement strategy.

2.1.1 Collecting Qualitative and Quantitative Data

When gathering user perspectives, it is important to collect both qualitative and quantitative data. Quantitative data helps to provide a sense of prevalence or proportion and is measured using numbers and values. Qualitative data, on the other hand, is descriptive and is used to understand underlying causes and considerations. It is important to be aware of what kind of information will be provided based on the questions that are asked.

• Survey questions that use a rating scale or check boxes will result in quantitative For example, they might show how many users are satisfied with the court’s processes or how many times a user appeared before a judge before their matter was resolved. When collected on an on-going basis, quantitative data can help measure whether an anticipated result has occurred. For example, if a court changed its processes with the intention of improving user satisfaction, it can use numerical assessments of satisfaction before and after the change to measure whether satisfaction has, in fact, improved.
• Qualitative data comes from open-ended questions. These questions give users the space to explain the “why” behind quantitative responses and are important both for allowing users to tell their stories and for testing assumptions about causation. For example, while a checkbox can show whether levels of user satisfaction have changed, it won’t be possible to understand why without qualitative questions. Without open-ended questions, a court may end up missing significant context to guide its reforms.

Written surveys can be designed to collect either quantitative or qualitative data, while personal approaches such as interviews or focus groups are most effective for gathering qualitative information.

2.1.2 Developing Effective Questions

The way in which the questions are developed can influence the success of an engagement process. Here are a few tips to keep in mind:

• Keep language simple, clear, and neutral. Be careful about leading questions and avoid negative sentences or emotive words.
• Ensure any scaled questions include equal positive, neutral, and negative options. For example, a question might ask a respondent to rank a statement based on whether they strongly agree, agree, are neutral, disagree, or strongly disagree.
• Be specific and avoid vague terms that might be interpreted differently by different people. Consider briefly defining terms so that all respondents will have a common understanding of the words that are used.
• When appropriate, include a reference period to situate the question, such as “in the past twelve months” or “the last time you were at the Registry counter.”
• Provide participants with the opportunity to qualify or expand upon their answer.

Useful questions may be drawn from court survey exercises in other jurisdictions. Potentially inspiring models include the United States’ Center for Court Innovation’s Can Courts be more User-Friendly? How Satisfaction Surveys can Promote Trust and Access to Justice and the United States’ State Justice Institute’s Court Voices Project.

2.1.3 Structuring the Engagement

In addition to asking clear questions, it is important to consider how they are presented, as well as what additional information should be provided to participants.

• Consider how to minimize any harm that the engagement process could cause court users. Focusing on respect, accountability, and transparency in both the design and delivery of the engagement can help prevent re-traumatization or users feeling compelled to answer due to the power imbalance between themselves and the court. Steps may include developing a plan to protect privacy and paying particular attention to the potential vulnerabilities of users you are planning to engage with.
• Begin with a brief statement explaining the purpose of the engagement, the parameters around the input being sought, who will see the data, and how the information being collected will be stored, aggregated, and used. This statement serves two functions. First, it ensures that participants are informed about how the information they share will be revealed. And second, it helps to avoid an incorrect understanding of the scope of the project, which can make participants less willing to engage in the future because they are disillusioned their suggestions are not reflected in a final report or eventual reforms.
• Begin with general questions, and then build to more specific ones. In all formats, but especially in focus groups and interviews, this approach can help build trust and comfort by the time more difficult questions are asked.
• Consider whether to include demographic questions and, if so, whether to include a “prefer not to answer” option. On the one hand, this information can help in identifying trends and understanding who uses the services under review. However, these questions can also make a survey more cumbersome or discourage participation by individuals who are uncomfortable with personal information being linked to their comments.

2.1.4 Engaging with Indigenous People and Marginalized Court Users

It is important to seek input from all court users, but those who have the greatest challenges accessing justice may be the most difficult to reach. As such, a court may wish to pay particular attention to ensuring it hears from Indigenous people from both urban and remote communities and marginalized court users.

• Work with community organizations to reach marginalized groups. For groups that may be distrustful of justice sector institutions and are over-represented in the criminal justice system, such as Indigenous communities and Black people, it is often most effective for individuals who already have relationships of trust to lead the engagement. In addition, working with existing groups and networks can minimize the added pressure that consultations could place on busy community members.
• Whenever possible, meet with Indigenous communities on their own territory and ensure there is adequate time to listen and engage in meaningful dialogue. This may include returning to the community to share what was heard during consultations and how that information will be used.
• Consider how to gain access to the voices of accused or convicted persons who are in custody.

2.2 Specific Engagement Approaches

There are many ways to engage with court users. In some situations, it may work well to provide respondents with options for how they would like to answer the questions. For example, some participants may prefer to fill out a written survey while others would rather participate in an interview in which someone will ask them the questions and record their answers. A list of considerations to assist in selecting methods of engagement in a given situation is included as Annex 1.

2.2.1 Written Surveys

Surveys are one of the most common ways that institutions, including courts, seek user input. They can range in complexity from a few short questions that are always available on the court’s web portal asking users about their experience, to a longer and more targeted questionnaire designed to elicit feedback on a specific procedure or initiative.

Strengths of this Approach

• While potentially labour-intensive to develop, surveys can require fewer resources to administer than other engagement methods, particularly when administered online. The results of online surveys that do not include open-form questions can be automatically tabulated.
• Works well in situations where the goal is to:
  • Get a general sense of user satisfaction, detect overall issues with service delivery, or fine-tune the approach once a general reform direction has been chosen.
  • Gather the type of information that can be reduced to an answer scale or other “check box” answers. This may include, for example, asking court users to rate their satisfaction with court processes or to rank the importance of different services.
  • Continue to gather feedback on the same matter on an on-going basis, from different users. On-going feedback can make it possible to identify trends in topics like user satisfaction or service times.

Limitations of this Approach

• Because there is no direct interaction with the respondent, it does not provide an opportunity to clarify answers or facilitate relationship-building.
• The impersonal mode of distribution can make it difficult to both gather a significant number of responses and reach marginalized court users.
• Absent demographic data, it can be difficult to know whether the survey results are reflective of the population the court serves.

Development and Delivery Tips

When planning a survey of users, a court may wish to:

• Make surveys available through a variety of means. This could include having hard copies on hand at the registry office; conducting a “blitz” where staff hand out copies and encourage people to fill them out on the spot; or circulating a link to an online survey with communications from the court or through its website or social media account.
• Keep surveys short and simple. Once a draft has been developed, consider whether all the questions are necessary, and workshop the questionnaire with court staff or other collaborators.
• Create space for open-ended remarks so that respondents can expand upon or qualify their responses if they wish.
• Use surveys with a Likert scale (for example, rating on a scale of 1 to 5) to gather quantitative information (numbers). This type of question can help in understanding user satisfaction or general public perception of justice services, and it is easy to collate the data from multiple surveys to get an overall sense of how the court is are doing.
• Use longer-form survey questions to gather more qualitative information (stories), which are essential in understanding the grounds for satisfaction or dissatisfaction or the reasons for an individual user’s stated perspectives or priorities.
• If encouraging people to answer a survey on the spot, offer them a small incentive for participating like a coffee or small gift card.

2.2.2 Town Halls

Hosting an open house or townhall meeting can be an opportunity to both share information on what a court has been doing and create space for input. If the goal of a townhall meeting is to build relationships with the community, court leaders may decide to open the floor for participants to provide input on whatever is on their mind. However, if the goal is to receive feedback on a planned or recently implemented change to processes, they may wish to ask more targeted questions about participants’ interest, priorities, or experiences.

Strengths of this Approach

• Can reach many respondents, especially if it is open to the public.
• Creates a reciprocal relationship with participants since the request for input is paired with the sharing of information from the court.
• Can create an opportunity for the public to speak about whatever is on their minds without the constraints of pre-determined questions.
• Works well in situations in which the goal is to:
  • Introduce a planned or recently implemented change to court procedures
  • Both share information and receive input
  • Hear from as many people as possible
  • Create a space for open-ended input from the community.

Limitations of this Approach

• Due to this format’s open nature and potential size, some attendees may not feel comfortable speaking about their experience or have enough time to be heard fully.
• May not provide the opportunity to follow up on or clarify input.
• If input is open-ended, may create frustration when ideas that are beyond the court’s ability to implement are ultimately not acted upon.

Development and Delivery Tips

When planning a townhall meeting, a court may wish to:

• Send targeted invitations to representatives of specific populations they would like to hear from in addition to a general open invitation.
• To manage participant expectations, establish the parameters of the input before opening the floor for responses.
• Make sure that there is a way for participants to know how their input is being used, such as through a “what we heard” report.

2.2.3 Focus Groups

A focus group is a small group of people who participate in a guided discussion. There are a range of methodologies and tools that can be used to facilitate discussion in this format. Targeted groups can also be used to test new products such as online portals and other web services.

Strengths of this Approach

• Makes it possible to target a specific population, which can be valuable to either gather feedback on the needs of a particular group of users or address a gap in the court’s understanding of needs and preferences.
• Works well in situations in which the goal is to:
  • Address complex matters that require probing or follow-up questions.
  • Generate ideas: participants can build off each other’s responses.

Limitations of this Approach

• A court will need to either dedicate staff or hire someone to conduct the focus group sessions.
• To be most effective, a focus group may require several hours of participants’ time. This can be a disincentive for busy people to participate.

Development and Delivery Tips

When designing a focus group, courts may wish to:

• Meet each constituency separately before bringing together groups with opposing views. These separate meetings create an opportunity to identify where perspectives and interests both differ and converge and can support consensus-building when the two sides are brought together to solve a common problem.
• Consider whether it will be more effective to meet with the focus group in person or virtually. Virtual meetings may be easier to attend, especially if potential participants are geographically spread out. However, it may be easier to build the necessary rapport with the people being consulted if through face-to-face interactions.
• Consider engaging a third party to conduct the focus groups, particularly in situations where the neutrality of the facilitator may add credibility to the findings.

2.2.4 Interviews

Interviews involve a one-on-one exchange to gather information. They can create space to ask probing or follow-up questions to dig into the “why” behind initial responses.

Strengths of this Approach

• Creates an opportunity to gather feedback on the needs of a particular group of users or address a gap in a court’s understanding of user needs and preferences.
• Creates space to listen, without sticking to an overly formulaic “script”. This can be particularly valuable for seeking input from Indigenous people, who may find it difficult to fully tell their story within the linear nature of a planned questionnaire, or court users who have special communication needs, such as those who live with cognitive, hearing or speech disabilities or whose first language is not English or French.
• Works well in situations in which the goal is to:
  • Address complex matters that require probing or follow-up questions.
  • Address sensitive subjects that give rise to strong emotions participants may not feel comfortable discussing in front of other people.
  • Gather qualitative information about user satisfaction, including understanding the “why” behind an initial response.
  • Develop relationships, since there is more space for open exchange in a one-on-one setting.

Limitations of this Approach

• A court will need to either dedicate staff or hire someone to conduct interviews.
• It can be time-consuming, particularly without training, to make sense of the information that comes out of interviews.

Development and Delivery Tips

When designing user interviews, courts may wish to:

• Collaborate with a law school program or academic to recruit assistance to conduct interviews.
• Ensure whoever is conducting interviews is trained in trauma-informed approaches, as relevant.
• Consider engaging a third party to conduct the interviews, particularly in situations where the neutrality of the facilitator may add credibility to the findings.

3. Inspiring Examples

Many Court leaders across Canada have reached out to a range of individuals and organizations, including end users, to learn about their experience. Information gathered from these engagements has supported a greater understanding of what is working in the courts and what needs to be improved to increase access to justice for people in Canada.

3.1 Surveying Users on Pandemic-Related Adaptations (British Columbia Court of Appeal)

In the summer of 2021, the British Columbia Court of Appeal conducted a survey to take stock of user experience with its pandemic-related adaptations. The goal was to support longer-term decisions on which new processes to retain as public health measures were lifted. Usually, the Court collects ad hoc and incremental feedback on changes to its procedures through individual letters to the registry or other informal means of communication. However, considering the drastic and extensive nature of pandemic-related changes, the court wanted to both ensure it was effectively communicating court processes to users and understanding how these processes were being received.

The survey was developed by a small group of judges, with support from the Registrar, and then distributed online using the Google Forms application. The Court also accepted printed responses, entering them into the online system so that all results were consolidated. In preparing the survey, the court developed a mix of open-ended and closed questions, and law clerks tested it to ensure that it would not be too long. The survey was available for 3 months, with the link advertised on the court’s social media and website, as well as on cards at the Registry counter.

The Court received 120 responses from a range of individuals that, when tabulated, helped it to identify trends in user preferences for keeping, discarding, or modifying procedures that were initiated during the pandemic. The survey was an effective way to find out how the Court’s recent reforms were being received, and was instrumental in supporting decisions about which new technological solutions would be maintained. 

3.2 Engaging with End Users to Identify Reform Options (Manitoba Family Law Modernization)

Focus groups were a major element of Manitoba’s recent family law modernization engagement strategy. Drawing from several engagement methodologies, the Family Law Modernization team co-designed solutions for family justice services with Manitobans. After a first round of consultations, the team mapped out the journeys that family law clients take through the justice system, and then used these journey maps to develop a strategy and draft legislation.

This strategy guided two additional half-day sessions, which aimed to address the issues identified in phase one. Attendants to these sessions included subject matter experts, individuals going through separation or divorce, interested contributors unconnected to the topic, and organizations representing clients with unique access needs.

During the sessions, participants took part in a variety of activities that led to nearly 250 ideas for improving access to family justice. By clearly defining the scope of the project, the team tried to ensure all ideas generated during the engagement sessions were eligible for eventual consideration.

When these ideas were analyzed, six “big ideas” were identified for reform: a responsive web application; timelines, incentives, and penalties; support and training for parents; support and education for children; mandatory mediation; and improved service coordination. The engagement sessions allowed the government to identify both problems and solutions that might not have emerged if it had only engaged with professionals in the family justice system.

More information about family law reforms in Manitoba can be found in the Action Committee’s Case Study on this topic. Further details about the Family Law Modernization Public Engagement exercise are contained in the Government of Manitoba’s Report on the process.

3.3 Seeking Missing Voices (Nova Scotia Courts’ All Courts Virtual Courts Committee)

During the pandemic, the Nova Scotia Courts established the All Courts Virtual Committee to oversee the launch of fully virtual court proceedings in that province. Concerned with how the transition to virtual proceedings may be affecting historically marginalized court users, the Committee launched a project to gather their perspectives. The Committee collaborated with Pro Bono Dalhousie at the Schulich Faculty of Law, working with three student volunteers to gather feedback from the community, based on a questionnaire developed by in the judicial members of the committee. The questionnaire was designed to be concise and to focus on the study's primary objectives.

The Committee, with support from the Executive Office of the Judiciary, provided the student interviewers with a script to contact community organizations that work with the target populations. Students were also given key messages to guide their discussions and ensure that participants understood the scope of the project and how the Committee intended to use the information they gathered. The students met regularly with the judicial lead for the project to address any problems. Participants could choose between a short interview to answer the questions or submit written responses. Ultimately, one of the greatest challenges was finding court end-users to participate in the study, but valuable information received from community leaders with experience supporting court users was used to fill this gap.

The project resulted in a What we Heard summary with highlights from the full report produced by the student volunteers. This project provided valuable insights to guide the courts of Nova Scotia in their post-pandemic modernization plans.

3.4 Making User Feedback a Reflex (British Columbia Court Services Branch)

In addition to surveys and formal evaluations of services and programs, the British Columbia Court Services Branch conducts informal interviews – which it calls "interventions" – to gather in-the-moment feedback from court users. This feedback is sought right after their court transactions while their experiences are still fresh. Staff conducting the interventions ask brief questions, such as the purpose of the individual’s visit that day and whether they required assistance filling out forms. A small token, such as a gift card for coffee, is offered to participants to thank them for their time.

ANNEX 1
CONSIDERATIONS FOR CHOOSING AN ENGAGEMENT APPROACH

As outlined above, an effective feedback strategy will likely involve using multiple engagement approaches. Asking the following questions can help a court to identify which approach(es) will work best in a given context.

Is the goal to collect on-going feedback, or to capture a given period in time?
Spontaneous communication	Written Surveys	Townhalls	Focus Groups	Interviews
Use to gather on-going feedback about whatever is on users’ minds	Use to gather information about a specific initiative or event	Use to capture information at a specific time
	Use to gather on-going feedback on user experiences and satisfaction

Would stories or numbers be more useful?
Spontaneous communication	Written Surveys	Townhalls	Focus Groups	Interviews
Letters and other spontaneous channels allow users to provide as much background information as they would like	Use surveys with a Likert scale (e.g., rating on a scale of 1 to 5) to gather quantitative information that can be easily collated to help understand user satisfaction or experiences	Use to primarily ask open-ended questions
	To understand the grounds for satisfaction or dissatisfaction, use open-ended survey questions to gather more qualitative information	Use to maximize the number of people who can share their stories	Use when relationship-building is an important aspect of the engagement process

What resources are available to devote to gathering user perspectives?
Spontaneous communication	Written Surveys	Townhalls	Focus Groups	Interviews
Can be done with relatively few resources	Requirements will depend on the scale of the survey: can require fewer resources to administer than other models, but development can be labour-intensive	Can provide an opportunity to hear from many respondents with relatively low human resource needs	Can be more resource-intensive unless done on a limited or ad hoc scale
			Collaborating with a law school program or academic can be a way to reduce the financial requirements of these methods

Are there specific sub-populations that it is important to reach?
Spontaneous communication	Written Surveys	Townhalls	Focus Groups	Interviews
Not possible to target specific populations	Can be difficult to reach specific target populations	Can be difficult to reach specific populations unless the townhall specifically targets their community	More likely to reach specific target populations
			Working with trusted intermediaries in the community can help to reach populations that are marginalized, have special needs, or mistrust the justice system

What is the nature of the topic on which input is being sought?
Spontaneous communication	Written Surveys	Townhalls	Focus Groups	Interviews
Allows users to share feedback about whatever is on their minds	Well suited for queries that can be reduced to an answer scale or other “check box” answers	Can allow users to share feedback about whatever is on their minds	Work well for complex matters that require probing or follow up questions
		Works well for presenting a specific initiative and then immediately receiving feedback on it	Ideal for generating ideas, as they allow participants to build off each other	Appropriate for sensitive subjects that participants may not feel comfortable discussing in front of other people