Evaluation of the immigration and refugee panel standards implementation: Executive summary

Prepared by WSCS Consulting Inc.
December 2016

Last updated: June 2017

The full report is available upon request.

1. Executive summary

Legal Aid Ontario (LAO) is a public agency responsible for providing access to justice for low income individuals throughout Ontario. It has a statutory mandate to provide high quality legal aid services to clients. An important element to achieving this mandate is the empanelment process for lawyers who represent legal aid clients.

Following an extensive consultation process, LAO adopted new standards for the immigration and refugee panel. In January 2015, it began to implement these standards in a very new and unique way. Empanelment has typically been relatively simple with an application process at any time. Once a lawyer has been admitted to the panel, they are on indefinitely unless they resign or are removed through a panel removal process. The implementation of the new immigration and refugee panel standards was different. In order to protect its vulnerable clients and attempt to mitigation organizational risk faced by LAO, it was decided that every lawyer, existing or new, was required to apply to be on the panel. The process would include a rigorous assessment of the lawyer's work against the standard.

This report provides an evaluation of the implementation of the immigration and refugee panel standards with some options and recommendations regarding the development and implementation of panel standards related to other LAO service provider panels.

In summary, the implementation of the new standards was considered to be a success by most including the external stakeholders. It appears that the short-term outcomes desired by LAO have been achieved. The challenge for LAO is to ensure that the ongoing oversight mechanisms that have been set up are effective, sustainable and efficient. This is an area where LAO needs to spend its energy. It seems on the surface, that many elements are in place to better manage the panel and ensure quality. However, the program needs to be better supported with policies, procedures and tools to ensure ongoing success. In terms of ongoing analysis, it will be very difficult for LAO to determine if it achieves its expected outcomes without some effort to capture the knowledge it has gained through this process.

In terms of the process itself, there was, in our view, an unnecessary administrative burden placed upon staff. Bottlenecks were created at the front end of the process due to the application deadline without sufficient resources, policies, systems or tools to manage the large workload. Any new initiatives of this nature need to have a very detailed project plan and processes that do not create large inventories of work, some of it for no value. Data collection and file management are particularly problematic but with some effort these can be resolve.

The new approach, which envisions a five-year empanelment for approved lawyers, needs to be assessed and a new process contemplated in order to ensure that a similar workload backlog is not created at that time.

While we believe that the exercise resulted in some positive outcomes, we caution LAO to think about how it would apply similar approaches to other panels. The reliance on staff and external resources to assess the applications was not without its cost and may not be sustainable. However, this report did not include an assessment of the current costs to manage the other panels in the current form. Therefore, this would need to be undertaken in order to determine if it would be more or less costly than the current approach, both in terms of dollars and quality. Similarly, the number of lawyers with conditional empanelment represents a significant amount of work both from mentors (external) and staff for ongoing oversight and management (internal). Should an implementation of this type be undertaken in the other panels and similar proportion of lawyers be put on conditions, LAO may not have the resources necessary to undertake such an endeavor.

All that being said, LAO should be commended for taking on this project and the success that they have achieved. Now the challenge for LAO is to review the implementation, assess the lessons learned and make adjustments. This report has many findings and recommendations that are intended to assist LAO in improving its processes and knowledge management practices.

2. Summary of findings

  1. Early signs indicate that the implementation of the new immigration and refugee panel standards will result in the achievement of longer term outcomes. LAO recognized that there were concerns with the quality of representation for immigration and refugee clients and it needed to act expeditiously. While some processes, procedures and analysis could have been improved, the overall implementation appears to be successful.

  2. By undertaking a very vigorous approach to implementation and requiring all lawyers to re-apply to the panel, LAO has heightened its knowledge of the panel and has a better understanding of the quality of representation.

  3. Other long term expected outcomes such as the targeted reduction in COS, audit breaches, investigations, complaints and appeals of 50% was not supported with a benchmark prior to the implementation, nor are they tracked. Therefore, it will be very difficult to assess if these expected outcomes are realized. However, the applicability of these outcomes to the implementation of panel standards is not clear and are questioned. These outcomes should be reviewed to ascertain whether or not they are applicable in determining the performance of the implementation of panel standards.

  4. The RLA was very supportive of the development and implementation of the new standards. The involvement of external stakeholders in the consultations as well as the implementation through PRC membership and the mentorship program, was a key to success.

  5. The panel standards, coupled with the best practices guide and quality service expectations, are very comprehensive and should allow lawyers to "self-assess" their work against the standard.

  6. While the implementation of the new panel standards was generally considered a success, it may not be feasible to implement new standards in a similar way for all of the other panels for several reasons. In particular, the cost may be prohibitive given the size of some of the other panels and the documentation approach may not be sufficient to assess the quality of representation in other areas of law. Concerns were raised during the implementation of immigration and refugee panel standards, that it was difficult to assess a lawyer's work on documents alone. While some felt that oral representations are often more important, there was a conscious decision not to go in that direction as it was believed that the cost prohibitive.

  7. The approach could work in other small panels such as CCB or CFSA with some tweaks to policies, procedures and process. It is evident that a "documentation only" assessment approach may need to be supplemented with other assessment tools.

  8. There is a distinct possibility that complaints will increase with the new panel standards. More detailed requirements, best practices and quality service expectations lend itself to more scrutiny by clients and lawyers who may be more critical if these are not followed.

  9. The governance structure for overseeing the implementation of the panel standards was generally viewed as effective, interviews revealed that there was a desire for a more structured approach to the implementation. A more detailed project plan with timelines and a budget was needed. As there was no project specific budget or project cost, we were unable to determine if the project came in on time or on budget.

  10. The implementation and application process resulted in a significantly reduced panel which should allow better ongoing oversight. However, the standards are more vigorous and the requirements to continue to meet standards are more complex, therefore, the amount of oversight required per lawyer will increase.

  11. At the time of the report, 88 lawyers (or 21% of the panel) were conditionally empaneled representing significant cost and ongoing workload for regular review by Panel Officers and Director Generals.

  12. While many conditions do not carry direct costs for LAO, they do require that ongoing monitoring take place which requires LAO resources. The number and expanse of some conditions seems to be excessive and burdensome. There does not seem to be consistency in applying conditions or which lawyer receives which conditions.

  13. The mentorship program has had mixed results and is at risk of losing dedicated mentors. According to the panel standards data, seventeen (17) mentees remain on conditions without a mentor, in part due to awaiting additional information. Some mentors indicated that they may not continue due to workload pressures and the processes. Some mentees have violated their conditions and placed mentors in a difficult position. Guidelines are needed to enforce conditions and assist both mentor and mentee to fulfil their obligations. LAO needs to assess the capacity of mentors, the criteria for mentees to be put in the mentoring program and the implications for long term sustainability.

  14. Not all policies and procedures were not in place prior to implementation but were developed over time. The detailed rating system introduced in August 2015, significantly improved LAOs approach to assessment of the applicant's work.

  15. The implementation was very detailed and required a significant number of documents and analysis. This was a very important "knowledge management" exercise which allowed LAO to gain significant and very in depth understanding of the immigration and refugee panel. Unfortunately, it missed a very big opportunity to capture that knowledge in a useable framework that will allow it to monitor its progress, not just on implementation but ongoing quality of representation. This may have been due to a lack of front end analysis and planning. However, it would have been worth the effort to find a more efficient manner to collect the data throughout the implementation as opposed to following the implementation. It will require significant and focused effort to bring the data together now.

  16. The rationale for a "one time" deadline for all applications is not evident. All panel members were required to apply on the same date creating significant administrative burden. Alternative risk based approaches or processing methods would result in reduced workload and improved efficiency.

  17. Administrative targets were not set in terms of each process step, only for the entire process. While advertised targets for application processes was six to eight weeks, these turnaround times were not tracked and there is no data to determine if they were achieved. The applicant was not made aware of the time it would take for the SRC review, decision regarding whether or not the application would be forwarded to the PRC or when a decision may be made.

  18. There are several non-value added steps in both the internal and external processes. Time per application and step was not tracked and therefore, cost per application processed are estimates only.

  19. The application itself contains important information that could be submitted electronically and populate a database directly without staff intervention. Documents that LAO already is in possession of would not need to be resubmitted.

  20. At the beginning of the process, there is a bottleneck as the Panel Support Administrative Assistant was responsible for all of the document management as well as capturing the data. LAO needed to provide additional front end resources and improved procedures to handle the inventory and/or change application approach.

  21. Data collection and file management practices need to be revamped. Key data elements have not been captured and documents are not catalogued appropriately nor secured to ensure that chain of decisions are protected.

  22. Policies and procedures are lacking in terms of ongoing oversight. LAO is at risk of returning to prior approaches of complaint based oversight without a strategy and resources. Policies are needed for managing conditions, assessing the effectiveness of each type of condition and managing the ongoing quality of representation.

  23. The dedication, commitment and teamwork demonstrated by LAO staff and management across the organization to ensure the success of this implementation was commendable. It provided LAO with an excellent pilot project on a "different" approach to panel management. As with any new venture, there will be lessons learned and areas for improvement but on the whole, the implementation was successful. LAO now needs to make some process changes to ensure that the program continues to be successful and data is available for ongoing monitoring.

3. Recommendations

The recommendations that follow are intended to assist the Immigration and Refugee Panel Services improve on its panel application process and ongoing panel management oversight obligations. In terms of other panels, the recommendations provide some guidance should LAO decide to take a similar approach to improved panel standards.

Recommendation 1

Undertake a lessons learned focus group with staff to develop improved processes and support panel.

Recommendation 2

Undertake a detailed data collection strategy and supporting technology for historical and ongoing analysis.

Recommendation 3

Review and formalize all policies and procedures to ensure consistent, efficient approach to application processing.

Recommendation 4

Revamp processes and file management practices to support ongoing panel management.

Recommendation 5

The Complaints policy and tracking of types of complaints should be updated to determine if panel standards implementation has raised the expectations and awareness of best practices.

Recommendation 6

Develop a mechanism to isolate audit breaches and investigations by area of law.

Recommendation 7

Develop a Logic Model reporting mechanism for ongoing analysis and measurement of outcomes. Review the stated outcomes in terms of applicability to panel standards. In particular, are reductions in COS requests, Audit Breaches and Investigations directly correlated to the implementation of panel standards?

Recommendation 8

Policies and procedures to address ongoing oversight and monitoring of conditions need to be formalized. Tracking of conditions, their cost of delivery and the effectiveness needs to be implemented. This will allow for the determination of the conditions that have impact on the quality of representation and those that do not.

Recommendation 9

Detailed mentoring guidelines should be developed to address obligations and consequences as well as ongoing oversight of the conditions.

Recommendation 10

Review the approach of assessing work on documents alone. Engage the mentors in an approach that may result in improved outcomes in terms of quality of work for mentees. Develop self-assessment tools for the panel to assess their own work against standards.

Recommendation 11

Undertake an assessment to determine the panel that creates the most risk for LAO in terms of panel management for the next panel standards review. It seems reasonable that the CCB or CFSA would be most similar to immigration and refugee panel.

Recommendation 12

Design the application process to allow for staggered application dates and pilots.

Recommendation 13

Develop comprehensive project plan with data collection strategy which includes technology solutions to support application processing, file and knowledge management.

4. Conclusions

In conclusion, the evaluators found that, overall the implementation of the new immigration and refugee panel standards achieved the short-term outcomes it had set out to achieve. The new panel is smaller and more manageable, there is greater knowledge of the panel's level of service and new supports for lawyers that are new to the area of law as well as dedicated panel members that may need assistance.

However, LAO needs to improve its ability to capture the knowledge it has gained from this exercise and put in place policies and procedures to assist in the ongoing oversight of this newly appointed panel. LAO needs to review the processes for conditional appointments, the criteria for which panel members are put on conditions and provided for supports that have a cost. There needs to be analysis of the effectiveness of the different types of supports and conditions to determine if they make a difference in the quality of services.

In terms of applicability to the other panels, a risk analysis is needed in order to determine which panel (s) need to have better management. The size and area law has a bearing on the approach but it became clear from the implementation of the immigration and refugee panel standards that proper supports, technology and planning is required in order to be successful. It would not be recommended to pursue a process requiring all panel members to apply on the same date which created an unnecessary administrative burden.

Appendix

Acronyms

BOC Basis of Claim Form
CCB Consent & Capacity Board
CCR Complex Case Rate
CFSA Child & Family Services Act
COS Change of Solicitor
DCO Designated Country of Origin
DG Director General
FTE Full-Time Equivalent
GTA Greater Toronto Area
H&C Humanitarian and Compassionate Considerations
IRB Immigration and Refugee Board
IRPA Immigration and Refugee Protection Act
JR Judicial Review
LAO Legal Aid Ontario
LASA Legal Aid Services Act
LSP Lawyer Services & Payments
LSUC Law Society of Upper Canada
POE Port of Entry
PRC Peer Review Committee
PRRA Pre-Removal Risk Assessment
RAD Refugee Appeal Division (IRB)
RLA Refugee Law Association
RPD Refugee Protection Division (IRB)
SRC Staff Review Committee