Performance Measures example of charts and reports
Published: July 28, 2016
Click here to download image.
Once the Clinic Information Management System (CIMS) has been implemented, clinics will be required to submit quarterly reports on their performance measures. Reporting of specified measures is mandatory for all clinics.
Purpose of performance measures
These clinic performance measures have been developed to meet LAO’s obligation under the Legal Aid Services Act and the Transfer Payment Accountability Directive, and address recommendations made by the Auditor General in his 2011 value for money audit. These performance measures are to be used by LAO and the clinics as a management tool, to provide the data for organizations to gauge whether they are achieving their goals and objectives in an effective and efficient manner. The reports should inform decisions and promote continuous improvement and fulfil obligations for accountability and transparency to the public. While the performance measures were designed to meet LAO’s legislative requirements, they will evolve over time and will provide a foundation for discussions between LAO RVPs and the clinics. You can get more detail on the development and intended use of the measures on the Legal Aid Ontario website.
The first full-year report will be available for the 2018/19 fiscal year. Development of reports are in progress to be delivered post CIMS go-live.
Reports will be submitted by the last day of the month following the quarter
All reports will be cumulative for the fiscal year, therefore the 4th quarter report will include a full year’s worth of data.
1. Performance measures generated by CIMS
LAO did not want to impose additional workload on clinics to docket administration functions. Clinics will be docketing for cases and initiatives, and administration will be calculated as the residual. Administration includes all activities not considered to be direct client services.
Direct client services include: referral, brief service, standard and test casework, and initiative files such as public legal education, community development, and law reform. Activities such as travel time, photocopying case file documents and case research are direct client services.
Administrative functions include activities not directly related to a case or initiative, such as: continued educational learning, admin tasks such as HR work, payroll, filing, and back office activities that are not considered to be direct client services.
1.1 Measure #1: Cases and initiatives: services provided, average cost per case, and cost of initiatives
1.1.1 Sample reports – Cases/Initiatives
Cost per case is calculated based on docketed time (user salary) plus disbursements, minus recoveries. (see definitions for standard case and salary). The report will be based on which case type is selected during the reporting period.
Complexity refers to cases that require significantly more time or resources than other cases of that type within an individual legal clinic’s practice. Complexity can result from a variety of factors, including complex client characteristics or barriers, legal complexities, challenging processes, requirements of multiple strategies to resolve issue, etc. You can “flag” complexity, but still docket less than two hours on a case.
Standard case summary data for reporting period
Total
Open cases at beginning of reporting period
425
New cases opened during reporting period
1,575
Total cases open during the period
2,000
Total cases remaining open at the end of the period (=active)
480
Total cases closed during period
1,520
Total cost of cases closed during period
$577,600
Closed cases with “flagged” complexity* indicator
812
Referrals during the period
900
Date of oldest open case
Jan 15, 2015
Test case summary data for reporting period
Total
Open cases at the beginning of reporting period
3
New cases opened during the reporting period
1
Total cases open during the period
4
Total cases remaining open at the end of the period (=active)
2
Total cases closed during period
2
Total cost of cases closed during period
$320,000
Average cost of cases closed during period
$160,000
Cost summary data in reporting period
Total
Total cost of all standard case work in period
$180,000
Total cost of all test case work in period
$200,000
Total cost of all initiatives in period
$120,000
The following table is based on the primary area of law selected in CIMS. The performance measures only report on these primary areas of law.
Standard case area of law
Standard cases opened during period
Standard cases closed during period
Housing
300
320
Social assistance
206
230
Workers compensation
550
600
Immigration/refugee/ citizenship
94
34
Employment
265
201
Other types of law
160
135
Total
1,575
1,520
Standard and test cases closing during period – Results achieved1
* Provide comments in performance measures submission report template.
1.1.2 Efficiency measure
Standard cases closed during period by primary area of law
Number
Housing cases
320
Social assistance cases
230
Workers compensation cases
600
Immigration/refugee/ citizenship cases
34
Employment cases
201
Other types of law cases
135
Total cases (closed)
1,520
Average time to close case (months)
6
Average cost per closed case
$380
Standard case primary area of law
Average cost per standard case
Housing cases
$425
Social assistance cases
$260
Workers compensation cases
$180
Immigration/refugee/ citizenship cases
$640
Employment cases
$250
Other types of law cases
$300
All closed cases
$380
Cost by initiative types during reporting period
Total cost in period
Public legal education/outreach
$32,000
Training
$14,000
Community development
–
Policy advocacy
$2,300
Law reform
$5,000
Systemic advocacy
$8,000
Partners/network/community groups
–
LAO/clinic committee & consultations
–
Inter-clinic groups
$4,500
Memberships
–
Media/communications
$12,000
Governance
$12,000
Professional development
–
Other
$2,050
Total initiatives
$91,850
1.1.3 Effectiveness measure
Measure based on standard cases closed in two previous fiscal years and current reporting period
2011
2012
Current reporting period
Average cost per standard case*
$300
$320
$380
Click here to download image.
1.2 Measure #2: Resource allocation
The measures below will be achieved through docketing.
Any Intake/referral/other service time that has been docketed to an Intake and/or associated Case that totals two hours or less will be allocated in the four components of direct client service. For example; if one hour has been docketed to an Intake and 30 minutes to the associated case, 1 hour and 30 minutes will be allocated to direct client service.
Standard case work = total time docketed to a standard case that is over two hours.
Test case work = total time docketed to a Test Case that is over two hours
Whether the Case is opened or closed or has been re-opened, the report is based on the time docketed falling within the reporting period.
1.2.1 Sample data
Hours docketed for open and closed cases (during period)
Number
Intake/referral/other service (2 hours or less of time docketed)
462
Standard case work (over 2 hours of time docketed)
5,950
Test case work (over 2 hours of time docketed)
1,130
Initiatives
5,300
Direct client service (sub-total)
12,842
# of FTEs
9
Total hours = 9 FTEx1561hrs
14,049
Administration and other (not docketed, but by subtraction), includes administration plus other activities such as continued educational learning, admin tasks, filing, etc.)
1,207
# of referrals
1,450
1.2.2 Efficiency measure
Resource allocation
2011 staff time (%)
2012 staff time (%)
2013 staff time (%)
Intake/referral/other service (two hours or less of time docketed)
4
4
3
Standard case work (over two hours of time docketed)
39
30
42
Test case Work (over two hours of time docketed)
15
12
8
Initiatives
31
41
38
Administration and other
11
13
9
Click here to download image.Chart 2. Resource allocation – 2013
1.2.3 Effectiveness measure
Click here to download image.Chart 3. Resource allocation
1.3 Measure #3: Service outcomes and client feedback
1.3.1 Efficiency measure
This is based on closed standard cases (over 2 hours of time docketed). The information will be captured when the case is closed.
Note: Withdrawn/discontinued = matter is withdrawn or legal action is discontinued by the clinic or client.
Service outcomes of closed standard cases in 12-month period as perceived by clinic staff
Click here to download image.Chart 4. Average standard case cost by service – Outcome 2013
Service outcomes of standard cases as perceived by clinic staff
Average cost per case, 2011
Average cost per case, 2012
Average cost per case, 2013
Successful or partially successful
$1,535
$1,620
$1,750
Unsuccessful, withdrawn, discontinued, or unknown
$1,380
$1,300
$1,283
Click here to download image.Chart 5. Cost per standard case by outcome (2011-2013)
1.4 Measure #4: Served versus denied and referred
1.4.1 Sample data
Intakes served and denied during reporting period
#
%
New cases opened (proxy for clients served)
2021
80%
Not financially eligible
186
7%
Not part of the clinic’s area(s) of law
52
2%
Services not offered by clinic
92
4%
Not within catchment area
138
5%
Conflict of interest found
14
0.6%
Other
26
1.1%
Total denied or not eligible
508
20%
Grand total
2,529
100%
1.4.2 Analytical information
Click here to download image.Chart 6. Percentage of clients services vs. denied service or not eligible
1.4.3 Effectiveness – to be calculated from previous table
Year of service
# of standard cases opened
# of intakes denied or referred
Total
Percent cases opened to total
2011
2,062
688
2,750
75%
2012
1,900
660
2,560
74%
2013
2,014
520
2,529
80%
Click here to download image.Chart 7. Standard cases opened as a percent of total intake, 2011-2013
1.5 Measure #5: Stage when case file outcomes are achieved with clinic involvement
Before Hearing includes: initiating legal action, mediation, negotiations, case assessment, pre-hearing
After Hearing includes: the hearing, post hearing, enforcement
Cases withdrawn or lost contact includes: service not provided
1.5.1 Sample data
Resolution stange
# of closed standard cases
%
Before hearing
3,500
64%
After hearing
2,000
36%
Cases withdrawn or lost contact
165
1.5.2 Efficiency – chart to be calculated from data in 2.5.1 and 2.5.3
Resolution stange
# of closed standard cases
Cost per standard case
Before hearing
3,500
$1,725
After hearing
2,000
$3,500
Click here to download image.Chart 8. Average standard case cost by resolution stage (2013)
1.5.3 Effectivness
Year of standard case closing
Average cost If standard case closed before hearing
Average cost if standard case closed after hearing
2011
$1,700
$3,200
2012
$1,860
$3,700
2013
$1,725
$3,500
Click here to download image.Chart 9. Average standard case cost by resolution stage, 2011-2013
1.6 Definitions
Admin time*
Admin time for a time period is the residual time calculated by taking the time docketed for referral and brief service, standard case work, test case work, and initiatives from the total hours of effective staff time calculated from the FTE available for the time period.
Annual percentage resource allocation
For section 2.2.2, the annual percentage resource allocation, the percentage of staff time per activity type (referral and brief service, standard case, test case, initiatives, and admin time), are calculated as the percentage of the total hours docketed during the year against each of the four categories compared to the total effective annual hours available for all clinic staff.
Areas of law
Areas of law in which a clinic may provide service. See the complete Area of Law list in CIMS.
Case file
The record of the clinic’s response to a client’s request for assistance. This includes the services offered, work completed and relevant details and notes about a client’s matter. A Case File is not created for situations where the clinic has only offered a referral elsewhere for service or has declined to provide services.
Case file closing
Date when all matters laid out in the retainer agreement (written or verbal) have been completed.
Case file outcome
Result of client’s matter. For example: hearing order, settlement, eviction, non-eviction,-financial recovery, immigration status settled.
Case file work
Case file work includes legal research, drafting submissions, preparing for and appearing at hearings/court, travel, client meetings, case conferences, negotiations, letters, document-drafting, scheduling appointments, etc.
Client
Persons served by any clinic staff. Person to whom a clinic has agreed to or is obligated to provide a service or anyone to whom a lawyer owes a duty of confidentiality whether or not a solicitor/client relationship exists.
Client objective
Client’s expected/desired remedy to the matter
Conflict of interest
Whenever confidential information is received from a prospective client, identifying information is checked against the clinic client database to avoid a conflict of interest (For definition of conflict of interest see Rule 2.04 of The Rules of Professional Conduct of the LSUC).
Direct legal services
The services provided to a client that are directly related to a case.
Eligibility / referral / other service
For the purposes of Table 2.2.1, Resource Allocation includes the early service conducted by the clinic once a file is set up and information from the client has been entered into CIMS. It may include eligibility testing, referral and a limited level of service, and is calculated as the total number of hours in the reporting period for all client services where time docketed by all staff relating to the case in the period is less than two hours. It includes the activity of recording a potential client’s information along with reason for their contact with the legal clinic.
Financial eligibility
Whether the client meets the clinic’s financial eligibility guidelines.
Full-time equivalent (FTE)
The total FTE for a time period is calculated as the sum of hours allocated to all staff in the period as a proportion of the annual total hours available excluding estimated holiday, vacation and sick days (e.g. 1561 hrs per year for a full-time person). For example, if A works 200 hours or 200/1561 FTE (temporary worker), B works 780 hours or 780/1561 FTE (half time), and C works 1561 hours or 1561/1561 FTE (full time), the total FTE for all 3 staff for the year is (200+780+1561)/1561= 1.63 FTEs. For a 3-month period, it would be: 1.63/4=0.41FTE.
Initiative file
Work done or services provided in support of the initiative files (Initiative Type field in CIMS) such as; Public legal education/Outreach, Training, Community development, Policy advocacy, Law reform, Systemic advocacy, Partners/Network/Community Groups, LAO/Clinic Committee and Consultations, Inter-Clinic Groups, Memberships, Media/Communications, Governance, Professional development. These files may have a defined start and end date or be on-going with no finite end date.
Initiative file closing
Date when initiative is deemed to be completed (Close Date in CIMS).
Intake
This refers specifically to the process of initial data collection – the activity of recording a potential client’s information along with the reason for their contact with the legal clinic. The goal of this collection is to inform a clinic’s decision making about whether and/or how a clinic can best assist this person. It does not include any services provided to the client including drafting paperwork, assisting with forms or providing advice. It does include “straight referrals”. “Intake” for CIMS purposes just refers to the initial data collection.
Matter
A client’s concern/issue/questions.
Referral
Service cannot be provided. Applicant given alternative source for assistance (this may be in combination with other services provided)
Salary
CIMS calculates the cost per case based on the personnel cost of docketed time, plus disbursements, minus recoveries. A salary or hourly rate must be entered and updated in the CIMS Salary Profile area to enable calculation of performance measures. Note: A value of $0.00 can be used (e.g. volunteers). Benefits should be included as applicable.
Service outcome definitions (to be used by clinic staff to determine case outcomes):
Successful: Meets all of the Client’s Objectives
Partially successful: Meets some of the client’s objectives
Unsuccessful: Does not meet client’s objectives
Withdrawn/discontinued: Matter is withdrawn or legal action is discontinued by the clinic or client
Unknown: Clinic loses contact with client or outcome is unknown
Standard case
Is the service provided to a client where total docketed time by all staff is greater than two hours and the service relates to a specific situation or situations addressing a client’s need.
Test case
Is service provided to a client, clients, or public interest matter where total docketed time is greater than two hours and the case is intended to have impact beyond the single case, such as public interest or law reform, and advances important public interests, transcends individual interests, and addresses a serious issue that fundamentally affects low-income Ontarians or disadvantaged communities whose perspective would be unlikely to come before the courts but for the clinic’s involvement.
2. Performance measures to be reported outside of CIMS
2.1 Measure #6: Service outcomes and client feedback
2.1.1 Efficiency measure
Category
Satisfaction level (1-5) with case service as indicated by client
Timeliness of services provided
4.2
Ease of access to service
4
Responsive to needs
3.75
Treatment received from staff
4.1
2.1.2 Effectiveness measure
Average satisfaction level, on a scale of 1 (very dissatisfied) to 5 (very satisfied)
Category
2011
2012
2013
Client satisfied with amount of time to get service from the clinic
4
4.3
4.1
Found it easy to get service and assistance client was looking for
3
3.5
3.8
Client got what he/she needed
3.5
3
3.75
Client was treated with courtesy and respect by clinic staff
3.5
3.75
4.0
Click here to download image.Chart 10. Client feedback on service on a scale fo 1 (very dissatisfied) to 5 (very satisfied)
2.2 Measure #7: Complaints filed and founded
2.2.1 Sample data
Resolution stage of complaint
# of complaints
Abandoned
12
Resolved by clinic
10
Resolved by board
2
Resolved by LAO
3
Not resolved To date
1
Total received
28
Total founded
3
2.2.2 Efficiency
Time to resolve complaint
# of complaints
%
Under 30 days
20
71%
Over 30 days
8
29%
Total complaints received
28
Click here to download image.Chart 11. Time required to resolve comlplaints (2013)
2.2.3 Effectiveness
Category
2011
2012
2013
Complaints received
31
15
28
Complaints founded
5
2
3
% of complaints founded
16%
13%
11%
Click here to download image.Chart 12. Trend in founded complaints (2011-2013)
Category
2011
2012
2013
Complaints received
31
15
28
Resolved in < 30 days
25
9
20
% resolved in < 30 days
81%
60%
71%
Click here to download image.Chart 13. Percentage of complaints resolved in less than 30 days (2011-2013)
Click here to download image.
Click here to download image.
Click here to download image.
Click here to download image.
Click here to download image.
Click here to download image.
Click here to download image.
Click here to download image.
Click here to download image.
Click here to download image.
Click here to download image.
Click here to download image.
Click here to download image.
Click here to download image.
Click here to download image.