Public Service Commission 5-Year Evaluation Plan 2011-2016

Prepared by: Evaluation Division, Corporate Management Practices and Evaluation Directorate, Corporate Management Branch

Table of Contents

1.0 Introduction

At the Public Service Commission (PSC), the Evaluation Division provides advice to senior management that supports decision making. It generates timely, neutral and evidence-based information on the relevance and performance (effectiveness, efficiency and value for money) of PSC policies, programs and initiatives and it meets the requirements of the PSC's Policy on Evaluation1.

This document presents the PSC's rolling Five-Year Evaluation Plan for the period of 2011-2016. In doing so, it:

  • Describes the context of evaluation planning at the PSC;
  • Highlights the accomplishments of the Evaluation Division in 2010-2011;
  • Details the process of developing the risk-based Five-Year Evaluation Plan; and
  • Outlines the particulars of the PSC Five-Year Evaluation Plan 2011-2016.

2.0 Evaluation planning and context

In the Government of Canada, evaluation is the systematic collection and analysis of evidence on the outcomes of programs in order to make judgments about their relevance and performance and to propose alternative, more cost-effective ways to achieve the same results.

Evaluation at the Public Service Commission

For organizational purposes, the Director General of the Corporate Management Practices and Evaluation Directorate (CMPED) is the head of evaluation at the PSC. It is the head of evaluation's responsibility to ensure that the PSC's Five-Year Evaluation Plan aligns with and supports the PSC's Management Resources and Results Structure, including information requirements for strategic reviews and that the Plan supports the PSC's Policy on Evaluation.

Upon approval by the Executive Management Committee (EMC), the PSC Five-Year Evaluation Plan 2011-2016 acts as a record of the evaluation function's contribution to results-based management and PSC accountability. The head of evaluation presents evaluation issues to the EMC and is responsible for the Plan's implementation2.

  • The Evaluation Division within the CMPED is mandated to conduct program evaluations and formulate evaluation frameworks. It provides oversight and advice on performance measurement within the PSC and helps ensure that management responses and action plans for evaluations are undertaken and followed up as required. It also supports the adoption of best practices and lessons learned that can improve management practices and program activities.

3.0 2010-2011 evaluation activity accomplishments

In 2010-2011, the Evaluation Division undertook the following evaluation activities:

  • Evaluation Framework and Evaluation of the Assessment Policy Requirements Specific to the EX Category;
  • Evaluation Framework and Evaluation of Priority Administration;
  • Evaluation Framework for the Federal Student Work Experience Program (FSWEP), Research Affiliate Program (RAP) and Cooperative Education and Internship Program.
  • Follow-up Study on the Evaluation of Post-Secondary Recruitment; and
  • Evaluation of the Communications and Parliamentary Affairs Directorate.

Other completed products or contributions included the following:

  • Update of the Five-Year Risk Based Evaluation Plan;
  • Performance advice specific to information management;
  • Input to the PSC Performance Management Framework; and
  • Delivered performance measurement information sessions.

4.0 Risk-based methodology

Developing the risk universe

The initial step in developing the PSC Evaluation Plan 2009-2014 was the identification of the PSC's evaluation universe. The Evaluation Division followed a comprehensive risk assessment strategy that included the following:

  • Extensive consultation with members of the Executive Management Committee (EMC) and some senior managers;
  • Benchmarks for risk-based assessments;
  • Identification of high priority areas by senior management;
  • Integration of other PSC risk assessments, including the internal audit risk assessment; and
  • A risk-based analysis with detailed priority-setting exercises to identify the evaluation projects to be included in the evaluation plan.

Assessing risk levels

The PSC Program Activity Architecture (PAA), as presented in the Operational Plan, was used as the framework of analysis for risk assessment. Criteria used to assess the levels of risk included Management Accountability Framework (MAF) results and performance measurement issues; program maturity; financial materiality and corporate risk profile: capacity to manage, monitor and evaluate in a changing environment and context; management interest; risk events; operational impacts; previous, current or ongoing planned internal or external evaluations/audits and risk events and political and/or public sensitivities.

Medium to high-risk PAA sub-sub-activities are shown as priorities and determine the sequence of evaluations and the level of effort, e.g., the required number of lines of evidence. Although the intent is to utilize internal capacity, there are expected to be some instances of outsourcing.

Past commitments and activities undertaken as a result of the PSC's last approved Five-Year Evaluation Plan, in this case 2009-2014, were reviewed for completeness and continued relevance.

Upon the completion of the risk assessment, final comments from senior management and the President were incorporated into the PSC Five-Year Evaluation Plan 2009-2014. Once approved by the EMC, the Plan will officially be published, forwarded to the Treasury Board Secretariat and posted on the PSC Web site.

5.0 Resources and capacity

Corporate Management Branch resources and capacity

The Evaluation Division of the CMPED has five full-time equivalents (FTEs), including the manager, who are dedicated to the function. Evaluation team capacity is estimated at an average of three evaluations and three evaluation frameworks per year, including additional operations and maintenance that may be made available from other sources for priority projects3.

Given the current FTE complement, combined with inevitable turnover, the Evaluation Division meets the projected coverage outlined for 2011-2012 in the PSC Five-Year Evaluation Plan 2011-2016 that was developed with its current capacity in mind. Should additional requests emerge, further resources may be required or priorities may need to be re-aligned. Variances resulting from a lack of capacity will be dealt with using contract or casual resources, as deemed necessary.

The Evaluation Division must also maintain the capacity to provide other evaluation and performance measurement advice in order to support evaluation results and to monitor and participate in PSC initiatives that may arise on a priority basis. These efforts can amount to an estimated 20% of a program evaluator's time.

Monitoring the Five-Year Evaluation Plan

It is inevitable that circumstances occur that could not have been reasonably predicted when the PSC's Five-Year Evaluation Plan 2009-2014 was finalized. As per the TBS Policy on Evaluation, the PSC's rolling Five-Year Evaluation Plan will be formally reviewed on an annual basis by the PSC's Evaluation Division and will be recommended for approval by the President and the EMC.

It should be noted that the activities included in the planning of subsequent years surpasses the capacity of the unit, with the intention of staying ahead of commitments.

6.0 Public Service Commission Five-Year Evaluation Plan 2011-2016

The PSC Five-Year Evaluation Plan 2011-2016 includes a total of 20 evaluations and 19 frameworks and 2 other evaluation activities (performance measurement and peer reviews) over the five-year period, all focused on the PSC's key programs and policies. Annex B details planned evaluation activities for 2011-2012.

7.0 Glossary

Evaluation Framework3
Plan that describes how a program will be measured (performance measurement framework), evaluated (key-questions and methodology) and reported (program profile).

Evaluation4
The application of systematic methods to periodically and objectively assess effectiveness of programs in achieving expected results, their impacts, both intended and unintended, continued relevance and alternative or more cost-effective ways of achieving expected results.

Peer review
The process by which a professional's practice is reviewed by members of the same profession for compliance with professional standards.

Performance measurement
The processes and systems of selection, development and ongoing use of performance measures to guide decision-making. These are comprised of indicators (quantitative or qualitative) used to determine to what extent an organization, a policy, a program or an initiative has achieved its objectives. Quantitative performance measures are made up of a number and a unit.

Annex A: Public Service Commission Five-Year Evaluation Plan — 2011-2016

  • F: Framework
  • E: Evaluation
  • P: Peer review
  • PM: Performance measurement

Level of risk:

  • L: Low
  • M: Medium
  • H: High
Public Service Commission Five-Year Evaluation Plan - 2011-2016
  PAA / Program Titles 11-12 12-13 13-14 14-15 15-16 Risk Level
2011-2012 1.1.2 - Delegated Authorities and Strategic Staffing F- E         M
Provide strategic staffing advice to organizational corporate HR teams
1.1.4 -Political Activities F E       M
Provide expertise on matters related to political activities in the federal public service (e.g., requests for permission, awareness and understanding of rights and obligations, guidance, information, tools and monitoring)
1.2.2 - Learning Strategy F E       M
Implementation of a learning strategy for ADSB auditors
1.3.1 Recruitement of Policy Leaders F E       M
Recruitment of Policy Leaders Program (RPL) is designed to give organizations the opportunity to attract and recruit exceptional leaders from across Canada, including Canadians living abroad, who have a proven track record as thinkers and self-starters and are passionate about getting involved in shaping public policy.
1.3.1 - General Recruitment F   E     M
Provide staffing planning, screening and assessment services
1.3.1 - Virtual Career Fair F-E         M
The PSC intends to replicate a real-world or in person career fair. Fair attendees will experience through an immersive and interactive medium, the ability to communicate with public servants, watch online presentations, review information, videos, etc., inside a custom built three-dimensional (3D) online environment.
1.3.1 - Federal Student Work Experience Program (FSWEP), Research Affiliate Program (RAP), Co-operative Education and Internship Program * E         M
1.3.2 - Personnel Psychology Centre F E       H
Provide evaluation and assessment services and tools for recruitment, selection and development at all levels of the public service (incl. R&D, product line, duty to accommodate, test defence)
2.1.1.1C - Program Evaluation P         L
Conduct neutral evaluation of PSC programs as outlined in the Five-Year Plan
2.1.2.3 A - Information Management Office F-E         M
Implement a sound information management (IM) framework for making of timely and informed decisions, supporting business activities and plans and preserving corporate memory
2012-1013 1.1.4 - Political Activities F E       M
Provide expertise on matters related to political activities in the federal public service (e.g., requests for permission, awareness and understanding of rights and obligations, guidance, information, tools and monitoring)
1.2.2 - Data Services   F   E   H
Produce statistical information and products and disseminate that information through the information highway
1.2.2 - Learning Strategy F E       M
Implementation of a learning strategy for ADSB auditors
1.2.3 - Investigations   F E     M
Investigate specific appointment processes in order to protect the merit principle and also investigate allegations of improper political activities in order to ensure the non-partisanship of the federal public service.
1.3.1 Recruitement of Policy Leaders F E       M
Recruitment of Policy Leaders Program (RPL) is designed to give organizations the opportunity to attract and recruit exceptional leaders from across Canada, including Canadians living abroad, who have a proven track record as thinkers and self-starters and are passionate about getting involved in shaping public policy.
1.3.1 - Post-Secondary Recruitment Program   F-E       M
Enable federal organizations to meet recruitment challenges by accessing and hiring qualified university and college graduates for entry-level positions across Canada
1.3.1 - EX-Resourcing   F E     L
Provide executive staffing services to departments and agencies for EX-1 to EX-5, on a cost recovery basis
1.3.2 - Personnel Psychology Centre F E       H
Provide evaluation and assessment services and tools for recruitment, selection and development at all levels of the public service (incl R&D, product line, duty to accommodate, test defence)
2.1.2.3B - Library   F   E   M
Provide access to information from reliable and authoritative sources in support of PSC priorities and activities
2013-2014 1.1.1 - Appointment Policies and Tools     F   E M
Policy instruments interpretation, application and development
1.1.1 - Annual Report     F E   M
Develop the concept and content for the PSC's Annual Report - process efficiency, effectiveness of the report
1.2.1 - Staffing Management Accountability Framework (SMAF) and Departmental Staffing Accountability Report (DSAR)     F   E H
Continuous improvement of oversight monitoring through the SMAF and DSAR. Implementation of monitoring and reporting on results through the DSAR Cycle
1.2.3 - Investigations   F E     M
Investigate specific appointment processes in order to protect the merit principle and also investigate allegations of improper political activities in order to ensure the non-partisanship of the federal public service
1.3.1- EX-Resourcing   F E     L
Provide executive staffing services to departments and agencies for EX-1 to EX-5, on a cost recovery basis
1.3.1 - General Recruitment F   E     M
Provide staffing planning, screening and assessment services
1.3.1 - Public Service Resourcing System     F   E H
Enhance, support and promote the use of PSRS functionalities by departments and agencies
1.3.1 - Staffing and Assessment Services Branch Business Plan - Action Plan     F   E H
Further streamline, integrate and standardize SASB's processes, products, systems and services to achieve efficiencies. This also provides the opportunity to better serve clients by providing them with more efficient and innovative options to facilitate their staffing and assessment activities
1.3.1 - Cost recovery operations     F   E H
Manage the current cost recovery business situation regarding outreach, sales, revenue forecasting and expenditure management
2.1.1.1C - Integrated Planning Process     P     M
Develop, implement and coordinate an integrated operational planning and reporting process that integrates human resource planning, resource allocation, performance measurement, information technology and information management
2014-2015 1.1.1 - Annual Report     F E   M
Develop the concept and content for the PSC's Annual Report - process efficiency, effectiveness of the report
1.2.1 - Oversight Coordination       F-E   M
Coordination of oversight activities at the PSC, planning and identifying priorities
1.2.2 - Data Services   F   E   H
Produce statistical information and products and disseminate that information through the information highway
2.1.2.4 - Information Technology (IT)       F   M
Provide IT services to the PSC (incl. Infrastructure and Operations, Client Services, IT Security, Corporate IT Management, Applications)
2.1.2.3B - Library   F   E   M
Provide access to information from reliable and authoritative sources in support of PSC priorities and activities
2015-2016 1.1.1 - Appointment Policies and Tools     F   E M
Policy instruments interpretation, application and development
1.2.1- Staffing Management Accountability Framework (SMAF) and Departmental Staffing Accountability Report (DSAR) (DAD) - PAA 611     F   E H
Continuous improvement of oversight monitoring through the SMAF and DSAR. Implementation of monitoring and reporting on results through the DSAR Cycle
1.3.1 - Public Service Resourcing System (including Publiservice)     F   E H
Enhance, support and promote the use of PSRS functionalities by departments and agencies
1.3.1 - Staffing and Assessment Services Branch Business Plan - Action Plan     F   E H
Further streamline, integrate and standardize SASB's processes, products, systems and services to achieve efficiencies. This also provides the opportunity to better serve clients by providing them with more efficient and innovative options to facilitate their staffing and assessment activities
1.3.1 - Cost recovery operations     F   E H
Manage the current cost recovery business situation regarding outreach, sales, revenue forecasting and expenditure management
1.3.1 - Staffing and Assessment Services Branch Systems         F M
Optimize e-systems and tools

* This evaluation is on hold pending policy changes to the CO-OP and Internship programs. The Evaluation Framework will be used as part of the policy review process.

Legend:

Annex B: Planned Evaluation Activities - 2011-2012

Evaluation frameworks

In total, the Evaluation Division will undertake eight evaluation frameworks in 2011-2012, as follows:

  • Information Management (T1);
  • Political Activities (T1);
  • Delegated Authorities and Strategic Staffing (T2);
  • Virtual Career Fair (T2);
  • Personnel Psychology Centre (T2);
  • Recruitment of Policy Leaders (T2)
  • General Recruitment (T3); and
  • Audit and Data Services Learning Strategies (T3).

Evaluations

We are planning three or four evaluations for the current year including:

  • Delegated Authorities and Strategic Staffing involves two branches (Policy and Staffing and Assessment Services), thus adding to the level of complexity. The purpose of the evaluation will be to analyze the roles and responsibilities of strategic consultants, identify possible duplication with other functions and the impact of their services on hiring organizations. (T2-4)
  • The evaluation of the Virtual Career Fair will assess through multiple lines of evidence the relevance and performance of this initiative in support of the PSCs mandate and the recruitment to the public service.
  • The Information Management Office, in order to ensure that they are well-positioned to support the transition to Gatineau in 2013 and to meet Library and Archives Canada requirements.
  • In addition, there is a possibility that a horizontal evaluation of three student employment programs will be undertaken, following the implementation of a new operational policy. These programs are the Federal Student Work Experience Program, the Research Affiliate Program and the Cooperative Education and Internship Program. (T4)

Performance measurement activities

The Evaluation Division will coordinate and participate in the development of the Annual Report on the State of Performance Measurement at the Public Service Commission (PSC), as per the Policy on Evaluation. This report will be presented to the Executive Management Committee.

Other evaluation activities

Among other evaluation activities, the Evaluation Division must undertake a peer review this year. At the beginning of the year, the team will develop a logic model and performance measurement indicators to facilitate this exercise and further refine its protocols.

The Evaluation Division ensures semi-annually that progress is made in the implementation of action plans stemming from evaluation recommendations.

Codification of protocols

The evaluation function has now been in place for several years and a culture of monitoring and evaluation is gradually evolving at the PSC. Focus is being placed on developing capacity and evaluation knowledge across the Division. To help this happen, the Evaluation Division is codifying a series of protocols to bring additional rigour and discipline to its function and increase its credibility. Processes will be described in greater detail and effort will be made to better manage the quality of deliverables.

It is also increasing its collaboration with other oversight areas of the PSC such as Access to Information and Privacy and internal audit. Examples of publications are the PSC's Policy on Evaluation and the update and planned implementation of the integrated PSC Five-Year Evaluation Plan 2011-2016.


  1. It also meets the needs of the Treasury Board of Canada Secretariat (TBS) Policy on Evaluation, effective April 2009, and satisfies the implications of the Government of Canada's Federal Accountability Act. (Return)
  2. For full details about the role of the head of evaluation, please refer to the PSC's Policy on Evaluation. (Return)
  3. Evaluation Guidebook for Small Agencies (Return)
  4. Results-Based Management Lexicon (Return)