Chapter 5 - Audit Examination Phase

Audits should have appropriate and sufficient evidence to support the contents of the audit report.

Audit Evidence

Evidence is information that is collected and used to provide a factual basis for developing observations and concluding against audit objectives. Evidence provides grounds for believing that a particular thing is true or not by providing persuasive support for a fact or a point in question. As such, it is evidence that must support the contents of an audit report, including any descriptive material and, more importantly, all observations and conclusions leading to recommendations.

Auditing can be described as an iterative decision-making process. The gathering of evidence is in line with this overall process. The auditor gathers information, evaluates it for its appropriateness, and determines whether it is sufficient to support observations, make conclusions with respect to audit objectives, and make useful recommendations. If not, additional evidence may be needed.

The evidence-gathering process involves:

  • designing the audit procedures or tests (part of the audit examination plan);
  • carrying out the audit procedures or tests to gather evidence;
  • analyzing evidence and drawing conclusions which may also involve evaluating performance against the audit criteria; and
  • making decisions about whether additional information is required and can be obtained, or whether appropriate and sufficient evidence exists.

It is not unusual for audits to be redesigned during the examination phase as teams encounter unforeseen difficulties in gathering sufficient evidence of appropriate quality.

Auditors have to be alert to any signs that the evidence gathering process may not be achieving the level of assurance required for the audit assignment, and must take appropriate corrective action. Audit observations, conclusions and recommendations included in the report must be able to withstand critical examination. Therefore, they must be supported by appropriate and sufficient evidence.

In determining whether evidence of appropriate quality and sufficient quantity has been gathered, the auditors need to be satisfied that, in their professional judgment, there is a low level of risk of making erroneous observations, faulty conclusions, or inappropriate recommendations.

By definition, and consistent with the auditing standards established by the Canadian Institute of Chartered Accountants, audits provide a high level of assurance. The auditors provide a high, though not absolute, level of assurance in gathering evidence by designing inspection, enquiry, confirmation, computation, and analysis and discussion procedures so that the risk of an inappropriate conclusion is reduced to a low level.

Although decisions about whether there is appropriate and sufficient evidence are ultimately matters for the auditor's professional judgment, there are several considerations to bear in mind in making these decisions.

Appropriate Evidence

For evidence to be appropriate, the information must be relevant, reliable and valid.

Relevance refers to the extent to which the information bears a clear and logical relationship to the audit criteria and objectives. If information is not relevant, it cannot be evidence.

Reliability concerns the likelihood of coming up with the same answers either when the audit test is repeated or when information is obtained from other sources. A measurement or evidence-gathering process is considered more reliable if repeated measures or performance of the process produce the same result or a consistent result that is minimally affected by measurement errors (random distribution of measurement errors).

Validity has to do with whether the information actually is what it purports to be in relation to content, origin and timing. An audit rarely involves the authentication of documentation or information. An auditor is not trained as or expected to be an expert in authentication. However, the auditor has to consider the validity of the information to be used as audit evidence (for example, photocopies, facsimiles, filmed, digitalized, scanned, or other electronically produced documents, including consideration of controls over their preparation, where relevant).

The following rules should be considered in judging the appropriateness of evidence:

  • documentary evidence is usually better than testimonial evidence;
  • audit evidence is more reliable when the auditor obtains consistent evidence from different sources or of a different nature (e.g. testimonial evidence that is corroborated by other sources is better than testimonial evidence alone);
  • original documents are better than photocopies;
  • evidence from credible third parties may be better than evidence generated within the audited organization;
  • the quality of information generated by the audited organization is directly related to the strength of the organization's internal controls (the auditors should have a good understanding of internal controls as they relate to the objectives of the audit); and
  • evidence generated through the auditor's direct observation, inspection and computation is usually better than evidence obtained indirectly.

Sufficient Evidence

The concepts of sufficient (quantity) and appropriate (quality) evidence are interrelated. The quantity of evidence is sufficient if, when taken as a whole, its weight is adequate to provide persuasive support for the contents of the audit report. In exercising professional judgment, auditors should ask themselves whether the collective weight of the evidence that exists would be enough to persuade a reasonable person that the observations and conclusions are valid and that the recommendations are appropriate. Important factors to consider in making judgments include the:

  • quality of the evidence—its relevance, reliability, and validity;
  • level of significance of the observation or conclusion—in general, the higher the level of significance, the higher the standard that evidence will have to meet;
  • risk involved in making an incorrect observation or reaching an invalid conclusion; and
  • cost of obtaining additional evidence relative to likely benefits in terms of supporting observations and conclusions.

It is often the case in audits that important "facts" are not singular but instead are made up of a collection of interrelated facts. In the assessment of the quality and quantity of evidence, the auditor has to consider that the strength of the sum of the facts may be as important as the strength of the individual facts.

Auditors frequently face the challenge of providing sufficient and appropriate evidence that something does not exist. Not finding something begs the question of where and how hard one has looked. In these circumstances, it is particularly important for auditors to use multiple sources of evidence to corroborate observations, and to document the approach taken to look for evidence.

The clearance of the audit report by the audited entity does not replace the need for appropriate and sufficient evidence. Such evidence must be on hand before the audit report is drafted so that the report's observations, conclusions and recommendations are based on evidence. The purpose of sending a draft report to the audited entity is to obtain confirmation (not evidence) that the facts in the report are accurate and that the report presents a fair perspective.

A limitation exists on the scope of an audit that, despite best efforts, is unable to meet the standard of obtaining appropriate and sufficient evidence. The available evidence and its limitations could be reported, but observations and conclusions would not be drawn. If the PSC decides to report the matter, it would be reported as a qualification to the conclusion, indicating that a lack of evidence prevented a certain part of subject matter from being evaluated. When, in the judgment of the PSC, a qualification would not be sufficient due to the significance and extent of the limitation in the evidence, the audit report would express a denial of the conclusion. A denial states that a conclusion cannot be made on the subject matter.

Sources and Types of Audit Evidence

There are three broad sources for the information that constitutes audit evidence.

Information gathered by the auditors (primary evidence). Auditors can gather information themselves using interviews, surveys, and direct inspection or observation. In these cases the auditors have control over the methods employed and the quality of the information gathered. However, the auditors must have the necessary skills and experience to apply the methods competently.

Information gathered by the audited entity (secondary evidence). Auditors can use information gathered by the audited entity, including the reports of internal audit and program evaluation groups, as well as information found in the audited entity's files, databases, reports and documents. Auditors should determine the quality of this information by evaluation and corroboration, as well as by tests of the effectiveness of the entity's internal controls over the quality of information. Auditors can reduce tests of information quality if they find that the internal controls are effective.

Information gathered by third parties (secondary evidence). Audit evidence can also include information gathered by third parties. In some cases others may have audited information, or the auditors may be able to audit the information themselves. In some cases, third party information cannot be audited, but its quality is known (for example, data from Statistics Canada). The extent to which third-party evidence can be used as evidence will depend on the extent to which its quality can be established.

Forms of Evidence

Audit evidence can take a variety of forms.

  • Physical evidence
    Physical evidence is typically obtained by the auditor's direct inspection or observations, and supported by field notes, photographs or videotapes wherever possible. An inherent risk of observation is that the observer's presence may alter what occurs in the setting, and as a consequence the evidence collected can be less valid. The observer should disturb the setting as little as possible.
  • Testimonial evidence
    Testimonial evidence includes oral or written statements obtained in response to the auditor's inquiries. Examples include interviews with entity staff and surveys (either by telephone or mail) of clients of the service. Inquiry has always been one of the significant audit techniques. Careful preparation and briefing beforehand, and debriefing and documentation afterwards, improve the effectiveness of an interview. Wherever possible, evidence from individual interviews should be corroborated with evidence from other people or other sources.
  • Documentary evidence
    Documentary evidence is that obtained from such sources as files, performance reports, databases, minutes of meetings, organization charts and correspondence. Documentary evidence can be obtained from the audited entity or from third-party sources, and includes both electronic and hard copy information.
  • Analytical evidence
    The auditor produces analytical evidence by manipulating other types of evidence using analytical techniques such as computations, comparisons, and content analysis of qualitative data.

Although evidence analysis follows evidence-gathering in chronological terms, the audit team members need to know what specific analytical techniques they will use before they start to design the strategy for capturing evidence. Otherwise, the auditors may find that the evidence collected is not susceptible to the appropriate forms of analysis.

When gathering information during the examination phase, the auditor thinks forward to the reporting phase and the need to communicate the audit message in a persuasive manner. The auditor needs to look for opportunities to use case studies, as these often provide a convincing way to illustrate an issue in the audit report.

Quantification is an important means of demonstrating the significance of an audit's observations and recommendations. A focus on quantification should be built in at the planning phase.

Relying on the Work of Others

In the interest of audit efficiency, auditors should rely on the relevant work of internal audit and evaluation, internal and external experts, and specialists whenever possible. When the work of others is the main or sole evidentiary support for particular observations, conclusions and recommendations, auditors should evaluate and corroborate the specific work on which they intend to rely. The purpose will be to determine whether the work meets the PSC's audit policies with respect to appropriate and sufficient evidence, and that an adequate basis for reliance exists.

Auditors can determine the quality of others' work by assessing their reputation, qualifications and independence, and by reviewing their reports, programs and working papers. The nature and extent of the evaluation and corroboration will depend on the significance of this work in relation to the PSC's audit objectives and the extent to which the auditors will rely on it.

Where auditors use the work of others, the audit team should evaluate and corroborate the supporting evidence to assure the validity of the findings. Normally, when such matters are included in the PSC audit report, the source of findings is clearly indicated.

Documentation of Evidence

One of the PSC's audit conduct policies requires the audit team to maintain appropriate documentation and files. Audit working papers and files are used to document key audit decisions and work. The documentation of evidence is a vital aspect of auditing, and it should be completed before the Transmission Draft is issued.

Good documentation of evidence helps ensure that:

  • an adequate and defensible basis exists for the audit observations, conclusions and recommendations;
  • the observations, conclusions and recommendations can be explained in response to internal or external enquires;
  • an effective link exists between successive audits; and
  • an appropriate basis exists for quality control in carrying out an audit and for third-party reviews.

Auditors need to exercise professional judgment in documenting evidence. A guiding principle is that the audit files and working papers must include either the evidence or the description of the evidence examined, sufficient to allow the audit managers and others who examine all of the evidence to come to the same conclusions as the auditors.

In addition to being complete, accurate and clear, the files and working papers containing the evidence need to be structured in a logical way, to provide ready access to the evidence.

Although the documentation will usually include most of the evidence itself, it is not always necessary to copy and file every document examined or to list detailed information from all such documents. For example, when evidence includes the audit entity's records, it may be enough to note that a particular document was examined and to provide the information required to identify and locate that document.

Complete, indexed and cross-referenced working papers are critically important when reviewing findings with management, briefing the President, providing support at parliamentary committee hearings, answering subsequent queries from the audited entity and others, and planning future assignments. Clear indexing and cross-referencing ensures the evidence is readily accessible.

The audit team should prepare substantiation binders that contain the audit evidence most pertinent to the audit report content. These binders are prepared as a means of providing assurance as to the quality of the audit. Gathering together the evidence specific to a report for easy access also allows the PSC to respond to internal or external enquiries (e.g. a hearing by the Government Operations and Estimates Committee or other standing committees).

The goal of having a substantiation binder is to ensure observations, conclusions and recommendations flow logically from the evidence available and are well supported. The evidence in the substantiation binders should be persuasive, so that a review of it by a reasonably knowledgeable person will result in similar observations, conclusions and recommendations.

Substantiation covers all aspects of the report. In addition to the evidence needed to support factual statements, the substantiation binders include support for the judgments, assumptions and conclusions made by the auditor. For example, it would include a paper setting out the logical arguments and supporting evidence for the auditor's decisions. Usually, only some small part of a document or a working paper summary is needed as proof for a particular statement.

Auditors use their professional judgment in deciding what to include in the substantiation binder to support the report, ensuring that appropriate and sufficient evidence for the more contentious, sensitive and highly visible issues is included. For other matters, such as the background information on an entity, the audit team can choose to include a cross-reference to the evidence found in other audit working paper files, rather than putting a copy of the evidence itself in the substantiation binder. The binders are to be carefully indexed and cross-referenced to supporting details.

Before issuing the external draft, the audit team gathers together the documentation to enable the audit director general to determine that sufficient appropriate evidence was obtained to support the major observations, recommendations and conclusions. The substantiation binders should be completed and reviewed prior to the issuance of the transmission draft report.

Audit Observations

Audits should objectively evaluate evidence against the criteria to develop observations and conclusions.

The audit team gathers evidence to support a description of a staffing activity under review and make an assessment of the actual performance of that activity against the audit criteria. Where the auditor finds that performance does not meet the criteria, further examination should be carried out to gain assurance that any resulting observations and conclusions are significant, fair and well founded, and that recommendations have the potential to result in important performance improvements.

Audit observations confirm satisfactory performance or disclose the level, nature and significance of deviations from criteria, the cause, if determinable, of the problem and its effect on the subject matter of the audit.

Gathering additional evidence and/or discussing the matter with management of the entity being audited may be necessary to:

  • determine whether the deficiency is an isolated instance or represents a generic or systemic problem;
  • assess the impact or potential impact of the deficiency on the results of the staffing activity. Wherever possible, the effect of the problem should be quantified or otherwise identified to illustrate the "so what" in the audit report;
  • identify the cause of the deficiency to increase assurance that recommendations will be appropriate;
  • determine whether the problem can be fixed by the audit entity, or whether it results from circumstances beyond its control;
  • gather further evidence (for example, cases, statistics, etc.) to illustrate the nature and importance of the issue, where appropriate;
  • determine who is affected by the issue—for example, other units within the organization, central agencies, and third parties; and
  • determine management's awareness of the issue. If management of the audited entity is aware of the issue and has corrective action under way, the issue may have less significance for reporting purposes. Certainly, it will change how the matter is reported.

The comparison of evidence against criteria, and further examination work into the nature and significance of the issue, will result in the development of observations. Audit observations confirm satisfactory performance or disclose the level, nature and significance of deviations from criteria. Observations may declare who is responsible, and may disclose the cause and effect of the problem. In reaching their decisions on observations, auditors may need to look at the collection of interrelated facts and evidence to assess them against the corresponding criteria, as well as considering them individually.

The observations, in turn, are the basis for forming conclusions against each of the audit objectives. The auditor should assess the significance of the observations in relation to the audit objectives. Auditors will use their professional judgment in drawing conclusions against an audit objective.

Audit Recommendations

When deficiencies are reported, audits should include recommendations to guide necessary corrective actions.

Audits include recommendations to prompt corrective action where the potential for significant improvement of a staffing activity is demonstrated by the report findings. A recommendation may address a single deficiency or a number of related observations or deficiencies.

There may be circumstances where making a recommendation is not the best way to achieve the intended result. In those circumstances, exceptions to the audit standard should be justified on a case-by-case basis and approved by the Vice-President, Audit and Data Services Branch. The audit can still make a major contribution in such cases by bringing a highly professional analysis of the situation to the attention of the entity being audited and Parliament.

Where corrective action is underway, it is good practice to point this out in the report.

Recommendations should be:

  • fully supported by, and flow from, the associated observations and conclusions;
  • related to the underlying causes of the deficiency;
  • clear, succinct, straightforward and sufficiently detailed to make sense on their own;
  • broadly stated (i.e., stating what needs to be done, while leaving the specifics of how to the management of the entity being audited);
  • action-oriented (i.e., presented in the active voice and addressed to the organization with the responsibility to act on them);
  • positive in tone and content;
  • practical (i.e., able to be implemented in a reasonable timeframe, taking into account legal and other constraints);
  • cost-effective (i.e., the costs of implementing them will not outweigh the benefits);
  • results-oriented (i.e., giving some indication of what the intended outcome is, ideally in measurable terms);
  • able to be followed up (i.e., able to determine whether it has been acted upon); and
  • coherent and consistent with the other recommendations in the audit report.

The audit team, at any time during the examination phase, may come across situations where immediate action is required. These situations should be brought to the attention of the Vice-President, Audit and Data Services Branch. If the audit team concludes that:

  • there is a possible breach of legislation or fraudulent action, Legal Services should be consulted;
  • there is a staffing action which is not in compliance with the PSEA, the Director General, Audit Operations will determine if the action should be brought to the attention of the deputy head of the audit entity or the PSC Investigations Branch for an investigation;
  • there are serious deficiencies in the exercise of delegated authority by the deputy head and that conditions should be imposed, the Vice-President, Audit and Data Services Branch should consult with the Vice-President, Policy Branch of the PSC to obtain the President's approval. The audit report should reflect these actions. The auditors may also recommend conditions under which the delegated authority could be retained or returned; and
  • there are deficiencies in the application of the PSC's own non-delegated authority, the auditors may make recommendations to the branch where the deficiencies occurred.

A recommendation for changes to legislation is highly sensitive. If observations point to the need for such changes, the matter should be discussed with Legal Services.

Audit Conclusions

Audits should have necessary and sufficient observations to support the conclusions reached against each audit objective.

The process of dividing the audit into component parts does not remove the need to make conclusions in relation to the overall audit objectives. Planning decisions have identified the audit issues. Audit evidence has been gathered and performance in the critical areas has been assessed against each of the criteria. Actual performance has been found to be satisfactory or deviations from the criteria have been identified. Further examination of the deviations from satisfactory results of good practices has led to the development of observations.

The auditor should assess the significance of the observations in relation to the audit objectives. At the extreme ends of the performance spectrum–fully satisfactory performance or highly unsatisfactory performance–concluding with respect to the overall objective may not pose a problem. In these cases, the audit report would contain an unqualified (positive) conclusion or an adverse conclusion. An adverse conclusion is used when the significance and extent of the deviations from satisfactory performance are persuasive. In the majority of cases, the auditor will have to use judgment in forming a qualified conclusion. Qualified conclusions are made when there are significant deviations from satisfactory performance for one or more aspects of the subject matter. A qualified conclusion contains an "except for" statement, either stated explicitly or implicitly, to disclose the deviations in relation to the audit objectives.

Issues Outside the Audit Jurisdiction of the PSC

During the course of an audit, the audit team may come across a potential issue that is outside the audit jurisdiction of the PSC. In such instances the audit team should consult with Legal Services to determine the next steps to be taken. This would include determining:

  • which authority would have jurisdiction over this issue;
  • whether this issue should be brought to the attention of the appropriate audit or investigative authority (for example, the Office of the Auditor General, the Canadian Human Rights Commission, or the PSC Investigations Branch);
  • how the issue should be raised with entity management; and
  • whether the issue should be raised in the audit report.

The audit team should collect sufficient evidence to be able to explain to the appropriate authority why the issue is being raised.

Examination Report

Having completed the field audit work, the audit team prepares the Examination Report. This Report includes an annotated outline of the audit report, as well as a summary of the audit observations, recommendations, and conclusions with respect to each audit objective. It is used to review facts and the results of the tests. At the same time, the audit team begins preparing the substantiation binder.

This Report is presented to the Vice-President, Audit and Data Services Branch, for advice and comments. After being reviewed, it is used as the basis for debriefing the management of the entity being audited. As well, it serves as the starting point for writing the internal draft report, as discussed in the next chapter.

Working Papers and Audit Files

Audit files and working papers should contain information about the approach and work undertaken to achieve the audit objectives. The key documentation covering the entire audit process should include:

  • the Commission's approval of the selection of the audit;
  • information gathered from the survey, including relevant analysis;
  • the terms of reference;
  • the summary examination plan and the examination plan;
  • where applicable, an explanation of major deviations from the original examination plan and approval for that deviation;
  • decisions by the Commission, President and Vice-President, Audit and Data Services Branch;
  • management's views of the objectives, criteria and other elements of the audit;
  • audit programs (if any) specifying the work to be carried out and the work completed;
  • comments and advice from advisors, the AAC and the quality reviewer, and significant audit decisions taken by the audit team and Vice-President, Audit and Data Services Branch based on this advice;
  • reporting phase signoffs;
  • significant correspondence with entity management;
  • entity management's comments on the audit report and steps taken to resolve any differences;
  • the internal draft audit report presented to the AAC, with the corresponding substantiation binder;
  • comments resulting from the various draft reviewers, and corresponding clearance of those comments; and
  • a summary for each audit objective, explaining how the methodology was employed, the nature and extent of the evidence collected, and the analysis to which it was subjected.

The audit team should maintain a PSC audit control file that contains the critical documents related to the management of the audit as a project. This file contains the most significant reports, approvals and decisions throughout the life cycle of the audit.

The auditors are required to comply with the National Archives of Canada Act in the retention of their working papers. All working papers are confidential documents belonging to the PSC. Audited organizations, Parliament and the public do not have automatic right of access to working papers. All requests for working papers from outside the Audit and Data Services Branch should be forwarded to the PSC Access to Information and Privacy (ATIP) Office.