Quick Links
Customer Centres
Inter-rater Reliability Assessment of an Audit Tool for OH&S Management Systems in Healthcare*
RS2006-DG06
Final Report Date: November 2007
| Principal Investigator: | Waqar Mughal (Fraser Health) |
| Co-Investigator: | Anna Matheson (Fraser Health) |
For more information about this project, please contact Waqar Mughal.
*In partnership with the Saskatchewan Workers’ Compensation Board
Issue
This project assessed the reliability of a new, standardized health and safety program audit tool designed for use in health care environments. Two independent auditors used the tool to audit the health and safety programs of four departments in a B.C. health authority, and the consistency of the resulting audits was evaluated.
Key findings
- Despite using the same audit tool, the two audits resulted in inconsistent responses about whether health and safety program components were in place in the health care units examined.
- The auditors also identified challenges with the tool, including layout, usability, length and the lack of a method (weights) to capture the relative importance of different program characteristics.
- Based on these findings, the researchers conclude that the tool has low reliability and is not yet suitable for implementation.
Objectives
- To evaluate the validity and reliability of an occupational health and safety program audit tool for use in health care environments
Methods
The audit tool assessed in this study was developed by the Audit Task Group of the Health Care Safety Professionals Association of British Columbia. It has 270 questions grouped under the following areas:
- Occupational Health and Safety Policy
- Planning
- Implementation and Operation
- Control and Improvement
- Management Review
The tool assesses whether program components are “In Place”, with response options consisting of: Yes, No, Partial or Not Applicable. The form also suggests which “Source” information to use for assessing different components, with the options consisting of: documentation, interviews, observation or a combination of these.
Two experienced, independent auditors conducted audits of four units: an acute care medical unit, an extended care unit, a sterile processing department, and a laboratory department. The auditors visited the units at the same time but did not interact.
Data from the completed audits were analyzed to assess inter-rater reliability for both “In Place” responses and “Source” information used.
Auditors also provided feedback on the audit tool’s complexity, comprehensiveness, detail, length, and usability.
Results
- The two audits were substantially variable in responses about whether program components were in place, although quite consistent in terms of “Source” data used for assessing program components.
- The above finding was based on a statistical calculation called a Kappa analysis. Low Kappa values (.285 - .448, p=<0.05) were obtained for “In Place” responses for three of the four units audited. High Kappa values resulted for “Source” responses for all four units (.720 – 1.000, p=<0.05).
- The auditors’ feedback suggests that the tool is comprehensive but cumbersome, overly long, and lacks a weighting scheme to capture the relative importance of different program criteria within the overall assessment.
Conclusions
The audit tool was shown to have low reliability in the environments audited. Further refinement of the tool is needed.
Future directions
The survey instrument will be revised to address the challenges noted by the auditors including the layout and usability issues. Future research could include a more in depth evaluation of the tool’s reliability.
Future studies could also examine the relationship between audit findings and occupational health and safety program performance, as well as the reliability of health and safety program audits in general.
Publications and Presentations
Mughal, Waqer. Presentation: British Columbia Healthcare Workplace Health, Safety & Wellness Conference, held September 17– 18, 2007 in Kelowna, B.C.